Identifiers with Astral Unicode characters are unsupported

[Alhadis]

TypeScript Version

3.5.1

Search Terms

utf8, unicode

Code

const 𝑚 = 4;
const 𝑀 = 5;
console.log(𝑀 + 𝑚); // 9

Expected behaviour

The above is legal JavaScript, so 9 should be printed to the console.

Actual behaviour

The TypeScript compiler goes completely apeshit:

(click to see warnings)

test.ts:1:7 - error TS1127: Invalid character.

1 const 𝑚 = 4;
        

test.ts:1:8 - error TS1127: Invalid character.

1 const 𝑚 = 4;
         

test.ts:1:10 - error TS1134: Variable declaration expected.

1 const 𝑚 = 4;
           ~

test.ts:1:12 - error TS1134: Variable declaration expected.

1 const 𝑚 = 4;
             ~

test.ts:2:7 - error TS1127: Invalid character.

2 const 𝑀 = 5;
        

test.ts:2:8 - error TS1127: Invalid character.

2 const 𝑀 = 5;
         

test.ts:2:10 - error TS1134: Variable declaration expected.

2 const 𝑀 = 5;
           ~

test.ts:2:12 - error TS1134: Variable declaration expected.

2 const 𝑀 = 5;
             ~

test.ts:3:13 - error TS1127: Invalid character.

3 console.log(𝑀 + 𝑚); // 9
              

test.ts:3:14 - error TS1127: Invalid character.

3 console.log(𝑀 + 𝑚); // 9
               

test.ts:3:18 - error TS1127: Invalid character.

3 console.log(𝑀 + 𝑚); // 9
                   

test.ts:3:19 - error TS1127: Invalid character.

3 console.log(𝑀 + 𝑚); // 9

... and generates obviously unusable JavaScript:

var ;
4;
var ;
5;
console.log(+); // 9

More info

• ECMA-262, § 11.6 Names and Keywords.

[RyanCavanaugh]

Title is underspecific; 𝑚 is a surrogate pair which is why we don't recognize it. The broad class of other non-ASCII Unicode characters are correctly parsed.

[Alhadis]

𝑚 is a surrogate pair

Uhm, no it isn't.

If it was a surrogate character, then it wouldn't be valid UTF-8, let alone valid as a JavaScript identifier.

[Alhadis]

I've amended the title with what I think you meant.

[weswigham]

Our scanner is still only following the 5.1 spec which lists explicitly accepted unicode groups for identifiers - the 6.0 spec allows any Unicode code point with the Unicode property “ID_Start” or “Other_ID_Start” as an IdentifierStart and any Unicode code point with the Unicode property “ID_Continue”, “Other_ID_Continue”, or “Other_ID_Start” as an IdentifierPart.

cc @rbuckton - we have to have looked at this before, right? I mean, it'd be nice if we could just use /\p{ID_Start}/u, /\p{Other_ID_Start}/u, /\p{ID_Continue}/u, and /\p{Other_ID_Continue}/u, but that'd be host-dependent.

[weswigham]

Oh, our scanner also iterated by char code and not by code point, so every time we call isIdentifierPart we do so with only part of an astral plane character rather than the whole thing, in addition to the list of accepted charactersets being out of date.

[Alhadis]

Oh, our scanner also iterated by char code and not by code point

Yup, and there's the culprit. 👍

Seems like a pretty easy fix if it's just swapping string.charCodeAt for string.codePointAt.

[weswigham]

Seems like a pretty easy fix if it's just swapping string.charCodeAt for string.codePointAt.

Not quite - iterating by string index is by char code, so you need to handle the second "char code" of a multi-code code point - and to top it off, using exclusively the builtin codePointAt would cause us to break on es5 or below runtimes, so we need to polyfill it. In any case, #32096 should do juuuust enough to support parsing this all correctly.

[Alhadis]

Ah right, I forgot about supporting legacy environments. Makes sense, then.

Thanks for the fast PR!

[weswigham]

Ehhhh, it's just a first draft - it'll probably take us a bit to figure out how we want to take it.

[fatcerberus]

Uhm, no it isn't.

To be fair, he did say surrogate pair - i.e. it would have to be encoded as two codepoints in UTF-16. It's true that astral characters can't legally be encoded as surrogate pairs in UTF-8, though (that's what CESU-8 is for).

"Astral" is the correct term, of course, since "surrogate pair" describes a particular encoding scheme, not something inherent to the character itself. But it is relevant here, as astral characters are represented in JS strings per spec as surrogate pairs--that's why we needed fromCodePoint.