U+2028 and U+2029 maybe should be treat as line ending characters for line comments

[chirsz-ever]

Current statement of JSONC:

• Line ending: Refers to the sequence of (often invisible) control characters that indicate the end of a line. In JSONC, this sequence can be represented by a line feed (LF) or a carriage return followed by a line feed (CRLF). In terms of Unicode, LF corresponds to U+000A and CR corresponds to U+000D.

Single-line comments start with // and continue until a line ending is encountered.

But ECMA-262 treat U+2028 and U+2029 as line terminators. They can be used to end line comments.

Under the current definition, the following represents { "a": 1, "b": 2 } as JavaScript, but {"b": 2} as JSONC.

{
    // U+2028 here -> 
"a": 1,
    "b": 2
}

And the following is valid JSONC, but not a valid JavaScript object literal:

{
    // U+2028 here -> 
xxx
    "b": 2
}

A similar issue is that JSON allows U+2028 and U+2029 characters to be directly included in string literals, but JavaScript did not. Therefore, JSON was not a subset of JavaScript until ES2019 made it so.

The solutions adopted by other similar standards:

• JSON5: Consistent with JavaScript
• HJSON: Only treat LF as line terminator
• JWCC: Only treat LF as line terminator
• HOCON: Only treat LF as line terminator
• HanSON: Only treat LF as line terminator
• json-next: Only treat LF as line terminator
• RapidJSON: Only treat LF as line terminator
• nlohmann/json: Treat LF and CR as line terminators

[chirsz-ever]

Note that JSON does not allow U+2028 and U+2029 as whitespace characters. If we include them in the definition of line endings but not in the definition of whitespace characters, then for this example:

{
    // U+2028 here -> 
"a": 1,
    "b": 2
}

The lexical parser will fail to parse U+2028 as the next character of the line comment. This makes JSONC a subset of JavaScript: all valid JSONC data is valid JavaScript literals.

[DecimalTurn]

Thanks for raising this issue, I had started to think about it as well and that is what I'm going with in the current version of the ABNF that I've started here: #25

Basically, LS (U+2028) and PS (U+2029) are accepted as single-line comment terminators, but not as white space characters to try to preserve as much compatibility with JSON as possible.

[chirsz-ever]

In the ABNF description introduced in #25, U+2028 and U+2029 can be used as line comment terminators, but will cause an error if used as whitespace characters. This is fine.

[DecimalTurn]

After thinking about it some more and seeing the discusion in tailscale/hujson#43, we should also consider the option of treating LS and PS as invalid characters inside a single-line comment.

This way we avoid the security issue mentioned in #25 (comment), but also the other security issues of allowing something that looks like a single-line comment visually, but hides information/code.

Considering that VS Code doesn't support them either (ie. they render as � and don't cause a line break), banning them seems like a sensible approach.