[Hee-San]

Fixes #279. Downstream reports: ruby/psych#371, ruby/psych#508, glebm/i18n-tasks#291.

Problem

With yaml_emitter_set_unicode(emitter, 1), 3-byte UTF-8 characters are written as is, but 4-byte characters (U+10000–U+10FFFF, e.g. emoji) are still escaped. A single such character also forces the whole scalar into double quoted style:

$ cat in.yaml
sun: ☀
globe: 🌐
text: |
  hello 🌐
  world

$ ./tests/example-reformatter --unicode < in.yaml     # before
sun: ☀
globe: ! "\U0001F310"
text: "hello \U0001F310\nworld\n"

$ ./tests/example-reformatter --unicode < in.yaml     # after
sun: ☀
globe: 🌐
text: |
  hello 🌐
  world

Output with unicode = 0 (the default) is unchanged.

Cause

IS_PRINTABLE_AT implements c-printable only up to #xFFFD. The production includes [#x10000-#x10FFFF] in YAML 1.1 as well as 1.2:

[1] c-printable ::=   #x9 | #xA | #xD | [#x20-#x7E]            /* 8 bit */
                    | #x85 | [#xA0-#xD7FF] | [#xE000-#xFFFD]  /* 16 bit */
                    | [#x10000-#x10FFFF]                      /* 32 bit */

PyYAML's analyze_scalar has had this range since 2017 (yaml/pyyaml@cf1c86c).

Changes

1. Fix low surrogate in UTF-16 output (src/writer.c). The high byte of the low surrogate was masked with 0xFF instead of 0x03; U+1F310 came out as D83C CF10 instead of D83C DF10. This was unreachable until now, because characters above U+FFFF were always escaped. This commit comes first, because with only the macro change PyYAML's test suite fails in 6 tests with UnicodeDecodeError: illegal UTF-16 surrogate.
2. Add the 4-byte range to IS_PRINTABLE_AT (src/yaml_private.h), in the same style as the existing clauses. The upper bound is exact (F4 90 .. and above stay escaped), because yaml_check_utf8() does not reject sequences above U+10FFFF.
3. Add tests/test-emitter.c, registered for both CMake and autotools. It compares the emitted bytes for the range boundaries, both code paths (analyze_scalar and write_double_quoted_scalar), unicode = 0, and UTF-16LE/BE. The yaml-test-suite cannot cover this, since run-emitter-test-suite runs with unicode = 0.

No dependency is added and no character property tables are needed, because c-printable is defined by ranges. This is an alternative to #150, which solves the same problem using ICU.

Testing

• make test-all and the CMake tests pass with gcc and clang on Linux and macOS: https://github.com/Hee-San/libyaml/actions/runs/35680870522
• yaml-test-suite: 452 tests pass, no change in results
• Psych (master) built against this branch: test suite results identical to master (6 tests error on both, due to missing core test helpers in my setup); {"globe" => "🌐"}.to_yaml now gives globe: "🌐"
• PyYAML (main) built against this branch: 2616 tests pass