yaml conformance tests - #4
Merged
Merged
Conversation
Vendors the community YAML Test Suite (351 files, 406 cases) and measures where
YL actually stands. This is a "see where we are" instrument, not a pass/fail
gate: very few YAML parsers reach 100%, and YL is not trying to — it reads YAML
AS JSON, so anything the JSON data model cannot express is out of scope by
construction rather than a bug.
That framing is what makes the suite usable for us: its `json` field is the JSON
a document is equivalent to, which is exactly our expectation. A case is lexed
twice — the YAML with YL, the expected JSON with the JSON lexer L — and
conformance means the two token streams are identical. Organised like the JSON
conformance test (buckets + xfail inventory + golden snapshot of the parts that
can only be observed).
Baseline
--------
accept + token stream matches the expected JSON 204 / 272 (75%)
invalid document rejected 85 / 94 (90%)
recorded only (no single-root JSON equivalent) 63
known divergences 54
The 54 divergences group into five causes, listed per case in
conformanceXFail and analysed in CONFORMANCE.md:
23 non-scalar mapping keys. Mostly keys that RESOLVE to a string (`*alias :
v`, `&a k : v`, `!!str k : v`) — the suite's own JSON shows the resolved
string as the key. A feature gap, not a scope boundary, and the biggest
single win available: it would take the accept rate to ~82%.
14 multiple documents. Out of scope until ND-JSON lands.
9 invalid documents we ACCEPT. The only group where we are wrong in the
dangerous direction; all inherited from goccy (flow collections, comment
placement, tabs) rather than from our walk.
5 accepted but the token stream differs — scalar resolution (!!binary,
non-specific tags, trailing whitespace). Subtlest group: nothing errors.
3 valid documents goccy will not parse at all.
Notes on the harness
--------------------
The suite renders invisible characters visibly and they MUST be translated
back — a fixture that hinges on a trailing space or a hard tab silently becomes
a different, usually valid, document otherwise. My first implementation matched
a fixed set of tab widths instead of upstream's /—*»/ and mangled any run of
four or more em-dashes; fixing it moved 7 cases from "divergent" to passing. The
rule is now a regexp and is documented in SOURCE.md.
The third bucket is not a weasel. A multi-document stream's `json` field holds
several values in sequence and an empty document's is empty; neither is a JSON
token stream, so there is nothing to compare and only the verdict is recorded.
testdata/conformance_recorded.golden pins those verdicts, so "out of scope"
cannot quietly become "unnoticed regression" — the same blind spot the JSON
suite had with its i_ cases, where ten documents carrying ill-formed UTF-8 were
accepted for months with the suite green.
Both drive modes (bytes and reader) run every case and must agree with each
other before either is compared to the suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Frederic BIDON <fredbi@yahoo.com>
The YAML Test Matrix (matrix.yaml.info) publishes a "JSON output" comparison over the same suite — is the loaded data equal to the `json` field — which is the same question our harness asks, so the numbers are meaningfully comparable. On its basis we score 289/366 = 79% (or 71% counting the 40 cases with no JSON equivalent as misses), which puts us in the same band as the mainstream loaders — PyYAML 75%, go-yaml 76%, ruby psych 76%, ruamel 78% — and clearly behind the four leaders: libfyaml 93%, JS yaml 92%, YAML::PP 92%, HsYAML 91%. Recorded with the caveats that stop it being a ranking: the matrix measures LOADERS (parse, serialize, compare) while we measure a lexer's token stream; we forfeit points by design on multi-document streams and complex keys; and our floor is goccy's, since 12 of our 54 divergences are its behavior rather than ours and goccy is not in the matrix at all (its `go-yaml` entry is a different project). The actionable part: closing group 1 alone — resolved-scalar mapping keys, 23 cases, one coherent feature — reaches 85%, above every implementation in the middle band. Groups 3 and 4 on top reach 89%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Frederic BIDON <fredbi@yahoo.com>
Conformance fixtures are DATA: their exact bytes are the test. A checkout with core.autocrlf=true (the Windows default) rewrites every LF to CRLF and silently changes what is being measured — a YAML fixture that hinges on a trailing space or a hard tab becomes a different document, and the failure looks like a lexer bug rather than a checkout artefact. The repo-wide fix already landed on master (.gitattributes marks testdata/**, *.golden and generated *.s as -text, so git never translates them). This is the other half, for the YAML harness: a suite that arrived some other way — an unmarked clone, a zip, a copy through a Windows tool — must still yield the documents upstream wrote, so the loader normalises CRLF to LF on read. Ordering matters and is the subtle part: normalisation runs BEFORE unescaping, so the suite's "←" (an intentional carriage return) is still the only way a CR enters a case. Normalising after would have eaten it. No vendored fixture contains a raw CR byte, which is what makes the normalisation safe. Tested as a property rather than assumed: the whole suite is copied to CRLF into a temp dir, re-loaded through the real loader, and every case asserted byte-identical to its LF original — that would catch a leak anywhere in the path, including inside the YAML parser the fixtures are read with. Plus unit coverage for the narrow contract (CRLF collapses, a lone CR survives) and a guard that the committed fixtures are still stored with LF. Mutation-checked: stubbing normalizeLineEndings to the identity fails the line-ending tests, so they have teeth. Signed-off-by: Frederic BIDON <fredbi@yahoo.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Frederic BIDON <fredbi@yahoo.com>
Signed-off-by: Frederic BIDON <fredbi@yahoo.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change type
Please select: 🆕 New feature or enhancement|🔧 Bug fix'|📃 Documentation update
Short description
Fixes
Full description
Checklist