Repository navigation
feat: add the DSL markup lexer - #24
Merged
Merged
Conversation
Layer three of the BKRS importer: turning a line of card body into text and markup. Pure, in the core, and validated against the real 350 MB set. Escapes are handled before tags, and that ordering is the whole design. DSL writes a literal bracket as `\[`, and the set contains 55,414 of them — a dictionary of Chinese grammar is full of bracketed glosses. The predecessor's `stripDSL` recognised tags first, so every `[` began a tag and everything to the next `]` was deleted: `\[обладать\]` lost the word inside it. Every "unknown tag" found while investigating that prototype was an escape artefact, and handling the backslash first makes the whole category vanish. Nothing is ever deleted. A span the lexer cannot identify is emitted as literal text *and* reported, because a dictionary that silently drops what it does not understand is worse than one that shows a stray bracket — the reader cannot tell "no entry here" from "the importer ate it". Running it over all three files found a bug worth the exercise. A **zero-width non-joiner binds to the character before it**, so `]` followed by U+200C is a single Swift `Character` that does not compare equal to `"]"`. A character-based scanner walks past the closing bracket and swallows the rest of the line into one unknown tag. Five tags in 13,739,066 tokens are written that way — a rate no hand-written fixture would ever reach. The lexer now scans unicode scalars, which is right anyway: markup is made of scalars, only text is made of characters. It is the same trap as CRLF being one grapheme cluster, which is what stops `split(separator: "\n")` splitting a CRLF file. Validated on the full set: 3,434,224 cards, 13,739,066 tokens, **zero** unknown tags and zero unterminated brackets, in 27 seconds. Two numbers from the plan are corrected by that scan. Escaped brackets are 55,414, not ~175k — 175,480 is the `[*]` tag count, which the plan appears to have conflated. And 62 `[*]` spans contain no `[ex]`, not 6; that path still must not crash, which is the card builder's problem. No BKRS data is committed. The plan called for a 200-card slice of the real dictionary as a fixture, which CONTRIBUTING forbids outright, so the fixtures are hand-written to cover every tag and the validation above runs against a local copy instead. The hand-written cases cannot prove the parser handles the real file; the full-set run is what proves that, and it is reproducible by anyone who has the dictionary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PR Summary by QodoAdd a lossless DSL markup lexer for BKRS card bodies
AI Description
Diagram
High-Level Assessment
Files changed (2)
|
Code Review by Qodo
1.
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
Three ways the lexer accepted things DSL never writes, each turning an unparsed fragment into valid-looking markup — which is the worst outcome available, because the result reads correctly and has silently dropped whatever it could not understand. `[/m1]` closed a sense. A sense ends with a bare `[/m]`; a levelled close is not one, and accepting it admitted markup this format does not produce while hiding it from the diagnostics. `[m٥]` opened a sense at level five. `Character.wholeNumberValue` answers for Arabic-Indic digits and Roman numerals too, so the level test now requires an ASCII digit. The accepted range is the format's 1–9 rather than the 1–4 BKRS happens to use: the lexer implements DSL, and which levels a dictionary uses is the card builder's business. `[b extra]` became bold with an argument. Colour is the only tag that carries one, so an argument anywhere else is now reported. All three are reported and kept as text, the same as any unknown tag. Re-running the full set confirms none of these forms occur in it: 3,434,224 cards, 13,739,066 tokens, still zero diagnostics. The stricter rules cost nothing against real data and close three holes against data that is not this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First pull request for M6, the BKRS importer. This is layer three of four: turning a line of card body into text and markup. Pure, in the core, validated against the real 350 MB set.
Escapes before tags, which is the whole design
DSL writes a literal bracket as
\[, and the BKRS set contains 55,414 of them — a dictionary of Chinese grammar is full of bracketed glosses.The predecessor's
stripDSLrecognised tags first: every[began a tag and everything to the next]was deleted, so\[обладать\]lost the word inside it. Every "unknown tag" found while investigating that prototype turned out to be an escape artefact, and handling the backslash first makes the whole category disappear.Nothing is deleted
A span the lexer cannot identify is emitted as literal text and reported. A dictionary that silently drops what it does not understand is worse than one showing a stray bracket: the reader cannot tell "no entry here" from "the importer ate it".
A real bug this found
A zero-width non-joiner binds to the character before it, so
]followed by U+200C is a single SwiftCharacterthat does not compare equal to"]". A character-based scanner walks past the closing bracket and swallows the rest of the line into one unknown tag.Five tags in 13,739,066 tokens are written that way — a rate no hand-written fixture would ever reach. The lexer now scans unicode scalars, which is correct anyway: markup is made of scalars, only text is made of characters.
Same trap as CRLF being one grapheme cluster, which is what stops
split(separator: "\n")splitting a CRLF file. That one was caught in M2; this is its sibling.Validation
Lexing all three files, start to finish:
Two corrections to the plan
The plan's figures came from an earlier survey; a full scan of this file set disagrees on two:
[*]tag count, which the plan appears to have conflated.[*]spans contain no[ex], not 6. That path still must not crash — the card builder's problem, carried forward.Confirmed unchanged: 3,434,224 cards, 77% with
_for pinyin, and the complete tag set is[m1]–[m4],[b],[i],[p],[c],[ex],[*],[ref]with nothing else.Also noted for the builder:
[mN]opens outnumber[/m]closes by 2 across 3.4 M cards, so it cannot assume balance.A deviation worth your approval
The plan called for committing a ~200-card slice of the real BKRS as a fixture.
CONTRIBUTING.mdforbids committing BKRS data outright, and the two instructions contradict each other.I chose the conservative reading: fixtures are hand-written to cover every tag and structure, and the full-set validation above runs against a local copy that is never committed. The hand-written cases cannot prove the parser handles the real file — the full-set run is what proves that, and anyone with the dictionary can reproduce it.
If you would rather have the real slice committed, say so and I will add it with an extraction script.
Next
Layer four, the card builder:
[mN]nesting on a level-indexed stack, examples, part-of-speech labels and style runs. Then the byte reader and#INCLUDEdiscovery, then the resumable pipeline.Local: 22 new tests, 421 total, lint clean, both app bundles build.
🤖 Generated with Claude Code