Repository navigation
feat: read DSL files and group their lines into cards - #26
Merged
Merged
Conversation
Layers one and two of the importer: bytes to lines, lines to cards. With the lexer and the sense builder already in, this reads the real 350 MB set end to end — 3,434,224 cards, matching the known count exactly, in 2.3 seconds under 10 MB of resident memory. **The line terminator is a code unit, not a byte pair.** In UTF-16LE a line feed is `0x000A`, which is the bytes `0A 00` *at an even offset*. `ਅ昀` — U+0A05 then U+6600 — is `05 0A 00 66`, so a scanner that searches the raw bytes breaks the line between the two bytes of a single character. The predecessor scanned for `0A 00` with no alignment check and got away with it: across all three files there are 10,302,687 aligned line feeds and **zero** misaligned occurrences of the pair, because a Chinese-Russian dictionary contains no Gurmukhi. So this is not a fix for an observed failure — it is the difference between code that is correct and code that happens to work on one input, and the aligned scan costs nothing. A short read is looped over rather than trusted, so the same misalignment cannot arrive by that route either. **`pread`, not `FileHandle`, and that is a measured decision.** Reading 116 MB through `FileHandle.read(upToCount:)` took peak resident size to 124 MB — the size of the file — because Foundation maps a large read instead of copying it, so resident size follows the input. The import has a 60 MB ceiling. The same read through `pread` into a reusable buffer costs 1 MiB whatever the file size, which is the entire point of reading in chunks. **UTF-16 is transcoded by hand, which is 7× faster.** Over 3.4 million lines, gathering the code units into a `[UInt16]` costs 0.13 s and handing them to `String(decoding:as: UTF16.self)` costs a further 3.05 s; writing UTF-8 directly, so `String` is handed its own native storage, costs 0.43 s. Output is byte-identical — 85,997,869 UTF-8 bytes either way — and it takes reading the whole set from 10 seconds to 1.2. Line endings are mixed *within* each file: exactly two CRLF lines, `#NAME` and `#INDEX_LANGUAGE`, among three million LF ones. Both are handled on every line. And all three files end with a line feed, which splitting the text turns into an empty final element — an unindented empty line is indistinguishable from a headword, and that is a headword this dictionary does not contain. **The card reader is positional and nothing else.** An indented line is article text, an unindented `#` line is a directive, anything else unindented is a headword. Indentation is tested first, so an indented `#` stays article text. Consecutive headwords are one card: BKRS never does it, but refusing it would be a parser bug waiting for the next dictionary. Each card records the byte offset of its headword line, which is the only place an interrupted import can restart without losing a card or writing one twice. **The file set is discovered, never hardcoded.** `dabkrs_1.dsl` names the other two in its own `#INCLUDE` directives; the predecessor hardcoded all three filenames and the dictionary's name, so it imported this one dictionary and no other and stopped working if a file was renamed. Includes are authoritative and followed with a visited set so a cycle cannot loop; only when a file declares none does discovery look at the directory, and then only for a contiguous run of numbered siblings. Picking volume two restarts from volume one, because that is where the directives live. The seam repair the plan asks for is here as two properties rather than a third of an implementation: a headword left dangling at the end of a file is reported and handed out as `trailingHeadwords`, and a reader can be given `leadingHeadwords` to complete it. Both are zero for this set, which is card-aligned; they are insurance against a set someone split with `split(1)`. **A 203-card fixture of real data**, with an extraction script so its provenance is documented and it can be regenerated. Thirteen cards are picked deliberately — 爱, 了, 一 and 打 for their sense structure, one per piece of markup, and the single card whose article breaks a line inside an open `[m1]` — and 190 are a contiguous run, which covers the shapes nobody thought to pick. Committed as UTF-8 so it stays diffable; the tests re-encode it to UTF-16LE+BOM and assert both readings produce identical cards, so the bytes are reviewable and the real encoding path is still exercised. 52 KB, well inside the 256 KB limit CI enforces, and permitted by the one exception CONTRIBUTING names — 大БКРС is not ours to redistribute. `DSLDiagnostic` moves to its own file now that two layers produce them, and gains a byte offset for the ones found at card level. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
widgetii
enabled auto-merge (squash)
October 4, 2026 20:30
PR Summary by QodoStream DSL files into resumable cards and discover included volumes
AI Description
Diagram
High-Level Assessment
Files changed (13)
|
Code Review by Qodo
1.
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
Six correctness fixes raised in review, each measured against the full 3,434,224-card set. **A reading line is a convention, not a rule of the format.** The card reader took the first indented line as the reading unconditionally, so a DSL dictionary that writes no readings — the format does not require them — lost the first line of every article to a field it does not have, and `DSLCardBuilder` received a card with no senses in it. It is now detected, by a rule the whole set agrees with: the first indented line is the reading unless it contains a `[mN]` tag. **Zero of 3,434,224 reading lines do**, and an article's first line effectively always does. The test has to be the sense tag specifically rather than markup in general, because 287 reading lines *do* carry other markup — `lùzhóu, [i]разг.[/i] liùzhóu`, a register note between two readings — and a markup test would have read all 287 as articles. A single indented line is now article text rather than a reading. That case is genuinely ambiguous and the two mistakes are not equal: a reading with no definition is not an entry worth having, while a definition read as a reading leaves the entry saying nothing at all. No BKRS card has one, so nothing in this set changes. **A card completed across a file seam offered offset zero as its resume point**, which restarts at the next file's byte-order mark and loses the carried headword entirely. `DSLCardRecord.offset` is now optional and nil for exactly that card: no offset in the file it was completed in restarts it, and the one that does is in the previous file, which an importer already has. **Each dangling headword now reports its own line.** A run of them at the end of a file all carried the first one's offset, pointing every diagnostic but the first at the wrong place in the file — worse than no offset at all. **Discovery says when it stops at its file limit.** Returning 64 volumes of a 65-volume dictionary as though it were complete is the worst of the three possible behaviours: it imports, it looks right, and the missing volume's words simply cannot be found. Both the include walk and the numbered-sibling scan now report `tooManyFiles` instead. **A directive may be separated by a tab.** `#INCLUDE\t"b.dsl"` was read as an unknown directive, which leaves the include list empty and the other volumes undiscovered — the one failure that turns a three-volume dictionary into a third of one with no error at all. **A file that ends before the length it reported is refused.** `fill()` took a zero-byte read as end of file even when the file said there was more, so a source shortened between being measured and being read produced a dictionary quietly missing its tail — and in UTF-16, a final half-character dropped without the odd-length check ever running. Counts over the whole set are unchanged: 3,434,224 cards, 1 multi-line article, 0 trailing headwords, 0 diagnostics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`discover` was doing three things in one body — reading the header, walking the `#INCLUDE` graph, and scanning for numbered siblings — and the walk is the part with the loop, the visited set and the file limit in it. Moved to its own function, which is also what gets it back under the length the linter enforces. No behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layers one and two of the BKRS importer: bytes to lines, lines to cards, plus
#INCLUDEdiscovery. With the lexer and the sense builder already in, this reads the real 350 MB set end to end — 3,434,224 cards, matching the known count exactly, in 2.3 seconds under 10 MB of resident memory.The line terminator is a code unit, not a byte pair
In UTF-16LE a line feed is
0x000A— the bytes0A 00at an even offset.ਅ昀(U+0A05 then U+6600) is05 0A 00 66, so a scanner that searches the raw bytes breaks the line between the two bytes of a single character.The predecessor scanned for
0A 00with no alignment check, and got away with it: across all three files there are 10,302,687 aligned line feeds and zero misaligned occurrences of the pair, because a Chinese-Russian dictionary contains no Gurmukhi. So this is not a fix for an observed failure — it is the difference between code that is correct and code that happens to work on one input, and the aligned scan costs nothing. A short read is looped over rather than trusted, so the same misalignment cannot arrive by that route either.Two measured performance decisions
pread, notFileHandle. Reading 116 MB throughFileHandle.read(upToCount:)took peak resident size to 124 MB — the size of the file — because Foundation maps a large read instead of copying it, so resident size follows the input. The import has a 60 MB ceiling. The same read throughpreadinto a reusable buffer costs 1 MiB whatever the file size, which is the entire point of reading in chunks.UTF-16 is transcoded by hand, 7× faster. Over 3.4 M lines:
[UInt16]String(decoding:as: UTF16.self)String(decoding:as: UTF8.self)Output is byte-identical (85,997,869 UTF-8 bytes either way). It takes reading the whole set from 10 seconds to 1.2.
Line endings and the trailing newline
Line endings are mixed within each file: exactly two CRLF lines —
#NAMEand#INDEX_LANGUAGE— among three million LF ones. Both are handled on every line. And all three files end with a line feed, which splitting the text turns into an empty final element; an unindented empty line is indistinguishable from a headword, and that is a headword this dictionary does not contain.The card reader is positional and nothing else
An indented line is article text, an unindented
#line is a directive, anything else unindented is a headword. Indentation is tested first, so an indented#stays article text. Consecutive headwords are one card — BKRS never does it, but refusing it would be a parser bug waiting for the next dictionary. Each card records the byte offset of its headword line, which is the only place an interrupted import can restart without losing a card or writing one twice.The file set is discovered, never hardcoded
dabkrs_1.dslnames the other two in its own#INCLUDEdirectives. The predecessor hardcoded all three filenames and the dictionary's name, so it imported this one dictionary and no other and stopped working if a file was renamed.Includes are authoritative and followed with a visited set so a cycle cannot loop. Only when a file declares none does discovery look at the directory, and then only for a contiguous run of numbered siblings — a gap ends the run, and a file outside the run is imported alone, because importing files the reader did not pick needs a better reason than a coincidence of naming. Picking volume two restarts discovery from volume one, because that is where the directives live.
The seam repair the plan asks for is here as two properties rather than a third of an implementation: a headword left dangling at the end of a file is reported and handed out as
trailingHeadwords, and a reader can be givenleadingHeadwordsto complete it. Both are zero for this set, which is card-aligned; they are insurance against a set someone split withsplit(1).A 203-card fixture of real data
With
Scripts/extract-dsl-fixture.pyso its provenance is documented and it can be regenerated. Thirteen cards are picked deliberately — 爱, 了, 一 and 打 for their sense structure, one card per piece of markup, and the single card in 3,434,224 whose article breaks a line inside an open[m1]— and 190 are a contiguous run, which covers the shapes nobody thought to pick at a realistic density.Committed as UTF-8 so it stays diffable in review; the tests re-encode it to UTF-16LE+BOM and assert both readings produce identical cards, so the bytes are reviewable and the real encoding path is still exercised. 52 KB, well inside the 256 KB limit CI enforces, and permitted by the one exception
CONTRIBUTING.mdnames — 大БКРС is not ours to redistribute.Validation
73 new tests. Core suite at 308, lint clean, no warnings under
-warnings-as-errors, both app bundles build.🤖 Generated with Claude Code