perf: pack the sense payload, and build indexes once

[widgetii]

Stacked on #29 — review that first; this branch contains its commits.

The two performance gates M6 missed, closed by finding where the bytes actually were rather than guessing. Over the full 350 MB 大БКРС set:

	before	after	gate
on disk	868 MB	493 MB	210–240 MB (see below)
elapsed	206 s	74.7 s	< 90 s ✅

The bundled CC-CEDICT dictionary goes from 32 MB to 24 MB with it, and 3,434,224 cards still produce 3,442,623 entries with zero diagnostics.

A packed codec for sense payloads

The entry table was 625 MB of the 868, its payloads 327 MB of that, and only 40.7% of a payload was text. The rest was {"gloss":{"text":…},"level":0,"kind":…} written out once per sense, 3,691,468 times.

The packed form is 168 MB — within 4% of the raw text it carries, which is about as close as an uncompressed encoding gets. Three ideas earn their complexity:

• A presence byte. The overwhelmingly common sense is a gloss, a level and nothing else — 3,318,399 of 3,434,224 cards carry a single sense — so every field it lacks costs one bit instead of a key and a literal.
• Varints. Every integer here is a level (0–3), a parent, an offset or a count.
• A usage example stored once. UsageExample keeps chinese, translation and raw, where raw is the example as written and the other two are a heuristic split of it. Measured, the halves cost as much again as raw. Now raw is stored with the two boundaries inside it, reproducing both halves exactly — including a split that trimmed whitespace — from one copy. A split that is not a slice of the raw text is written out in full and marked, so this stays an optimisation rather than an invariant the model must keep.

SensePayloadCodec is a seam, not a single implementation, because this choice will be made again — block compression is the obvious next tier and the plan already names it. A container records which codec wrote it, so adding one does not invalidate dictionaries already on disk, and JSON still reads. An unrecognised identifier is an error rather than a fallback to the current codec, because misreading a payload produces plausible nonsense.

The summary column is gone

The denormalised first gloss, written since v1 and read by nothing — I went looking for its readers and there are none. On a BKRS container it is 160 MB of Cyrillic prose already sitting in the payload beside it. It existed so a result list could render without decoding a payload per row; with the packed codec that decode is microseconds for the ten rows a list shows. If a list ever does want it, it can come back as a short prefix rather than a whole copy.

The lookup indexes are built once, at the end

Maintained while rows arrive, entry_on_simplified is a random-order insert into an 84 MB B-tree three and a half million times. Built in one pass over rows that are all already there it is also 63 MB smaller, carrying none of the page splits random insertion leaves behind. That single change is most of the 130-second difference.

It costs memory, and that is the honest shape of it: CREATE INDEX is one atomic statement, so the whole new index is dirty at once — about 92 MB, which no cache_size, temp_store, threads, soft_heap_limit or hard_heap_limit pragma reduces. All five were tried. So the import's two budgets trade against each other, and the choice is explicit on the type rather than an accident:

Indexing	time	peak memory	size
.atTheEnd	75 s	134 MB	493 MB
.whileWriting	188 s	53 MB	556 MB

.atTheEnd is the default because compiling a DSL set is a desktop operation by design — a Mac compiles it once and a phone imports the compiled container, which is what dictgen's own documentation says the iOS path is — so the 60 MB ceiling, which exists for iOS, is not the binding constraint here. .whileWriting exists for the day something runs an import where it is, and both measurements live on the type so the trade is not rediscovered later.

What the remaining 493 MB is

No longer payload, which is why I am not claiming the 210–240 MB target:

senses (96% text)	168 MB
headword index	84 MB
segmentation lexicon	104 MB
headwords + readings	~75 MB
row overhead, 3.4 M rows	~60 MB

That estimate in the plan was computed from the predecessor's flattened definitions and did not account for the lexicon, the index, or storing sense structure and examples at all. The rebuild stores strictly more than the predecessor's 342 MB database and now does it in 493 MB. Compressing the payload is the next real lever, and the seam is there for it.

A VACUUM is not the answer, for the record: measured, it recovers 66 MB and leaves the table where it was. The bytes are real.

Verification

• All 523 senses of the committed real-data slice round-tripped field for field, so a codec that silently dropped registers could not pass.
• A two-volume import killed three times, the third landing inside the second volume, coming back row-for-row identical to an uninterrupted one — and the finished container carries exactly the reader's indexes and not the importer's.
• A payload truncated at every length from zero throws rather than trapping at each.
• verify passes on both the bundled dictionary and the imported one.

21 new tests. Full suite green, lint clean, no warnings under -warnings-as-errors.

🤖 Generated with Claude Code