perf(fast): pack candidates into u128 keys with spill-table fallback#60
Merged
Conversation
Candidate keys were Vec<GraphV> in every trainer map — a clone plus deep hash per touch, and (it turns out) the dominant memory cost, duplicated across global counts, index shards, and per-word candidate maps. CandKey = u128, gigatoken ShortPretokenCache-style: all-Node candidates with 2-4 elements totalling <= 13 bytes pack bijectively into the integer (tag + count + length nibbles + payload) — lock-free, table-free, and covering >99.5% of hot candidates (23 of 5000 chosen Boundless merges spill). The rest goes to a 16-shard intern table whose id embeds the candidate length so merge_score never touches the table. pair_key builds boundary-pair keys straight from two Node byte slices without a Vec. CandOp is now Copy; PickQueue's private intern tables are deleted (score comes from key bits); seq_recount emits through a callback. Full scale, cold subprocess per case, all digests unchanged: - 5000 merges: BPE 3.22->2.79s, BNE 4.89->3.43s, Boundless 13.11->8.83s, SuperBPE 16.74->13.88s - Peak RSS: BNE 1.31->0.74GB, Boundless 2.54->1.09GB, SuperBPE 2.12->0.95GB — the vector keys were the memory hogs all along. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #59 (merge that first, normal merge). Finishes the interning #59 started, gigatoken-ShortPretokenCache style.
Design: pack-first, spill-table fallback — packing won decisively
CandKey = u128: all-Node candidates with 2–4 elements totalling ≤13 bytes pack bijectively into the integer itself (tag + element count + length nibbles + payload) — no table, no locks, no growth;unpackreconstructs the exactVec<GraphV>only where a value is needed (pick winners, tie sets). Long/n-ary/non-Node candidates spill to a 16-shard intern table whose id embeds the candidate length, somerge_scorenever touches the table. Measured decider: only 23 of 5000 chosen Boundless merges spill (0.46%); a pure id-table design (#59's PickQueue tables, +130MB RSS) is strictly dominated and was deleted.Every trainer map is now integer-keyed — global counts, index shards, per-word/per-doc candidate maps, word-delta lists,
WordRegistry, ops (Copynow — bucketing moves 40-byte tuples instead of cloning vectors).Vec<GraphV>survives only at the intern/unpack boundary.Results (cold subprocess per case, full split; digests unchanged at all three scales)
The gate was "don't make RSS worse" — instead it halved or better the big cases: the vector keys duplicated across maps were the memory hogs all along. 500-merge and 2000-row digests all unchanged/matching; both suites (properly aliased) 145 passed, 2 skipped; clippy unchanged.
Remaining (ranked)
🤖 Generated with Claude Code