perf(fast): incremental candidate deltas, sharded counts, and a premerge registry (10× scale round)

[AmitMY]

Rust-only round at 10× the calibrated workload (20,000 rows / 100 merges), per the plan on #53. Profile first, then one measured change at a time; output byte-identical to the reference throughout (digests verified at 20k and 500 samples).

What 10× scale exposed that 2k rows hid

Case	20k before	Dominant costs
Boundless	13.15s	4.79s freeing old candidate maps at scope end (invisible until drop(changes) was timed explicitly), apply+recount 3.6s, diff 1.1s, serial delta apply 1.1s
SuperBPE	9.30s	2.35s replaying 50 phase-1 merges serially, phase-1 BPE 2.0s, setup ~1s ×2
BPE	1.93s	Python-side setup ~1.0s dominates — the Rust loop is small

Changes (fast/src/trainer.rs, + one pub(crate) in graph.rs)

1. Old candidate maps drop inside the parallel diff workers, not serially at scope end (Boundless train 11.6→7.1s just from this).
2. Initial recount deferred + parallel; a double recount removed.
3. try_apply_incremental — for minimal pair merges on a Seq with no top-level match, decrement/increment deltas are emitted directly from the changed children and their adjacent boundary pairs, patching candidates in place: no full recount, no full-map diff, nothing to drop. Any other shape falls back to the byte-identical old path.
4. Corpus-wide canonicalization + entry-local merge memo (a step-wide mutex-sharded memo measured slower — lock overhead beat the cross-doc dedup — and was rejected; three-way A/B in the report).
5. Global counts + inverted index sharded 16 ways by candidate hash; diff workers tag ops per shard, shards apply in parallel (safe: updates additive, zero only on final decrement, index reads via min()).
6. Premerge registry: Trainer.apply_merge and the training loop merge each unique word once in parallel and rebuild docs by pointer lookup mirroring seq_merge order exactly; unknown pointers fall back to direct try_merge, so the registry is purely an accelerator (SuperBPE's phase-1 replay 2.35→0.78s).

Results (20k rows, best of 3, digests = reference exactly)

Case	fast before	fast after	Δ	reference	fast vs reference
BPE	1.93s	1.81s	1.1×	2.40s	1.3×
BNE n=4	2.90s	2.17s	1.3×	5.21s	2.4×
Boundless	13.15s	3.56s	3.7×	68.3s	19×
SuperBPE	9.30s	4.77s	2.0×	42.9s	9×

Independently re-verified: digests 0684e7286c / d5af577a2f / 7178c58d28 / 7178c58d28 at 20k, all four matching the reference; 141 tests + 2 skipped; clippy warning set identical to main.

Ranked next steps (measured)

1. Python-side setup now dominates BPE and SuperBPE: 0.60s pretokenize + 0.39s units/object-crossing per _build_graphs at 1.19M words (SuperBPE pays twice). A Rust batch entry point taking pretokenized words for the non-streaming path would cut SuperBPE ~1.5–2s, BPE ~0.5s.
2. Serial shard bucketing (~0.2s recoverable).
3. Extending the incremental delta to n-ary windows (BNE) — intricate bookkeeping, moderate payoff.

🤖 Generated with Claude Code