perf(fast): full-scale round — index sets, changed-only registry, in-place apply, shared word deltas

[AmitMY]

The full-scale profiling round (entire wikitext-2 train split: 23.8k docs / 10.9M chars, 500 merges), extracted from the on-hold #55 branch so it can merge independently — all five changes target machinery that's already on main (#54's registry/premerge/incremental paths). Cherry-picks cleanly; verified from scratch on this base.

Changes (each was digest-verified in isolation on the profiling branch)

1. Inverted-index values Vec<u32> → FxHashSet<u32> — removal was iter().position(), a linear scan of up to ~23k doc ids, paid per affected doc per step.
2. Changed-only registry emission — the per-step scan emits only words that actually merge; staying in the registry is the known-unchanged signal (kills a serial ~74k-entry map rebuild × 500 steps).
3. apply_premerge rewrite — changed-only map, scan-first without cloning untouched docs, in-place slot writes via Arc::make_mut (SuperBPE's 50-merge phase-1 replay 4.65s → 1.58s on the profiling branch).
4. In-place incremental apply — try_apply_incremental collects (position, replacement) and writes in place instead of rebuilding each affected doc's children Vec (~2.4M Arc clones/step early on), with boundary deltas read through a small position→replacement overlay.
5. Shared per-step word deltas — each changed word's (candidate, ±1) list is computed once per step; docs combine the pre-built lists weighted by occurrence count. This was the sleeper: thread-summed profiling showed delta emission at 17.1s vs 4.5s for the scans everyone suspected.

Results on this base (main + this PR; cold subprocess per case, 500 merges, full split)

Case	main	PR	Δ	digest
BPE	3.54s	3.63s	flat	2af4c7edca ✓
BNE n=4	4.32s	4.46s	flat	967201e51c ✓
Boundless	11.46s	7.65s	1.5×	10318b5e1c ✓
SuperBPE	12.89s	9.31s	1.4×	2a6baff51d ✓

(On top of #55's batch ingestion the same changes measure 1.9×/1.7× — the two PRs compound.) Default 2000-row benchmark digests all match the reference (20d930a435 / 6598b73bb3 / 9999e69912 ×2). 141 tests + 2 skipped; clippy unchanged.

Cumulative full-scale history: Boundless 179s → 7.7s on main (23×), → 5.6s with #55 (32×); SuperBPE 216s → 9.3s / 5.1s (23× / 43×).

Note: full-scale digest methodology is cold-process-per-case because of #56 (training mutates the global cluster cache — pre-existing, fix coming separately).

Tried and reverted this round: serial map build for changed-words (no measurable change vs rayon reduce).

🤖 Generated with Claude Code