perf(fast): parallelize per-step apply, recount, and delta diff

[AmitMY]

Profiled the calibrated benchmark workload (2,000 rows / 100 merges). Phase timers showed all four cases route through train_entries_delta, and the connected loop's cost was apply+recount 2.8s (of it seq_recount 1.8s) + delta bookkeeping 0.9s — all serial. SuperBPE additionally spent 2.35s serially replaying its 50 phase-1 merges in make_trainer.

Rust-side only, per the project rule: the reference stays clean, fast/ gets the machinery. (A set of Python micro-optimizations from the same profiling round bought ~9% on connected mode but was dropped as not worth the added complexity in the reference.)

Changes (fast/src/trainer.rs)

1. The per-step affected-entry apply+recount loop runs via par_iter_mut (threshold 64 entries, so small steps skip rayon overhead).
2. The old-vs-new candidate diff is computed per entry in parallel; only the few real changes are applied serially to the shared global/index (updates are additive, and pick_best_entries reads index vectors order-independently).
3. Trainer.apply_merge parallelizes over subgraphs with memoized try_merge — SuperBPE's phase-1 replay hits this 50 times.

Results (benchmark defaults, before → after)

Case	fast (Rust)	vs reference
BPE	0.26s → 0.27s (unchanged)	2.0×
BNE n=4	0.54s → 0.49s	2.2×
Boundless BPE	4.16s → 1.40s (3.0×)	3.6×
Super BPE	4.27s → 1.08s (4.0×)	3.9×

Digests unchanged and identical to the reference at 2,000 and 500 samples (20d930a435 / 6598b73bb3 / 9999e69912). Both suites 141 passed, 2 skipped; clippy clean.

Tested and rejected during profiling

• Python incremental counting for connected multi-doc graphs: digests held but 5.0s → 9.5s (early merges touch nearly every doc; per-doc bookkeeping loses to the memoized C-level recount).
• Rust dedup/canonicalize and write-back loops: measured 78ms/60µs — left alone.

Ranked next steps (measured, not done)

1. Incremental seq_recount for connected mode: recount remains the largest piece of what's left; only children adjacent to a merge change.
2. Token interning to u32 ids: candidate keys are deep-hashed Vec<GraphV>; parallelism hid that cost rather than removing it.
3. Setup/pretokenization: ~0.1s at this scale — a full-scale-only lever.

🤖 Generated with Claude Code