Found while benchmarking at full scale (wikitext-2, 500 merges): the fast implementation's training mutates the global cluster cache (apply_merge_to_cluster_cache in fast/src/units.rs), so the second training run in the same process starts from mutated multi-byte-cluster graphs and can produce different merges than the first (e.g. full-scale BPE cold 2af4c7edca vs warm 2a0889a340). Deterministic per cache state, invisible at small scales/merge counts.
The reference implementation has no such statefulness — training the same tokenizer twice in one process yields identical results — so this is a divergence from the source-of-truth contract, and it silently affects any benchmark or user code that trains more than once per process (e.g. SuperBPE's two phases are only correct because they intend to share the cache).
Likely fix direction: make the cluster cache immutable during training (snapshot per trainer, as the streaming path already does with snapshot_word_cache/replace_word_cache), or key it per-trainer rather than global.
All current benchmark methodology (bench_tokenizers.py subprocess-per-case, and the PR #55 numbers) uses cold processes, so published numbers are unaffected.
🤖 Generated with Claude Code
Found while benchmarking at full scale (wikitext-2, 500 merges): the fast implementation's training mutates the global cluster cache (
apply_merge_to_cluster_cacheinfast/src/units.rs), so the second training run in the same process starts from mutated multi-byte-cluster graphs and can produce different merges than the first (e.g. full-scale BPE cold2af4c7edcavs warm2a0889a340). Deterministic per cache state, invisible at small scales/merge counts.The reference implementation has no such statefulness — training the same tokenizer twice in one process yields identical results — so this is a divergence from the source-of-truth contract, and it silently affects any benchmark or user code that trains more than once per process (e.g. SuperBPE's two phases are only correct because they intend to share the cache).
Likely fix direction: make the cluster cache immutable during training (snapshot per trainer, as the streaming path already does with
snapshot_word_cache/replace_word_cache), or key it per-trainer rather than global.All current benchmark methodology (
bench_tokenizers.pysubprocess-per-case, and the PR #55 numbers) uses cold processes, so published numbers are unaffected.🤖 Generated with Claude Code