Merges round-trip through lossy UTF-8 strings, silently dropping non-aligned merges in SuperBPE phase 2

[AmitMY]

Found while attempting phase-1 trained-state reuse for SuperBPE (#61): the phase-1 → phase-2 handoff carries merges as strings (get_merges → add_merges, str(node) decoding with errors='replace'). Any merge whose tokens are not UTF-8-aligned — e.g. the (0xE2, 0x80) byte pair learned inside an en-dash — round-trips through U+FFFD, and the replayed merge (EF BF BD, EF BF BD) matches nothing: the merge is silently dropped from phase-2's starting state.

Minimal example: after phase 1 learns (e2, 80), the word " –":

• true trained state: Seq[' ', Seq[Node(e2 80), Node(93)]]
• state after string round-trip replay: Seq[' ', Seq[e2, 80, 93]] (merge lost)

Both implementations behave identically — this is the canonical semantics of the current API, and all published digests embody it. At real scale phase 1 always learns non-UTF8-aligned pairs, so SuperBPE's phase 2 systematically starts from a weaker state than phase 1 actually reached (and any user persisting merges via get_merges and reloading via add_merges loses the same information).

Decision needed (output-changing, both implementations together per the dual-impl rule):

1. Keep strings as the API but make the encoding lossless (e.g. latin-1/byte-escape representation), or
2. Carry merges as bytes end-to-end, with strings only for display, or
3. Document the lossiness as intended.

Telling detail: the pre-#58 SuperBPE digests were accidentally closer to true-state semantics because the cluster-cache mutation bug leaked applied merges into phase-2 builds — i.e., two bugs were partially masking each other.

🤖 Generated with Claude Code