`lookup_compound(transfer_casing=True)` returns invalid UTF-8 / uninitialised bytes

[chrisjbryant]

Hi there. I've had this small issue on my plate for a while, and finally got around to diving into it.

It's not very common, but has been responsible for some crashes in our logs, so I figured I'd share the AI report that will allow you to reproduce.
Hope it helps!

Summary

With transfer_casing=True, lookup_compound returns a std::string containing bytes
that are not valid UTF-8. pybind11 decodes the result as UTF-8 on the way back to Python,
so accessing .term raises UnicodeDecodeError.

Input and dictionary are pure ASCII, so the bad bytes originate inside the casing-transfer
step. Reproduces on this repo's own frequency_dictionary_en_82_765.txt with ordinary
lowercase typos. Deterministic.

Reproduction (realistic — uses this repo's bundled dictionary)

import SymSpellCppPy

sym = SymSpellCppPy.SymSpell()
sym.load_dictionary(
    corpus="resources/frequency_dictionary_en_82_765.txt",
    term_index=0, count_index=1, separator=" ",
)

for tok in ["utuxgent", "spswattle", "forgertrsts", "rollelicy", "cicuris"]:
    ok = sym.lookup_compound(tok, max_edit_distance=1)[0].term
    print(f"{tok!r:14} transfer_casing=False -> {ok!r}")
    try:
        bad = sym.lookup_compound(tok, max_edit_distance=1, transfer_casing=True)[0].term
        print(f"{tok!r:14} transfer_casing=True  -> {bad!r}")
    except UnicodeDecodeError as e:
        print(f"{tok!r:14} transfer_casing=True  -> UnicodeDecodeError: {e}")

Actual

'utuxgent'     transfer_casing=False -> 'it urgent'
'utuxgent'     transfer_casing=True  -> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 6: invalid start byte
'spswattle'    transfer_casing=False -> 'up seattle'
'spswattle'    transfer_casing=True  -> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 6: invalid start byte
'forgertrsts'  transfer_casing=False -> 'forget rats'
'forgertrsts'  transfer_casing=True  -> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 10: invalid start byte

Each of these crashed 10/10 runs. The position is stable per token, but the invalid
byte value varies between processes — the same script rerun gave 0xee ("invalid
continuation byte") for all five tokens instead of 0xdf.

Expected

A valid str, as transfer_casing=False returns. Both input and dictionary are ASCII, so
there is no multi-byte sequence anywhere in the data to be mishandled.

Minimal reproduction (single-word dictionary, no data files)

The bug does not need a real dictionary at all — one entry is enough:

import SymSpellCppPy

sym = SymSpellCppPy.SymSpell()
sym.create_dictionary_entry("a", 1000000)

# raises UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 4
sym.lookup_compound(">^v<G", max_edit_distance=1, transfer_casing=True)[0].term

Secondary symptom: silent corruption

The same code path also injects NUL bytes into results that do not raise, so callers
receive corrupted strings with no error at all:

tok="%'?Y`Hi"      ed=1 -> 'a a \x00Hi'
tok='8Qp>=U'       ed=0 -> '8Qp \x00U'
tok=']j:5(!-vF4f&' ed=0 -> ']j 5 \x00-vF4f'

This is arguably worse than the exception, since it is undetectable downstream.

Evidence this is uninitialised memory

• The invalid byte varies between processes for the same input, while the position
  stays fixed: ">^v<G" gave 0xfd then 0xff; the five tokens above gave 0xdf on one
  run and 0xee on the next.
• Across fuzzing, 13 distinct invalid bytes appeared: 0x80 0x90 0x97 0xc8 0xc9 0xca 0xcb 0xcc 0xe0 0xee 0xf6 0xf9 0xff.
• Input and dictionary are 100% ASCII, so the data contains no source of non-ASCII bytes.

This points at a buffer sizing / bounds issue in the casing-transfer routine, rather than
any UTF-8 handling problem.

Trigger conditions

• Requires transfer_casing=True. With it omitted/False, 240,000 calls across the same
  input distributions produced zero crashes and zero corruption.
• Overwhelmingly at max_edit_distance=1. At max_edit_distance=0 with alphabetic-only
  input, 180,000 probes were clean.
• Rate depends on input character class: roughly 1 in 10,000 plain alphabetic tokens
  against frequency_dictionary_en_82_765.txt, rising to ~11.6% for tokens containing
  punctuation.

Environment

SymSpellCppPy	0.0.18
Python	3.12.13
Platform	Linux-6.8.0-1029-gcp-x86_64-with-glibc2.35

Workaround

Pass transfer_casing=False and reapply casing in Python. Note that a naive
character-positional casing transfer is not equivalent — it drifts when the correction
inserts or deletes a character (e.g. ofcMultiple -> of multiple maps M onto u,
giving of mUltiple), so a difflib-style alignment is needed to match current behaviour.