Repository navigation
[Python][C++] utf8_normalize ignores its form option: NFC and NFKC output is decomposed #51225
Description
Describe the bug
pyarrow.compute.utf8_normalize returns decomposed text for every form. With form="NFC" the
output is the NFD form (base letter plus combining mark), whether the input was precomposed or
decomposed. unicodedata.normalize from the standard library gives the expected result.
pyarrow 25.0.1, Python 3.13.9, macOS 26.6 (arm64), wheel from PyPI.
import pyarrow as pa, pyarrow.compute as pc, unicodedata
s = "héllo" # precomposed e-acute, U+00E9
a = pa.array([s], pa.string())
for form in ("NFC", "NFD", "NFKC", "NFKD"):
out = pc.utf8_normalize(a, form=form).to_pylist()[0]
exp = unicodedata.normalize(form, s)
print(form, [hex(ord(c)) for c in out], "expected", [hex(ord(c)) for c in exp])Output:
NFC ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0xe9', '0x6c', '0x6c', '0x6f']
NFD ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f']
NFKC ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0xe9', '0x6c', '0x6c', '0x6f']
NFKD ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f']
The same happens when the input is already decomposed ("héllo"): NFC and NFKC return it
unchanged instead of composing to U+00E9.
Expected behavior
form="NFC" and form="NFKC" compose, as unicodedata.normalize does; NFD and NFKD are correct
today.
Component(s)
Python, C++
Activity
Upstream tracker: utf8_normalize form option filed as apache/arrow#51225
singhpratech commentedon Sep 8, 2026
Reading the kernel on main (cpp/src/arrow/compute/kernels/scalar_string_utf8.cc,
Utf8NormalizeBase):MakeDecomposeOptionsreturnsUTF8PROC_COMPOSEfor NFC and NFKC, but the only utf8proc call thekernel makes is
utf8proc_decompose, which decomposes regardless of that flag; composition happensin
utf8proc_reencode, which is never called. So NFC and NFKC come out as NFD and NFKD. Callingutf8proc_reencodeon the scratch buffer with the same options after the decompose (asutf8proc_mapdoes internally) would give the composed forms.Santoshkumarpuppala commentedon Sep 8, 2026
Confirmed on
main(397b3d00). Worth knowing before anyone patches this: the test that covers it cannot fail.scalar_string_test.cc:1248declaresjson_composedas a literalá, but its bytes are61 cc 81—a+ U+0301, the decomposed form, not thec3 a1the comment on:1247promises.:1249writes those same three bytes as escapes, so both fixtures are the same string and the compose loop at:1250-1255is an identity assertion.Three consequences:
:1251fail. The patch has to change:1248to"\xc3\xa1"(escape form, as at:1265) or add a distinct composed fixture.:1263-1295is compatibility-axis only — U+00B2 and U+FDFA. Every assertion there passes under the bug, so:1250-1255is the whole compose coverage.python/pyarrow/tests/test_compute.py:3822-3827has the same hole: its only input is U+00B2, whose NFC is itself, soform="NFC"andform="NFD"assert against the same array.On the suggested fix —
utf8proc_reencodereturns a length in bytes and NUL-writes UTF-8 into the buffer, so it replaces the encode loop at:566-575rather than composing with it, and its documented one-spare-byte precondition isn't met on the regrow path, where:543/:545makecapacity() == resexactly.utf8proc_normalize_utf32is the drop-in:reencodeis that function plus the encoding step, and it leaves the existing loop untouched.Happy to open the PR with the fixture fix and a composed-form pyarrow case if nobody has started.
singhpratech commentedon Sep 8, 2026
Thank you for the fixture analysis, that is exactly the trap. I have a branch with the
utf8proc_normalize_utf32()call, the fixture fix and a pyarrow case ready and will open the PRnow; happy to have your review on it.
apacheGH-51225: [C++] utf8_normalize: add a Hangul pair to the compos…
apacheGH-51225: [C++] utf8_normalize: drop the utf8proc detail from t…
GH-51225: [C++] utf8_normalize: compose for NFC and NFKC (#51237)
pitrou commentedon Sep 10, 2026
Issue resolved by pull request 51237
#51237