Repository navigation
Field metadata for dictionary-encoded extension types is lost #7982
Description
Describe the bug
The representation of Dictionary in the data types enum seems to exclude field metadata, so extension types are dropped when they go through arrow-rs structures:
arrow-rs/arrow-schema/src/datatype.rs
Line 359 in a7f3ba8
The definition of RunEndEncoded and others seem to use a FieldRef and I'm wondering if it was a deliberate choice not to do this or whether it's just never come up.
To Reproduce
I used arro3 to reproduce:
import arro3.core as a3
import geoarrow.pyarrow as ga
import nanoarrow as na
import pyarrow as pa
c_schema = na.c_schema(pa.dictionary(pa.int32(), ga.wkb()))
c_schema.metadata is None
#> True
c_schema.dictionary.metadata
#> <nanoarrow._schema.SchemaMetadata>
#> - b'ARROW:extension:name': b'geoarrow.wkb'
#> - b'ARROW:extension:metadata': b'{}'
c_schema2 = na.c_schema(a3.DataType.dictionary(pa.int32(), ga.wkb()))
c_schema2.metadata is None
#> True
c_schema2.dictionary.metadata is None
#> TrueExpected behavior
I would have expected the metadata to roundtrip through the arrow-rs data type representation
Additional context
Occasionally Parquet readers will return dictionary-encoded arrays on read whose representation is not entirely in control of the user.
Activity
paleolimbot commentedon Apr 7, 2026
Just updating this while the details are fresh in my head from apache/arrow-nanoarrow#861 :
The IPC flatbuffers format doesn't allow for field metadata on the dictionary value type, which I'm guessing why it never made it to the DataType enum. Arrow C++ handles this ambiguity at IPC deserialization time via the extension registry (if the extension type rejects Dictionary as a storage type, value type gets the extension metadata instead and is then deserialized). Functionally there are no extension types that actually support dictionary storage (except maybe arrow.opaque, which can have any storage type but is usually binary), so the value type is almost always the thing that should have the extension. metadata.
Add `custom_metadata` to IPC `DictionaryEncoding`
Add `custom_metadata` to IPC `DictionaryEncoding`