Repository navigation
[C++][Format] Can't roundtrip Dictionary of ExtensionType over IPC #49704
Description
Describe the bug, including details regarding any error messages, version, and platform.
When implementing nanoarrow's IPC reader ( apache/arrow-nanoarrow#861 ) I was surprised at the interaction of dictionary encodings and extension types. My personal summary (which could use checking!) is:
- ExtensionType with dictionary storage roundtrips over the C Data interface AND IPC
- DictionaryType with extension value type roundtrips over the C Data interface but NOT IPC
Specifically for IPC, the DictionaryType with extension type values is exported identically to ExtensionType with dictionary storage (i.e., extension metadata at the top level). Because almost no extension type (except possibly arrow.opaque, which can support any extension type in theory) actually supports DicionaryType storage (i.e., will error when deserialized), this is a somewhat confusing default.
For now I will probably expose this as an option since I've implemented both deserialization pathways already (and the one that passes the integration tests is the least realistic version).
A related Rust issue is apache/arrow-rs#7982 (where the DataType is not capable of representing a DictionaryType with extension value type).
Details
import pyarrow as pa
import nanoarrow as na
class DictEncodedExtensionType(pa.ExtensionType):
def __init__(self):
storage_type = pa.dictionary(pa.int32(), pa.string())
super().__init__(storage_type, "example.dict_encoded_ext")
def __arrow_ext_serialize__(self):
return b""
@classmethod
def __arrow_ext_deserialize__(cls, storage_type, serialized):
assert isinstance(storage_type, pa.DictionaryType)
return cls()
# Register once per process
try:
pa.register_extension_type(DictEncodedExtensionType())
except pa.ArrowKeyError:
pass
# Create both cases as Arrow types
extension_dictionary_storage = DictEncodedExtensionType()
extension_dictionary_storage
#> DictEncodedExtensionType(DictionaryType(dictionary<values=string, indices=int32, ordered=0>))
dictionary_of_extension = pa.dictionary(pa.int32(), pa.uuid())
dictionary_of_extension
# DictionaryType(dictionary<values=extension<arrow.uuid>, indices=int32, ordered=0>)
na_extension_dictionary_storage = na.c_schema(extension_dictionary_storage)
na_extension_dictionary_storage
# <nanoarrow.c_schema.CSchema example.dict_encoded_ext{dictionary(int32)<string>}>
# - format: 'i'
# - name: ''
# - flags: 2
# - metadata:
# - b'ARROW:extension:name': b'example.dict_encoded_ext'
# - b'ARROW:extension:metadata': b''
# - dictionary: <nanoarrow.c_schema.CSchema string>
# - format: 'u'
# - name: ''
# - flags: 2
# - metadata: NULL
# - dictionary: NULL
# - children[0]:
# - children[0]:
na_dictionary_of_extension = na.c_schema(dictionary_of_extension)
na_dictionary_of_extension
# <nanoarrow.c_schema.CSchema dictionary(int32)<arrow.uuid{fixed_size_binary(16)}>>
# - format: 'i'
# - name: ''
# - flags: 2
# - metadata: NULL
# - dictionary: <nanoarrow.c_schema.CSchema arrow.uuid{fixed_size_binary(16)}>
# - format: 'w:16'
# - name: ''
# - flags: 2
# - metadata:
# - b'ARROW:extension:name': b'arrow.uuid'
# - b'ARROW:extension:metadata': b''
# - dictionary: NULL
# - children[0]:
# - children[0]:
# Can roundtrip both over C Data Interface
pa.DataType._import_from_c_capsule(na_extension_dictionary_storage.__arrow_c_schema__())
# DictEncodedExtensionType(DictionaryType(dictionary<values=string, indices=int32, ordered=0>))
pa.DataType._import_from_c_capsule(na_dictionary_of_extension.__arrow_c_schema__())
# DictionaryType(dictionary<values=extension<arrow.uuid>, indices=int32, ordered=0>)
schema = pa.schema(
{
"extension_dictionary_storage": extension_dictionary_storage,
"dictionary_of_extension": dictionary_of_extension,
}
)
schema
# extension_dictionary_storage: extension<example.dict_encoded_ext<DictEncodedExtensionType>>
# dictionary_of_extension: dictionary<values=extension<arrow.uuid>, indices=int32, ordered=0>
schema_bytes = schema.serialize()
with pa.ipc.open_stream(schema_bytes) as s:
print(s.schema)
# ArrowInvalid: Invalid storage type for UuidType: dictionary<values=fixed_size_binary[16], indices=int32, ordered=0>Component(s)
Format, C++
Activity
[-]Can't roundtrip Dictionary of ExtensionType over IPC[/-][+][C++] Can't roundtrip Dictionary of ExtensionType over IPC[/+][-][C++] Can't roundtrip Dictionary of ExtensionType over IPC[/-][+][C++][Format] Can't roundtrip Dictionary of ExtensionType over IPC[/+]pitrou commentedon Apr 14, 2026
Ongoing discussion at https://lists.apache.org/thread/b3qn9o88wbsgysslzswc9bmhc5mrz7l3
pitrou commentedon Jun 11, 2026
It seems we reached a lazy consensus in https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz , do you want to work on the proposed resolution @paleolimbot ?
paleolimbot commentedon Jun 12, 2026
I do but need to carve out time to make the changes, which took longer than expected 😬
rustyconover commentedon Oct 16, 2026
We hit this in DuckDB's nanoarrow extension (duckdb/duckdb-nanoarrow#129), which is adding dictionary-encoded columns to its Arrow IPC stream writer. DuckDB's C Data export puts
arrow.uuidon the dictionary's value schema (dictionary<values=extension<arrow.uuid>>). nanoarrow's IPC encoder only writes the index field's metadata, so the extension name is dropped. Moving it to theField, as Arrow C++ does, gives the ambiguous form described above, which pyarrow 26 can't read back (Invalid storage type for UuidType: dictionary<values=fixed_size_binary[16], indices=int32, ordered=0>). For now we reject dictionary encoding for columns written as extension types.We'd like to help get option 1 from the dev@ thread implemented: an optional
custom_metadata: [KeyValue]onDictionaryEncodingthat holds the value type's metadata. @paleolimbot, would you welcome help with this, or have you already started? Since a format change needs two reference implementations with integration tests, we could put together draft PRs for theSchema.fbsand Columnar docs change, with C++, Go (arrow-go) and Rust (arrow-rs) implementations and an archery integration case. We'd follow your lead on naming and semantics, and don't want to duplicate work you have in progress.The semantics we'd propose for the drafts:
ARROW:extension:nameandARROW:extension:metadata) inDictionaryEncoding.custom_metadata, and keepField.custom_metadatafor the field's own type, i.e. an extension type with dictionary storage.DictionaryEncoding.custom_metadatato the value type before wrapping it in the dictionary type, then applyField.custom_metadatato the result as they do today. A reader that doesn't know the new member still getsdictionary<storage type>, which is the graceful degradation Antoine described.paleolimbot commentedon Oct 16, 2026
I haven't started but would be happy to review, and happy to implement in nanoarrow as a "second implementation", if we count 🙂
rustyconover commentedon Oct 16, 2026
Draft PRs for option 1 are up:
Schema.fbs, Columnar/Versioning/Integration docs, archerydictionary_extensioncase): GH-49704: [Format][C++] Add custom_metadata to DictionaryEncoding for the dictionary value type #53994custom_metadatato IPCDictionaryEncodingarrow-rs#11622They add
custom_metadata: [KeyValue]at the end ofDictionaryEncoding(format 1.6,MetadataVersionstays V5). Writers should put a value extension type's keys there. Readers apply it to the value type before the dictionary wrap, then applyField.custom_metadataas before. Readers that predate the change seedictionary<storage>. With the drafts, C++ ⇄ Go pass archery over IPC and C Data arrays. arrow-rs reads both correctly, but can't produce the case, because it can't represent a dictionary of an extension type (apache/arrow-rs#7982). Until it can, it maps the value metadata into the field's metadata when the field has no extension of its own.One point we'd like feedback on before a vote: what readers should do with value metadata they can't apply. C++ and Go can only attach a registered extension type to a dictionary's values. Their
DictionaryTypehas nowhere to keep other keys or unregistered extension names, so those are dropped on read, and pass-through tools (Flight servers, file↔stream converters) lose them. At the field level, the same keys survive inField.custom_metadata. The draft spec text currently says:Is that acceptable, or should the spec restrict the member to the
ARROW:extension:*keys, or require more? We also deliberately left out a reader fallback for files written by existing writers (field-levelarrow.uuidon a dictionary still fails to read, as today), to keep the old form unambiguous. Happy to revisit that if people prefer it.@paleolimbot, nanoarrow would need the same encoder/decoder change once the format lands. Happy to help there too.
rustyconover commentedon Oct 16, 2026
nanoarrow draft: apache/arrow-nanoarrow#972. Because nanoarrow works on
ArrowSchemadirectly, it mapsDictionaryEncoding.custom_metadatalosslessly toArrowSchema.dictionary->metadata, all keys included. With the new archery case enabled for nanoarrow, C Data passes in both directions between C++ (#53994) and nanoarrow. nanoarrow's IPC writer still has existing dictionary gaps (apache/arrow-nanoarrow#622), so its IPC integration stays skipped for dictionary cases as before.While testing this we also found that nanoarrow's IPC encoder truncated all metadata keys and values at NUL bytes. That has a separate fix: apache/arrow-nanoarrow#971.
apacheGH-49704: [Format][C++] Add custom_metadata to DictionaryEncodi…
rustyconover commentedon Oct 16, 2026
Update on the open question in my earlier comment: we've dropped it. The draft spec in #53994 now says
DictionaryEncoding.custom_metadatasimply has the same semantics asField.custom_metadata, including the reservedARROW:extension:*keys, applied to the dictionary values instead of the field. There are no special rules about ignoring or losing keys.C++ and Go applying only registered extension types and dropping other value-type metadata is an implementation limitation of their dictionary types, documented as such in their PRs. nanoarrow keeps all keys. arrow-rs maps them into field metadata until apache/arrow-rs#7982 is resolved.
feat(ipc): Read and write DictionaryEncoding.custom_metadata for dict…