GH-49704: [Format][C++] Add custom_metadata to DictionaryEncoding for the dictionary value type

[rustyconover]

Rationale for this change

In IPC a dictionary-encoded column is a single Field whose type is the value type. The only metadata slot was Field.custom_metadata, so dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> serialized identically. Arrow C++ reads both as the latter, so pyarrow cannot read back its own stream of pa.dictionary(pa.int32(), pa.uuid()): Invalid storage type for UuidType: dictionary<...>. The dev@ thread (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding KeyValue metadata for the value type to DictionaryEncoding (option 1).

This is a format change. It needs a [VOTE] on dev@ and two reference implementations, so this PR is a draft. Companion PRs:

• Go, the second reference implementation: feat(arrow/ipc): store dictionary value type metadata in DictionaryEncoding arrow-go#1381
• Rust, which reads the new member and maps it into Field metadata until Field metadata for dictionary-encoded extension types is lost arrow-rs#7982 is fixed: Add custom_metadata to IPC DictionaryEncoding arrow-rs#11622
• nanoarrow, which maps the member losslessly to ArrowSchema.dictionary->metadata: feat(ipc): Read and write DictionaryEncoding.custom_metadata for dictionary values arrow-nanoarrow#972

What changes are included in this PR?

• Format: an optional custom_metadata: [KeyValue] appended to DictionaryEncoding. It is the custom_metadata of the dictionary value type, with the same semantics as Field.custom_metadata (including the reserved ARROW:extension:* keys) but applied to the dictionary values. It is the IPC counterpart of ArrowSchema.dictionary->metadata in the C Data Interface.
  • Writers should put a value extension type's ARROW:extension:name/ARROW:extension:metadata there rather than in Field.custom_metadata.
  • Readers apply it to the value type before wrapping it in the dictionary type, then apply Field.custom_metadata as before.
  • C++ applies only the extension keys: its DictionaryType cannot hold value-type metadata, so other keys and unregistered extension names are not preserved. That is an implementation limitation, not a spec rule.
  • Format version 1.6, recorded in the Schema.fbs history and the Columnar, Versioning and Integration docs. MetadataVersion stays V5, and the file footer and Flight need no other change.
• Generated code: Schema_generated.h regenerated with flatc 24.3.6, limited to the new member.
• C++: the IPC writer and reader, and the integration JSON reader and writer ("dictionary": {..., "metadata": [...]}).
• Archery: a new dictionary_extension case, run for C++ only for now.
  • Go is skipped until the companion arrow-go PR lands, because integration CI builds arrow-go from main. With that PR, C++ ⇄ Go passes over IPC.
  • Rust is skipped because arrow-rs cannot represent a dictionary of an extension type (Field metadata for dictionary-encoded extension types is lost arrow-rs#7982). It reads the case correctly but cannot produce it, and archery has no producer-only skip.
• Not included: the Ruby flatbuffers bindings (ruby/red-arrow-format/.../dictionary_encoding.rb) were not regenerated, because no Ruby toolchain was available. They need rake flat_buffers:generate.
• Follow-ups:
  • DictionaryFieldMapper and DictionaryCollector (ipc/dictionary.cc) don't look through extension value types, so dictionaries nested inside extension dictionary values are not supported.
  • A Parquet ARROW:schema round-trip test.

Are these changes tested?

Yes.

• C++ IPC tests:
  • schema and record batch round trips, including nested types and the file footer;
  • the flatbuffer layout;
  • delta and replacement dictionaries of an extension value type;
  • the current C++ limitation that non-extension keys and unregistered extension names are dropped on read;
  • legacy field-level metadata, which behaves as before.
• Also: JSON integration tests, archery C++ IPC and C Data runs, and a pyarrow regression test.
• Cross-implementation, run manually: C++ ⇄ Go passes over IPC with the companion arrow-go PR, and arrow-rs reads the C++ and Go output in every format.

Are there any user-facing changes?

Yes:

• C++ and pyarrow now round-trip a dictionary whose value type is an extension type (e.g. pa.dictionary(pa.int32(), pa.uuid())) through IPC.
• The C++/pyarrow IPC output for such columns changes: the extension keys move from Field.custom_metadata to DictionaryEncoding.custom_metadata.
  • Readers that predate this change, including older C++/pyarrow, read such columns as a plain dictionary of the storage type and lose the extension type.
  • That includes types like arrow.opaque, which older readers used to keep, though with the wrong nesting (as an extension type with dictionary storage).
• Extension types whose storage type is a dictionary are serialized as before.
• There is deliberately no fallback for data written by existing writers. A field-level arrow.uuid on a dictionary still fails to read, as it does today.

Was AI used for this PR?

PR code and description written by:

• Human
• AI (Claude Code)

Reviewed before submission by:

• Human (maintainer)
• AI (AI reviewer)
• Not reviewed

• GitHub Issue: [C++][Format] Can't roundtrip Dictionary of ExtensionType over IPC #49704

🤖 Generated with Claude Code

[github-actions]

⚠️ GitHub issue #49704 has been automatically assigned in GitHub to PR creator.