feat(arrow/ipc): store dictionary value type metadata in DictionaryEncoding

[rustyconover]

Rationale for this change

Closes #1380. Part of apache/arrow#49704.

In IPC a dictionary-encoded field is one Field whose type is the value type, so dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> serialize identically. In arrow-go:

• a dictionary of arrow.uuid values is written with the UUID keys on the Field, and reading it back fails (invalid storage type for UUIDType: dictionary);
• writing the schema of an extension type with dictionary storage panics.

The dev@ discussion (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding custom_metadata: [KeyValue] to DictionaryEncoding for the dictionary value type's metadata.

Companion drafts: Rust apache/arrow-rs#11622, nanoarrow apache/arrow-nanoarrow#972.

This depends on the format change in apache/arrow#53994 (Schema.fbs, columnar spec, archery integration case), which is a draft pending a dev@ vote. It is the second reference implementation for that vote, so it should not be merged before the format change is accepted.

What changes are included in this PR?

• Generated code: arrow/internal/flatbuf/DictionaryEncoding.go regenerated (flatc 25.12.19) with the new DictionaryEncoding.custom_metadata member.
• IPC writer:
  • The ARROW:extension:name/ARROW:extension:metadata keys of a dictionary's extension value type are written to DictionaryEncoding.custom_metadata instead of Field.custom_metadata.
  • An extension type with dictionary storage keeps writing its keys to Field.custom_metadata, and its schema no longer panics on write.
• IPC reader:
  • Applies DictionaryEncoding.custom_metadata to the value type, wraps the result in the dictionary type, then applies Field.custom_metadata as before.
  • Only registered extension types are applied. Go's DictionaryType has nowhere to keep other value-type metadata, so other keys and unregistered extension names are dropped on read. That is a known limitation of this implementation; the format gives the member the same semantics as Field.custom_metadata.
  • Removing the extension keys from field metadata no longer mutates slices shared with the caller's arrow.Metadata.
• Integration JSON: an optional "metadata" list in a field's "dictionary" object. The reader also now honours "isOrdered".
• dictutils.Mapper: finds nested dictionaries inside extension value types.
• New arrow/internal/integration package: registers the canonical extension types. It is imported by arrow-json-integration-test, arrow-file-to-stream, arrow-stream-to-file and the Flight integration client and server. The C Data integration library doesn't import it, because its importer does not use the extension registry.

Are these changes tested?

Yes.

• IPC unit tests:
  • schema serialization, checking the keys sit in DictionaryEncoding and not Field;
  • stream and file data round trips of dictionaries of UUID and of an extension type with metadata;
  • extension types with dictionary storage, including a dictionary of extension values as storage;
  • a dictionary nested inside an extension value type;
  • delta dictionaries of dict<uuid>;
  • unregistered, non-extension and wrong-storage dictionary metadata on read;
  • legacy field-level metadata.
• Integration JSON round trips: dictionary metadata, opaque<dict<uuid>>, list<dict<uuid>> and ordered dictionaries.
• Archery, run manually:
  • C++ (GH-49704: [Format][C++] Add custom_metadata to DictionaryEncoding for the dictionary value type arrow#53994) ⇄ Go passes over IPC and C Data arrays for the extension, dictionary_extension, dictionary, nested_dictionary and extension_union cases.
  • Go's C schema import is skipped there, because the C Data importer does not use the extension registry.
  • Go-to-Go Flight integration passes for the same cases.

Are there any user-facing changes?

Yes:

• Changed IPC output: for a dictionary whose values are an extension type, the extension keys move from Field.custom_metadata to the new DictionaryEncoding.custom_metadata.
  • Such columns now round-trip as dictionary<extension<...>> in Go and in other implementations that support the new member.
  • Readers that predate the format change ignore the member and see a plain dictionary of the storage type. Previously they saw the keys on the field, which typically failed or produced an extension type with dictionary storage.
• Integration JSON format: gains an optional "metadata" key in the "dictionary" object.
• Integration programs: the IPC JSON integration, file/stream conversion and Flight integration client/server now register the canonical extension types.

This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer and an AI reviewer.

🤖 Generated with Claude Code