[Parquet] Unsigned Variant typed_value materializes as an invalid unsigned Parquet shredded column #1202

Describe the bug, including details regarding any error messages, version, and platform.

Description

Arrow Go permits unsigned Arrow integer types as typed_value fields in the arrow.parquet.variant extension type. When such an array is written to Parquet, the unsigned Arrow type is materialized directly as an unsigned Parquet integer column.

For example, an Arrow Uint32 Variant typed_value produces:

optional int32 typed_value (Int(bitWidth=32, isSigned=false));

This is not a valid shredded Variant type according to the Parquet Variant shredding specification. Its supported integer representations are:

• int8 → INT32 / INT(8, true)
• int16 → INT32 / INT(16, true)
• int32 → INT32
• int64 → INT64

Unsigned Parquet integer logical types are not supported.

The current Arrow canonical Variant extension specification permits unsigned Arrow storage types through widening mappings:

• Uint8 → Variant Int16
• Uint16 → Variant Int32
• Uint32 → Variant Int64

Interoperability test

The parquet-testing interoperability suite includes case 127, which expects INTEGER(32,false) to be rejected:

{
  "case_number": 127,
  "test": "testUnsignedInteger",
  "parquet_file": "case-127.parquet",
  "error_message": "Unsupported shredded value type: INTEGER(32,false)"
}

Arrow Go currently skips that case with:

s.T().Skip("Skipping case 127: test says uint32 should error, we just upcast to int64")

The Parquet column is not upcast to int64. It is materialized as Arrow Uint32, retained as array.Uint32, and written back to Parquet as INT32 / INT(32,false).

Variant reconstruction may encode the value as a signed Variant integer, using Int64 where necessary, but that does not make the shredded Parquet schema valid.

Expected behavior

At minimum, Arrow Go should cast unsigned Arrow typed_value columns to their signed, widened Variant representations when writing Parquet:

• Uint8 → signed Int16
• Uint16 → signed Int32
• Uint32 → signed Int64

The resulting Parquet typed_value columns must use the corresponding signed shredded types rather than unsigned integer logical annotations.

The more complete resolution is to drop unsigned typed_value support from the Arrow Variant extension type altogether. If the Arrow specification change below is accepted, Arrow Go can reject unsigned Variant typed_value schemas directly, without requiring fine-grained conversions in the Parquet writer.

The interoperability test for case 127 should no longer be skipped. Its skip comment should also be removed or corrected because no Parquet schema upcast currently occurs.

Related Arrow specification change

The current Arrow canonical extension specification permits these unsigned mappings, creating a representation that cannot map directly to a conforming Parquet shredded schema.

The following PR proposes aligning Arrow’s Variant primitive mappings with the Parquet shredding specification by removing unsupported mappings such as unsigned integers and Null:

• GH-50622: [Docs][Format] Align Variant typed_value primitive type mappings with the Parquet shredding spec arrow#50810
• Related issue: [Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec arrow#50622

Relevant links

• Parquet Variant shredded-value types
• Current Arrow canonical Variant primitive mappings
• Proposed Arrow specification correction
• parquet-testing case 127
• Arrow Go skips case 127
• INTEGER(32,false) maps to Arrow Uint32
• The shredded reader retains array.Uint32
• Arrow Go’s Variant mappings include unsigned types
• The shredded builder accepts unsigned Arrow builders
• The writer emits unsigned Parquet annotations

Generated with codex and proof read

Component(s)

Parquet