Repository navigation
[Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec #50622
Description
Describe the enhancement requested
The Primitive Type Mappings table in the Variant canonical extension is the normative list of Arrow types allowed for typed_value. The only place the table is referenced from is the typed_value storage rule, so its whole role is defining legal typed_value types:
The table appears to be derived from the Variant encoding types table (https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types), but its role corresponds to the shredding spec's Shredded Value Types table (https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types), which defines what a typed_value column may be. The two tables differ, so the Arrow table currently allows storage with no valid Parquet shredded representation:
Uint8 -> Int16,Uint16 -> Int32,Uint32 -> Int64: the shredding spec only permits signed integertyped_valuecolumns, so unsigned Arrow storage can never round trip (Arrow Uint -> Parquet INT -> Arrow Int). AUint8column also cannot represent the full value domain of its claimed variant type (int16), so it can only act as a partial storage that forces fallback tovalue. See the discussion in variant: Support Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10417 (issue: Parquet Variant rejects canonical Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10416).Null -> Null: the shredding table has no null row; a variant null must be encoded in thevaluecolumn, so aNulltyped_value column has no Parquet representation either.
In practice the unsigned path already produces non-conformant files: Arrow Go writes unsigned typed_value columns to Parquet with INT(8/16/32, false) annotations, which the shredding spec does not allow.
Proposal: re-derive the table from the shredding spec's Shredded Value Types table. Each row maps one variant type to its Parquet shredded type and the Arrow storage type(s) able to represent its full value domain. The table then defines reconstruction (a typed_value column of the listed Arrow type holds values of exactly the listed variant type) and the write-side lowering (the Parquet column is the only valid Parquet representation).
| Variant type | Parquet type | Arrow typed_value type |
|---|---|---|
| boolean | BOOLEAN | Boolean |
| int8 | INT32, INT(8, true) | Int8 |
| int16 | INT32, INT(16, true) | Int16 |
| int32 | INT32 | Int32 |
| int64 | INT64 | Int64 |
| float | FLOAT | Float32 |
| double | DOUBLE | Float64 |
| decimal4, 1 <= P <= 9, 0 <= S <= P | INT32, DECIMAL(P, S) | Decimal32(P, S) |
| decimal8, 10 <= P <= 18, 0 <= S <= P | INT64, DECIMAL(P, S) | Decimal64(P, S) |
| decimal16, 19 <= P <= 38, 0 <= S <= P | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S) | Decimal128(P, S) |
| date | INT32, DATE | Date32 |
| time | INT64, TIME(false, MICROS) | Time64(us) |
| timestamptz(6) | INT64, TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
| timestamptz(9) | INT64, TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
| timestampntz(6) | INT64, TIMESTAMP(false, MICROS) | Timestamp(us) |
| timestampntz(9) | INT64, TIMESTAMP(false, NANOS) | Timestamp(ns) |
| binary | BYTE_ARRAY | Binary / LargeBinary / BinaryView |
| string | BYTE_ARRAY, STRING | Utf8 / LargeUtf8 / Utf8View |
| uuid | FIXED_LEN_BYTE_ARRAY[len=16], UUID | UUID extension type |
The decimal precision bands follow the decimal table in VariantEncoding.md: precision alone selects the physical type (the bands are disjoint, so the narrowest type is required), and scale must satisfy 0 <= S <= P. P and S are shared across a row, so the bands constrain both the Parquet DECIMAL annotation and the Arrow type. Arrow otherwise permits negative scale and low-precision wide decimals, and Parquet permits both low-precision wide columns and BYTE_ARRAY decimals above precision 38.
This also resolves three existing ambiguities: the current Time64 row gains its required unit (variant time is micros only), the timestamp rows gain explicit units, and the decimal rows gain their precision bounds.
Related: #50620, apache/parquet-format#591
Component(s)
Format
Activity
[-][Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec[/-][+][Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec[/+]wgtmac commentedon Jul 24, 2026
I think this is possible. I'm not familiar with arrow-rs. For arrow-cpp, the Parquet writer serializes the Arrow schema into footer when the writer property
store_schemais enabled (which is the default value) and then Parquet reader deserializes that schema to restore their original form on read.This is a good argument. Parquet indeed has a UNKNOWN logical type to represent a column with no known type. It would be useful when user-hinted column cannot be shredded. So it seems like a missing piece in the Parquet shredding spec?
cc @zeroshade @alamb
zeroshade commentedon Jul 24, 2026
arrow-go does the same using the same property information and is thus compatible with arrow-cpp (and the impls based on it like pyarrow) writing the arrow schema into the footer.
Can you file this as an issue on arrow-go so we can track and fix this? We should definitely make sure that arrow-go is spec conformant
alamb commentedon Jul 24, 2026
Arrow-rs does the same thing
It looks to me like @sdf-jkl observation is that there are certain arrow types that can not be round tripped through a Parquet Variant column without some sort of additional metadata
Specifically something like
UInt8-->Variantmust be stored as Variant::Int16 (so the knowledge that the column only has UInt8 values is lost)As mentioned above, the normal Parquet logical types have the same problem, so the C++ writer adds extra metadata in the file so even though a value might be stored as Parquet type Int 16, Arrow readers know to try and read it as UInt8. This metadata convention seems to be commonly followed.
Maybe we can think about additional metadata writers could add if they want to recreate the original schema.
I am not sure how important this case is, though, because if you had data that was originally highly typed and structured you probably wouldn't be using Variant 🤔
sdf-jkl commentedon Jul 24, 2026
@alamb That's the point I was trying to make. Variant's physical types only include signed integers, same goes for the
typed_valuetype support in the Shredding spec.I am proposing to drop unsigned integers from the
typed_valueallowed types for Arrow since they always have to be written as signed.@wgtmac Same goes for the Null type, I'm proposing to drop it from the Arrow table rather than add a null row to the Parquet spec. A null in
typed_valuealready carries meaning: the row's value couldn't be shredded into the column type (or the field is missing, for object fields). An UNKNOWN column is always null, so it could never mark a row as shredded, and a Variant null would still have to be encoded invalueas00. Worse, for object fieldstyped_valuenull +valuenull means the field is missing, so shredding null intotyped_valuewould make "field present and null" indistinguishable from "field missing". I don't think the null row is a missing piece of the shredding spec, that state is reserved by the null-signaling protocol itself.@zeroshade before filing I want to pin the correct behavior. I think Arrow should not support unsigned int
typed_valueat all, which would make the fix different.alamb commentedon Jul 24, 2026
so is the proposed fix just an additional caveat to the spec that says typed_value has to be an arrow type that has a 1:1 mapping with the parquet types?
sdf-jkl commentedon Jul 24, 2026
Yes, that would be it
HuaHuaY commentedon Jul 27, 2026
So, is the final conclusion that we should update the Arrow documentation? How should we go about driving this forward?
HuaHuaY commentedon Jul 27, 2026
I’ve found another confusing issue. The Arrow specification maps
decimal32todecimal4,decimal64todecimal8,decimal128todecimal16. However, the Parquet documentation maps types based on decimal precision at https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types. Given the earlier discussion, I feel we should defer to the Parquet standard here as well?sdf-jkl commentedon Jul 27, 2026
@HuaHuaY yes, that makes sense. I updated the proposed table in the issue description, the precision bounds now constrain both the Parquet and the Arrow columns.
There is a related gap upstream: the shredding spec never states the decimal precision rules at all. The bands only appear in the Decimal table in VariantEncoding.md, while the decimal rows in VariantShredding.md are unbounded DECIMAL(P, S). So the Arrow table has to carry the bounds explicitly, and VariantShredding.md could use the same clarification or a cross-reference.
Update the Arrow doc and make an upstream Parquet PR to add decimal precision clarification/cross-reference to the shredding schema.
sdf-jkl commentedon Jul 29, 2026
@alamb @zeroshade @wgtmac Please confirm if you think this is the correct path and I'll start working on the Arrow spec PR and submit an issue in arrow-go.
sdf-jkl commentedon Aug 4, 2026
I filed a PR with the proposed changes - #50810
apacheGH-50622: Clarify Variant integer and null mappings
fix(arrow/extensions): reject Null Variant typed_value (#1243)
apacheGH-50622: [Docs][Format] Align Variant `typed_value` primitive …