Repository navigation
feat: apply write defaults in Parquet and Avro writers - #942
Open
huan233usc wants to merge 5 commits into
Open
huan233usc wants to merge 5 commits into
huan233usc wants to merge 5 commits into
Conversation
huan233usc
force-pushed
the
feat/write-default-e2e
branch
7 times, most recently
from
September 18, 2026 22:45
b824448 to
ddb87cc
Compare
Extend Project with ProjectionOptions so the same field-id matching serves both paths: reads fill missing fields with initial-default and allow type promotion, writes fill with write-default and require exact types.
Add arrow::AlignBatchForWrite which projects an input batch onto the write schema via Project with write-default options: present columns are reordered by field id (recursively for structs), missing columns are materialized from write-default or null, and unknown input columns are rejected rather than silently dropped.
DataWriter aligns batches to the write schema when the input schema differs, via a new Write(data, input_schema) overload; schemas that already match keep the zero-conversion Write(data) path. Adds Parquet and Avro end-to-end tests where a column added with write-default reads back the default for rows written without it.
huan233usc
force-pushed
the
feat/write-default-e2e
branch
from
September 18, 2026 22:57
ddb87cc to
8e01c0e
Compare
14 of 21 tasks
8 tasks
Member
|
Is this ready for review? |
Contributor
Author
|
I plan to do some final passes and then will send for review -- thanks |
- Preserve sliced struct offsets and align collection children by field id. - Materialize unknown inputs as typed nulls. - Configure optional input schemas without extending the writer interface.
- Always align batches while retaining the declared or default input schema. - Forward equal-schema batches without importing or exporting Arrow C Data.
huan233usc
marked this pull request as ready for review
October 12, 2026 16:48
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Writing an Arrow batch with an older schema currently requires callers to materialize every column added by schema evolution. This PR lets Parquet and Avro writers align batches to the target schema and populate omitted columns with Iceberg
write-defaultvalues.For a newly added required column with
initial-default = 42andwrite-default = 7:The file is written after
addedexists, so the omitted column useswrite-default, whileinitial-defaultremains a read-time rule for older files. This addresses the write-path portion of #637 and includes the work in #941.input_schematoDataWriterOptionsandWriterOptions. Omitting it defaults to the target schema. TheWriter::Writeinterface stays unchanged; both writers always call the shared alignment helper, which forwards the original Arrow C Data batch unchanged when schemas match.Projectwith write-default selection and disabled type promotion. Missing optional fields become null, missing required fields without a write-default fail, and explicitly supplied nulls stay null.ArrayData; materialize unknown inputs as null arrays of the target type. Reject unknown input field IDs at every nesting level.Review pointers
Start with
MissingColumnUsesWriteDefaultandUnknownInputWritesTypedNullsintest/default_value_test.cc, which write and read through the publicDataWriterfor Parquet and both Avro backends.AlignBatchForWriteinarrow/arrow_c_data_util.cchandles Arrow materialization;ProjectionOptionsinschema_util.hselects the read/write policy.The alignment regressions cover forwarding identical schemas without import/export or changes to C Data ownership, existing collections when a scalar default is added, defaults and field reordering inside nested collections, sliced structs with parent nulls, unknown field IDs inside collections, and typed nulls. The new alignment tests fail against the previous implementation and pass with these changes.
Validation
Built all five test targets and ran their complete test binaries:
All suites completed with zero failures. Skips cover codec availability and unsupported format/version cases.
clang-format,clang-tidyon the five changed implementation files, andgit diff --checkpassed.