Skip to content

feat: apply write defaults in Parquet and Avro writers - #942

Open
huan233usc wants to merge 5 commits into
apache:mainfrom
huan233usc:feat/write-default-e2e
Open

huan233usc wants to merge 5 commits into
apache:mainfrom
huan233usc:feat/write-default-e2e

Conversation

@huan233usc

@huan233usc huan233usc commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Writing an Arrow batch with an older schema currently requires callers to materialize every column added by schema evolution. This PR lets Parquet and Avro writers align batches to the target schema and populate omitted columns with Iceberg write-default values.

For a newly added required column with initial-default = 42 and write-default = 7:

input schema:  required int id
write schema:  required int id, required int added
input rows:    [[1], [2]]
stored rows:   [[1, 7], [2, 7]]

The file is written after added exists, so the omitted column uses write-default, while initial-default remains a read-time rule for older files. This addresses the write-path portion of #637 and includes the work in #941.

  • Add optional input_schema to DataWriterOptions and WriterOptions. Omitting it defaults to the target schema. The Writer::Write interface stays unchanged; both writers always call the shared alignment helper, which forwards the original Arrow C Data batch unchanged when schemas match.
  • Reuse Project with write-default selection and disabled type promotion. Missing optional fields become null, missing required fields without a write-default fail, and explicitly supplied nulls stay null.
  • Align struct, list, and map children recursively by field ID. Preserve existing array buffers, offsets, and validity through ArrayData; materialize unknown inputs as null arrays of the target type. Reject unknown input field IDs at every nesting level.

Review pointers

Start with MissingColumnUsesWriteDefault and UnknownInputWritesTypedNulls in test/default_value_test.cc, which write and read through the public DataWriter for Parquet and both Avro backends. AlignBatchForWrite in arrow/arrow_c_data_util.cc handles Arrow materialization; ProjectionOptions in schema_util.h selects the read/write policy.

The alignment regressions cover forwarding identical schemas without import/export or changes to C Data ownership, existing collections when a scalar default is added, defaults and field reordering inside nested collections, sliced structs with parent nulls, unknown field IDs inside collections, and typed nulls. The new alignment tests fail against the previous implementation and pass with these changes.

Validation

Built all five test targets and ran their complete test binaries:

Suite Passed Skipped
data_test 193 0
schema_test 566 0
avro_test 238 0
parquet_test 110 1
manifest_test 200 18

All suites completed with zero failures. Skips cover codec availability and unsupported format/version cases. clang-format, clang-tidy on the five changed implementation files, and git diff --check passed.

@huan233usc
huan233usc force-pushed the feat/write-default-e2e branch 7 times, most recently from b824448 to ddb87cc Compare September 18, 2026 22:45
Extend Project with ProjectionOptions so the same field-id matching
serves both paths: reads fill missing fields with initial-default and
allow type promotion, writes fill with write-default and require exact
types.
Add arrow::AlignBatchForWrite which projects an input batch onto the
write schema via Project with write-default options: present columns
are reordered by field id (recursively for structs), missing columns
are materialized from write-default or null, and unknown input columns
are rejected rather than silently dropped.
DataWriter aligns batches to the write schema when the input schema
differs, via a new Write(data, input_schema) overload; schemas that
already match keep the zero-conversion Write(data) path. Adds Parquet
and Avro end-to-end tests where a column added with write-default
reads back the default for rows written without it.
@wgtmac

wgtmac commented Oct 9, 2026

Copy link
Copy Markdown
Member

Is this ready for review?

@huan233usc

Copy link
Copy Markdown
Contributor Author

I plan to do some final passes and then will send for review -- thanks

- Preserve sliced struct offsets and align collection children by field id.
- Materialize unknown inputs as typed nulls.
- Configure optional input schemas without extending the writer interface.
- Always align batches while retaining the declared or default input schema.
- Forward equal-schema batches without importing or exporting Arrow C Data.
@huan233usc
huan233usc marked this pull request as ready for review October 12, 2026 16:48

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants