GH-51220: [Python][Parquet] Accept write_table options in dataset writer - #51461
Conversation
|
|
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Append the new parameter to preserve existing positional ParquetWriter compatibility.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
What changed in this PR
Adds support for additional Parquet write options in dataset writers and ParquetWriter.
Changes:
- Supports
bloom_filter_options,store_schema, andwrite_time_adjusted_to_utc. - Adds tests for option acceptance and behavior.
- Exposes Bloom filter options through
ParquetWriter.
| File | Summary |
|---|---|
python/pyarrow/tests/parquet/test_dataset.py |
Tests new dataset writer options. |
python/pyarrow/parquet/core.py |
Adds Bloom filter option support to ParquetWriter. |
python/pyarrow/_dataset_parquet.pyx |
Registers and propagates new writer options. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
587a8ff to
b951a92
Compare
|
@github-actions crossbow submit preview-docs |
|
Revision: b951a92 Submitted crossbow builds: ursacomputing/crossbow @ actions-d493102f4e
|
|
So the new preview-docs pyarrow.parquet.ParquetWriter signature now explicitly includes: (not listed in 25.0 : https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html) |
|
fyi @jorisvandenbossche, if you'd also like to check this, especially whether passing |
rok
left a comment
There was a problem hiding this comment.
One minor comment but good to merge already. We should merge this before release freezes in argument order!
|
@AlenkaF, do you have time to do a pass here? Otherwise I'd merge tomorrow to get this into the release. |

Rationale for this change
Fix #51220 - Parquet
write_tableoptions are not all accepted in datasetParquetFileWriteOptionssopyarrow.parquet.write_to_dataset( ...)raisesTypeError: unexpected parquet write option:for
bloom_filter_options,write_time_adjusted_to_utcorstore_schema(Similar to past #37469)
What changes are included in this PR?
Add
bloom_filter_options,store_schema,write_time_adjusted_to_utctoParquetFileWriteOptions,also explicitly expose
bloom_filter_optionsinpyarrow.parquet.ParquetWriter.Test writer options in
make_write_options, and test three newly supported options inwrite_to_datasetAre these changes tested?
Yes. CI also passes.
Are there any user-facing changes?
Yes, users may now use
bloom_filter_options,write_time_adjusted_to_utcandstore_schemain
pyarrow.parquet.write_to_dataset(...)and inParquetFileFormat.make_write_options(...)without raising TypeError.Was AI used for this PR?
PR code and description written by:
Reviewed before submission by:
write_tableoptions are accepted by dataset writer #51220