Skip to content

GH-51220: [Python][Parquet] Accept write_table options in dataset writer - #51461

Merged
rok merged 4 commits into
apache:mainfrom
tadeja:51220-write_dataset
Sep 29, 2026
Merged

rok merged 4 commits into
apache:mainfrom
tadeja:51220-write_dataset

Conversation

@tadeja

@tadeja tadeja commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

Fix #51220 - Parquet write_table options are not all accepted in dataset ParquetFileWriteOptions so pyarrow.parquet.write_to_dataset( ...) raises TypeError: unexpected parquet write option:
for bloom_filter_options , write_time_adjusted_to_utc or store_schema

(Similar to past #37469)

What changes are included in this PR?

Add bloom_filter_options, store_schema, write_time_adjusted_to_utc to ParquetFileWriteOptions,
also explicitly expose bloom_filter_options in pyarrow.parquet.ParquetWriter.
Test writer options in make_write_options, and test three newly supported options in write_to_dataset

Are these changes tested?

Yes. CI also passes.

Are there any user-facing changes?

Yes, users may now use bloom_filter_options , write_time_adjusted_to_utc and store_schema
in pyarrow.parquet.write_to_dataset(...) and in ParquetFileFormat.make_write_options(...) without raising TypeError.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@github-actions github-actions Bot added the awaiting committer review Awaiting committer review label Sep 22, 2026
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51220 has been automatically assigned in GitHub to PR creator.

@tadeja
tadeja marked this pull request as ready for review September 24, 2026 09:08
Copilot AI lite review requested due to automatic review settings September 24, 2026 09:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Append the new parameter to preserve existing positional ParquetWriter compatibility.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds support for additional Parquet write options in dataset writers and ParquetWriter.

Changes:

  • Supports bloom_filter_options, store_schema, and write_time_adjusted_to_utc.
  • Adds tests for option acceptance and behavior.
  • Exposes Bloom filter options through ParquetWriter.
File Summary
python/​pyarrow/​tests/​parquet/​test_dataset.py Tests new dataset writer options.
python/​pyarrow/​parquet/​core.py Adds Bloom filter option support to ParquetWriter.
python/​pyarrow/​_dataset_parquet.pyx Registers and propagates new writer options.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread python/pyarrow/parquet/core.py
@github-actions github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 24, 2026
Copilot AI review requested due to automatic review settings September 25, 2026 13:12
@tadeja
tadeja force-pushed the 51220-write_dataset branch from 587a8ff to b951a92 Compare September 25, 2026 13:12
@github-actions github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 25, 2026
@tadeja

tadeja commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit preview-docs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

No unresolved blocking issues were identified.

Review effort: Lite
Findings: None

Resolved since last review (1)

@github-actions

Copy link
Copy Markdown

Revision: b951a92

Submitted crossbow builds: ursacomputing/crossbow @ actions-d493102f4e

Task Status
preview-docs GitHub Actions

@tadeja

tadeja commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

So the new preview-docs pyarrow.parquet.ParquetWriter signature now explicitly includes:
..., max_rows_per_page=None, bloom_filter_options=None, use_content_defined_chunking=False, **options)

(not listed in 25.0 : https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html)

@tadeja

tadeja commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

fyi @jorisvandenbossche, if you'd also like to check this, especially whether passing store_schema makes sense here.

@rok rok left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor comment but good to merge already. We should merge this before release freezes in argument order!

Comment thread python/pyarrow/tests/parquet/test_dataset.py
Copilot AI review requested due to automatic review settings September 25, 2026 16:43
@github-actions github-actions Bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Sep 25, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

All reviewed changes are covered by tests with no unresolved blocking issues.

Review effort: Lite
Findings: None

@rok

rok commented Sep 28, 2026

Copy link
Copy Markdown
Member

@AlenkaF, do you have time to do a pass here? Otherwise I'd merge tomorrow to get this into the release.

@rok
rok merged commit dc90604 into apache:main Sep 29, 2026
53 of 55 checks passed
@rok rok removed the awaiting changes Awaiting changes label Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python][Parquet] Check write_table options are accepted by dataset writer

3 participants