Skip to content

[Python][Docs] use_content_defined_chunking not shown correctly for ParquetWriter / write_table #47477

Description

@wingkitlee0

Describe the bug, including details regarding any error messages, version, and platform.

Was reading about Content-Defined Chunking which sounds very interesting!

Then I came across
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html etc

use_content_defined_chunking was documented in the docstring, but was not part of the APIs on the top of the page.

Also, it will be useful to have a note about the consequence of setting use_content_defined_chunking=True on systems that do not support CDC.

(final personal thought: should this filesystem-specific option to be set somewhere else? for now this is only relevant for Hugging face now?)

Component(s)

Parquet, Python

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions