Describe the bug, including details regarding any error messages, version, and platform.
Was reading about Content-Defined Chunking which sounds very interesting!
Then I came across
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html etc
use_content_defined_chunking was documented in the docstring, but was not part of the APIs on the top of the page.
Also, it will be useful to have a note about the consequence of setting use_content_defined_chunking=True on systems that do not support CDC.
(final personal thought: should this filesystem-specific option to be set somewhere else? for now this is only relevant for Hugging face now?)
Component(s)
Parquet, Python
Describe the bug, including details regarding any error messages, version, and platform.
Was reading about
Content-Defined Chunkingwhich sounds very interesting!Then I came across
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html etc
use_content_defined_chunkingwas documented in the docstring, but was not part of the APIs on the top of the page.Also, it will be useful to have a note about the consequence of setting
use_content_defined_chunking=Trueon systems that do not support CDC.(final personal thought: should this filesystem-specific option to be set somewhere else? for now this is only relevant for Hugging face now?)
Component(s)
Parquet, Python