to_arrow_batch_reader() aborts on a filtered scan with a map column (pyarrow bug apache/arrow#51029) #4084

Not a pyiceberg bug: it's apache/arrow#51029, fixed in apache/arrow#51093 (unreleased as of pyarrow 25.0.1). Closing.

DataScan.to_arrow_batch_reader() aborted the process on a filtered scan of a table with a map column:

/arrow/cpp/src/arrow/array/array_nested.cc:912:  Check failed: _s.ok() Operation failed: ValidateChildData(data->child_data)
Bad status: Invalid: Map array keys array should have no nulls
Fatal Python error: Aborted

The batches pyiceberg produces are valid (validate(full=True) passes). The abort comes from the reader's final .cast(target_schema), which casts map<string, …> to map<large_string, …>. pyarrow aborts on that cast when the map array has been filtered. Pure pyarrow, pyarrow 25.0.1:

import pyarrow as pa

m = pa.array([[("a", 1)], [("b", 2)], [("c", 3)]], type=pa.map_(pa.string(), pa.int64()))
filtered = m.filter(pa.array([False, True, True]))
filtered.validate(full=True)  # passes
filtered.cast(pa.map_(pa.large_string(), pa.int64()))  # aborts the process

to_arrow() on the same scan doesn't hit that cast, which is why it works. pyiceberg users on pyarrow ≤ 25.0.1 can avoid it with to_arrow() on filtered scans of tables with map columns.