Skip to content

Fix(parquet): Fix spurious refill failure for multi-page DATA_PAGE_V2 chunks - #2341

Open
r7raul1984 wants to merge 1 commit into
IBM:branch-1.7-dftfrom
r7raul1984:fixv2
Open

Fix(parquet): Fix spurious refill failure for multi-page DATA_PAGE_V2 chunks#2341
r7raul1984 wants to merge 1 commit into
IBM:branch-1.7-dftfrom
r7raul1984:fixv2

Conversation

@r7raul1984

Copy link
Copy Markdown

Summary

PageReader::prepareDataPageV2's row == kRepDefOnly branch called
skipBytes() on a page's compressed bytes that had already been fully
consumed a few lines above via readBytes(). When a column chunk holds
more than one DATA_PAGE_V2 page and the first page's compressed size
exceeds the bytes remaining in the chunk after it, that redundant skip
drained the stream past the true end of the already-fetched buffer.
PageReader's logical byte offset (pageStart_) still believed more
data existed, so preloadRepDefs() issued one extra readPageHeader()
call, which failed with Thrift deserialize error: Empty buffer returned when refilling once the stream had nothing left to give.

This surfaced as a deterministic (non-flaky) failure when exploding a
nested list/array column whose chunk spans multiple data pages — e.g.
Spark's ParquetIOSuite test "explode nested lists crossing a rowgroup boundary" against packed-list-vectorized.parquet.

prepareDataPageV1 does not have this bug — it has no analogous extra
skip after consuming the page via readBytes().

The fix removes the redundant skipBytes() call; readBytes() already
advances past the whole page.

@r7raul1984 r7raul1984 closed this Aug 1, 2026
@r7raul1984 r7raul1984 reopened this Aug 1, 2026
@r7raul1984
r7raul1984 force-pushed the fixv2 branch 2 times, most recently from 0331a7e to db0fd1e Compare August 2, 2026 06:08
… chunks

Signed-off-by: jijtang <8199300+r7raul1984@users.noreply.github.com>
@r7raul1984

Copy link
Copy Markdown
Author

@majetideepak hi, could u kindly take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant