Skip to content

GH-51641: [C++][Parquet] fix int32 offset overflow in dictionary SetDict - #51642

Closed
Arawoof06 wants to merge 1 commit into
apache:mainfrom
Arawoof06:dict-setdict-offset-overflow
Closed

Arawoof06 wants to merge 1 commit into
apache:mainfrom
Arawoof06:dict-setdict-offset-overflow

Conversation

@Arawoof06

Copy link
Copy Markdown
Contributor

Rationale for this change

DictDecoderImpl<ByteArrayType>::SetDict and DictDecoderImpl<FLBAType>::SetDict concatenate the decoded dictionary values into byte_array_data_, which is sized from a 64-bit total_size, but they walk that buffer with a 32-bit offset. When the concatenated dictionary exceeds INT32_MAX bytes the offset wraps negative and memcpy(bytes_data + offset, ...) writes outside the allocation. The size is controlled by the dictionary page of an untrusted Parquet file, so a DICTIONARY_PAGE carrying more than 2 GB of decoded values reaches SetDict and corrupts the heap.

What changes are included in this PR?

For FLBA the values are addressed directly as index * type_length and a >2 GB concatenation is valid, so the offset accumulator is widened to int64_t. For BYTE_ARRAY the values are exposed through int32 offsets (byte_array_offsets_), so a concatenation past INT32_MAX cannot be represented; that case now throws instead of wrapping. total_size was already 64-bit, so this only closes the narrow accumulator/limit left behind it.

Are these changes tested?

Reaching the wrap needs a dictionary page above 2 GB, which is not worth adding as a unit test, so there is no new test. I built libparquet locally with the change to confirm it compiles; the valid-input paths are unchanged.

Are there any user-facing changes?

A BYTE_ARRAY dictionary carrying more than 2 GB of data now raises a ParquetException instead of writing out of bounds. Wide FIXED_LEN_BYTE_ARRAY dictionaries above 2 GB now decode correctly rather than corrupting memory.

This PR contains a "Critical Fix". An untrusted Parquet dictionary page above 2 GB triggers a heap out-of-bounds write through the wrapped 32-bit offset.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

AI tooling helped locate the narrow accumulator and draft this change; I reviewed the patch and verified it builds against the current tree.

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

This pull request has been automatically closed because you currently have 10 open pull requests, which is more than the limit of 3.

Due to the increase in pull requests opened by AI bots, and in order to keep the review queue manageable, Apache Arrow limits contributors without repository access to at most 3 concurrently open pull requests. This helps make sure each pull request gets the attention it needs and that work in progress does not go stale.

Once one of your other open pull requests has been merged or closed, you are welcome to reopen this one.

See also:

@github-actions github-actions Bot closed this Sep 29, 2026
@github-actions github-actions Bot added the awaiting review Awaiting review label Sep 29, 2026
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51641 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant