Skip to content

GH-51548: [Python] Fix silent corruption in non-native-endian dataframe imports - #51557

Open
tam3tamtam wants to merge 1 commit into
apache:mainfrom
tam3tamtam:fix-interchange-endianness
Open

tam3tamtam wants to merge 1 commit into
apache:mainfrom
tam3tamtam:fix-interchange-endianness

Conversation

@tam3tamtam

@tam3tamtam tam3tamtam commented Sep 27, 2026 •

Copy link
Copy Markdown

Rationale for this change

Fixes #51548

The dataframe interchange protocol includes byte order in each dtype description. The PyArrow importer currently wraps these buffers directly, so non-native-endian values can be interpreted incorrectly and silently corrupted.

What changes are included in this PR?

The importer now converts non-native-endian data and string offset buffers to native byte order when needed, using array.array's C-level byteswap().
If that conversion would require a copy and allow_copy=False, it raises a RuntimeError.

Regression tests cover numeric values, string offsets, sentinel nulls, 16-bit
and 64-bit values, and the allow_copy=False and invalid-buffer error paths.

Are these changes tested?

Yes. The focused tests for these changes passed:

python -m pytest pyarrow/tests/interchange/test_conversion.py -k "non_native_endian or unsupported_endianness or misaligned_buffer_size or int8_endianness" -q
9 passed, 1792 deselected

Are there any user-facing changes?

Non-native-endian columns are now imported with their correct values. Importing
them may copy their buffers; with allow_copy=False, conversion raises an error when a copy is required.

This PR contains a "Critical Fix". Non-native-endian values could previously be interpreted with the wrong byte order and silently corrupted during dataframe interchange imports. This change preserves the producer's values.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51548 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python] DataFrame interchange silently corrupts non-native-endian columns.

1 participant