Skip to content

GH-53519: [Python] Fix StringArray/LargeStringArray.from_buffers to accept int64 arguments - #53728

Closed
Asthenia0412 wants to merge 1 commit into
apache:mainfrom
Asthenia0412:GH-53519-fix-from-buffers-int64
Closed

Asthenia0412 wants to merge 1 commit into
apache:mainfrom
Asthenia0412:GH-53519-fix-from-buffers-int64

Conversation

@Asthenia0412

@Asthenia0412 Asthenia0412 commented Oct 14, 2026 •

Copy link
Copy Markdown

Background

GH-53519 reports that StringArray.from_buffers() and LargeStringArray.from_buffers() reject valid 64-bit array counters with OverflowError: value too large to convert to int. The issue is that these specialized factory methods declare their length, null_count, and offset parameters as Cython int (32-bit C int), while the underlying Arrow ArrayData counters are int64_t.

In contrast, the generic Array.from_buffers() method accepts unbounded Python integers and passes them through correctly to the C++ layer, which accepts int64_t parameters in MakeWithChildren().

Problem

When users pass valid values above INT32_MAX (2^31) to the specialized StringArray.from_buffers() or LargeStringArray.from_buffers(), Cython throws an OverflowError because the Python integer cannot fit into a 32-bit C int. This unnecessarily limits the API compared to the generic Array.from_buffers().

Solution

Widen the three parameters (length, null_count, offset) from Cython int to int64_t in both StringArray.from_buffers() and LargeStringArray.from_buffers(). This aligns with:

  • The underlying C++ ArrayData::MakeWithChildren() signature which uses int64_t
  • The generic Array.from_buffers() which already accepts large integers
  • Other specialized factories in the same file (e.g., DictionaryArray.from_buffers() already uses int64_t length)

Testing

The change is a straightforward type widening:

  • The underlying C++ code already accepts int64_t, so no C++ changes are needed
  • No buffer layout changes
  • Normal string/binary buffer round-trips continue to work as before
  • Values up to INT64_MAX will now be accepted (though practical use is limited by address space)

This fix was verified by code analysis against:\n- The issue's reproduction case\n- The _ensure_compression() patterns\n- Cython declaration conventions used elsewhere in the same file\n\n\nCloses: #53519

…s to accept int64 arguments

The StringArray.from_buffers() and LargeStringArray.from_buffers() methods
declared their length, null_count and offset parameters as C int (32-bit),
but the underlying C++ ArrayData counters are int64_t. This caused
OverflowError when passing values above INT32_MAX.

Widen all three parameters to int64_t in both methods so that the
specialized factories accept the same valid arrays as the generic
Array.from_buffers() does.

Closes apache#53519
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant