Skip to content

GH-50847: [Python] Add Type_RUN_END_ENCODED to _NESTED_TYPES set - #50848

Merged
AlenkaF merged 1 commit into
apache:mainfrom
Nishuuzz:gh-50847-is-nested-ree
Aug 27, 2026
Merged

GH-50847: [Python] Add Type_RUN_END_ENCODED to _NESTED_TYPES set#50848
AlenkaF merged 1 commit into
apache:mainfrom
Nishuuzz:gh-50847-is-nested-ree

Conversation

@Nishuuzz

@Nishuuzz Nishuuzz commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

pyarrow.types.is_nested() said a run-end encoded type wasn't nested, while the C++ arrow::is_nested() says it is:

>>> import pyarrow as pa
>>> t = pa.run_end_encoded(pa.int32(), pa.string())
>>> t.num_fields
2
>>> pa.types.is_nested(t)
False

The Python side reads from a hardcoded _NESTED_TYPES set in python/pyarrow/types.py rather than from the C++ trait, so it has to be kept in step by hand. Lining that set up against is_nested() in cpp/src/arrow/type_traits.h, run-end encoded was the only type the two still disagreed about — the list-view types and fixed-size list are already there.

It's also out of step with the type itself. Run-end encoded has two children, the run ends and the values, and every other type in pyarrow that has children answers True here.

This is the same thing that happened to fixed-size list in #40171, fixed by #40172, and the list-view types were added after that. This looks like the last one missed when run-end encoding went in.

What changes are included in this PR?

Adds Type_RUN_END_ENCODED to _NESTED_TYPES, which is the whole fix.

In the test I've added the run-end encoded case, and while I was there also a map and a dictionary. Map was nested already but wasn't asserted anywhere, and dictionary is the interesting negative — it wraps a value type but is deliberately not nested in either implementation, so pinning it means a later change can't quietly sweep it in.

Are these changes tested?

Yes. test_is_nested_or_struct fails on the current code and passes with the change.

I don't have a local C++ build, so I checked this by running the updated test_types.py against an installed pyarrow 25.0.0 with the same one-line change applied to its types.py. Before the change that file had 87 passing with test_is_nested_or_struct failing; after it, 88 passing. The two errors and one failure I see in both runs are environmental on my machine and unrelated — the errors are the pickle_module fixture, which comes from a conftest I wasn't loading, and the failure is test_pytz_timezone_roundtrip.

Are there any user-facing changes?

Yes, though it's small. pa.types.is_nested() now returns True for run-end encoded types where it previously returned False. Anything branching on that predicate will take the nested path for these types, which is the intended answer and what the C++ implementation has always given. Nothing inside pyarrow reads is_nested or _NESTED_TYPES, so the effect is limited to callers.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50847 has been automatically assigned in GitHub to PR creator.

@uros-b

uros-b commented Aug 13, 2026

Copy link
Copy Markdown
Member

Nice, thank you @Nishuuzz!

@Nishuuzz

Copy link
Copy Markdown
Contributor Author

Hi @uros-b any update on this PR?

@github-actions github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 25, 2026

@rok rok left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @Nishuuzz. This change looks reasonable.
Before we merge, can you please rebase on main so we get an updated linter check?

pyarrow.types.is_nested() answered False for run-end encoded types while
the C++ arrow::is_nested() answers True for them. The Python predicate
reads from a hardcoded _NESTED_TYPES set that never had
Type_RUN_END_ENCODED added to it; run-end encoded was the only type the
two definitions still disagreed about.

Also extend test_is_nested_or_struct with map and dictionary, so the set
is pinned at both ends rather than only for the types that are nested.
@Nishuuzz
Nishuuzz force-pushed the gh-50847-is-nested-ree branch from efe5537 to 5041fdf Compare August 27, 2026 06:16
@AlenkaF
AlenkaF merged commit 3db7076 into apache:main Aug 27, 2026
37 checks passed
@AlenkaF AlenkaF removed the awaiting committer review Awaiting committer review label Aug 27, 2026
@rok

rok commented Aug 27, 2026

Copy link
Copy Markdown
Member

Thanks for contributing @Nishuuzz and thanks for another review @AlenkaF ! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants