GH-50027: [Format][C++][Python] Add fixed and variable closedness range canonical extension types by Hoeze · Pull Request #50028 · apache/arrow

Hoeze · 2026-05-24T13:35:11Z

Rationale for this change

This PR implements #50027. It adds two canonical extension types for ranges, i.e. intervals with a lower and an upper bound:

arrow.fixed_closedness_range: the closedness (left, right, both or neither) is a type parameter shared by all values. This fits discrete ranges such as PostgreSQL's int4range or daterange, and pandas' IntervalArray.
arrow.variable_closedness_range: the inclusivity of each bound is stored per value. Continuous ranges such as PostgreSQL's numrange or tstzrange need this, because one column can hold both [1, 5] and (1, 5).

The names follow the existing arrow.fixed_shape_tensor / arrow.variable_shape_tensor pair.

What changes are included in this PR?

Spec text for both types in CanonicalExtensions.rst, and a row for each in the status table. Both use a Struct<lower: T, upper: T> storage, and the variable type adds two non-nullable booleans lower_inc and upper_inc. The spec defines only this storage layout: T may be any orderable type, bounds are compared with the order of T, and only a null bound means an unbounded side.
A C++ reference implementation in arrow/extension/range.h: FixedClosednessRangeType, VariableClosednessRangeType, their array classes, and the factories fixed_closedness_range() and variable_closedness_range(). Both types are registered in the global extension type registry. The JSON metadata uses the simdjson helpers and JsonWriter, like the other canonical extension types.
PyArrow bindings: pa.fixed_closedness_range(), pa.variable_closedness_range() and the matching type, array and scalar classes.
C++ and Python API docs.

Are these changes tested?

Yes, locally: the C++ tests in arrow-canonical-extensions-test (suites FixedClosednessRangeType and VariableClosednessRangeType) and the PyArrow tests in test_extension_type.py pass, including ranges over strings and pickling. I did not try the types in any other project yet.

Are there any user-facing changes?

Yes, but only additions: two new canonical extension types and their C++ and PyArrow APIs. There are no breaking changes.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

Human
AI

Reviewed before submission by:

Human
AI
Not reviewed

Note that I made heavy use of AI to create this PR and copied many structures from the fixed shape tensor extension type. I reviewed each change and hope the changes I made are meaningful.
Nevertheless, I am not sure whether the C++ parts are comprehensive or if I missed anything; this is my first contribution to Arrow.

GitHub Issue: [Format] Add canonical extension types for bounded ranges #50027

github-actions · 2026-05-24T13:35:36Z

⚠️ GitHub issue #50027 has been automatically assigned in GitHub to PR creator.

rok · 2026-05-24T19:50:45Z

See comment.

Copilot

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Copilot

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved wire-contract mismatches and validation and pickle-preservation issues block approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 1

Open (3)

Use canonical arrow.range extension name · New
Reconcile range wire format with canonical issue requirements · New
Preserve allow_unbounded in range type pickling · New

+                                    RangeClosed closed)
+      : ExtensionType(std::move(storage_type)), closed_(closed) {}
+
+  std::string extension_name() const override { return "arrow.fixed_closedness_range"; }


+   convention: SQL uses ``INTERVAL`` for durations and ``RANGE`` (or
+   ``PERIOD``) for bounded sets.
+
+* Extension name: ``arrow.fixed_closedness_range``.


Add a canonical extension type for bounded ranges (mathematical intervals), distinct from Arrow's calendar Interval (duration) type. - Spec: docs/source/format/CanonicalExtensions.rst adds the Range section. Storage is Struct<lower, upper> with both bounds nullable (null = +/-infinity, treated as exclusive). A closed parameter (left/right/both/neither, pandas vocabulary) is carried as JSON extension metadata; the subtype is read from storage. Disambiguates from the calendar Interval type per DB convention (INTERVAL = duration, RANGE/PERIOD = bounded set). - C++ reference impl: cpp/src/arrow/extension/range.{h,cc} (RangeType/RangeArray) with serialize/deserialize, storage validation, registration in the global registry, tests, and CMake/meson wiring.

The closedness is no longer defaulted on the wire: empty metadata or a JSON object without a "closed" key is now rejected by Deserialize, so a serialized arrow.range is always unambiguous. The C++ convenience default argument for constructing a RangeType in code is left-closed ([lower, upper)), matching the PostgreSQL/Rust/Python range convention. Spec and tests updated.

Verified by building the arrow-canonical-extensions-test target (50/50 pass, 10/10 RangeType). Two fixes to the previously-uncompiled test: - include arrow/array/array_nested.h for the full StructArray definition (it is only forward-declared in type_fwd.h). - wrap the CheckDeserialize helper in an anonymous namespace to avoid a link-time collision with the identically named helper in opaque_test.cc.

Add a sibling canonical extension type to arrow.range that stores bound inclusivity per value via non-nullable boolean lower_inc/upper_inc fields, storage Struct<lower:T, upper:T, lower_inc:bool, upper_inc:bool>. arrow.range carries a single type-level closed parameter, sufficient for discrete ranges that canonicalize to one closedness (int4range, int8range, daterange). Continuous ranges (numrange, tsrange, tstzrange) cannot be canonicalized, so closedness must travel with each value. arrow.range_inc mirrors PostgreSQL's internal range representation for that case; both types coexist. The type has no metadata parameters: inclusivity lives in storage, so Serialize emits {} and Deserialize accepts empty/{}/extra keys. A null (infinite) bound is always exclusive regardless of its flag. Covers C++ (type, array, registration, tests), pyarrow bindings and tests, and the format spec, status table, and C++/Python API docs.

Under CMAKE_UNITY_BUILD (Windows CI), range_test.cc and opaque_test.cc are merged into one translation unit. Both declared a CheckDeserialize helper (range's in an anonymous namespace, opaque's in namespace arrow), making the unqualified call ambiguous and failing the MSVC build with C2668. Rename the range helper to CheckRangeDeserialize to remove the collision.

Use JsonWriter for Serialize and the simdjson DOM helpers for Deserialize, matching the other canonical extension types.

Name the pair after arrow.fixed_shape_tensor and arrow.variable_shape_tensor: closedness is either one type parameter or stored per value. - arrow.range -> arrow.fixed_closedness_range - arrow.range_inc -> arrow.variable_closedness_range - C++: RangeType/RangeArray -> FixedClosednessRangeType/Array, RangeIncType/RangeIncArray -> VariableClosednessRangeType/Array, range()/range_inc() -> fixed_closedness_range()/variable_closedness_range() - pyarrow: range_/range_inc and the Range* classes follow the same names

Allow any orderable type as the range subtype and compare bounds with the order of that type; the spec defines only the storage layout. State that only null means unbounded and that all empty ranges denote the same set. Add string subtype cases to the C++ and Python tests.

__reduce__ rebuilt both range types from their parameters only, so a type with non-nullable bounds came back nullable after unpickling. Rebuild them from the storage type through the C++ Deserialize instead. This also keeps a type where only one bound is nullable, and tests cover both cases.

Copilot

Copilot review overview

🔵 Needs a closer look

Broad cross-language changes remain, including unresolved orderability validation in range factories and deserialization.

Review effort: Lite
Findings: 2

Open (2)

Reconcile range wire format with canonical issue requirements
Use canonical arrow.range extension name

Resolved since last review (1)

Preserve allow_unbounded in range type pickling

The emptiness rule compared lower and upper without saying what happens with a null bound. A null bound means unbounded and cannot be compared, so the rule now applies only when both bounds are non-null.

_fixed_closedness_range_from_storage put closed into the JSON metadata without escaping. It now builds its prototype with closed, so fixed_closedness_range() rejects any value other than left, right, both and neither first.

Copilot

Copilot review overview

🔵 Needs a closer look

Four moderate unresolved issues need to be addressed before approval.

Review effort: Lite
Findings: 2

Open (2)

Reconcile range wire format with canonical issue requirements
Use canonical arrow.range extension name

github-actions Bot added Component: C++ Component: Python Component: Documentation awaiting review Awaiting review labels May 24, 2026

Hoeze marked this pull request as ready for review May 24, 2026 13:36

Hoeze requested review from AlenkaF, raulcd and rok as code owners May 24, 2026 13:36

rok mentioned this pull request May 24, 2026

[Format] Add canonical extension types for bounded ranges #50027

Open

Copilot AI lite review requested due to automatic review settings June 4, 2026 19:52

Copilot AI reviewed Jun 4, 2026

rok reviewed Jun 4, 2026

View reviewed changes

Comment thread cpp/src/arrow/extension/range.cc Outdated

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Jun 8, 2026

hoeze-minion force-pushed the feat/arrow-range-extension branch from f3ec0b6 to c414320 Compare September 13, 2026 00:24

hoeze-minion requested a review from pitrou as a code owner September 13, 2026 00:24

Copilot AI review requested due to automatic review settings September 13, 2026 00:24

Copilot AI reviewed Sep 13, 2026

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 13, 2026

Hoeze changed the title ~~GH-50027: [Format][C++][Python] Add arrow.range canonical extension type~~ GH-50027: [Format][C++][Python] Add fixed and variable closedness range canonical extension types Sep 13, 2026

Copilot AI review requested due to automatic review settings September 21, 2026 19:11

Copilot started reviewing on behalf of Hoeze September 21, 2026 19:12 View session

Hoeze changed the title ~~GH-50027: [Format][C++][Python] Add fixed and variable closedness range canonical extension types~~ GH-50027: [Format][C++][Python] Add fixed and variable closedness range canonical extension types Sep 21, 2026

Copilot AI reviewed Sep 21, 2026

View reviewed changes

Hoeze added 3 commits September 26, 2026 21:13

Hoeze added 13 commits September 26, 2026 21:13

style(format): align range extension type with neighbor C++ conventions

8eb0a3f

docs(format): refine arrow.range spec and add status and C++ API entries

6671cf0

feat(python): add pyarrow bindings for arrow.range extension type

96a70f1

feat(format): allow non-nullable bounds in arrow.range

604fb38

feat(python): add allow_unbounded option to pyarrow range_

dbf4614

style(format): apply clang-format and cmake-format to range extension

ba99eb9

refactor(format): migrate arrow.range JSON metadata to simdjson

16f3384

Use JsonWriter for Serialize and the simdjson DOM helpers for Deserialize, matching the other canonical extension types.

docs(format): clarify range spec wording and add examples

31b44c7

hoeze-minion force-pushed the feat/arrow-range-extension branch from 5ff2abd to ffcfd4d Compare September 26, 2026 21:38

Copilot AI review requested due to automatic review settings September 26, 2026 21:38

Copilot started reviewing on behalf of Hoeze September 26, 2026 21:38 View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

Hoeze added 2 commits September 26, 2026 22:00

docs(format): limit the range emptiness rule to non-null bounds

0feb4af

The emptiness rule compared lower and upper without saying what happens with a null bound. A null bound means unbounded and cannot be compared, so the rule now applies only when both bounds are non-null.

Copilot AI review requested due to automatic review settings September 26, 2026 22:24

Copilot started reviewing on behalf of Hoeze September 26, 2026 22:25 View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

This was referenced Sep 26, 2026

feat(c-glib): add fixed and variable closedness range GLib bindings Hoeze/arrow#2

Draft

feat(arrow-schema): add fixed and variable closedness range canonical extension types Hoeze/arrow-rs#2

Open

Navigation Menu

GH-50027: [Format][C++][Python] Add fixed and variable closedness range canonical extension types - #50028

GH-50027: [Format][C++][Python] Add fixed and variable closedness range canonical extension types#50028
Hoeze wants to merge 18 commits into
apache:mainfrom
Hoeze:feat/arrow-range-extension

Hoeze commented May 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 24, 2026

Uh oh!

rok commented May 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Hoeze commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Was AI used for this PR?

Uh oh!

github-actions Bot commented May 24, 2026

Uh oh!

rok commented May 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Hoeze commented May 24, 2026 •

edited

Loading