feat(format): add arrow.range canonical extension type - #1
Closed
hoeze-minion wants to merge 9 commits into
Closed
hoeze-minion wants to merge 9 commits into
hoeze-minion wants to merge 9 commits into
Conversation
…_of]` on older pre-full-C++20 SDKs (apache#50020) ### Rationale for this change Fixes the macos-CRAN failure in our crossbow jobs. ### What changes are included in this PR? Use the same workaround we have been to define the old(er) approach only when we are on an SDK that doesn't have full C++20 support ### Are these changes tested? Yes, crossbow ### Are there any user-facing changes? No * GitHub Issue: apache#50019 Authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Jonathan Keane <jkeane@gmail.com>
Add a canonical extension type for bounded ranges (mathematical intervals),
distinct from Arrow's calendar Interval (duration) type.
- Spec: docs/source/format/CanonicalExtensions.rst adds the Range section.
Storage is Struct<lower, upper> with both bounds nullable (null = +/-infinity,
treated as exclusive). A closed parameter (left/right/both/neither, pandas
vocabulary) is carried as JSON extension metadata; the subtype is read from
storage. Disambiguates from the calendar Interval type per DB convention
(INTERVAL = duration, RANGE/PERIOD = bounded set).
- C++ reference impl: cpp/src/arrow/extension/range.{h,cc} (RangeType/RangeArray)
with serialize/deserialize, storage validation, registration in the global
registry, tests, and CMake/meson wiring.
The closedness is no longer defaulted on the wire: empty metadata or a JSON object without a "closed" key is now rejected by Deserialize, so a serialized arrow.range is always unambiguous. The C++ convenience default argument for constructing a RangeType in code is left-closed ([lower, upper)), matching the PostgreSQL/Rust/Python range convention. Spec and tests updated.
Verified by building the arrow-canonical-extensions-test target (50/50 pass, 10/10 RangeType). Two fixes to the previously-uncompiled test: - include arrow/array/array_nested.h for the full StructArray definition (it is only forward-declared in type_fwd.h). - wrap the CheckDeserialize helper in an anonymous namespace to avoid a link-time collision with the identically named helper in opaque_test.cc.
hoeze-minion
force-pushed
the
feat/arrow-range-extension
branch
from
May 24, 2026 11:44
6a2752b to
6038f2d
Compare
4 of 5 tasks
hoeze-minion
force-pushed
the
feat/arrow-range-extension
branch
from
May 24, 2026 12:59
78f3593 to
8198463
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Draft a new canonical Arrow extension type
arrow.rangefor bounded ranges(mathematical intervals), distinct from Arrow's calendar
Interval(duration)type. This PR provides the spec text, a C++ reference implementation, PyArrow
bindings, and the supporting documentation. An independent Rust implementation
lives in a companion arrow-rs PR, and the GLib (C) bindings live in a stacked
branch (see Status below).
Design
arrow.range. Follows database convention:INTERVALmeans a duration(and Arrow already has a calendar
Intervaltype), whileRANGE/PERIODmeans a bounded set with a lower and upper endpoint (PostgreSQL range types,
SQL:2011 periods).
Struct<lower: T, upper: T>. Each bound is optionally nullable,independently: a nullable bound may hold null for an infinite endpoint
(-inf / +inf, always exclusive), while a non-nullable bound gives a
finite-only range. Nullability is only needed to represent an unbounded side.
The outer struct validity bit marks a missing/absent range.
closed: one ofleft,right,both,neither(pandasvocabulary;
left= lower bound inclusive,right= upper bound inclusive).{"closed": "..."}.closedis required onthe wire (no default), so a serialized
arrow.rangeis always unambiguous.The C++ convenience default argument when constructing in code is left-closed
[lower, upper), matching the PostgreSQL/Rust/Python range convention.Tis read from the storage struct and is not duplicated in themetadata (single source of truth).
Make/range()and PyArrowrange_()take anallow_unboundedflag (default true): true builds nullable(infinity-capable) bounds, false builds non-nullable finite-only bounds.
Changes
C++:
cpp/src/arrow/extension/range.{h,cc}:RangeType/RangeArraywithSerialize/Deserialize, storage validation, andMakefactory +range()helper (with the
allow_unboundedoption).cpp/src/arrow/extension/range_test.cc: serialize/deserialize round-trip,equality, default-closed, invalid storage/metadata, and an IPC batch
round-trip.
extension_type.cc); CMake andmeson build wiring.
Spec and docs:
docs/source/format/CanonicalExtensions.rst: newRangesection (storage,parameters, serialization, semantics, and a note disambiguating from the
calendar
Intervaltype), following the four-bullet template used by theother canonical types.
docs/source/status.rst: newRangerow in the canonical extension matrix(C++ supported).
docs/source/cpp/api/extension.rst:RangeType/RangeArray/range().Python (PyArrow):
RangeType,RangeArray,RangeScalar, and therange_()factory (trailingunderscore to avoid shadowing the builtin
range, matchingbool_/list_).Wired through
libarrow.pxd,lib.pxd,types.pxi,array.pxi,scalar.pxi,public-api.pxi, and exported from__init__.py.docs/source/python/api/{arrays,datatypes}.rst: new class/factory entries.python/pyarrow/tests/test_extension_type.py: parametrized round-trip,equality,
closedhandling, and bad-value tests.Testing
C++: built the
arrow-canonical-extensions-testtarget from a minimal Arrow C++configuration (
-DARROW_BUILD_TESTS=ON -DARROW_JSON=ON -DARROW_DEPENDENCY_SOURCE=BUNDLED) and ran it:RangeTypetests (coveringnon-nullable and asymmetric bounds, and the
allow_unboundedfactory option).opaque, tensors).
Python: a full PyArrow build against a local Arrow C++ build (with
compute/csv/filesystem enabled) was completed, and the
arrow.rangetests rungreen:
pytest test_extension_type.py -k rangegives 14 passed, 12 skipped, 0failed (the skips are the optional cloudpickle variants). This covers
construction, the
closedparameter across all four values and three valuetypes, repr, equality, pickle round-trip, IPC round-trip, storage<->extension
casts, the invalid-
closederror path, and theallow_unboundedoption.Conformance with Arrow conventions
Rangesection now matches the canonical-type template (four top-levelbullets, notes at the section edges rather than interleaved).
range.{h,cc}aligned to neighbor extension types (include style,this->extension_name()inToString).matrix, C++ API docs, full PyArrow binding set, Python API docs).
Status / caveats
CanonicalExtensions.rst, a real canonical typerequires an Arrow dev mailing-list discussion and vote, and a parameterized
type should ideally ship a second, independent (non-C++) implementation. This
PR provides the spec, a C++ reference implementation, and PyArrow bindings; an
independent Rust implementation (which does not wrap the C++ core) is provided
in the companion arrow-rs PR feat(arrow-schema): add arrow.range canonical extension type arrow-rs#1, satisfying the
two-implementation recommendation. The GLib (C) bindings are kept in a stacked
branch
feat/arrow-range-glib(PR feat(c-glib): add fixed and variable closedness range GLib bindings #2) on top of this one.dev/archery/.../datagen.py) is intentionally nottouched: adding
arrow.rangethere asserts cross-language interop, whichcannot pass until a second-language implementation exists. It should be added
together with that implementation.
expects
GH-<issue>: [Component] ...titles tied to a GitHub issue; titleswould be reworded for an upstream PR.