Skip to content

feat(format): add arrow.range canonical extension type - #1

Closed
hoeze-minion wants to merge 9 commits into
mainfrom
feat/arrow-range-extension
Closed

hoeze-minion wants to merge 9 commits into
mainfrom
feat/arrow-range-extension

Conversation

@hoeze-minion

@hoeze-minion hoeze-minion commented May 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Draft a new canonical Arrow extension type arrow.range for bounded ranges
(mathematical intervals), distinct from Arrow's calendar Interval (duration)
type. This PR provides the spec text, a C++ reference implementation, PyArrow
bindings, and the supporting documentation. An independent Rust implementation
lives in a companion arrow-rs PR, and the GLib (C) bindings live in a stacked
branch (see Status below).

Design

  • Name arrow.range. Follows database convention: INTERVAL means a duration
    (and Arrow already has a calendar Interval type), while RANGE/PERIOD
    means a bounded set with a lower and upper endpoint (PostgreSQL range types,
    SQL:2011 periods).
  • Storage Struct<lower: T, upper: T>. Each bound is optionally nullable,
    independently: a nullable bound may hold null for an infinite endpoint
    (-inf / +inf, always exclusive), while a non-nullable bound gives a
    finite-only range. Nullability is only needed to represent an unbounded side.
    The outer struct validity bit marks a missing/absent range.
  • Parameter closed: one of left, right, both, neither (pandas
    vocabulary; left = lower bound inclusive, right = upper bound inclusive).
  • Metadata is a JSON object {"closed": "..."}. closed is required on
    the wire (no default), so a serialized arrow.range is always unambiguous.
    The C++ convenience default argument when constructing in code is left-closed
    [lower, upper), matching the PostgreSQL/Rust/Python range convention.
  • Subtype T is read from the storage struct and is not duplicated in the
    metadata (single source of truth).
  • Factory the C++ Make/range() and PyArrow range_() take an
    allow_unbounded flag (default true): true builds nullable
    (infinity-capable) bounds, false builds non-nullable finite-only bounds.

Changes

C++:

  • cpp/src/arrow/extension/range.{h,cc}: RangeType / RangeArray with
    Serialize/Deserialize, storage validation, and Make factory + range()
    helper (with the allow_unbounded option).
  • cpp/src/arrow/extension/range_test.cc: serialize/deserialize round-trip,
    equality, default-closed, invalid storage/metadata, and an IPC batch
    round-trip.
  • Registered in the global extension registry (extension_type.cc); CMake and
    meson build wiring.

Spec and docs:

  • docs/source/format/CanonicalExtensions.rst: new Range section (storage,
    parameters, serialization, semantics, and a note disambiguating from the
    calendar Interval type), following the four-bullet template used by the
    other canonical types.
  • docs/source/status.rst: new Range row in the canonical extension matrix
    (C++ supported).
  • docs/source/cpp/api/extension.rst: RangeType / RangeArray / range().

Python (PyArrow):

  • RangeType, RangeArray, RangeScalar, and the range_() factory (trailing
    underscore to avoid shadowing the builtin range, matching bool_/list_).
    Wired through libarrow.pxd, lib.pxd, types.pxi, array.pxi,
    scalar.pxi, public-api.pxi, and exported from __init__.py.
  • docs/source/python/api/{arrays,datatypes}.rst: new class/factory entries.
  • python/pyarrow/tests/test_extension_type.py: parametrized round-trip,
    equality, closed handling, and bad-value tests.

Testing

C++: built the arrow-canonical-extensions-test target from a minimal Arrow C++
configuration (-DARROW_BUILD_TESTS=ON -DARROW_JSON=ON -DARROW_DEPENDENCY_SOURCE=BUNDLED) and ran it:

  • 51 / 51 tests pass, including 11 / 11 RangeType tests (covering
    non-nullable and asymmetric bounds, and the allow_unbounded factory option).
  • No regressions in the other canonical-extension suites (bool8, uuid, json,
    opaque, tensors).

Python: a full PyArrow build against a local Arrow C++ build (with
compute/csv/filesystem enabled) was completed, and the arrow.range tests run
green: pytest test_extension_type.py -k range gives 14 passed, 12 skipped, 0
failed
(the skips are the optional cloudpickle variants). This covers
construction, the closed parameter across all four values and three value
types, repr, equality, pickle round-trip, IPC round-trip, storage<->extension
casts, the invalid-closed error path, and the allow_unbounded option.

Conformance with Arrow conventions

  • RST Range section now matches the canonical-type template (four top-level
    bullets, notes at the section edges rather than interleaved).
  • range.{h,cc} aligned to neighbor extension types (include style,
    this->extension_name() in ToString).
  • Added the cross-cutting files that the other canonical types touch (status
    matrix, C++ API docs, full PyArrow binding set, Python API docs).

Status / caveats

  • Canonical-process: per CanonicalExtensions.rst, a real canonical type
    requires an Arrow dev mailing-list discussion and vote, and a parameterized
    type should ideally ship a second, independent (non-C++) implementation. This
    PR provides the spec, a C++ reference implementation, and PyArrow bindings; an
    independent Rust implementation (which does not wrap the C++ core) is provided
    in the companion arrow-rs PR feat(arrow-schema): add arrow.range canonical extension type arrow-rs#1, satisfying the
    two-implementation recommendation. The GLib (C) bindings are kept in a stacked
    branch feat/arrow-range-glib (PR feat(c-glib): add fixed and variable closedness range GLib bindings #2) on top of this one.
  • Integration datagen (dev/archery/.../datagen.py) is intentionally not
    touched: adding arrow.range there asserts cross-language interop, which
    cannot pass until a second-language implementation exists. It should be added
    together with that implementation.
  • Commit/PR titles use Conventional Commits here. Apache Arrow upstream
    expects GH-<issue>: [Component] ... titles tied to a GitHub issue; titles
    would be reworded for an upstream PR.

jonkeane and others added 7 commits May 23, 2026 08:47
…_of]` on older pre-full-C++20 SDKs (apache#50020)

### Rationale for this change

Fixes the macos-CRAN failure in our crossbow jobs.

### What changes are included in this PR?

Use the same workaround we have been to define the old(er) approach only when we are on an SDK that doesn't have full C++20 support

### Are these changes tested?

Yes, crossbow

### Are there any user-facing changes?

No

* GitHub Issue: apache#50019

Authored-by: Jonathan Keane <jkeane@gmail.com>
Signed-off-by: Jonathan Keane <jkeane@gmail.com>
Add a canonical extension type for bounded ranges (mathematical intervals),
distinct from Arrow's calendar Interval (duration) type.

- Spec: docs/source/format/CanonicalExtensions.rst adds the Range section.
  Storage is Struct<lower, upper> with both bounds nullable (null = +/-infinity,
  treated as exclusive). A closed parameter (left/right/both/neither, pandas
  vocabulary) is carried as JSON extension metadata; the subtype is read from
  storage. Disambiguates from the calendar Interval type per DB convention
  (INTERVAL = duration, RANGE/PERIOD = bounded set).
- C++ reference impl: cpp/src/arrow/extension/range.{h,cc} (RangeType/RangeArray)
  with serialize/deserialize, storage validation, registration in the global
  registry, tests, and CMake/meson wiring.
The closedness is no longer defaulted on the wire: empty metadata or a JSON
object without a "closed" key is now rejected by Deserialize, so a serialized
arrow.range is always unambiguous. The C++ convenience default argument for
constructing a RangeType in code is left-closed ([lower, upper)), matching the
PostgreSQL/Rust/Python range convention. Spec and tests updated.
Verified by building the arrow-canonical-extensions-test target (50/50 pass,
10/10 RangeType). Two fixes to the previously-uncompiled test:
- include arrow/array/array_nested.h for the full StructArray definition
  (it is only forward-declared in type_fwd.h).
- wrap the CheckDeserialize helper in an anonymous namespace to avoid a
  link-time collision with the identically named helper in opaque_test.cc.
@hoeze-minion
hoeze-minion force-pushed the feat/arrow-range-extension branch from 78f3593 to 8198463 Compare May 24, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants