Skip to content

feat[next]: NeighborConnectivity[Domain, Codomain] declarations, NeighborTableType, and shared local dimensions on every backend - #2907

Draft
egparedes wants to merge 13 commits into
connectivities-as-types-2-dimension-classesfrom
connectivities-as-types-3-neighbor-connectivity
Draft

egparedes wants to merge 13 commits into
connectivities-as-types-2-dimension-classesfrom
connectivities-as-types-3-neighbor-connectivity

Conversation

@egparedes

@egparedes egparedes commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #2899. Part 2 of 4 of the connectivities as types stack; see ADR 0030 in this PR (numbered 0030 because main's #2808 took 0028 and #2899's ADR moved to 0029).

What

A neighbor connectivity can be declared as a class, with its local dimension nested in it:

class V2E(gtx.NeighborConnectivity[Vertex, Edge], max_neighbors=6, min_neighbors=5):
    class Local(gtx.LocalDimensionIndex): ...

@gtx.field_operator
def f(a: Field[Dims[Edge], float], s: Field[Dims[Vertex, V2E.Local], float]) -> Field[Dims[Vertex], float]:
    return neighbor_sum(s * a(V2E), axis=V2E.Local) + a(V2E[0])

Additive for user code: FieldOffset and tag-keyed offset providers keep working. Migrating the tree to the declarations, class-keyed providers and the removal of FieldOffset are #2910. The renames below (NeighborTableType, TableTypes, ts.ShiftType) do change names of common and the type system.

Declarations

  • LocalDimensionIndex: owner (the declaring connectivity, or None), sharers, and the optional max_neighbors / min_neighbors counts. A local dimension with no table (class LsqCoeff(LocalDimensionIndex, size=3)) has owner is None. common.ConstList, the local dimension of make_const_list, is one of those, of size 1, and cannot be adopted.
  • NeighborConnectivity[Domain, Codomain] (metaclass ConnectivityMeta): for each Domain element, a list of Codomain neighbors; the class attributes are V2E.domain and V2E.codomain. A bound table is a field over (Domain, Local) with values in Codomain, the same use of "domain" as Connectivity.domain and CartesianConnectivity.domain_dim; "origin" is avoided because gt4py already uses it for a buffer's start (__gt_origin__). The declaration is never instantiated, and is checked at class creation (a Local, both dimensions non-local, consistent counts).
  • A declaration can adopt a module-level local dimension or share another connectivity's, written Local: typing.TypeAlias = C2E.Local -- ICON4Py's flattened sparse offsets (C2CE, E2ECV, ...) need this. The owner stays C2E; a sharer must have the owner's domain, and is named in the IR by its own tag (offset_tag), since the local dimension's tag names the owner's table.
  • Local is annotated nowhere: an annotation on the base or the metaclass makes V2E.Local a variable for the checkers, so Field[Dims[V, V2E.Local], float] would be rejected (by pyright for a nested Local, by mypy for an adopted or shared one). Library code reads it through common.local_dimension_of(conn). pyright now runs in the typing nox session over typing_tests/pyright_probes.py; it is what catches this regression, mypy accepts it.
  • common.check_neighbor_table(V2E, table_or_type) checks a table against the declaration (domain, codomain, integral dtype, neighbor counts, skip values) and returns its NeighborTableType.
  • Fingerprinting: a declaration is fingerprinted by its name and by its dimensions and counts, so redefining it under the same name (a re-run notebook cell) does not reuse stale artifacts.

Localness is the class: DimensionKind.LOCAL is removed

A dimension is local if and only if it subclasses LocalDimensionIndex (common.is_local_dimension(dim)), so DimensionKind is HORIZONTAL | VERTICAL. A local dimension's kind is None, and kind= on one is a TypeError. None, not HORIZONTAL, so that every backend comparison against HORIZONTAL / VERTICAL keeps its meaning. order_dimensions sorts by an explicit rank (horizontal, local, vertical), so sparse-field layouts do not move, and displays derive the label from the class (Local[local], the ₗ IR suffix, DaCe's _gtx_localdim map variables). The tree's DimensionIndex(kind=LOCAL) declarations become owner-less LocalDimensionIndex subclasses here.

The type of a bound table: NeighborTableType

common.NeighborConnectivityType becomes common.NeighborTableType: connectivity (the declaration the table is bound to), dtype, skip_value and max_neighbors, with domain and codomain derived from the declaration. OffsetProviderType becomes common.TableTypes (Mapping[Tag, NeighborTableType]). ADR 0019's rule stands: transformations and code generation see these records, never tables.

A table cannot name its declaration -- a sharer's table has the same domain as its owner's -- so the record is built from the offset-provider key: offset_provider_to_type finds the declaration whose offset_tag is the key among the owner and the sharers of the table's local dimension, and check_neighbor_table returns it. NeighborTable.__gt_type__() returns only what the table knows, the structural common.ConnectivityType. A table bound under a name no declaration answers to -- hand-written IR, and every FieldOffset -- keeps that structural type as its connectivity, so the IR level works unchanged. The records are dataclasses fingerprinted through their fields, so the declaration is part of the key of everything compiled for it.

Frontend

  • ts.OffsetType(source, target) becomes ts.ShiftType(codomain, domain, tag), printed Shift[<tag>: <codomain> -> <domain>]. domain has one dimension for a Cartesian shift and for V2E[i], two for V2E. V2E.__gt_type__() is the shift type of the equivalent FieldOffset, whose tag is the offset_tag.
  • V2E.Local in DSL code types as the local dimension; embedded premap accepts the class; grid-type deduction treats it as unstructured. DSL attribute access is limited to attributes that are types, so V2E.domain in DSL code is an error rather than a leak of the type's fields.

Backends: shared local dimensions work everywhere

A sharer brings back an offset tag that differs from its local dimension's tag, which #2898's regression matrix had marked xfail on several backends. They all work now:

  • common.connectivity_key_over(provider, local_dim) finds the key of a bound table over a local dimension: the owner's if bound, else a sharer's (the smallest key, so the choice does not depend on provider order). It replaces the lookups by the local dimension's tag in embedded reductions, unroll_reduce, gtfn sparse arguments, iterator-embedded sparse lists and list materialization, and DaCe make_field, reductions with skip values, map_list, if/scan/concat_where and const lists.
  • DaCe sizes a connectivity array's local dimension from that connectivity's own table (sdfg_args.local_dimension_size).
  • The markers uses_offset_tag_differing_from_local_dim and …_in_reduction are removed with their skip-list entries: every cell they marked passes.

Tests

  • unit_tests/test_neighbor_connectivity.py: owner and sharer wiring, counts, owner-less locals, identity, pickling, resolve, declaration errors, check_neighbor_table mismatches, NeighborTableType bound by key (owner, sharer, undeclared), connectivity_key_over, fingerprints.
  • feature_tests/ffront_tests/test_neighbor_connectivity.py: a shift, a reduction, a sparse argument and a program, on every backend; shift and reduction through a shared local dimension, with and without the owner bound, on different tables so using the wrong one fails.
  • typing_tests: V2E.Local in annotations for all three spellings, under mypy and pyright. The pyright probes also pin feat[next]: a concrete Dimension is a class, identified by its qualified name #2899's Cartesian axis levels (staggering or shifting a mesh location or a local dimension is rejected), with reportUnnecessaryTypeIgnoreComment so that a rejection that stops firing fails the session.
  • DimensionKind.LOCAL removal: is_local_dimension, kind is None, display and layout order.

Verification

Run locally (CPU) on 4dfda4856 (the head adds only the rebase onto that ADR-table fix); 0 failures in every row that ran:

Check 3.12 3.13 3.14
test_next (internal, cpu, nomesh): embedded, roundtrip, gtfn 5059 passed passed (local) passed (local, CI)
test_next (dace, cpu, nomesh) 1513 passed passed (local) passed (CI)
src/gt4py/next doctests (in each session) 119 passed
test_typing_exports (mypy cases, pyright probes) 20 passed, pyright 0 errors 20 passed, pyright 0 errors 20 passed, pyright 0 errors
pre-commit run --all-files clean

GitHub CI on the pushed head passes every check: test_next (internal and dace; nomesh and atlas) on 3.12 and 3.14, test_cartesian, test_eve, test_storage, notebooks, package, typing-exports and code quality. The local 3.13/3.14 runs were stopped in favour of CI.

@egparedes
egparedes added this pull request to stack #2900 September 22, 2026 06:59
@egparedes
egparedes force-pushed the connectivities-as-types-3-neighbor-connectivity branch from d1e1aeb to 8c55110 Compare September 23, 2026 16:09
@egparedes
egparedes force-pushed the connectivities-as-types-3-neighbor-connectivity branch 2 times, most recently from bf5c950 to 90252b3 Compare September 24, 2026 10:12
@egparedes
egparedes removed this pull request from stack #2900 September 24, 2026 10:14
@egparedes
egparedes added this pull request to stack #2917 September 24, 2026 10:14
@egparedes egparedes changed the title feat[next]: NeighborConnectivity declarations and local dimensions that know their owner feat[next]: NeighborConnectivity[Domain, Codomain] declarations, NeighborTableType, and shared local dimensions on every backend Sep 24, 2026
Comment thread src/gt4py/next/common.py
/,
*,
max_neighbors: Optional[int] = None,
min_neighbors: Optional[int] = None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not has_skip_values. Feels like the better concept, because there might not be a natural minimum (e.g. if you use it to express missing values close to the boundary).

- check_neighbor_table: min_neighbors exceeding the table, bool tables
- descriptive errors for non-integer indices, the undeclared base, DSL attributes
- FieldOffset.Local, so the spelling works for legacy offsets in embedded too
- fingerprint a declaration by its dimensions and counts, not only its name
- negative counts are a ValueError
…mension

Flattened sparse patterns (ICON4Py's C2CE, E2ECV, ...) index the same neighbor
axis as another connectivity. A declaration can now adopt an owned local
dimension; it is then named in the IR by its own tag (offset_tag), since the
local dimension's tag already names the owner's table.
An annotated 'Local' -- on 'NeighborConnectivity' or on its metaclass -- makes
every declaration's local dimension a *variable*: pyright then rejects
'Field[Dims[V, V2E.Local], float]', and mypy rejects an adopted or shared one.
Annotate it nowhere; library code reads it through 'common.local_dimension_of',
and adoption and sharing are written 'Local: TypeAlias = ...'.

Also from the review: a redefined declaration re-owns an adopted local dimension
instead of becoming a sharer, and its counts are checked against the local
dimension's own 'size='. The missing-'Local' error names both spellings.

Adds pyright over 'typing_tests/pyright_probes.py' to the typing session, which
is what catches a regression here: the mypy cases cannot.
Now that local dimensions can state their size, the local dimension of
'make_const_list' results is one; a declaration cannot adopt it, since it
belongs to no connectivity.
Reductions, sparse arguments and list materialization used to find a
connectivity by the local dimension's tag, which only names the owner's table.
They now look up a table over the local dimension (common.connectivity_key_over),
so a connectivity sharing another one's local dimension works on every backend,
bound on its own or together with its owner. DaCe sizes a connectivity array's
local dimension from its own table. Lifts the PR 1 xfail markers.
- DaCe if/scan/concat_where/const-list sites find the table over the local dimension
- connectivity_key_over takes a tag, avoids resolve, and picks deterministically
- embedded map_list compares lists by local dimension, not by offset
- a sharer must have its owner's origin; counts are compared one by one
- ADR 0029: sharers must have the owner's neighbor structure
…Type and ts.ShiftType

- 'NeighborConnectivity[Origin, Codomain]' becomes '[Domain, Codomain]', and
  the class attribute 'origin' becomes 'domain': a bound table is a field over
  (Domain, Local) with values in Codomain. 'origin' already names a buffer
  origin ('__gt_origin__').
- 'NeighborConnectivityType' becomes 'NeighborTableType': the declaration the
  table is bound to, dtype, skip value and max_neighbors, with domain and
  codomain derived from the declaration. It is built from the offset-provider
  key ('offset_provider_to_type', 'check_neighbor_table', which now returns
  it), since a sharer's table looks like its owner's; local dimensions record
  their sharers for that. A table under a key no declaration answers to keeps
  its structural 'ConnectivityType', which is also what
  'NeighborTable.__gt_type__()' returns now. 'OffsetProviderType' becomes
  'TableTypes'.
- 'ts.OffsetType(source, target)' becomes 'ts.ShiftType(codomain, domain)',
  printed 'Shift[<tag>: <codomain> -> <domain>]'.
- ADR 0029 describes the result; ADR 0019 names the new record.
ADR 0028 on main is now 'Plain Builders Instead of factory-boy Factories' (#2808),
so the two ADRs of this stack move up by one.
A dimension is local if and only if it subclasses `LocalDimensionIndex`
(`common.is_local_dimension`), so `DimensionKind` loses its `LOCAL` member and
is `HORIZONTAL | VERTICAL`. A local dimension's `kind` is `None`, and declaring
one with `kind=` is a `TypeError`; `None` rather than `HORIZONTAL` keeps every
backend comparison against `HORIZONTAL` / `VERTICAL` meaning what it meant.
`order_dimensions` sorts by an explicit rank (horizontal, local, vertical), so
sparse-field layouts do not move. Displays derive the label from the class:
`Local[local]`, the `ₗ` IR suffix and DaCe's `_gtx_localdim` map variables are
unchanged.

The tree's `DimensionIndex(kind=LOCAL)` declarations become owner-less
`LocalDimensionIndex` subclasses. pyright probes pin the Cartesian axis levels
of ADR 0029, including staggering and shifting a local dimension, with
`reportUnnecessaryTypeIgnoreComment` so a rejection that stops firing fails
the session. ADR 0030 records the change.
@egparedes
egparedes force-pushed the connectivities-as-types-3-neighbor-connectivity branch from 90252b3 to e565d9c Compare October 2, 2026 02:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants