feat[next]: NeighborConnectivity[Domain, Codomain] declarations, NeighborTableType, and shared local dimensions on every backend - #2907
Draft
egparedes wants to merge 13 commits into
Conversation
This was referenced Sep 21, 2026
egparedes
added this pull request to stack #2900
September 22, 2026 06:59
egparedes
force-pushed
the
connectivities-as-types-3-neighbor-connectivity
branch
from
September 23, 2026 16:09
d1e1aeb to
8c55110
Compare
egparedes
force-pushed
the
connectivities-as-types-3-neighbor-connectivity
branch
2 times, most recently
from
September 24, 2026 10:12
bf5c950 to
90252b3
Compare
egparedes
removed this pull request from stack #2900
September 24, 2026 10:14
egparedes
added this pull request to stack #2917
September 24, 2026 10:14
This was referenced Sep 24, 2026
havogt
reviewed
Sep 25, 2026
| /, | ||
| *, | ||
| max_neighbors: Optional[int] = None, | ||
| min_neighbors: Optional[int] = None, |
Contributor
There was a problem hiding this comment.
Why not has_skip_values. Feels like the better concept, because there might not be a natural minimum (e.g. if you use it to express missing values close to the boundary).
…at know their owner
- check_neighbor_table: min_neighbors exceeding the table, bool tables - descriptive errors for non-integer indices, the undeclared base, DSL attributes - FieldOffset.Local, so the spelling works for legacy offsets in embedded too - fingerprint a declaration by its dimensions and counts, not only its name - negative counts are a ValueError
…mension Flattened sparse patterns (ICON4Py's C2CE, E2ECV, ...) index the same neighbor axis as another connectivity. A declaration can now adopt an owned local dimension; it is then named in the IR by its own tag (offset_tag), since the local dimension's tag already names the owner's table.
An annotated 'Local' -- on 'NeighborConnectivity' or on its metaclass -- makes every declaration's local dimension a *variable*: pyright then rejects 'Field[Dims[V, V2E.Local], float]', and mypy rejects an adopted or shared one. Annotate it nowhere; library code reads it through 'common.local_dimension_of', and adoption and sharing are written 'Local: TypeAlias = ...'. Also from the review: a redefined declaration re-owns an adopted local dimension instead of becoming a sharer, and its counts are checked against the local dimension's own 'size='. The missing-'Local' error names both spellings. Adds pyright over 'typing_tests/pyright_probes.py' to the typing session, which is what catches a regression here: the mypy cases cannot.
Now that local dimensions can state their size, the local dimension of 'make_const_list' results is one; a declaration cannot adopt it, since it belongs to no connectivity.
Reductions, sparse arguments and list materialization used to find a connectivity by the local dimension's tag, which only names the owner's table. They now look up a table over the local dimension (common.connectivity_key_over), so a connectivity sharing another one's local dimension works on every backend, bound on its own or together with its owner. DaCe sizes a connectivity array's local dimension from its own table. Lifts the PR 1 xfail markers.
- DaCe if/scan/concat_where/const-list sites find the table over the local dimension - connectivity_key_over takes a tag, avoids resolve, and picks deterministically - embedded map_list compares lists by local dimension, not by offset - a sharer must have its owner's origin; counts are compared one by one - ADR 0029: sharers must have the owner's neighbor structure
…Type and ts.ShiftType
- 'NeighborConnectivity[Origin, Codomain]' becomes '[Domain, Codomain]', and
the class attribute 'origin' becomes 'domain': a bound table is a field over
(Domain, Local) with values in Codomain. 'origin' already names a buffer
origin ('__gt_origin__').
- 'NeighborConnectivityType' becomes 'NeighborTableType': the declaration the
table is bound to, dtype, skip value and max_neighbors, with domain and
codomain derived from the declaration. It is built from the offset-provider
key ('offset_provider_to_type', 'check_neighbor_table', which now returns
it), since a sharer's table looks like its owner's; local dimensions record
their sharers for that. A table under a key no declaration answers to keeps
its structural 'ConnectivityType', which is also what
'NeighborTable.__gt_type__()' returns now. 'OffsetProviderType' becomes
'TableTypes'.
- 'ts.OffsetType(source, target)' becomes 'ts.ShiftType(codomain, domain)',
printed 'Shift[<tag>: <codomain> -> <domain>]'.
- ADR 0029 describes the result; ADR 0019 names the new record.
ADR 0028 on main is now 'Plain Builders Instead of factory-boy Factories' (#2808), so the two ADRs of this stack move up by one.
A dimension is local if and only if it subclasses `LocalDimensionIndex` (`common.is_local_dimension`), so `DimensionKind` loses its `LOCAL` member and is `HORIZONTAL | VERTICAL`. A local dimension's `kind` is `None`, and declaring one with `kind=` is a `TypeError`; `None` rather than `HORIZONTAL` keeps every backend comparison against `HORIZONTAL` / `VERTICAL` meaning what it meant. `order_dimensions` sorts by an explicit rank (horizontal, local, vertical), so sparse-field layouts do not move. Displays derive the label from the class: `Local[local]`, the `ₗ` IR suffix and DaCe's `_gtx_localdim` map variables are unchanged. The tree's `DimensionIndex(kind=LOCAL)` declarations become owner-less `LocalDimensionIndex` subclasses. pyright probes pin the Cartesian axis levels of ADR 0029, including staggering and shifting a local dimension, with `reportUnnecessaryTypeIgnoreComment` so a rejection that stops firing fails the session. ADR 0030 records the change.
egparedes
force-pushed
the
connectivities-as-types-3-neighbor-connectivity
branch
from
October 2, 2026 02:14
90252b3 to
e565d9c
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2899. Part 2 of 4 of the connectivities as types stack; see ADR 0030 in this PR (numbered 0030 because main's #2808 took 0028 and #2899's ADR moved to 0029).
What
A neighbor connectivity can be declared as a class, with its local dimension nested in it:
Additive for user code:
FieldOffsetand tag-keyed offset providers keep working. Migrating the tree to the declarations, class-keyed providers and the removal ofFieldOffsetare #2910. The renames below (NeighborTableType,TableTypes,ts.ShiftType) do change names ofcommonand the type system.Declarations
LocalDimensionIndex:owner(the declaring connectivity, orNone),sharers, and the optionalmax_neighbors/min_neighborscounts. A local dimension with no table (class LsqCoeff(LocalDimensionIndex, size=3)) hasowner is None.common.ConstList, the local dimension ofmake_const_list, is one of those, of size 1, and cannot be adopted.NeighborConnectivity[Domain, Codomain](metaclassConnectivityMeta): for eachDomainelement, a list ofCodomainneighbors; the class attributes areV2E.domainandV2E.codomain. A bound table is a field over(Domain, Local)with values inCodomain, the same use of "domain" asConnectivity.domainandCartesianConnectivity.domain_dim; "origin" is avoided because gt4py already uses it for a buffer's start (__gt_origin__). The declaration is never instantiated, and is checked at class creation (aLocal, both dimensions non-local, consistent counts).Local: typing.TypeAlias = C2E.Local-- ICON4Py's flattened sparse offsets (C2CE,E2ECV, ...) need this. The owner staysC2E; a sharer must have the owner's domain, and is named in the IR by its own tag (offset_tag), since the local dimension's tag names the owner's table.Localis annotated nowhere: an annotation on the base or the metaclass makesV2E.Locala variable for the checkers, soField[Dims[V, V2E.Local], float]would be rejected (by pyright for a nestedLocal, by mypy for an adopted or shared one). Library code reads it throughcommon.local_dimension_of(conn). pyright now runs in the typing nox session overtyping_tests/pyright_probes.py; it is what catches this regression, mypy accepts it.common.check_neighbor_table(V2E, table_or_type)checks a table against the declaration (domain, codomain, integral dtype, neighbor counts, skip values) and returns itsNeighborTableType.Localness is the class:
DimensionKind.LOCALis removedA dimension is local if and only if it subclasses
LocalDimensionIndex(common.is_local_dimension(dim)), soDimensionKindisHORIZONTAL | VERTICAL. A local dimension'skindisNone, andkind=on one is aTypeError.None, notHORIZONTAL, so that every backend comparison againstHORIZONTAL/VERTICALkeeps its meaning.order_dimensionssorts by an explicit rank (horizontal, local, vertical), so sparse-field layouts do not move, and displays derive the label from the class (Local[local], theₗIR suffix, DaCe's_gtx_localdimmap variables). The tree'sDimensionIndex(kind=LOCAL)declarations become owner-lessLocalDimensionIndexsubclasses here.The type of a bound table:
NeighborTableTypecommon.NeighborConnectivityTypebecomescommon.NeighborTableType:connectivity(the declaration the table is bound to),dtype,skip_valueandmax_neighbors, withdomainandcodomainderived from the declaration.OffsetProviderTypebecomescommon.TableTypes(Mapping[Tag, NeighborTableType]). ADR 0019's rule stands: transformations and code generation see these records, never tables.A table cannot name its declaration -- a sharer's table has the same domain as its owner's -- so the record is built from the offset-provider key:
offset_provider_to_typefinds the declaration whoseoffset_tagis the key among the owner and the sharers of the table's local dimension, andcheck_neighbor_tablereturns it.NeighborTable.__gt_type__()returns only what the table knows, the structuralcommon.ConnectivityType. A table bound under a name no declaration answers to -- hand-written IR, and everyFieldOffset-- keeps that structural type as itsconnectivity, so the IR level works unchanged. The records are dataclasses fingerprinted through their fields, so the declaration is part of the key of everything compiled for it.Frontend
ts.OffsetType(source, target)becomests.ShiftType(codomain, domain, tag), printedShift[<tag>: <codomain> -> <domain>].domainhas one dimension for a Cartesian shift and forV2E[i], two forV2E.V2E.__gt_type__()is the shift type of the equivalentFieldOffset, whose tag is theoffset_tag.V2E.Localin DSL code types as the local dimension; embeddedpremapaccepts the class; grid-type deduction treats it as unstructured. DSL attribute access is limited to attributes that are types, soV2E.domainin DSL code is an error rather than a leak of the type's fields.Backends: shared local dimensions work everywhere
A sharer brings back an offset tag that differs from its local dimension's tag, which #2898's regression matrix had marked xfail on several backends. They all work now:
common.connectivity_key_over(provider, local_dim)finds the key of a bound table over a local dimension: the owner's if bound, else a sharer's (the smallest key, so the choice does not depend on provider order). It replaces the lookups by the local dimension's tag in embedded reductions,unroll_reduce, gtfn sparse arguments, iterator-embedded sparse lists and list materialization, and DaCemake_field, reductions with skip values,map_list,if/scan/concat_whereand const lists.sdfg_args.local_dimension_size).uses_offset_tag_differing_from_local_dimand…_in_reductionare removed with their skip-list entries: every cell they marked passes.Tests
unit_tests/test_neighbor_connectivity.py: owner and sharer wiring, counts, owner-less locals, identity, pickling,resolve, declaration errors,check_neighbor_tablemismatches,NeighborTableTypebound by key (owner, sharer, undeclared),connectivity_key_over, fingerprints.feature_tests/ffront_tests/test_neighbor_connectivity.py: a shift, a reduction, a sparse argument and a program, on every backend; shift and reduction through a shared local dimension, with and without the owner bound, on different tables so using the wrong one fails.typing_tests:V2E.Localin annotations for all three spellings, under mypy and pyright. The pyright probes also pin feat[next]: a concrete Dimension is a class, identified by its qualified name #2899's Cartesian axis levels (staggering or shifting a mesh location or a local dimension is rejected), withreportUnnecessaryTypeIgnoreCommentso that a rejection that stops firing fails the session.DimensionKind.LOCALremoval:is_local_dimension,kind is None, display and layout order.Verification
Run locally (CPU) on 4dfda4856 (the head adds only the rebase onto that ADR-table fix); 0 failures in every row that ran:
test_next(internal, cpu, nomesh): embedded, roundtrip, gtfntest_next(dace, cpu, nomesh)src/gt4py/nextdoctests (in each session)test_typing_exports(mypy cases, pyright probes)pre-commit run --all-filesGitHub CI on the pushed head passes every check:
test_next(internal and dace; nomesh and atlas) on 3.12 and 3.14,test_cartesian,test_eve,test_storage, notebooks, package,typing-exportsand code quality. The local 3.13/3.14 runs were stopped in favour of CI.