Type-safe filter expression API

[shivamka1]

Type-safe filter expression API

What this PR does

A filter is one thing everywhere: an expression tree. The Rust builder, Python and
GraphQL all produce it, one compiler turns it into typed expression structs and their
ops, the remote client sends it, and permission grants store it. The old composite and
builder paths are deleted, and so is the per-entity GraphQL grammar
(nodeFilter: { property: { name, where } }, window: { …, expr }).

Vocabulary used throughout: a term is a value taken from the node or edge under test
(property, temporalProperty, metadata, field, degree, isActive, an endpoint's
term through src/dst); a literal is a constant you write; an aggregate
collapses a list (sum, avg, min, max, first, last, len, earliest,
latest); a view scopes what a term sees.

g.filter(NodeFilter.degree().gt(5))?;
g.filter(EdgeFilter.src().name().eq("alice"))?;
g.filter(NodeFilter.property("score").temporal().max().ge(10i64))?;
g.filter(EdgeFilter.window(1, 3).property("p").eq(1u64))?;   // the view is on the term

filter.Node.property("score") > 4                 # a literal on the right
filter.Node.degree() > filter.Node.in_degree()    # another term on the right
filter.Node.window(0, 5).property("score") > 4    # a term inside a window

filter(expr: { node: { cmp: { op: GT,
                             lhs: { read: { property: { name: "score",
                                                        views: [{ window: { start: 0, end: 5 } }] } } },
                             rhs: { const: { f64: 4.0 } } } } })

The layers

1. Rust: filter/model/expr/mod.rs defines the tree. FilterExpr is
   node/edge/explodedEdge/view/and/or/not; each entity leg is an
   Expr<Leaf> over that entity's own terms, so an edge-only operation inside a node
   expression is a compile error. expr/builder.rs is the typed API: NodeFilter,
   EdgeFilter, ExplodedEdgeFilter, GraphFilter and their methods build the tree,
   so Rust callers, Python and GraphQL meet in the same data. expr/compile.rs turns a
   tree into the typed expression structs in node_expr/ (BinaryCmpExpr, StringExpr,
   PropValueSetExpr, UnaryExpr, AnyExpr/AllExpr, the aggregates,
   AndExpr/OrExpr/NotExpr), each of which owns its type rules and compiles to an
   op once. expr/stream.rs evaluates aggregates and any/all over a history without
   collecting it. expr/split.rs rewrites a tree, before it is compiled, into the
   views to apply and the two questions it answers (which nodes stay, which edges
   stay); compile.rs compiles that once for every place a filter runs.
2. Python: a filter object holds the tree and nothing else. Every builder method
   calls the Rust builder of the same name, running compiles the tree, repr() prints
   it back. Comparisons, membership, presence
   and any()/all() are refused where they are written when the left side's type is
   known statically.
3. GraphQL: one @oneOf input per entity (NodeExpr, EdgeExpr,
   ExplodedEdgeExpr), each declared plainly with the crate's derives and mirroring the
   core tree one variant to one type: const, read, cmp { op },
   str { op }, isIn/isNotIn, presence { op }, agg { op }, quantified { op },
   and/or/not; operators are enum values, a read is a one-of naming its term
   (property: { name, views }, field: { name: NAME, views }, src: { expr, views },
   …), so each term carries its own views and nothing else, and the argument-less view
   operations are one kind enum. The conversion to
   and from the tree is written once over a small per-entity trait. Every filter(expr:)
   and select(expr:) takes it. Policy placeholders (var, claim) are Value variants.
4. Remote Python: sends the tree as a FilterExpr! variable. The one filter that
   cannot travel, by_state_column, raises ValueError at the client.

Breaking changes

• GraphQL callers and stored grants written in the per-entity grammar must be
  rewritten as trees; nothing converts on load. The matching pometry-storage change is
  Pometry/pometry-storage#405: the store keeps trees, rejects the old grammar with a
  message that says what to rewrite, and its ABAC scanner reads placeholders out of the
  tree.
• applyViews is removed everywhere, with the ViewCollection, NodesViewCollection
  and GraphViewCollection inputs. filter(expr: { view: [ …ViewOps… ] }) does what they
  did, in the same order; a predicate that narrowed a collection is select(expr:). The
  view operations now include excludeNodes, subgraph, subgraphNodeTypes and valid,
  so nothing the collections offered is lost. Two behaviours move with it: layers and
  excludeLayers name layers that must exist, while validLayers and
  excludeValidLayers ignore unknown names as the collections did; and
  latest: false-style no-ops are refused. A property, degree or is_active term read
  through a view that hides nodes or edges answers for the hidden ones as it does for
  ones a window leaves out; a view written on name(), id() or node_type() is kept
  for display and ignored when the filter runs, since those fields have no time axis.
• An edge collection remembers what it holds. edges holds edges and
  edges.explode() / explode_layers() hold exploded edges; both stay one type,
  Edges (and NestedEdges, EdgesWindowSet) in Python and GraphQL and
  RemoteEdges / RemoteNestedEdges in the remote client. [] and select ask
  their question once per item of the collection they are called on: per edge on
  edges, per exploded edge on edges.explode(), whatever kind of predicate they
  are given. A view used as a predicate on edges therefore keeps whole
  edges and never trims their updates; graph.window(..).edges is the form that does.
• sum(), avg(), min() and max() require numeric elements, as before the tree;
  a string or boolean history is refused with the aggregate and element type named.
• The bundled UI (raphtory-graphql/resources/index.html) still issues the old
  grammar (nodeFilter/edgeFilter with where:) and applyViews, so the UI test
  jobs are red until the UI is rebuilt against schema.graphql.
• nodes.filter() defers uniformly for every filter kind (as Node ID and OR filter bug #2754 did upstream);
  membership narrowing is nodes[...], select, or graph.filter.

Semantics decided

• What a view leg means depends on where the filter is applied. On graph.filter
  a view leg of the top-level and applies first and the other legs run inside it, the
  way graph.window(..).filter(expr) does; nested ands flatten, and a view under or
  or not is refused there, since no graph is "windowed or filtered" or "not windowed".
  On a collection (nodes[..], edges[..], select) every leg is a test on the
  collection's own graph: a view leg asks that the item exist in the view, the other legs
  read the graph itself, and &, | and ~ have their plain set meanings, so
  edges[~Graph.window(0, 5)] is the edges that do not exist in the window. Two view
  legs are two tests on a collection and one chained view on a graph. Closes Negating a view-scoped filter is a no-op; the intended semantics are undefined #2718: a
  negated view is refused where it has no meaning and defined where it has one.
• A view on a term scopes only that term: Node.window(0, 5).property("score") is
  the score inside the window and the node comes back unwindowed. In GraphQL the views sit
  on the read itself and apply in list order. The view
  operations are window, at, after, before, latest, snapshotAt,
  snapshotLatest, layers, validLayers, defaultLayer,
  excludeLayers/excludeLayer, excludeValidLayers,
  shrinkStart, shrinkEnd, excludeNodes, subgraph, subgraphNodeTypes and valid,
  each equal to the graph view method of the same name.
• Literals keep db_v4's family rule: numbers compare by value across widths, a
  string never compares with a number, nothing is converted. is_in drops members
  outside the family. Two maps are comparable when every key they share is; a key only
  one side has does not get in the way, because a map property's declared type is the
  union of the shapes its values take.
• Endpoint terms keep the edge's views: Edge.window(0, 5).src().property("x")
  reads inside the window.
• Name equality resolves its node through the id index instead of scanning, as the
  id field does.
• Empty and/or/view are refused.
• A filter answers two questions, which nodes stay and which edges stay; a node
  predicate answers the second by "both ends stayed", an edge predicate keeps every
  node. and, or and not combine the answers question by question, so
  name == "b" | name == "c" keeps the edge b→c, ~ of a node predicate keeps the
  nodes that fail it and the edges between them, and ~(node & edge) negates each
  answer. An or between a node predicate and an edge predicate is the union of what each keeps: every node, and the edges either leg keeps (settled while fixing the grant union in Pometry/pometry-storage#405, where several roles' filters are combined with or). A node
  collection refuses any filter that tests edges. graph.edges[f] and
  graph.filter(f).edges agree for every tree.
• Invalid chains fail early, with the same message locally and remotely, and the
  message says what to write instead: a history compared without any/all, a
  qualifier on a single yes/no, an aggregate of a scalar, is_some on a term that
  always has a value, two terms of different types.

Performance

• Aggregates and any/all over a temporal history stream the history; nothing is
  collected into a list first.
• A node term decides the node's membership in its view once, and the erased graph it
  reads from is not re-boxed on every evaluation.
• Node-id and name filters resolve their domain through the storage index instead of
  scanning (the Node ID and OR filter bug #2754 narrowing, carried onto expressions).

Follow-ups filed

• Nodes created only through add_edge vanish from views that hide all their edges #2821: nodes created only through add_edge vanish from views that hide all their
  edges. Pre-existing core behaviour, found while checking the guide's examples.
• Pometry/pometry-storage#410: a claim-fed grant filter is compiled twice per request.
• A per-update predicate inside an edge-level test #2841: a per-update predicate inside an edge-level test
  (Edge.filter(ExplodedEdge.property("p2") == 4).is_valid()).
• Declare the type of an edge read per layer and per exploded update #2842: declare the type of an edge read per layer and per exploded update, so
  comparisons on edge metadata are type-checked when the filter is built.

Verification

suite	result
Rust: raphtory + raphtory-graphql + raphtory-tests (merged with db_v4 at 1d6ac0b32)	all passing; test_filters 670 scenarios, raphtory-graphql 247
Python test_base_install (incl. GraphQL e2e and local/remote parity)	3420 passed · 15 skipped · 9 xfailed; mypy over the stubs clean
pometry-storage #405 against this head	raphtory-auth incl. the restriction-equivalence property tests; integration-tests 111
CI on this PR	Rust, Python, benchmark and doc jobs green; UI jobs red until the UI bundle is updated

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite	Current: f3be241	Previous: 20e0972	Ratio
lotr_graph/num_edges_temporal	350 ns/iter (± 13)	151 ns/iter (± 0)	2.32
lotr_graph/num_edges_temporal_cached	336 ns/iter (± 16)	154 ns/iter (± 1)	2.18
lotr_graph_window_100/num_edges_temporal	362 ns/iter (± 11)	162 ns/iter (± 4)	2.23
lotr_graph_window_100/num_edges_temporal_cached	361 ns/iter (± 10)	161 ns/iter (± 0)	2.24
lotr_graph_subgraph_10pc/num_nodes	15 ns/iter (± 0)	4 ns/iter (± 0)	3.75
lotr_graph_subgraph_10pc/num_nodes_cached	16 ns/iter (± 0)	4 ns/iter (± 0)	4

This comment was automatically generated by workflow using github-action-benchmark.

[ljeub-pometry]

we should have another think about this, do we want the early failure in rust as well?

[shivamka1]

I'd keep it as is: Python refuses early where the type is known without a graph (fields, degree, is_*, booleans), with the same messages the engine gives at filter(..); Rust keeps infallible builders and reports at filter(..)?, since there the error is already one ? away and making eq/gt/… return Result would put a ? on every Rust filter.

[ljeub-pometry]

This file needs significant work to simplify the implementation, a lot of it is reimplementing stuff that comes from Repr, also a lot of the extra traits ideally would also just delegate to an implementation of Repr wherever possible as that is our trait for defining the output of __repr__ in python.