Skip to content

perf(candid): de-quadratic subtype checking (#603) - #780

Open
parthod0x wants to merge 5 commits into
dfinity:masterfrom
parthod0x:fix/subtype-quadratic-603
Open

parthod0x wants to merge 5 commits into
dfinity:masterfrom
parthod0x:fix/subtype-quadratic-603

Conversation

@parthod0x

Copy link
Copy Markdown

Fixes #603.

Subtype checking was O(depth²) on nested types. Type derives a deep
structural PartialEq, and subtype_ / subtype_collect_ used
if t1 == t2 as a per-level short-circuit — so the deep compare re-ran at
every recursion level, re-traversing the surrounding structure each time
(vec^d(leaf) → O(d) + O(d−1) + … = O(d²)). That == is also load-bearing
for correctness: it is the only source of reflexivity for base types that
have no structural subtype arm, so it cannot simply be removed.

Fix

  • Add same_type_fast(t1, t2): Rc::ptr_eq fast-accept, then for leaves
    (and Var/Knot) an O(1) leaf == t2 check, while composites return
    false so the existing structural arms descend one level — no deep
    re-traversal. It only ever returns true when == would, so it is sound.
  • Use same_type_fast for the per-level short-circuit in subtype_ and
    subtype_collect_.
  • Keep a single deep if t1 == t2 { return } at the public entry points
    (subtype_with_config, subtype_check_all), before allocating
    Assumptions — so a wholly-equal pair is still decided in one linear pass
    and never enters the recursion.

This differs from the same_type_fast sketch in the issue thread, which
dropped the top-level deep ==: separately-allocated-but-equal composite
trees then fell through into the full structural recursion and regressed
3.8–6.9×. Restoring the single top-level == keeps the large unequal-case
speedups and avoids that equal-case regression. The decode data path
(de.rs → subtype_with_config) benefits from both.

Benchmark (examples/subtype_bench.rs, median ns/op/depth)

vec^d(leaf), depths 16–512, asserting the verdict on every run:

case before (d512) after (d512) shape
widening (succeeds) 607,669 ns 19,499 ns ns/op/depth 1187 → ~38 (flat = linear), ~31×
rejection (fails at leaf) 603,823 ns 17,001 ns 1179 → ~33 (flat), ~36×
equal, separately allocated 2,053 ns ~2,543 ns stays flat ~5 ns/op/depth = still the single-pass path, no regression

(Absolute timings are noisy — the machine was under full CPU load — but the
ns/op/depth shape is the signal: before grows with depth, after is flat.)

Tests

  • cargo test -p candid: 130 passed, 0 failed (subtype_assumptions,
    subtype_lookup, subtype_recursion_limit all green).
  • cargo clippy -p candid --all-targets -- -D warnings: clean.
  • Verified on the host target; CI will exercise the wasm32 build/bench. Happy
    to also port the repro into rust/bench as a canbench case if you'd like
    the speedup reflected in CI bench output.

@parthod0x
parthod0x requested a review from a team as a code owner October 28, 2026 14:47
@zeropath-ai

zeropath-ai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

✅ No security or compliance issues detected. Reviewed everything up to ca4c02b.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► rust/candid/examples/subtype_bench.rs
    Add micro-benchmark for deep subtype traversal (#603)
Enhancement ► rust/candid/src/types/subtype.rs
    Introduce memoized deep equality and per-type pointer identity cache to address quadratic subtype traversal (#603)
Enhancement ► rust/candid/tests/subtype_equal_subtree.rs
    Add tests for equal subtree handling under recursion guard (#603)

@cla-idx-bot

cla-idx-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

Dear @parthod0x,

In order to potentially merge your code in this open-source repository and therefore proceed with your contribution, we need to have your approval on DFINITY's CLA.

If you decide to agree with it, please visit this issue and read the instructions there. Once you have signed it, re-trigger the workflow on this PR to see if your code can be merged.

— The DFINITY Foundation

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Valid subtype checks can now fail with recursion-limit errors on deeply nested equal subtrees, including trees reached through different aliases in both subtype APIs.

1 open issue
What changed in this PR

Addresses #603 by removing repeated deep equality comparisons from Candid’s recursive subtype checks while retaining a fast path for wholly equal types.

Changes:

  • Adds pointer and leaf equality checks within both subtype implementations.
  • Keeps one deep equality check at each public entry point.
  • Adds a microbenchmark for widening, rejection, and separately allocated equal types.
File Description
rust/​candid/​src/​types/​subtype.rs Replaces per-level deep equality with a lightweight check.
rust/​candid/​examples/​subtype_bench.rs Measures subtype-checking performance across increasing nesting depths.

Review effort: Balanced

Comment thread rust/candid/src/types/subtype.rs Resolved
@parthod0x

Copy link
Copy Markdown
Author

Good catch — both cases were real. Fixed in fbd60df.

The per-level reflexivity check inside the recursion was a bare ptr_eq/leaf test. It's O(1), but it stops recognising separately-allocated equal composite subtrees. When such a subtree's parents differ — so the entry-point whole-type == can't fire — it was walked level by level into the depth guard, so a valid, reflexive check could return "recursion limit exceeded". The alias case is the same gap reached through two different Var names.

Fix: the per-level check is a deep structural equality again — it returns exactly what t1 == t2 would, so the acceptance is restored — but it's memoized by pointer identity, so each subtree pair is compared at most once across the whole check. That keeps #603's de-quadratic behaviour. The benchmark gains a subtree row for exactly this shape (record { 0: vec^d nat; 1: nat } <: record { 0: vec^d nat }), and it stays linear in depth:

## subtree   record{vec^d nat; nat} <: record{vec^d nat}
 depth |          ns/op |    ns/op/depth
    16 |         3306.6 |        206.661
   128 |        16544.9 |        129.257
   512 |        49531.3 |         96.741

Verification:

  • An inline oracle test asserts same_type agrees with the derived == across leaves, records, variants, funcs, services, classes and nested composites, so the memoized walk can't drift from ==.
  • Small-stack integration tests (tests/subtype_equal_subtree.rs) check that a deep equal subtree under differing parents, and one reached through two different aliases, is accepted at a depth where a mismatched chain of the same depth does hit the limit. Both assertions fail on the pre-fix check and pass now.
  • The memo's keys are pinned alive for the duration of the check, so a freed address can't be reused under a live key.
  • cargo test -p candid, cargo clippy -p candid --all-targets (no warnings) and cargo fmt --check are all green.

Subtype checking short-circuited every recursion level with `t1 == t2`,
a deep structural comparison over `Rc<TypeInner>`. Because it ran at each
level, a mismatch deep inside an otherwise-identical type tree re-compared
the surrounding structure on the way down, costing O(depth^2).

Replace the per-level deep compare with `same_type_fast`: an `Rc::ptr_eq`
fast-accept plus leaf/indirection equality, letting composite types descend
through the existing structural match arms instead of re-traversing the
whole subtree. Keep a single deep `t1 == t2` at the public entry points
(`subtype_with_config`, `subtype_check_all`), so a wholly-equal pair is
still decided in one linear pass and never enters the structural recursion.

Unlike a bare `Rc::ptr_eq`-only short-circuit, separately allocated but
structurally equal types stay on that fast path and are not regressed.

Native release micro-benchmark (vec-nesting by depth, median ns/op over 9
runs; see examples/subtype_bench.rs):

  widening  vec^d(nat) <: vec^d(int)   depth 512: 607669 -> 19499 ns  (31x)
  rejection vec^d(nat) <: vec^d(text)  depth 512: 603823 -> 17001 ns  (36x)
  equal     vec^d(nat) <: vec^d(nat)   depth 512:   2053 ->  2543 ns

ns/op/depth is roughly constant for all three after the change (widening
~40, rejection ~50, equal ~5), confirming linear behaviour; before, the
widening/rejection cases grew from ~60 to ~1180 ns/op/depth.
The per-level reflexivity check inside the subtype recursion was a bare
ptr_eq/leaf test. It is O(1), but it stopped detecting separately
allocated equal composite subtrees. When such a subtree's parents differ
-- so the entry-point whole-type `==` cannot fire -- it was walked level
by level into the recursion depth guard, turning a valid, reflexive
subtype check into a "recursion limit exceeded" error. The same gap
affected equal trees reached through different aliases.

Make the per-level check a deep structural equality again, so it returns
exactly what `t1 == t2` would and that acceptance is restored, but
memoize it by pointer identity so each subtree pair is compared at most
once across the whole check. This keeps the de-quadratic behaviour of
dfinity#603: the subtree benchmark stays O(depth), not O(depth^2).

The memo's keys are pinned alive for the check, so a freed address can
never be reused under a live key.

Tests: an inline oracle asserting same_type agrees with the derived `==`
across leaves, records, variants, funcs, services, classes and nested
composites; small-stack integration tests that a deep equal subtree
under differing parents, and via different aliases, is now accepted
where it previously hit the limit; and the subtree case in the bench.
@parthod0x
parthod0x force-pushed the fix/subtype-quadratic-603 branch from fbd60df to 170902e Compare November 10, 2026 07:11
Copilot AI balanced review requested due to automatic review settings November 10, 2026 07:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The equality fallback caches only outer function/service type pairs, leaving nested descendants uncached and causing O(depth²) equality work during subtype checks.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread rust/candid/src/types/subtype.rs Resolved
…near

same_type memoized deep structural equality for Opt/Vec/Record/Variant/Class
but left Func and Service to the derived `==`. The subtype recursion calls
same_type at every level, so a deeply nested func-returning-func (or service)
chain that differs only at the bottom leaf re-walked the whole tail with `==`
at each enclosing level -- the dfinity#603 quadratic, still live for these two types.

Add explicit Func/Service arms that compare the non-type metadata (modes and
method names, which hold no nested Type) directly and recurse child types
through same_type, so each nested pair is memoized and the traversal stays
linear. The verdict is unchanged -- same_type still returns exactly what `==`
would; only the cost drops.

Extends the agreement oracle with separately-allocated deep func and service
chains, equal and leaf-mismatched.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 07:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The recursive same_type traversal bypasses the recursion-depth guard and can overflow the stack and abort the process in both subtype APIs when comparing deep equal subtrees beneath unequal record parents.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread rust/candid/src/types/subtype.rs Resolved
same_type recursed one native frame per nesting level without the stack guard
every other structural recursion in this file uses. On a separately allocated
equal subtree too deep for the equality check to verify -- deeper than the
structural limit, since same_type's frames are lighter -- it could overflow the
native stack and abort the process, instead of failing with the graceful
"Recursion limit exceeded" the subtype checker returns elsewhere (PR dfinity#780
review).

Thread the same RecursionDepth through same_type: when the stack runs low it
stops and returns "not trivially equal", so the caller falls through to the
guarded structural rules, which end the whole check gracefully. The bail means
"undetermined here", not "unequal", so a verdict is cached only when no limit
was hit in the subtree.

Adds an integration test driving an equal subtree well past the equality
check's own stack limit and asserting a graceful error rather than a crash.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 07:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

When stack size cannot be detected, the calibration leaves insufficient depth budget for enclosing record and alias-resolution frames, causing both acceptance tests to exceed the 512-depth guard and fail.

1 open issue
1 resolved since last review

Review effort: Balanced

The deep-equal-subtree acceptance tests calibrated the subtree to the exact
recursion limit. Now that same_type shares the structural RecursionDepth, the
equality check of the subtree plus its enclosing record/alias frames must fit
under that one limit. On platforms where the stack size can't be measured the
guard is a flat 512-level count with no byte slack, so the wrapper frames pushed
these tests one or two levels over the limit and they failed there (PR dfinity#780
review).

Place the equal subtree a fixed headroom below the measured limit while keeping
the sanity check at the full depth, so the tests pass whether the guard measures
remaining stack or counts levels. The subtree is still hundreds of levels deep.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 08:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human review is needed to assess how equality probing interacts with the shared recursion guard and can incorrectly reject valid subtype relations.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@parthod0x

Copy link
Copy Markdown
Author

Summary for review — the automated passes are now clean (no open issues), and the latest Copilot note correctly defers one point to human judgment: how the reflexive-equality fast path (same_type) should interact with the shared recursion guard. Worth stating the trade-off explicitly since it's a design choice, not an accident.

What this PR does. subtype ran a deep structural t1 == t2 at every recursion level (#603), which is O(depth²). It's now replaced by same_type, a pointer-memoized deep equality that returns exactly what == would but compares each subtree pair at most once, so the per-level check is O(1)-amortized. Func/Service recurse through it like the other composites, and it shares the structural RecursionDepth.

The trade-off (the point Copilot flags for a human). Because same_type now shares the recursion budget, a reflexively-valid relation whose equal subtree is deeper than the budget is reported as Recursion limit exceeded — the same limit the structural rules already impose on every other too-deep type — rather than being accepted. The two coherent options:

  1. (this PR) Respect the shared limit. One recursion bound for all paths; a too-deep type is rejected uniformly and nothing can overflow the native stack. Simple and consistent with how subtype already treats depth. Cost: an equal subtree deeper than the limit is rejected even though it is reflexively a subtype.
  2. Let equality transcend the limit via an explicit heap-backed worklist, so arbitrarily deep equal subtrees are accepted without recursing. More permissive, but a second traversal mechanism alongside the existing guarded recursion, and only for the reflexive case.

I went with (1) as the smaller, more consistent change — candid already rejects over-deep types by design (utils::check_recursion_depth), so applying the same bound to the equality shortcut keeps one policy and removes the unguarded stack-overflow path entirely. Happy to switch to (2) if you'd rather deep equal types always be accepted; it's a self-contained follow-up.

Behaviour within the limit is unchanged (the oracle test pins same_type ≡ ==), and the deep-recursion tests cover both the accept case (equal subtree just under the limit) and graceful failure well past it, in both the stack-measured and the fixed-512-count guard modes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: Subtype checking should not perform quadratic traversal

2 participants