Skip to content

Walk long OR chains iteratively instead of recursing - #103

Merged
stevevanhooser merged 2 commits into
mainfrom
claude/did-base-id-index
Oct 24, 2026
Merged

stevevanhooser merged 2 commits into
mainfrom
claude/did-base-id-index

Conversation

@stevevanhooser

@stevevanhooser stevevanhooser commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Closes #100.

Problem

q = q | Query(...) in a loop nests one OR per term. Three places on the search path recursed once or more per level, so a few hundred to ~1,000 terms raised RecursionError before the search ran:

where frames per OR level failed at roughly
Query.to_search_structure 2 1,000 terms (any OR)
SQLiteDB._search_doc_ids (an OR that isn't all base.id lookups) 2–3 330–500 terms
datastructures.field_search (brute force, Database.search) 1 1,000 terms

On main, to_search_structure on a 3,000-term chain built in a loop raises RecursionError.

Change

  • datastructures.or_operands flattens a chain of nested ORs into its operands, left to right, using an explicit stack.
    • A negated OR (~or) stays a single operand, because the negation applies to its whole result.
    • Empty operands pass through unchanged; each caller treats them as before.
  • _search_doc_ids and field_search take the union over those operands instead of recursing.
  • Query._resolve_search_structure rebuilds the structure bottom-up with an explicit stack (_resolve_children / _resolve_node). The rule applied to each node is unchanged; the leaf rules moved to _resolve_leaf.

Tests

tests/test_deep_or.py runs at 3× the recursion limit (3,000 terms):

  • searching a loop-built OR of ids;
  • searching a loop-built OR of other terms;
  • the same, ANDed with a session;
  • field_search on a deep OR;
  • the new resolver compared against a copy of the old recursive one on 300 random query trees (ANDs, ORs, ~ors, isa, depends_on, ~depends_on, '*');
  • the same subquery reached twice;
  • or_operands: operand order and ~or handling.

Local runs:

  • black --check and ruff check are clean.
  • pytest: 830 passed.
  • Bridge coverage against the DID-matlab #232 branch: 0 problems.

DID-matlab

VH-Lab/DID-matlab#232 (0de9cf2) makes the same fix in MATLAB: it adds did.datastructures.orOperands, used by fieldsearch and did.database/search_doc_ids.

Bridge changes here:

  • new entry orOperands ↔ or_operands;
  • fieldsearch and database now point at 0de9cf2.

Merge DID-matlab #232 first, with a merge commit. Until then, 0de9cf2 exists only on that PR's branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia

q = q | Query(...) in a loop nests one OR per term. Three places on the
search path recursed once (or more) per level, so a few hundred to ~1,000
terms raised RecursionError before the search ran:

- Query.to_search_structure (_resolve_search_structure/_resolve_single):
  now an explicit-stack, post-order rebuild (_resolve_children /
  _resolve_node); per-node behaviour unchanged, leaf rules moved to
  _resolve_leaf.
- SQLiteDB._search_doc_ids for an OR that is not all base.id lookups: now
  unions over the flattened operands.
- datastructures.field_search for OR (brute force, Database.search): same.

datastructures.or_operands flattens a chain of nested ORs into its operands,
left to right, with a stack. A negated OR ('~or') is kept as one operand,
since its negation applies to its whole result; empty operands are passed
through for each caller to treat as before.

Tests (tests/test_deep_or.py), at 3x the recursion limit (3,000 terms):
an OR of ids and an OR of other terms, built in a loop, searched; the same
ANDed with a session; field_search on a deep OR; the iterative resolver
compared with a copy of the old recursive one on 300 random query trees
(ANDs, ORs, ~ORs, isa, depends_on, ~depends_on, '*'); a shared subquery;
or_operands' order and ~or handling. On main, to_search_structure on a
3,000-term chain raises RecursionError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
DID-matlab 0de9cf2 makes the same fix there: a new
did.datastructures.orOperands, used by fieldsearch and
did.database/search_doc_ids. New orOperands entry (ported as or_operands);
fieldsearch and database entries -> 0de9cf2, whose change this PR already
ports in field_search and SQLiteDB._search_doc_ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia

Copy link
Copy Markdown
Contributor Author

Bridge coverage vs DID-matlab is red on ada3e89, and this is expected until VH-Lab/DID-matlab#232 merges.

The check reports one problem:

orOperands: matlab_path does not exist: src/did/+did/+datastructures/orOperands.m

CI checks out DID-matlab main, and that file is added by #232. Every other check passes. Against #232's branch (0de9cf2) the bridge check reports 0 problems.

Dropping the entry from this PR wouldn't remove the red window, it would just move it. Once #232 merges, main here would go red with "MATLAB file has no bridge entry".

So the order is:

  1. Merge DID-matlab#232, with a merge commit.
  2. Re-run this check.
  3. Merge this PR.

Generated by Claude Code

@stevevanhooser
stevevanhooser merged commit 8c3ac14 into main Oct 24, 2026
12 of 13 checks passed
@stevevanhooser
stevevanhooser deleted the claude/did-base-id-index branch October 24, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Query.to_search_structure raises RecursionError on a long chain of ORs

2 participants