Repository navigation
Walk long OR chains iteratively instead of recursing - #103
Merged
Merged
Conversation
q = q | Query(...) in a loop nests one OR per term. Three places on the
search path recursed once (or more) per level, so a few hundred to ~1,000
terms raised RecursionError before the search ran:
- Query.to_search_structure (_resolve_search_structure/_resolve_single):
now an explicit-stack, post-order rebuild (_resolve_children /
_resolve_node); per-node behaviour unchanged, leaf rules moved to
_resolve_leaf.
- SQLiteDB._search_doc_ids for an OR that is not all base.id lookups: now
unions over the flattened operands.
- datastructures.field_search for OR (brute force, Database.search): same.
datastructures.or_operands flattens a chain of nested ORs into its operands,
left to right, with a stack. A negated OR ('~or') is kept as one operand,
since its negation applies to its whole result; empty operands are passed
through for each caller to treat as before.
Tests (tests/test_deep_or.py), at 3x the recursion limit (3,000 terms):
an OR of ids and an OR of other terms, built in a loop, searched; the same
ANDed with a session; field_search on a deep OR; the iterative resolver
compared with a copy of the old recursive one on 300 random query trees
(ANDs, ORs, ~ORs, isa, depends_on, ~depends_on, '*'); a shared subquery;
or_operands' order and ~or handling. On main, to_search_structure on a
3,000-term chain raises RecursionError.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
DID-matlab 0de9cf2 makes the same fix there: a new did.datastructures.orOperands, used by fieldsearch and did.database/search_doc_ids. New orOperands entry (ported as or_operands); fieldsearch and database entries -> 0de9cf2, whose change this PR already ports in field_search and SQLiteDB._search_doc_ids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
Contributor
Author
|
Bridge coverage vs DID-matlab is red on The check reports one problem: CI checks out DID-matlab Dropping the entry from this PR wouldn't remove the red window, it would just move it. Once #232 merges, So the order is:
Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #100.
Problem
q = q | Query(...)in a loop nests one OR per term. Three places on the search path recursed once or more per level, so a few hundred to ~1,000 terms raisedRecursionErrorbefore the search ran:Query.to_search_structureSQLiteDB._search_doc_ids(an OR that isn't allbase.idlookups)datastructures.field_search(brute force,Database.search)On
main,to_search_structureon a 3,000-term chain built in a loop raisesRecursionError.Change
datastructures.or_operandsflattens a chain of nested ORs into its operands, left to right, using an explicit stack.~or) stays a single operand, because the negation applies to its whole result._search_doc_idsandfield_searchtake the union over those operands instead of recursing.Query._resolve_search_structurerebuilds the structure bottom-up with an explicit stack (_resolve_children/_resolve_node). The rule applied to each node is unchanged; the leaf rules moved to_resolve_leaf.Tests
tests/test_deep_or.pyruns at 3× the recursion limit (3,000 terms):field_searchon a deep OR;~ors,isa,depends_on,~depends_on,'*');or_operands: operand order and~orhandling.Local runs:
black --checkandruff checkare clean.pytest: 830 passed.DID-matlab
VH-Lab/DID-matlab#232 (
0de9cf2) makes the same fix in MATLAB: it addsdid.datastructures.orOperands, used byfieldsearchanddid.database/search_doc_ids.Bridge changes here:
orOperands↔or_operands;fieldsearchanddatabasenow point at0de9cf2.Merge DID-matlab #232 first, with a merge commit. Until then,
0de9cf2exists only on that PR's branch.🤖 Generated with Claude Code
https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia