Skip to content

Query.to_search_structure raises RecursionError on a long chain of ORs #100

Closed

Description

@stevevanhooser

What happens

Building an OR of many queries one at a time, the way a loop naturally does, gives a search structure nested one level per OR:

q = Query("base.id", "exact_string", ids[0])
for i in ids[1:]:
    q = q | Query("base.id", "exact_string", i)
db.search(q)   # RecursionError with ~1,000 ids

SQLiteDB.search calls query_obj.to_search_structure(). That calls Query._resolve_search_structure → _resolve_single, and the or branch of _resolve_single recurses into param1/param2. With about 1,000 ids that exceeds Python's default recursion limit (1000), so the search fails before it runs.

Why it matters

ORing a batch of ids built in a loop is a normal pattern. NDI, for example, looks up several documents by id at once.

The search itself already handles this depth. Since #98, _base_id_lookup_values walks the OR tree with a stack, and ORed base.id lookups run as one SQL query per 1,000 ids. The recursion in to_search_structure is the only thing in the way.

Found by

Writing tests/test_search_indexes.py (#98). A test for a left-nested OR of 1,200 ids was dropped there because of this.

Possible fix

Make _resolve_search_structure / _resolve_single iterative: walk the tree with an explicit stack and rebuild the resolved copy bottom-up.

Or rebalance a long left-nested OR chain into a balanced tree when Query.__or__ builds it. That keeps depth at log₂(n), but it changes the shape of search_structure, which callers might inspect.

Then restore the dropped test: an OR of 1,200 ids built in a loop returns all of them.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions