[stevevanhooser]

Closes #104. Port of VH-Lab/DID-matlab#234 (for VH-Lab/DID-matlab#233). The bridge records DID-matlab e4e02dc, #234's head, which also contains DID-matlab#235.

Update: #106 / DID-matlab#235 merged in

• main (with Test that depends_on matches as in DID-matlab, case for case #106) is merged into this branch. The one conflict was the database entry's matlab_last_sync_hash: Test that depends_on matches as in DID-matlab, case for case #106 set 124cf5e, this PR sets e4e02dc. e4e02dc is #234 with DID-matlab#235 merged in, so its history contains 124cf5e. Test that depends_on matches as in DID-matlab, case for case #106's decision_log notes are kept.
• What DID-matlab#235 changes: MATLAB now brute-forces a depends_on that SQL can't match exactly, and #234 keeps such operands out of its OR batches. Python already did both: _query_struct_to_sql_str returns None for them, so _search_or_operands leaves them to _search_doc_ids. No code change was needed. test_batched_or.py now also covers depends_on / ~depends_on with an empty name.

What changed

SQLiteDB._search_doc_ids used to search an OR with one SQL query per operand. Now the OR's operands go to a new method, _search_or_operands, which sorts them into three groups:

• Single SQL terms are searched up to _or_batch_terms() = _MAX_OR_BATCH_TERMS = 200 per query. A single SQL term is a non-negated operand that isn't an OR and for which _query_struct_to_sql_str returns a clause.
• base.id lookups among the operands are collected into one _search_base_ids call, as an all-base.id OR already was.
• Everything else goes to _search_doc_ids one operand at a time, as before: ANDs, nested ~ors, negated terms, and terms with no SQL form (which are brute-forced).

If a batch's SQL fails, its operands are retried one at a time, so each one falls back to brute force exactly as it would on its own.

The SQL, and why

This is the same approach as the MATLAB PR:

• Over a whole branch: one SELECT per operand, each exactly the query the operand would run on its own, joined with UNION. SQLite plans each member separately, so each keeps its index, and isa keeps its LIMIT -1 distinct-superclass subquery.
• Restricted to candidates: one query with the restricted _search_query_head(candidates) (CROSS JOIN … INDEXED BY doc_data_doc_idx) and the conditions joined by OR.

I measured the alternative, one ORed WHERE over a whole branch, on the SQL in the DID-matlab PR (#234). It loses the indexes: about 1 s for 10 names, 6 s for 100, and at 1,000 terms it exceeds SQLite's expression-depth limit. Restricted to candidates the opposite holds: every term reads every candidate's rows, so reading them once with an ORed WHERE wins. Full numbers are in VH-Lab/DID-matlab#234.

Timings

These use the real SQLiteDB, opened on a synthetic database: DID schema, 100,000 docs × 30 fields (3M doc_data rows), with SQLiteDB adding its search indexes on open.

• "names" are base.name exact_string terms.
• "mixed" rotate through exact_string, exact_number, depends_on, isa, exact_string_anycase, base.id and greaterthan.
• "1,000 candidates" means the OR is ANDed with an OR of 1,000 base.ids.

"Before" is the same code with _or_batch_terms() returning 1, which is the old one-query-per-operand path.

OR of	where	before	after
10 names	whole branch	2.3 ms	0.8 ms
100 names	whole branch	6.4 ms	4.2 ms
1,000 names	whole branch	55 ms	53 ms
10 names	1,000 candidates	122 ms	32 ms
100 names	1,000 candidates	1,070 ms	46 ms
1,000 names	1,000 candidates	10,064 ms	232 ms
10 mixed	whole branch	945 ms	830 ms
100 mixed	whole branch	9,809 ms	7,799 ms
1,000 mixed	whole branch	92,149 ms	73,362 ms
10 mixed	1,000 candidates	123 ms	61 ms
100 mixed	1,000 candidates	1,091 ms	92 ms
1,000 mixed	1,000 candidates	11,441 ms	608 ms

In-process sqlite3 makes per-query overhead small, so the whole-branch gain is modest here; the restricted case is where Python gains. The "mixed" whole-branch rows are dominated by terms that match most of the database, which cost the same either way.

Tests

New tests/test_batched_or.py. Each comparison runs the search three ways — default batching, batches of 3, and one query per operand — and checks they give the same documents.

• test_every_operation_in_an_or: every operation _query_struct_to_sql_str turns into SQL (exact_string, exact_string_anycase, contains_string, regexp, exact_number, lessthan, lessthaneq, greaterthan, greaterthaneq, hasfield, isa, depends_on with a name and with *), each ORed with other terms. It also covers the negated forms and some brute-forced terms (hasmember, a non-numeric exact_number, hasanysubfield_contains_string, depends_on / ~depends_on with an empty name), which stay per operand.
• test_all_sql_operations_in_one_or: all the SQL operations in one OR, which is checked to run as one query.
• test_every_operation_in_an_or_inside_an_and: the same inside isa demoB AND (…), the restricted path. Also compared with the search over the whole branch, and checked to run as 2 queries.
• test_mixed_or: base.id lookups, an AND, a nested ~or, negated and brute-forced terms in one OR, alone and inside an AND.
• test_long_or_is_searched_in_batches: 450 names take 3 queries instead of 450, and 1 + 3 when restricted.
• test_without_search_indexes and test_failing_batch_falls_back_to_per_operand.

Bridge

• The database entry's matlab_last_sync_hash is now e4e02dc (was 0de9cf2 before Test that depends_on matches as in DID-matlab, case for case #106 and this PR).
• Its decision_log notes that the port is in did/implementations/sqlitedb.py, so did/database.py doesn't change.
• It also records two deviations, neither of which changes a result:
  1. Negated terms stay one query each, because Python takes a negated term's complement in Python; MATLAB batches them.
  2. A failed batch is retried per operand.
• It also records how DID-matlab#235 interacts with this change, as described in the update above.

Checks run locally (after the merge)

• black --check src tests and ruff check src tests (pinned versions): clean.
• pytest: 841 passed, 10 skipped, 1 xfailed.
• bin/check_bridge_coverage.py, all six checks: 0 problems, against both DID-matlab #234's head (e4e02dc) and DID-matlab main (2092b81) with full history, which is how the CI job runs it.

Merge order: VH-Lab/DID-matlab#234 first, with a merge commit so e4e02dc stays reachable from main, then this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Z8rXBFDpnZo3FeCE3ymyZ