[stevevanhooser]

Closes #233. DID-python counterpart: VH-Lab/DID-python#105 (closes VH-Lab/DID-python#104). Merge this one first.

Not run locally: there was no MATLAB where this was written, so I didn't run the MATLAB code or tests myself. CI is their first run: on ec95ef7, "Analyse and test code" passed with 461 tests (+6, the new TestBatchedOr) and 0 failures. Before that, I checked the SQL with Python's sqlite3 (details below). After #235 was merged in, CI runs again on the new head.

Update: #235 merged in

#235 (exact depends_on matching) changed the same part of database.m, so main is merged into this branch (4f4e9aa, followed by the fix e4e02dc).

• What Match depends_on in SQLite exactly as did.query does #235 changes for this PR: a depends_on that SQL can't match exactly, such as one with an empty name, is now brute-forced (isBruteForceDependsOn → brute_force_search), and query_struct_to_sql_str throws an error for it. search_or_operands now keeps those operands out of its SQL batches, so search_doc_ids brute-forces them one at a time.
• New test: testBruteForceDependsOnInAnOr covers depends_on and ~depends_on with an empty name, in an OR and in an OR inside an AND.
• Merge fix: resolving the merge first dropped the two ends that close search_or_operands. e4e02dc puts them back. The merged database.m was then compared with both sides: it removes nothing from main, and from this branch it removes only the code Match depends_on in SQLite exactly as did.query does #235 replaced.

What changed

did.database/search_doc_ids used to search an OR with one SQL query per operand, then union the results in MATLAB. Now the OR's operands go to a new method, search_or_operands, which sorts them into three groups:

• Single SQL terms (a scalar search structure that isn't an OR or a brute-forced depends_on) are searched up to maxOrBatchTerms = 200 per SQL query. That covers every operation query_struct_to_sql_str supports, negated ones included.
• base.id lookups among the operands are collected and searched together with search_base_ids. That's the same batching an all-base.id OR already gets, used only when hasSearchIndexes() is true.
• ANDs (struct arrays), nested ~ors and brute-forced depends_on queries are searched one at a time by search_doc_ids, as before.

The batch size comes from a new protected method, orBatchTerms(). The test helper QueryLogSqlitedb gets an orBatchSize property that overrides it. A value of 1 gives the old one-query-per-operand path, which the tests use for comparison.

The SQL, and why

Over a whole branch: one SELECT per operand, joined with UNION. Each member is exactly the query the operand used to run on its own, built by get_sql_query_str.

• SQLite plans each member separately, so each keeps its index.
• isa keeps its LIMIT -1 distinct-superclass subquery.

Restricted to candidates (an OR inside an AND, after the AND has narrowed to at most 1,000 documents): one query using the restricted head (CROSS JOIN … INDEXED BY doc_data_doc_idx … docs.doc_id IN (…)), with the operands' conditions joined by OR.

• Each restricted term reads every row of every candidate, whatever it searches for, so per-term SELECTs, UNIONed or not, pay that cost once per term.
• With one ORed WHERE the rows are read once.
• The plan is fixed by CROSS JOIN/INDEXED BY, so the OR can't make SQLite fall back to scanning all of doc_data.

Results are the same either way: a document matches when one of its doc_data rows satisfies one of the conditions. The 200-term cap keeps a batch well below SQLite's default limits of 500 compound-select terms and expression depth 1000.

Timings

Setup: synthetic DID database built with Python sqlite3 3.45.1 — DID schema, 100,000 docs × 30 fields (3M doc_data rows), with doc_data_value, doc_data_field_value and doc_data_doc_idx. The SQL is a Python mirror of get_sql_query_str / query_struct_to_sql_str. Times are SQL execution only.

Workloads:

• "names": base.name exact_string terms.
• "mixed": a rotation of exact_string, exact_number, depends_on, isa, exact_string_anycase, base.id and greaterthan terms.
• Restricted runs use 1,000 candidate documents.

OR of	where	per term (before)	UNION batches	one ORed WHERE
10 names	whole branch	0.2 ms	0.2 ms	908 ms
100 names	whole branch	1.8 ms	1.5 ms	5,983 ms
1,000 names	whole branch	40 ms	44 ms	error: expression tree too large (depth 1000)
10 mixed	whole branch	672 ms	597 ms	1,294 ms
100 mixed	whole branch	8,951 ms	7,363 ms	7,560 ms
1,000 mixed	whole branch	82,544 ms	70,880 ms	error
10 names	1,000 candidates	108 ms	75 ms	11 ms
100 names	1,000 candidates	1,097 ms	901 ms	23 ms
1,000 names	1,000 candidates	10,004 ms	10,998 ms	199 ms (5 × 200)
10 mixed	1,000 candidates	113 ms	126 ms	44 ms
100 mixed	1,000 candidates	1,038 ms	939 ms	84 ms
1,000 mixed	1,000 candidates	10,437 ms	10,662 ms	570 ms (5 × 200)

Over a whole branch, the ORed WHERE that #233 worried about really does lose the indexes. It took ~1 s for 10 names, so it's out.

• The UNION's SQL time matches the per-term SQL time.
• What it saves in MATLAB is the per-query overhead: the mksqlite call, run_sql_query's struct-to-cell conversion and the growing union. That's 1,000 round trips becoming 5.
• That overhead couldn't be measured here without MATLAB. OR searches run one SQL query per term; batch them into one statement #233 estimated it at 1–3 ms per term, so roughly 1–3 s for a 1,000-term OR.
• The "mixed" rows are dominated by terms that match tens of thousands of documents (greaterthan, exact_number). That cost is the same either way.

Restricted to candidates, the ORed WHERE is 10–50× faster than one query per term, before counting any MATLAB-side saving. VH-Lab/DID-python#105 has the same comparison timed on DID-python's real SQLiteDB: 1,000 names restricted to 1,000 candidates drop from 10.1 s to 0.23 s.

Tests

New tests/+did/+unittest/TestBatchedOr.m. Each comparison runs the search three ways — default batching, batches of 3 (to exercise chunking), and one query per operand — and checks the results are identical (verifyEqual, same order):

• testEveryOperationInAnOr: each of the 14 SQL operations, plus the negation of each that has one, ORed with other terms.
• testAllOperationsInOneOr: all of them in one OR. Also checks it runs as a single query.
• testEveryOperationInAnOrInsideAnAnd: the same inside isa demoB AND (…), which takes the restricted path. Also compared with the unrestricted search (noSearchIndexes), and checks the query count is 2.
• testMixedOr: base.id lookups, an AND operand, a nested ~or and plain terms in one OR, alone and inside an AND. Checks the batched search runs fewer queries.
• testLongOrIsSearchedInBatches: 450 names take 3 queries instead of 450, and 1 + 3 when restricted.
• testBruteForceDependsOnInAnOr: a brute-forced depends_on (empty name), plain and negated, in an OR and inside an AND.
• testWithoutSearchIndexes: the UNION path with no candidates and no id batching.

Outside MATLAB, I checked the two SQL forms with Python sqlite3 (with a regex() function registered). Both returned exactly the per-term results for:

• every operation and its negation, on the whole branch and restricted to candidates;
• the full all-operations OR, in batches of 3 and of 200;
• 200 of the deepest conditions (~hasmember, ~isa) in one restricted OR, which stays inside the expression-depth limit.

Noticed, not changed

• hasmember with a string value: it emits doc_data.value=<value> unquoted, so SQLite reads it as a column name ("no such column"). This happens on both paths. Only numeric values work today.
• Top-level ~or: search_doc_ids treats it as a plain or (the ~ is stripped). This is unchanged, and a nested ~or still goes through that same path.

Merge

Please merge with a merge commit, so e4e02dc (the hash VH-Lab/DID-python#105's bridge records) stays reachable from main. Then merge VH-Lab/DID-python#105.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Z8rXBFDpnZo3FeCE3ymyZ

[github-actions]

Test Results

511 tests  +7   511 ✅ +7   2m 47s ⏱️ +38s
 42 suites +1     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit e4e02dc. ± Comparison against base commit 2092b81.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.30%. Comparing base (2092b81) to head (e4e02dc).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #234      +/-   ##
==========================================
+ Coverage   66.91%   67.30%   +0.39%     
==========================================
  Files          67       67              
  Lines        4748     4793      +45     
==========================================
+ Hits         3177     3226      +49     
+ Misses       1571     1567       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.