[stevevanhooser]

Port of VH-Lab/DID-matlab#229 (dae960b..83a2bda).

What was slow

NDI's most common searches combine base.id or isa daqsystem with base.session_id, and sometimes a regexp. _search_doc_ids searched every ANDed term over the whole branch, then intersected the results. So:

• the session term returned every document of the session;
• isa ran the Python regexp() on every document's superclass list;
• a regexp term tested every document.

MATLAB measured 5.5 s for isa daqsystem AND session on ~100,000 documents.

What changed (src/did/implementations/sqlitedb.py)

• Term order in an AND (_search_rank):
  1. base.id lookups;
  2. other terms (such as isa);
  3. base.session_id;
  4. terms that test every value (regexp, contains_string, hasmember, and the operations Python brute-forces).
• Candidate restriction. The search stops at the first term that matches nothing. Once the result is down to ≤1000 documents, later terms only check those documents. The query uses docs.doc_id IN (...) and a CROSS JOIN order, with INDEXED BY doc_data_doc_idx (_search_query_head). A brute-force term given candidates loads only those documents.
• ORed base.id lookups, nested to any depth, run as one query per 1000 ids (_search_base_ids, _base_id_lookup_values).
• isa (unrestricted) runs its regexp only on the distinct superclass lists, read from doc_data_field_value. LIMIT -1 stops SQLite from pushing the regexp back onto every row.
• _ensure_indexes
  • It now records whether the indexes exist (_search_indexes_ready).
  • On a database it can't write to, it leaves the indexes off. It used to fail to open such a file. Searches there run exactly as before.
  • DID-matlab now creates the same indexes under the same names (doc_data_value, doc_data_field_value, doc_data_doc_idx), so a database touched by both carries one copy of each.

Three deviations from MATLAB, none of which changes a result. All are recorded in the sqlitedb bridge decision_log:

1. Python also ranks its brute-force operations last.
2. A brute-force term with candidates loads only those documents. MATLAB has no brute-force path.
3. A negated term with candidates is the complement within the candidates, because Python applies negation outside SQL.

Timings

Same synthetic database as the MATLAB PR: 100,000 docs, 3M doc_data rows. Best of 3, main vs this branch:

search	main	this PR
base.id AND session	0.177 s	0.0001 s
OR of 9 ids AND session	0.181 s	0.0006 s
isa daqsystem AND session (3 docs)	0.299 s	0.027 s
isa AND session AND base.name regexp (.*)	0.603 s	0.027 s

On first open, adding the indexes to an older database took 2.4–2.9 s. main creates them too.

Tests

tests/test_search_indexes.py mirrors DID-matlab's TestSearchIndexes. Its test database has 1,500 + 100 documents. It records the rows each search query returns. It covers:

• the indexes: present on a new DB, added to an older one, and a read-only DB without them still searches;
• id lookups:
  • alone;
  • with a session, in either order, where no query may return more than 1 row;
  • in the wrong session;
  • an unknown id, where the session term is skipped entirely;
• ORed ids:
  • one query for all of them;
  • more than 1000 ids;
• isa AND session, in either order: no query returns more rows than isa matched;
• isa/~isa give the same results with and without the indexes;
• a regexp runs last, in three term orders;
• a brute-force term loads only the candidate;
• restricted vs full search agree for each kind of term.

Locally, all pass:

• black --check and ruff check;
• pytest: 817 passed, run with DID_BRIDGE_CHECK_STRICT=1;
• bin/check_bridge_coverage.py, all six checks, run against a full DID-matlab checkout of the PR branch.

Bridge

sqlitedb and database entries → 83a2bda.

Ordering with the MATLAB PR:

• 83a2bda only exists on DID-matlab's claude/did-base-id-index branch. CI's fetch-depth: 0 checkout fetches all branches, so it resolves now. Merge DID-matlab#229 first.
• If #229 is squash-merged, re-point both entries at the squashed commit (see matlab_last_sync_hash is a commit in PORTING_INSTRUCTIONS.md).

One existing limit, not changed here: Query.to_search_structure recurses once per OR level. So an OR of ~1,000 ids built left-to-right in a loop raises RecursionError before the search runs.

---

Generated by Claude Code