[stevevanhooser]

What was slow

NDI's most common searches pair an id or class lookup with a session filter:

• base.id exact_string ID AND base.session_id exact_string SESSION
• isa daqsystem AND base.session_id exact_string SESSION, sometimes also AND a regexp

Measured in NDI on a ~100,000-document database:

• isa daqsystem + session took 5.5 s, returning 3 documents.
• Adding a base.name regexp (.*) term added +5.2 s.

Three causes:

1. The AND. did.database/search_doc_ids ran every ANDed term as its own query over the whole branch, then intersected the results in MATLAB. So the session term returned one row per document in the session, and a regexp tested every document.
2. isa. It matches meta.class exactly, OR runs regex() on the comma-joined meta.superclass list. The regex can't use an index. SQLite built a temporary index over all of doc_data, then ran the regex once per document: 3.0 s on a synthetic 100,000-document database.
3. No index on doc_data.doc_idx. A term could not be checked against just a few documents.

What changed

Indexes

sqlitedb.searchIndexes defines two indexes:

• doc_data_doc_idx on (doc_idx)
• doc_data_field_value on (field_idx, value)

DID-python's _ensure_indexes has created these with the same names and columns for some time. So a database touched by both libraries carries one copy of each.

• New databases get both.
• An older database gets them when it is opened (ensure_search_indexes).
• A file that can't be written is left alone. hasSearchIndexes() is then false, and searches run exactly as before.

AND terms (search_doc_ids)

• Terms are searched in this order:
  1. base.id lookups
  2. other terms (such as isa)
  3. base.session_id
  4. terms that test every value (regexp, contains_string, hasmember)
• The search stops at the first term that matches nothing.
• Once the result is down to at most 1000 documents, later terms only check those documents. That query uses docs.doc_id IN (...), a CROSS JOIN order, and INDEXED BY doc_data_doc_idx.

So isa daqsystem AND session AND regexp finds the 3 daqsystem documents first. It then checks their session, then runs the regexp on their 3 names, whatever order the query was written in.

isa and ORed ids

• isa (unrestricted, not negated) runs its regex only on the distinct superclass lists, read from doc_data_field_value. LIMIT -1 stops SQLite from pushing the regex back onto every row.
• ORed base.id lookups run as one query per 1000 ids (search_base_ids).

Timings

SQL only (Python sqlite3), on a synthetic copy of this schema: 100,000 docs, 3M doc_data rows.

	before	after
isa daqsystem AND session (3 docs)	2.44 s	~0.03 s
isa element AND session (99,999 docs)	2.92 s	~0.6–1.2 s
base.id AND session	0.167 s	0.0002 s
a term checked against 1000 candidates	—	0.03–0.08 s (hasfield 0.34 s)
one-time index build on open	—	~3 s

The before figures leave out MATLAB converting and intersecting the large result sets, which adds to them in NDI.

On the synthetic database, isa/~isa return the same results under the old and new SQL. For each other kind of term checked, the restricted search returns the full search's results limited to the candidates.

Tests

tests/+did/+unittest/TestSearchIndexes.m, with the helper did.test.helper.QueryLogSqlitedb (it records each query's row count).

• Indexes: present in new databases; added to an older one when it is opened.
• Id lookups:
  • alone;
  • ANDed with a session in either order, where no query may return more than 1 row;
  • in the wrong session;
  • unknown ids;
  • ORed ids, including more than 1000.
• isa AND session, in either order.
• regexp AND isa AND session, in three orders.
• isa/~isa: the same results with and without the indexes.
• Restricted vs full search: the same results for each kind of term.

CI on 83a2bda:

• 450 MATLAB tests pass, 13 more than main.
• The MATLAB ↔ Python symmetry tests pass.
• Codecov: 91% of the new lines are covered; project coverage rises from 65.7% to 66.3%.

The timings above have not been re-measured in NDI itself.

DID-python

The port is VH-Lab/DID-python#98. It uses the same index names, the same add-on-open behaviour and the same search order. Python's own timings, main → that branch:

• isa daqsystem AND session: 0.30 → 0.027 s
• with a regexp added: 0.60 → 0.027 s

Its bridge entries point at 83a2bda on this branch. If this PR is squash-merged, they need re-pointing to the squashed commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia

[github-actions]

Test Results

450 tests  +13   450 ✅ +13   2m 27s ⏱️ +9s
 39 suites + 1     0 💤 ± 0 
  1 files   ± 0     0 ❌ ± 0 

Results for commit 83a2bda. ± Comparison against base commit f4a4905.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

❌ Patch coverage is 91.34615% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.28%. Comparing base (f4a4905) to head (83a2bda).

Files with missing lines	Patch %	Lines
src/did/+did/database.m	90.80%	8 Missing ⚠️
src/did/+did/+implementations/sqlitedb.m	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #229      +/-   ##
==========================================
+ Coverage   65.71%   66.28%   +0.56%     
==========================================
  Files          66       66              
  Lines        4579     4674      +95     
==========================================
+ Hits         3009     3098      +89     
- Misses       1570     1576       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.