sqlitedb: make id, isa and regexp searches fast (port of DID-matlab#229)

NDI's most common searches -- base.id or isa daqsystem, ANDed with
base.session_id, sometimes with a regexp -- searched every term over the
whole branch and intersected the results, so the session term returned every
document of the session and the regexp tested every document.

Ported from DID-matlab#229 (dae960b..83a2bda):
- _search_doc_ids searches the terms of an AND in rank order: base.id
  lookups, other terms, base.session_id, then terms that test every value
  (regexp, contains_string, hasmember, and the brute-force operations). It
  stops at the first term that matches nothing, and once the result is down
  to 1000 documents checks later terms against those only
  (docs.doc_id IN (...), CROSS JOIN order, INDEXED BY doc_data_doc_idx).
  A brute-force term given candidates loads only those documents.
- ORed base.id lookups, nested to any depth, are one query per 1000 ids
  (_search_base_ids, _base_id_lookup_values).
- Unrestricted 'isa' runs its regexp on the distinct superclass lists only,
  read from doc_data_field_value, instead of on every document.
- _ensure_indexes now records whether the doc_data indexes exist
  (_search_indexes_ready) and leaves a database it cannot write to without
  them, where it used to fail to open; searches there run as before.
  DID-matlab now creates the same indexes under the same names.

Synthetic database, 100,000 documents / 3M doc_data rows, main -> this:
  base.id AND session               0.177 s -> 0.0001 s
  OR of 9 ids AND session           0.181 s -> 0.0006 s
  isa daqsystem AND session         0.299 s -> 0.027 s
  isa AND session AND regexp        0.603 s -> 0.027 s

Tests: tests/test_search_indexes.py (mirrors DID-matlab's
TestSearchIndexes). Bridge: sqlitedb and database entries -> 83a2bda.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia

Author: claude

src/did/did_matlab_python_bridge.yaml

@@ -40,12 +40,23 @@ classes:
   - name: database
     type: class
     matlab_path: "+did/database.m"
-    matlab_last_sync_hash: "bb39399"
+    matlab_last_sync_hash: "83a2bda"
     status: ported
     python_path: "did/database.py"
     python_class: "Database"
     inherits: "abc.ABC"
     out_of_sync: false
+    decision_log: >
+      DID-matlab#229 (dae960b..83a2bda, 2026-10-24) made database.m's search
+      fast. search_doc_ids searches the terms of an AND in rank order (base.id
+      lookups, other terms, base.session_id, then the terms that test every
+      value) and checks each later term against the documents found so far;
+      ORed base.id lookups become one query (search_base_ids); and 'isa'
+      matches its regex against the distinct superclass lists only. All of it
+      relies on the doc_data indexes that sqlitedb creates (hasSearchIndexes).
+      In Python the SQL search is SQLiteDB's, not Database's, so the port is in
+      did/implementations/sqlitedb.py -- see the sqlitedb entry. Nothing
+      changes in did/database.py.
     sync_notes: >
       Schema validation was ported on 2026-08-28, closing what had been the
       largest capability gap between the two languages. Python previously had

src/did/did_matlab_python_bridge_implementations.yaml

@@ -29,13 +29,32 @@ classes:
   - name: sqlitedb
     type: class
     matlab_path: "+did/+implementations/sqlitedb.m"
-    matlab_last_sync_hash: "bb39399"
+    matlab_last_sync_hash: "83a2bda"
     status: ported
     python_path: "did/implementations/sqlitedb.py"
     python_class: "SQLiteDB"
     inherits_matlab: "did.database"
     inherits_python: "Database"
     out_of_sync: false
+    decision_log: >
+      DID-matlab#229 (dae960b..83a2bda, 2026-10-24): fast id and isa searches,
+      ported to SQLiteDB (_search_doc_ids, _search_query_head,
+      _search_base_ids, _query_struct_to_sql_str, _ensure_indexes). MATLAB
+      adopted the index names Python already created -- doc_data_doc_idx ON
+      doc_data(doc_idx), doc_data_field_value ON doc_data(field_idx, value) --
+      so a database touched by both carries one copy of each. Both languages
+      now add them on open, leave a read-only file without them, and only then
+      (_search_indexes_ready / hasSearchIndexes) restrict terms to candidates,
+      collapse ORed base.id lookups, and match 'isa' against the distinct
+      superclass lists. Three deviations, none of which changes a result:
+      (1) Python also ranks the operations it brute-forces
+      (hasanysubfield_*, hassize, partial_struct) with regexp /
+      contains_string / hasmember as last-in-an-AND, since a brute-force term
+      loads every document it tests; (2) a brute-force term given candidates
+      loads only those documents (MATLAB has no brute-force path); (3) Python
+      applies negation as a complement in Python rather than in SQL, so a
+      negated term with candidates is the complement within the candidates
+      rather than within the branch -- the same set once intersected.
     sync_notes: >
       MATLAB 14439fb (2026-06-12, audit 6.1-3) added a private
       escapeSqlLiteral() helper and applied it at the 14 sites where