Skip to content

sqlitedb: search depends_on in SQL instead of loading every document - #102

Merged
stevevanhooser merged 1 commit into
mainfrom
claude/did-base-id-index
Oct 24, 2026
Merged

stevevanhooser merged 1 commit into
mainfrom
claude/did-base-id-index

Conversation

@stevevanhooser

Copy link
Copy Markdown
Contributor

Closes #101.

Problem

Query("", "depends_on", name, value) resolves to hasanysubfield_exact_string on depends_on. That operation had no SQL form, so every depends_on search loaded and JSON-parsed every document in the branch, then tested each one in Python.

NDI uses depends_on constantly: "all documents that depend on this element / epoch / probe".

Change

depends_on is now matched in SQL against meta.depends_on, which doc2sql stores as name1,value1;name2,value2;.

  • Match rule. An entry (name, value) is present exactly when ';' || meta.depends_on contains ;name,value;. For the * name, the stored string must contain ,value;.
  • Matcher. instr() is a plain, case-sensitive substring test, like brute force's ==. MATLAB's own SQL for this, LIKE '%name,value;%', is not copied: it ignores case, and it isn't anchored, so id,X; would also match subject_id,X;. The bridge decision_log records this difference.
  • Kept on brute force: a query whose name or value is empty, or contains , or ;. Neither case can be expressed exactly in that stored format.
  • Order within an AND. A depends_on that has an SQL form now ranks with the ordinary terms, ahead of base.session_id. It's one row per document, tested in SQL.

Also fixed (from #99)

Creating a new database printed three DID: adding search index … (about 0 rows) lines. _create_db_tables doesn't create the indexes, so _ensure_indexes builds all three on every new file. It now builds them silently when doc_data is empty.

Timings

50,000 documents, each with 2 dependencies; main → this PR:

search main this PR
depends_on element_id E7 (100 docs) 2.07 s 0.009 s
depends_on * E7 0.79 s 0.007 s
depends_on element_id E7 AND session 1.19 s 0.009 s

Tests

New file tests/test_depends_on_search.py. It compares the SQL path with brute force, for both depends_on and ~depends_on, on documents built to trip it up:

  • id vs subject_id;
  • E1 vs xE1 / E10;
  • case differences;
  • enumerated names;
  • the * name;
  • empty values;
  • documents with no dependencies.

It also checks that no document is loaded, and that what SQL can't represent still uses brute force.

Other test changes:

  • test_search_indexes: a new database is created without printing anything.
  • test_query_correctness: the depends_on SQL uses no LIKE wildcards.

Local results:

  • black --check and ruff check are clean.
  • pytest passes: 823 tests.
  • Bridge coverage, all six checks against DID-matlab main: 0 problems.

Generated by Claude Code

…101)

Query("", "depends_on", name, value) resolves to hasanysubfield_exact_string
on depends_on, which had no SQL form: every depends_on search loaded and
JSON-parsed every document of the branch and tested it in Python. NDI uses
depends_on constantly.

It is now matched against meta.depends_on, which doc2sql stores as
"name1,value1;name2,value2;": an entry (name, value) is present exactly when
';' || meta.depends_on contains ';name,value;' -- or ',value;' for the '*'
name. instr() is a plain, case-sensitive substring test, as brute force's ==
is; MATLAB's LIKE '%name,value;%' is case-insensitive and unanchored ('id,X;'
also matches 'subject_id,X;'), so Python does not copy it. An empty name or
value, or one containing ',' or ';', is not representable that way and keeps
the brute-force path. A depends_on term with an SQL form now ranks with the
ordinary terms in an AND (it is one row per document, tested in SQL), ahead
of base.session_id.

Also: _ensure_indexes no longer announces the indexes it builds on an empty
doc_data. Every new Python database got three "adding search index ...
(about 0 rows)" lines, because _create_db_tables does not create the
indexes and _ensure_indexes then does (since #99).

50,000 documents, two dependencies each, main -> this:
  depends_on element_id E7 (100 docs)       2.07 s -> 0.009 s
  depends_on * E7                           0.79 s -> 0.007 s
  depends_on element_id E7 AND session      1.19 s -> 0.009 s

Tests: tests/test_depends_on_search.py compares the SQL path with brute
force (depends_on and ~depends_on) on documents built to trip it up:
'id' vs 'subject_id', 'E1' vs 'xE1'/'E10', case, enumerated names, the '*'
name, empty values, no dependencies; checks no document is loaded; and that
what SQL cannot represent still brute-forces. test_search_indexes: a new
database is created quietly. test_query_correctness: the depends_on SQL has
no LIKE wildcards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
@stevevanhooser
stevevanhooser merged commit 8c8d661 into main Oct 24, 2026
7 checks passed
@stevevanhooser
stevevanhooser deleted the claude/did-base-id-index branch October 24, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

depends_on search loads every document of the branch (no SQL path)

2 participants