Repository navigation
sqlitedb: search depends_on in SQL instead of loading every document - #102
Merged
Merged
Conversation
…101) Query("", "depends_on", name, value) resolves to hasanysubfield_exact_string on depends_on, which had no SQL form: every depends_on search loaded and JSON-parsed every document of the branch and tested it in Python. NDI uses depends_on constantly. It is now matched against meta.depends_on, which doc2sql stores as "name1,value1;name2,value2;": an entry (name, value) is present exactly when ';' || meta.depends_on contains ';name,value;' -- or ',value;' for the '*' name. instr() is a plain, case-sensitive substring test, as brute force's == is; MATLAB's LIKE '%name,value;%' is case-insensitive and unanchored ('id,X;' also matches 'subject_id,X;'), so Python does not copy it. An empty name or value, or one containing ',' or ';', is not representable that way and keeps the brute-force path. A depends_on term with an SQL form now ranks with the ordinary terms in an AND (it is one row per document, tested in SQL), ahead of base.session_id. Also: _ensure_indexes no longer announces the indexes it builds on an empty doc_data. Every new Python database got three "adding search index ... (about 0 rows)" lines, because _create_db_tables does not create the indexes and _ensure_indexes then does (since #99). 50,000 documents, two dependencies each, main -> this: depends_on element_id E7 (100 docs) 2.07 s -> 0.009 s depends_on * E7 0.79 s -> 0.007 s depends_on element_id E7 AND session 1.19 s -> 0.009 s Tests: tests/test_depends_on_search.py compares the SQL path with brute force (depends_on and ~depends_on) on documents built to trip it up: 'id' vs 'subject_id', 'E1' vs 'xE1'/'E10', case, enumerated names, the '*' name, empty values, no dependencies; checks no document is loaded; and that what SQL cannot represent still brute-forces. test_search_indexes: a new database is created quietly. test_query_correctness: the depends_on SQL has no LIKE wildcards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #101.
Problem
Query("", "depends_on", name, value)resolves tohasanysubfield_exact_stringondepends_on. That operation had no SQL form, so everydepends_onsearch loaded and JSON-parsed every document in the branch, then tested each one in Python.NDI uses
depends_onconstantly: "all documents that depend on this element / epoch / probe".Change
depends_onis now matched in SQL againstmeta.depends_on, which doc2sql stores asname1,value1;name2,value2;.(name, value)is present exactly when';' || meta.depends_oncontains;name,value;. For the*name, the stored string must contain,value;.instr()is a plain, case-sensitive substring test, like brute force's==. MATLAB's own SQL for this,LIKE '%name,value;%', is not copied: it ignores case, and it isn't anchored, soid,X;would also matchsubject_id,X;. The bridgedecision_logrecords this difference.,or;. Neither case can be expressed exactly in that stored format.depends_onthat has an SQL form now ranks with the ordinary terms, ahead ofbase.session_id. It's one row per document, tested in SQL.Also fixed (from #99)
Creating a new database printed three
DID: adding search index … (about 0 rows)lines._create_db_tablesdoesn't create the indexes, so_ensure_indexesbuilds all three on every new file. It now builds them silently whendoc_datais empty.Timings
50,000 documents, each with 2 dependencies;
main→ this PR:depends_on element_id E7(100 docs)depends_on * E7depends_on element_id E7AND sessionTests
New file
tests/test_depends_on_search.py. It compares the SQL path with brute force, for bothdepends_onand~depends_on, on documents built to trip it up:idvssubject_id;E1vsxE1/E10;*name;It also checks that no document is loaded, and that what SQL can't represent still uses brute force.
Other test changes:
test_search_indexes: a new database is created without printing anything.test_query_correctness: thedepends_onSQL uses noLIKEwildcards.Local results:
black --checkandruff checkare clean.pytestpasses: 823 tests.main: 0 problems.Generated by Claude Code