depends_on search loads every document of the branch (no SQL path) #101

What happens

Query("", "depends_on", name, value) is resolved by Query._resolve_single into hasanysubfield_exact_string on the depends_on field. For that operation, SQLiteDB._query_struct_to_sql_str returns None, so every depends_on search goes through _brute_force_search. That loads and JSON-parses every document in the branch, then tests each one in Python.

On a large dataset (NDI, ~100,000 documents) that is seconds to minutes per search, and NDI uses depends_on constantly ("all documents that depend on this element/epoch/probe").

The depends_on branch of _query_struct_to_sql_str, which uses meta.depends_on LIKE '%name,value;%', is unreachable. Its own comment says so.

Since #98, a depends_on ANDed with a selective term (e.g. a base.id or a rare isa) only loads the candidate documents. Alone, or with only broad terms, it still loads everything.

Proposal

Give hasanysubfield_exact_string on depends_on an SQL form against meta.depends_on, which doc2sql stores as name1,value1;name2,value2;.

• Use an anchored, case-sensitive match such as instr(';' || doc_data.value, ';name,value;') > 0, or ',value;' for the * wildcard name. Not MATLAB's LIKE '%name,value;%', for two reasons:
  • LIKE is case-insensitive.
  • An unanchored pattern lets 'id,X;' also match 'subject_id,X;'.
• Results must stay identical to brute force. Add a test that compares the two over a set of documents.
• Keep brute force for anything the serialization can't represent unambiguously, e.g. a name or value containing , or ;.

Generated by Claude Code