Skip to content

Match depends_on in SQLite exactly as did.query does - #235

Merged
stevevanhooser merged 3 commits into
mainfrom
claude/did-depends-on-parity
Oct 24, 2026
Merged

stevevanhooser merged 3 commits into
mainfrom
claude/did-depends-on-parity

Conversation

@stevevanhooser

@stevevanhooser stevevanhooser commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem

did.query says depends_on(NAME, VALUE) is an exact, case-sensitive match of a dependency's name and value. Only NAME '*' is a wildcard. to_searchstructure and fieldsearch work that way.

The SQLite search did not. query_struct_to_sql_str searched meta.depends_on with LIKE "%NAME,VALUE;%", which is unanchored and ignores case. doc2sql stores that field as name1,value1;name2,value2;. As a result:

  • depends_on('', V) matched any dependency name. Brute force matches none.
  • depends_on('id', V) also matched subject_id,V;.
  • depends_on('ID', V) matched id,V;.
  • a * inside a name ('subject*') acted as a wildcard.

DID-python already matches exactly (DID-python#101), so the two languages disagreed.

Change

src/did/+did/database.m:

  • New SQL test. dependsOnSqlMatch(name, value) builds the test with instr, a plain case-sensitive substring test:

    • for a named dependency: instr(';' || ifnull(doc_data.value,''), ';NAME,VALUE;') > 0
    • for '*': instr(ifnull(doc_data.value,''), ',VALUE;') > 0

    This is the same test as DID-python's _depends_on_sql_clause. It is still one test on one meta.depends_on row per document, so it costs about the same as the old LIKE.

  • Negated form. ifnull makes ~depends_on include documents with no dependencies, as brute force does.

  • Brute-force fallback. A NAME or VALUE that is empty, or that contains , or ;, cannot be matched exactly in the stored string. search_doc_ids sends those queries to the new brute_force_search, which runs fieldsearch on the candidate documents, or on the whole branch if there are none. Inside an AND these terms are searched last, so they only test the documents the other terms found. DID-python does the same.

src/did/+did/+datastructures/fieldsearch.m:

  • hasanysubfield_* used to return early when the field held something other than a struct or cell. That skipped the negation, so ~depends_on left the document out.
  • A document with no dependencies has exactly that once it is read back from a database: depends_on = [], because an empty struct array is saved as [].
  • Now a non-struct value simply does not match, and ~ makes it a match. DID-python's field_search already behaved this way.

src/did/+did/query.m: the help text now states the rule.

Tests

tests/+did/+unittest/TestDependsOnParity.m uses five documents:

  • subject_id→X
  • id→X and element_id→Y
  • ID→X
  • no dependencies
  • element_id→x

It runs 16 cases on the SQLite search, on fieldsearch, and inside an AND. The cases cover:

  • ''
  • '*'
  • an exact name
  • a name that is a suffix of another name (id / subject_id)
  • name case and value case
  • the second of two dependencies
  • a name and a value that come from different dependencies
  • 'subject*' is not a glob
  • ~depends_on with an exact name, '*', '' and a case difference

There is also a test with two depends_on terms ANDed together. DID-python gets a matching tests/test_depends_on_parity.py with the same documents and the same expected results; DID-python main already passes it.

Compatibility

NDI-matlab only uses '*' as a wildcard, and NDI-python moved from '' to '*' (Waltham-Data-Science/NDI-python#344), so tightening '' does not affect NDI.

🤖 Generated with Claude Code

https://claude.ai/code/session_014BLjcMbQRXr5MDdivxzYEc

did.query documents depends_on(NAME, VALUE) as an exact, case-sensitive
match of a dependency's name and value, with only NAME '*' a wildcard, and
to_searchstructure/fieldsearch implement exactly that. The SQLite search
instead tested meta.depends_on with LIKE "%NAME,VALUE;%", which was neither
anchored nor case-sensitive: an empty NAME matched any name, 'id' also
matched 'subject_id', 'ID' matched 'id', and '*' inside a name acted as a
glob.

query_struct_to_sql_str now tests meta.depends_on (stored by doc2sql as
'name1,value1;name2,value2;') with instr: ';' || value must contain
';NAME,VALUE;', or, for '*', value must contain ',VALUE;'. That is still a
single SQL test of one row per document, as before. ifnull keeps documents
with no dependencies in the result of ~depends_on. A NAME or VALUE that is
empty or contains ',' or ';' cannot be matched exactly in that string, so
search_doc_ids tests such a query document by document with fieldsearch
(brute_force_search), and ranks it last in an AND so that it only sees the
documents the other terms found. This is what DID-python's
_depends_on_sql_clause does (DID-python#101).

TestDependsOnParity runs the same documents and cases on the SQLite search
and on fieldsearch; DID-python's tests/test_depends_on_parity.py asserts
identical expectations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BLjcMbQRXr5MDdivxzYEc
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

504 tests  +49   504 ✅ +49   1m 38s ⏱️ -1s
 41 suites + 1     0 💤 ± 0 
  1 files   ± 0     0 ❌ ± 0 

Results for commit 051f535. ± Comparison against base commit 6a398d0.

♻️ This comment has been updated with latest results.

did.query(STRUCT) rejected the search structure ('Field names of search
structure do not match expected fields'), so every depends_on that falls
back to brute force (an empty name) errored. Pass field, operation and
parameters to did.query instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BLjcMbQRXr5MDdivxzYEc
hasanysubfield_contains_string / _exact_string returned early when the
field held something other than a struct or cell, skipping the negation at
the end of fieldsearch, so the ~ form reported 'no match' as well. A
document with no dependencies has depends_on = [] once read back from a
database (an empty struct array is saved as []), so ~depends_on('', V) -
searched by brute force - left it out, where the SQL search, brute force on
the document as made, and DID-python all include it. A non-struct value now
simply does not match, and the negation applies as for any other value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BLjcMbQRXr5MDdivxzYEc
@codecov

codecov Bot commented Oct 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.69767% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.91%. Comparing base (6a398d0) to head (051f535).

Files with missing lines Patch % Lines
src/did/+did/database.m 90.47% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #235      +/-   ##
==========================================
+ Coverage   66.41%   66.91%   +0.50%     
==========================================
  Files          67       67              
  Lines        4710     4748      +38     
==========================================
+ Hits         3128     3177      +49     
+ Misses       1582     1571      -11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@stevevanhooser
stevevanhooser merged commit 2092b81 into main Oct 24, 2026
6 checks passed
@stevevanhooser
stevevanhooser deleted the claude/did-depends-on-parity branch October 24, 2026 19:48
stevevanhooser pushed a commit that referenced this pull request Oct 24, 2026
#235 brute-forces the depends_on queries SQL cannot match exactly
(isBruteForceDependsOn), and query_struct_to_sql_str now errors on them.
search_or_operands keeps such operands out of its SQL batches, so they
are searched one at a time by search_doc_ids, which brute-forces them.
TestBatchedOr covers depends_on with an empty name, alone in an OR and
inside an AND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Z8rXBFDpnZo3FeCE3ymyZ
stevevanhooser pushed a commit that referenced this pull request Oct 24, 2026
Resolving the merge with #235 dropped the 'end's that close the last loop
of search_or_operands and the function, so it ran on into
brute_force_search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Z8rXBFDpnZo3FeCE3ymyZ
jesshaley pushed a commit that referenced this pull request Oct 29, 2026
did-schema #235 adds the register rows. Each marker now names its row and
each note opens with the row's tag and kind instead of 'unplaced:'.
private/jModelFit.m tags carry-136 for fitcurve, carry-137 for vmspikefit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants