Repository navigation
sqlitedb: announce and time the one-time search index build - #231
Merged
Merged
Conversation
Opening a database that predates doc_data_doc_idx / doc_data_field_value builds them, silently. doc_data_field_value holds a copy of every field value, so for a large dataset (100,000 documents can be tens of millions of doc_data rows) on a slow or synced drive the build takes minutes and the open just appears to hang - reported on a real 100,000-document dataset that sat for over 2 minutes. ensure_search_indexes now prints, before each index it builds, its name, the database, the approximate number of rows (max(rowid), which costs nothing) and that this happens once and can take minutes; afterwards, the time it took; and, if it cannot be built (read-only file), that searches will run without it. A database that already has the indexes opens without printing anything. Test: TestSearchIndexes checks the announcement and timing for each index when an older database is opened, and silence on the next open. Not run here: no MATLAB in this environment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
It only formats the row count in ensure_search_indexes' message; as a public static method it would also need a DID-python bridge entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
Contributor
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #231 +/- ##
==========================================
+ Coverage 66.28% 66.29% +0.01%
==========================================
Files 66 66
Lines 4674 4694 +20
==========================================
+ Hits 3098 3112 +14
- Misses 1576 1582 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
stevewds
approved these changes
Oct 24, 2026
jesshaley
pushed a commit
that referenced
this pull request
Oct 29, 2026
…ad of adding a second testADeclaredCoordinateSystemEdgeIsReadFromTheSchema inserted a required coordinate_system_id into a copy of statement.json. Against did-schema #231, which declares the edge (optional, [site frame is a coordinate system]), the copy then held two declarations. did2.build took the later, optional one and raised no did2:build:missingEdge; the validator, which unions required edges, raised did2:validation:emptyRequiredDependency (Quick migrator tests run 40892458228 on that schema). When the schema already declares the edge, the test now sets that declaration's mustBeNonEmpty to true; otherwise it inserts as before. The assertions are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
jesshaley
pushed a commit
that referenced
this pull request
Oct 29, 2026
…egans's does did-schema #231 recorded openminds:species/caenorhabditisElegans as NCBITaxon:6239's equivalent ([species text is the openMINDS equivalent], Jess Haley, 2026-10-29). The fixture of testAnOpenmindsIdThatDoesNotRebuildIsCarried used exactly that id, so it now rebuilds and the document converts (Quick migrator tests run 40917656427: 0 passthroughs, 1 expected). The fixture keeps the NCBITaxon:6239 term and names an id with no recorded equivalent, which is the case the test is about. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #229.
Problem
Opening a database that predates the search indexes
doc_data_doc_idxanddoc_data_field_valuebuilds them, and it does so silently.doc_data_field_valueholds a copy of every field value.For a large dataset the build can take minutes, especially on a slow or synced drive: 100,000 documents can mean tens of millions of
doc_datarows. During that time the open just appears to hang. This was reported on a real 100,000-document dataset that sat for more than 2 minutes on open.While the build sorts, the
.sqlitefile doesn't grow, which makes it harder to recognise. SQLite sorts in memory or inetilqs_*temp files, and only writes the finished index into the database at the end.Change
ensure_search_indexesnow prints three kinds of message:max(rowid), which costs nothing;count(*)would scan the table.A database that already has the indexes opens without printing anything.
Example:
groupDigitsis a new private static helper that formats the row count.Tests
TestSearchIndexes/testIndexesAddedWhenOlderDatabaseOpenedchecks two things:evalc).Not run here: there's no MATLAB in this environment.
DID-python
The same change is in VH-Lab/DID-python's
claude/did-base-id-indexbranch; its PR follows. Its bridge entry points at5b7220b, the last commit here, so please merge with a merge commit, as for #229.Generated by Claude Code