Skip to content

sqlitedb: announce and time the one-time search index build - #99

Merged
stevevanhooser merged 2 commits into
mainfrom
claude/did-base-id-index
Oct 24, 2026
Merged

stevevanhooser merged 2 commits into
mainfrom
claude/did-base-id-index

Conversation

@stevevanhooser

Copy link
Copy Markdown
Contributor

Follow-up to #98. This ports VH-Lab/DID-matlab's f35bbb4..5b7220b, on the DID-matlab follow-up PR to #229.

Problem

When an older database is opened, _ensure_indexes builds doc_data_doc_idx and doc_data_field_value, and it does so silently.

For a large dataset on a slow or synced drive, that build takes minutes, and the open looks like it has hung. This was reported in DID-matlab on a real 100,000-document dataset.

Change

_ensure_indexes now prints to stderr:

  • Before each index it builds: the index name, the database, roughly how many rows it covers (max(rowid), which costs nothing), and that this happens once and can take minutes.
  • After the build: how long it took.
  • If a build fails: that searches will run without that index.

The text matches DID-matlab's. A database that already has the indexes opens without printing anything.

Tests

test_indexes_added_when_older_database_opened now checks two things:

  • the first open announces and times each index;
  • the next open prints nothing about indexes.

Local results:

  • black --check and ruff check are clean.
  • pytest: 817 passed.
  • bin/check_bridge_coverage.py (all six checks, against the DID-matlab branch): 0 problems.

Bridge

The sqlitedb entry now points at 5b7220b (the newest commit on the DID-matlab PR branch); its decision_log records the port. That commit only reaches DID-matlab main when the DID-matlab PR is merged, so merge that PR first, with a merge commit.


Generated by Claude Code

…DID-matlab f35bbb4)

Opening a database without doc_data_doc_idx / doc_data_field_value builds
them, silently - and for a large dataset on a slow or synced drive that
takes minutes, so the open appears to hang. _ensure_indexes now prints to
stderr, before each index it builds, its name, the database, the
approximate number of rows (max(rowid)) and that this happens once and can
take minutes; afterwards the time it took; and a build that failed. A
database that already has the indexes opens without printing anything.

Test: test_indexes_added_when_older_database_opened checks the messages on
the first open and silence on the next. Bridge: sqlitedb -> f35bbb4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
…hing to port)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv7SxVR8MCu3MKpUvwwUia
@stevevanhooser
stevevanhooser merged commit b224054 into main Oct 24, 2026
7 checks passed
@stevevanhooser
stevevanhooser deleted the claude/did-base-id-index branch October 24, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants