Skip to content

perf: open the database during the startup scan - #1566

Merged
thekevinbot merged 11 commits into
mainfrom
bot/1560-scan-insert-1102
Nov 3, 2026
Merged

thekevinbot merged 11 commits into
mainfrom
bot/1560-scan-insert-1102

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1567

Refs #1560

First M-sized slice of overlapping startup work: on a transient (non-persist) build with no extensions or declared functions, open SQLite and run the tables' DDL on a second thread while the directory scan runs, and carry the opened connection into ingest. With extensions or functions, the DDL still runs after they load. Row insertion during the scan is not in this PR.

Measurement

Local, 131072 frontmatter notes, SELECT status, count(*), sum(words) ... GROUP BY status, release CLI, 100 interleaved pairs against the merge base 05880bb: median 162.7 ms (base) vs 161.6 ms (branch), median paired ratio 0.999. That is neutral: opening the DB and creating the tables takes about a millisecond, so overlapping them with the scan saves nothing measurable.

Phase timing shows that the time after the scan goes to the per-table hook's output (the sharded awk runs), not to SQLite inserts. A prototype that also streamed the early rows into SQLite during the scan measured 1.037x, a little slower, because grouping no longer overlapped. It is not included here.

E2E Verification

  • uv run maturin develop --release, cargo build --release -p dirsql --features cli
  • Python: uv run python -m pytest tests/e2e/frontmatter_parse_speed_test.py tests/e2e/jsonl_parse_speed_test.py tests/e2e/cli_version_test.py tests/e2e/declared_table_name_test.py tests/e2e/extension_package_test.py tests/e2e/fanout_test.py tests/e2e/interpret_removed_test.py tests/e2e/on_file_query_test.py tests/e2e/plugin_discovery_test.py tests/e2e/plugin_extension_discovery_test.py tests/e2e/query_contract_test.py tests/e2e/repl_interactive_test.py tests/e2e/sigint_interrupt_test.py tests/e2e/use_case_dossier_test.py tests/e2e/use_case_join_test.py tests/e2e/use_case_notebook_test.py tests/e2e/use_case_transcripts_test.py -q: 41 passed (receipt committed)
  • TypeScript: pnpm test:e2e: 18 passed (receipt committed)
  • By hand: query results match the base on the 131072-note tree; a malformed DDL fails with the same error, and a DDL that creates an extra table gives the same output.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/ for each changed package
  • Migration fragment with the same slug added under packages/rust/migrations.d/ for each changed package (every subsection but Summary _None._ when nothing breaks)

@thekevinbot
thekevinbot merged commit 4e53086 into main Nov 3, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1560-scan-insert-1102 branch November 3, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: open transient SQLite and create tables during the scan

1 participant