Repository navigation
perf: open the database during the startup scan - #1566
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1567
Refs #1560
First M-sized slice of overlapping startup work: on a transient (non-persist) build with no extensions or declared functions, open SQLite and run the tables' DDL on a second thread while the directory scan runs, and carry the opened connection into ingest. With extensions or functions, the DDL still runs after they load. Row insertion during the scan is not in this PR.
Measurement
Local, 131072 frontmatter notes,
SELECT status, count(*), sum(words) ... GROUP BY status, release CLI, 100 interleaved pairs against the merge base 05880bb: median 162.7 ms (base) vs 161.6 ms (branch), median paired ratio 0.999. That is neutral: opening the DB and creating the tables takes about a millisecond, so overlapping them with the scan saves nothing measurable.Phase timing shows that the time after the scan goes to the per-table hook's output (the sharded
awkruns), not to SQLite inserts. A prototype that also streamed the early rows into SQLite during the scan measured 1.037x, a little slower, because grouping no longer overlapped. It is not included here.E2E Verification
uv run maturin develop --release,cargo build --release -p dirsql --features cliuv run python -m pytest tests/e2e/frontmatter_parse_speed_test.py tests/e2e/jsonl_parse_speed_test.py tests/e2e/cli_version_test.py tests/e2e/declared_table_name_test.py tests/e2e/extension_package_test.py tests/e2e/fanout_test.py tests/e2e/interpret_removed_test.py tests/e2e/on_file_query_test.py tests/e2e/plugin_discovery_test.py tests/e2e/plugin_extension_discovery_test.py tests/e2e/query_contract_test.py tests/e2e/repl_interactive_test.py tests/e2e/sigint_interrupt_test.py tests/e2e/use_case_dossier_test.py tests/e2e/use_case_join_test.py tests/e2e/use_case_notebook_test.py tests/e2e/use_case_transcripts_test.py -q: 41 passed (receipt committed)pnpm test:e2e: 18 passed (receipt committed)Changelog / Migrations
packages/rust/changelog.d/for each changed packagepackages/rust/migrations.d/for each changed package (every subsection but Summary_None._when nothing breaks)