Repository navigation
perf(rust): read a path table's content ahead, in parallel, when a statement names it - #1242
Merged
Merged
Conversation
5 tasks done
thekevinbot
force-pushed
the
claude/1241-content-prefetch
branch
from
October 5, 2026 00:50
c70d57c to
43719c5
Compare
A statement naming content has every selected row's file read before the first row is delivered, not one file at a time as SQLite steps the cursor. Stepping a statement with the tree deleted after its first row shows which: the rows that follow still carry the text only if it was read ahead. Covers a projection, a WHERE on content, and the side of a join that names it.
…atement names it A statement that names content had each row's file read on the SQLite cursor thread as the row was emitted: five syscalls per file, one file at a time, 28% of the join speed-of-light query. Now xBestIndex reads colUsed, and a filter whose statement names content reads the file of every row it selected before SQLite steps the first, eight files at a time; a statement that does not name it reads nothing, as before. The read is also sized from the size the scan already stat'ed, so it opens, reads and closes without stat'ing again. A lookup on a table whose content is named is declared dearer, so a join scans that side and probes the other: read ahead, the column covers the whole table at once rather than one probe at a time.
…e the rest read_all now has to grow geometrically for an outgrown file, and the default prefetch is an empty body with nothing to mutate. The file reads and the read-ahead itself are integration-tier behavior, and the bitwise-or in plan_idx_num cannot differ from xor on any column index.
thekevinbot
force-pushed
the
claude/1241-content-prefetch
branch
from
October 5, 2026 01:11
43719c5 to
f87ec67
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1241
Status
Rebased onto
mainafter #1239 (parallel path-table walk), #1236 (glob-prefix walk), #1248 (ingest perf), #1251 (one-leveldir) and #1245 (on-file cleanup) merged. Green locally:just preflight0 failing pairs (Rust mutation: 46 mutants tested, none surviving),cargo test --workspace --features cli1609 passed, both e2e receipts rewritten. Red commit623a580was confirmed on CI before the rebase: RustIntegration testsandWindows testsfailed on exactly the three new tests inpackages/rust/tests/path_table_content_prefetch.rs.Rebase notes:
PARITY.mdkeeps both the #1239 paragraph and this one; thetesting-conventions.tomlmutation waivers forvtab.rs/vtab_scaffold.rswere re-pinned to the merged line numbers the mutation gate reports (rows/columnat 360 / 371,read_text/ScanSpec::read/prefetchat 81 / 328 / 382, the equivalent|->^inplan_idx_numat 335).What changed
A statement that names a path table's
content-- in its projection, itsWHERE, or across a join -- has the bodies of the rows the filter selected read ahead, eight files at a time, before SQLite steps the first row. Detection isxBestIndex'scolUsedbitmask (rusqliteIndexInfo::col_used), sodb.rsis untouched. A statement that does not namecontentreads no file;SELECT *still excludes it; an unreadable or non-UTF-8 file is still aNULLcell. Each read is presized from thesizethe scan already recorded, so it is open / read / read / close with no second stat (strace: statx count unchanged from the walk,read= 2 x files).vtab_scaffold.rs:TableSource::PREFETCH_COLUMN/prefetch;PREFETCH_IDXflag bit inidx_num; a lookup that prefetches costsPREFETCH_LOOKUP_COST(100, between the bare lookup's 10 and a scan's 1000) so in a join the planner scans the content-bearing table and probes the other -- otherwise the read-ahead would run once per probe. Verified:EXPLAIN QUERY PLANon the join flips toSCAN t ... INDEX 1073741824/SCAN a ... INDEX 3.vtab.rs:FileRow.content: OnceLock<Option<String>>, filled by athread::scopeofREADERSworkers infilter;column()falls back to a lazy read for a row the prefetch did not cover.WHERE path = ?,WHERE path IN (...)) prefetches only the rows it selected, so a point lookup still reads one file.Trade-off: a non-lookup
WHEREorLIMITno longer shieldscontentreads -- naming the column reads the whole selected set, and its text is held for the statement's lifetime. That is the issue's accepted cost: reads were already going to happen for every row the cursor would step.Thread count
READERS = 8, no knob. Measured 4 / 8 / 16 on the 20k-paper fixture, warm cache, hyperfine 10 runs (pre-rebase, against the serial walk):SELECT content FROM './papers/*/title.md'Content delta over a bare
SELECT count(*)on the same table (~179 ms): 76.7 ms before, 30.2 / 25.8 / 25.5 ms at 4 / 8 / 16. 8 is the knee; 16 buys nothing.Benchmarks (hyperfine, 10 runs, 3 warmup, warm cache, 20k papers; pre-rebase)
a.dir, t.content, a.size ... JOIN ... ORDER BY a.dir)SELECT content FROM './papers/*/title.md'SELECT content FROM './papers/*/abstract.md'SELECT count(*) FROM './papers/*/title.md'Measured before #1239 merged, with it as a scratch stack:
Together the join lands under native (1.1 x native = 266 ms); either alone missed the bar. #1239 is now on
main, so this branch carries the combined number.Speed-of-light tests (#1205)
Run on the rebased branch (
mainwith #1239 + this PR),-q -rxX:use_case_dossier_test.pyuse_case_join_test.pyuse_case_notebook_test.pyuse_case_transcripts_test.pydirsql-plugin-embeddings/tests/e2e/speed_of_light_test.pyPython: 3 passed, 1 xfailed in 117.02s. Plugin: 1 passed in 5.82s. (A first pass of the Python four, taken while a cargo build was still running on the box, had two wall-clock misses; the quiet-box rerun above is the recorded result.)
E2E Verification
cargo test --workspace --features cli: 1609 passed)packages/python/e2e-attestations/<branch>.jsonreceipt written ifpackages/pythonchanged (just e2e-attest-python)packages/ts/e2e-attestations/<branch>.jsonreceipt written ifpackages/tschanged (just e2e-attest-ts)cargo test --workspace --features cli;cd packages/python; uv run maturin develop --release;cd packages/ts; pnpm build;cd packages/python; uv run python -m pytest tests/e2e/use_case_dossier_test.py tests/e2e/use_case_join_test.py tests/e2e/use_case_notebook_test.py tests/e2e/use_case_transcripts_test.py -q -rxX;cd plugins/dirsql-plugin-embeddings; uv run --with-editable ../../packages/python python -m pytest tests/e2e/speed_of_light_test.py -q -rxX;just e2e-attest-python;just e2e-attest-ts;just preflightpackages/python/e2e-attestations/claude-1241-content-prefetch.jsonandpackages/ts/e2e-attestations/claude-1241-content-prefetch.jsonChangelog / Migrations
<root>/<pkg>/changelog.d/for each changed package (or:skip-changelogtrailer on a commit with reason) --packages/rust/changelog.d/2026-10-02-path-table-content-prefetch.md<root>/<pkg>/migrations.d/(or: not required -- additive/bugfix only) --packages/rust/migrations.d/2026-10-02-path-table-content-prefetch.md, required changes_None._PARITY.md: paragraph added recording this as a Rust-crate-only surface with no SDK drift.docs/reference/path-tables.md: the laziness paragraph now describes the read-ahead.