Skip to content

perf: read a plain path-table projection without stepping SQLite - #1537

Merged
thekevinbot merged 11 commits into
mainfrom
bot/1475-rowpath-1016
Oct 20, 2026
Merged

thekevinbot merged 11 commits into
mainfrom
bot/1475-rowpath-1016

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1475

A plain projection of one path table (SELECT path FROM './a/b/**/*.md', any stat columns, aliases, SELECT *) now reads its rows straight from the statement's row cache instead of stepping SQLite once per row. The query is recognized by its EXPLAIN program: only the exact program SQLite compiles for a bare virtual-table scan qualifies. Anything else, including WHERE, ORDER BY, LIMIT, DISTINCT, expressions, joins, CTEs or the content column, still steps SQLite. Output is byte-identical: 18 queries were md5-compared against main, and a new integration test checks the rows read directly against the rows SQLite steps.

The red commit raises the ./a/b/**/*.md speed case's bar from find to the fastest native (fd), matching ./**/*.md.

Ratios (paired median, dirsql / fd)

glob main this PR #1531 this PR + #1531
./a/b/**/*.md (800k-file tree, 41 rounds) 1.253 0.961 1.084 0.909
./a/b/**/*.md (2M-item tree, 31 rounds) 1.263 0.973 - 0.929
./**/*.md 0.940 0.656 0.939 0.620
./**/*.js (no matches) 0.464 0.466 0.517 0.518

E2E Verification

  • Red on main core: uv run python -m pytest tests/e2e/glob_speed_recursive_test.py failed with prefix case median 1.731 vs fd.

  • With the fix: the prefix case passed at median 1.009 (load 6). It failed at 1.23 under load 22, so it is load-sensitive.

  • Receipt (python): uv run python -m pytest tests/e2e/{query_contract,declared_table_name,on_file_query,use_case_dossier,use_case_join,use_case_notebook,use_case_transcripts,stat_speed,glob_speed_one_level,glob_speed_recursive}_test.py -x -q: 23 passed.

  • Receipt (ts): pnpm test:e2e: pass.

  • cargo test --workspace --features cli: pass. path_table_parallel_scan failed once under load (timing assertion, count(*) query, not this path); it passed on rerun.

  • Re-run after merging main (refactor: read a directory's listing in chunks #1528, fix: run config tables' on-file commands in declaration order #1539, A refresh updates a keyed row in place #1542, Check foreign keys once per build and per watch batch #1543), on c7452a0: receipt (python) the 13 glob-speed/query-contract/fanout/use-case-join/stat/release-speed e2e files, 32 passed. Fastest vs fd medians: ./a/b/**/*.md 0.972, ./**/*.md 0.589. Receipt (ts) pnpm test:e2e: pass. cargo test --workspace --features cli: pass.

  • Hand check, 800k tree, 30 interleaved pairs vs fd: ./a/b/**/*.md 0.894, ./**/*.md 0.592.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/ for each changed package
  • Migration fragment with the same slug added under packages/rust/migrations.d/ for each changed package

@thekevinbot
thekevinbot merged commit 59f77b0 into main Oct 20, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1475-rowpath-1016 branch October 20, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: ./a/b/**/*.md is 4.2x the fastest native (fd)

1 participant