Skip to content

Stat path-table rows only when a stat column is read - #1319

Merged
thekevinscott merged 17 commits into
claude/1305-glob-speedfrom
claude/1318-lazy-stat
Oct 6, 2026
Merged

thekevinscott merged 17 commits into
claude/1305-glob-speedfrom
claude/1318-lazy-stat

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Stat a path-table row only when the statement reads size, mtime, ctime or content. Stacked on #1317.

Fixes #1318

  • Building rows no longer stats anything. A row's stat facts fill lazily.
  • The first stat a statement asks for stats that statement's whole row set at once, in parallel. Stats are never done one by one as SQLite steps.
  • Content prefetch is unchanged.

Red commit first: build_rows_stats_no_file asserts that building rows stats no file. CI on f51c5b9 failed on exactly that assertion (vtab.rs:713).

Measurements (release build, warm cache)

Query Base (#1317) This PR
SELECT path FROM './*', 1M flat files 2.4s 1.6s
Join of two 150k-row tables reading size and content 1.6–2.3s 1.6s

The output is byte-identical to the base in both cases.

An earlier version stat'ed lazily one row at a time. It made the join 7.8s, so it was replaced before push.

glob_speed_one_level_test (#1305) still fails: dirsql takes 1.30s against a 0.38s budget. The walk is the next fix (#1320).

E2E Verification

  • Python: uv run python -m pytest tests/e2e/ -q --deselect tests/e2e/glob_speed_one_level_test.py, 40 passed.
    • The deselected test is the known-red speed test this stack is working toward. The receipt records the deselect. Whether a receipt may exclude a known-red test is for the maintainer to decide.
  • TypeScript: just e2e-attest-ts (pnpm test:e2e) passed.
  • Rust: cargo test --workspace --features cli passed apart from flaky server watcher tests. post_query_returns_rows_over_http and server_drops_the_rows_of_a_directory_moved_out_of_the_root fail the same way on the base branch under load and pass alone.
  • By hand:
    • SELECT * returns the stat and content values.
    • SELECT path, size returns correct sizes.
    • SELECT path returns paths only.

Changelog / Migrations

  • packages/rust/changelog.d/2026-10-05-path-table-lazy-stat.md
  • packages/rust/migrations.d/2026-10-05-path-table-lazy-stat.md (nothing breaks)

Speed up the path-table walk on large directories
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants