Skip to content

perf: skip the startup walk when no tables are declared - #997

Merged
thekevinscott merged 6 commits into
mainfrom
claude/issue-986-red-test-e7sjrb
Aug 20, 2026
Merged

thekevinscott merged 6 commits into
mainfrom
claude/issue-986-red-test-e7sjrb

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Aug 20, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #986

Problem

Configless startup declares zero named tables — filesystem access goes through path-tables, which mint their own per-statement scan in PathTabCursor::filter. Yet prepare_resolved called scan_directory unconditionally, and that walk returns nothing by construction: compile_matcher over zero tables builds a TableMatcher with empty entries, and match_all filters entries — so every entry the walk visits yields nothing.

Fix

let scanned = if table_names.is_empty() {
    Vec::new()
} else {
    scan(&root, &matcher)
};

table_names was already in scope and discarded via let _ = table_names;, now removed.

Only the walk is skipped. reconcile_scan still runs with an empty scan, so deleted is derived exactly as before, and path-table queries never consulted the startup scan to begin with.

Red/green

commit
c26e698 red test + seam. CI red on Unit tests: 678 passed; 2 failed, both on their own assertions — assertion left == right failed: zero tables can match nothing, so the startup walk is pure waste / left: 1 / right: 0. Lint green.
34d8ba3 the fix. 680 passed; 0 failed.

The walk has no result a caller can tell apart from not walking, so whether it ran needs a seam: prepare_resolved delegates to a private prepare_resolved_with(resolved, scan: ScanFn<'_>) passing &scan_directory. Rust has no monkeypatching, and scan_directory is naturally a callable the unit receives, so this is the DI case AGENTS.md carves out. The seam landed in the red commit and is behaviour-preserving on its own.

Three unit tests, using a recording double that counts invocations and returns a canned result (hermetic — no real tree, so unit lint isolation passes):

test asserts at c26e698
prepare_does_not_walk_the_tree_when_no_tables_are_declared walk invoked 0 times RED
prepare_still_opens_the_persist_cache_when_no_tables_are_declared walk invoked 0 times, persist context still built RED
prepare_walks_the_tree_when_a_table_is_declared walk invoked once, its result carried into scanned_files green

Unit tier only, deliberately. The change alters no observable behaviour and no public API — the walk's contribution is provably empty — so integration/binding/e2e have nothing new to assert; existing coverage at those tiers proves the absence of regression. Per AGENTS.md's decision tree (no public-API change + adequate integration coverage → unit tests only) that is the right shape.

On the persist interaction

#986 flags reconcile_scan as the one real interaction and asks for confirmation rather than assumption, suggesting a narrowing to the non-persist path as a conservative first cut. Confirmed — and the narrowing is not needed, in both directions:

  • The fix skips only scan_directory. reconcile_scan(&root, scanned, ctx, &RealFs) still runs with scanned empty, so deleted is derived exactly as today.
  • ctx.cached is provably empty whenever tables is empty anyway: a non-empty cache requires a prior run whose compute_glob_config_hash(tables, ignore) matched, and a prior zero-table run wrote nothing (its scan was empty too). A prior run with tables hashes differently, so prepare_persist wipes the cache and hands back an empty index.

Manual verification

Debug build, 20k files across 10 dirs, warm cache, best of 5 runs of dirsql "SELECT 1 AS x" (which touches no filesystem, so all variation is the startup walk):

cwd before after
empty dir 7 ms 7 ms
20k files 45 ms 7 ms

The walk is gone; the 20k-file case now matches the empty-dir baseline exactly.

Correctness, same binary:

$ cd /tmp/big && dirsql "SELECT count(*) AS n FROM './**'"
[{"n":20000}]

$ dirsql "SELECT path, basename, ext FROM './**' ORDER BY path"
[{"basename":"data.csv","ext":"csv","path":"data.csv"},{"basename":"note.txt","ext":"txt","path":"nested/note.txt"}]

$ dirsql query "SELECT * FROM rows" -c .dirsql.toml     # declared table still indexes
[{"a":"1","b":"2"}]

$ dirsql query "SELECT path FROM './*'" --persist        # twice; cache.db created, results stable
[{"path":"a.txt"}]
[{"path":"a.txt"}]

Changelog / Migrations

  • Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (packages/rust/changelog.d/2026-08-20-skip-configless-startup-walk.md)
  • Migration fragment added under <root>/<pkg>/migrations.d/ (or: not required — additive/bugfix only) — not required: no public API change, and no behaviour change (the skipped walk contributed nothing)

PARITY.md is untouched: no SDK's public API surface changes. No parity drift.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (32 passed)
  • TypeScript SDK e2e: pass (16 passed, 5 files)
  • Rust core e2e (if applicable): N/A — exercised via the two binding suites
  • packages/python attestation refreshed (just e2e-attest-python equivalent)
  • packages/ts attestation refreshed (just e2e-attest-ts equivalent)
  • Command(s) run:
    • cd packages/python && uvx testing-conventions e2e attest 'uv run python -m pytest tests/e2e/ -x -q'
    • cd packages/ts && uvx testing-conventions e2e attest 'pnpm test:e2e'
  • Result summary: both green, re-run and re-attested after the fix commit (a packages/rust/src change stales both bindings, and the red-commit receipts were staled by the fix).

Gates run locally on the green tree: cargo fmt --check, cargo clippy --workspace -- -D warnings, cargo test -p dirsql --lib --features cli (680 pass), cargo test -p dirsql --features cli --tests, unit colocated-test, unit lint, integration lint, unit mutation --base origin/main (no surviving mutants). packaging needs a built artifact, so CI covers it.

One local-only note: tests/vtab.rs::unreadable_file_yields_null_content_without_erroring_the_row fails in this sandbox both with and without the change — it chmod 000s a file, and the sandbox runs as root, which bypasses that. Unrelated to this diff and green on CI's non-root runner.

claude added 3 commits August 20, 2026 20:26
With no `-c`, dirsql declares zero named tables: filesystem access goes
through path-tables, which mint their own per-statement scan. Yet
`prepare_resolved` runs `scan_directory` unconditionally, and with no
table globs to match against that walk returns nothing by construction —
78-85% of the runtime of a query that reads no files.

The walk has no result a caller can tell apart from not walking, so
whether it ran takes a seam: `prepare_resolved_with` takes the walk as an
injected callable, and `prepare_resolved` passes `scan_directory`. Purely
a seam — the walk still runs unconditionally, so the two zero-table tests
are RED and the declared-table test (which guards against over-skipping)
is green.

Refs #986

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E88j3ZKfp1bsLA9yxqFxwG
claude added 3 commits August 20, 2026 20:38
Zero tables compiles to a `TableMatcher` with no entries, and `match_all`
filters those entries — so `scan_directory` is guaranteed to return
nothing. Short-circuit it.

That is the configless case, where filesystem access goes through
path-tables and `PathTabCursor::filter` runs its own per-statement
`scan_glob`. The startup walk contributed nothing there and cost 78-85%
of the runtime of a query that reads no files.

Only the walk is skipped. `reconcile_scan` still runs with an empty
scan, so `deleted` is derived exactly as before — and `ctx.cached` is
provably empty with zero tables anyway, since a non-empty cache needs a
prior run whose `compute_glob_config_hash` matched, and a prior
zero-table run wrote nothing.

Measured, debug build, 20k files across 10 dirs, best of 5:

    SELECT 1, empty dir      7 ms -> 7 ms
    SELECT 1, 20k files     45 ms -> 7 ms

Fixes #986

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E88j3ZKfp1bsLA9yxqFxwG
@thekevinscott
thekevinscott merged commit 63d7860 into main Aug 20, 2026
77 checks passed
@thekevinscott
thekevinscott deleted the claude/issue-986-red-test-e7sjrb branch August 20, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Configless startup walks the entire tree for a provably empty result

3 participants