Skip to content

Speed up the path-table walk on large directories - #1321

Merged
thekevinscott merged 10 commits into
claude/1318-lazy-statfrom
claude/1320-walk-speed
Oct 6, 2026
Merged

thekevinscott merged 10 commits into
claude/1318-lazy-statfrom
claude/1320-walk-speed

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Speed up the path-table walk on large directories. Stacked on #1319.

Fixes #1320

  • Sort a directory's names on an 8-byte big-endian prefix first, falling back to the full name only on a tie. The order is byte order, unchanged.
  • Judge the entries of a directory holding 4096 or more entries across every core. Results are applied in name order, so visits and descents are unchanged. Smaller directories stay on one thread and never query the core count; that query reads cgroup files and cost 2s in a 150k-directory tree.
  • Run glob.step for a file only when a symlink is crossed, since only then is its result read. Build the full entry path only when a .gitignore is in force.
  • build_rows cuts its work from the back, so each path moves once instead of the remainder being re-copied on every cut.

Red commit first: judge_all_shares_a_large_directory_across_threads. CI on a3821fc failed on exactly that assertion (judged on 1 thread(s)) on Linux and Windows.

Measurements (release build, warm cache)

Query Base (#1319) This PR
SELECT path FROM './*', 1M flat files (find 0.42s) 1.66s 1.05s
SELECT count(*) FROM './*', same tree 1.22s 0.61s
Join of two 150k-row tables 1.6s 1.6s

The output is byte-identical to the base for:

  • ./**/*.rs, ./**/*, ./.github/**/*.yml, ./*/* and ./packages/**/*.md on this repo (with gitignores)
  • ./*, ./**/*.md and ./docs/** on the 1M tree

glob_speed_one_level_test (#1305) still fails. On its 2M-file tree, dirsql takes 1.78s after startup against native 0.71s (budget 0.78s). Most of what remains is SQLite stepping and JSON output, not the walk.

E2E Verification

  • Python: uv run python -m pytest tests/e2e/ -q --deselect tests/e2e/glob_speed_one_level_test.py, 40 passed.
    • The deselected test is the known-red speed test. The receipt records the deselect.
  • TypeScript: just e2e-attest-ts (pnpm test:e2e) passed.
  • Rust: cargo test --workspace --features cli passed, apart from the server watcher tests that flake under load. They fail the same way on the base branch and pass alone.
  • By hand: touch b.md a.md; dirsql query "SELECT path FROM './*'" returns [{"path":"a.md"},{"path":"b.md"}].

Changelog / Migrations

  • packages/rust/changelog.d/2026-10-05-path-table-walk-speed.md
  • packages/rust/migrations.d/2026-10-05-path-table-walk-speed.md (nothing breaks)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants