Skip to content

Path-table walk: read directory names into one buffer - #1324

Merged
thekevinscott merged 8 commits into
claude/1305-glob-speedfrom
claude/1322-walk-readdir
Oct 6, 2026
Merged

thekevinscott merged 8 commits into
claude/1305-glob-speedfrom
claude/1322-walk-readdir

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Read each directory's names into one shared buffer instead of one allocation per entry. Stacked on #1321.

Fixes #1322

  • On Unix, rustix::fs::Dir reads each directory, a safe API with no new unsafe. Each name is copied into one buffer per directory. Elsewhere, std::fs::read_dir fills the same structure.
  • Entries are sorted on an 8-byte prefix, then on the name bytes, so the order is byte order, as before.
  • d_type says what an entry is. An entry whose type the filesystem does not report is lstated, as DirEntry::file_type did. A symlink is followed only where it was followed before.
  • Full paths are built only for directories entered, symlinks followed and .gitignore checks.

Red commit first: a_listing_sorts_its_names_into_byte_order. CI on e55027b failed on exactly that assertion: names tied on their first 8 bytes stayed unsorted.

Measurements (release build, warm cache, 32 cores)

Query Base (#1321) This PR find
SELECT count(*) FROM './*', 1M flat files 0.62s 0.39s 0.43s
SELECT path FROM './*', same tree 1.05s 0.82s 0.43s
SELECT count(*) FROM './**/*.md', 1.6M files in 2.3k dirs 0.80s 0.73s 0.58s
SELECT path FROM './**/*.md', same tree 1.40s 1.34s 0.58s

A microbenchmark of 1M names gives the floor: std::fs::read_dir plus sort takes 0.26s, rustix Dir plus a shared buffer takes 0.17s, and raw getdents64 with a 1MB buffer takes 0.15s.

Output is byte-identical to the base for 12 queries:

  • ./**/*.rs, ./**/*, ./.github/**/*.yml, ./*/* and ./packages/**/*.md (with size) on this repo
  • ./*, ./**/*.md and ./docs/** on the 1M tree
  • ./**/*.md, ./a/b/**/*.md, ./**/.cache/* and ./*/*/*/*/* on the recursive tree

glob_speed_one_level_test (#1305) still fails, now at 0.69s after startup against a 0.40s budget (native 0.36s). On the 1M tree, SELECT path breaks down as:

  • scan 0.29s (readdir and sort 0.16s)
  • row build 0.06s
  • the CLI's output pipeline about 0.45s:
    • per-row HashMap in run_statement about 0.2s
    • result_to_json 0.13s
    • render 0.06s
    • drops 0.06s

The output pipeline is next.

E2E Verification

  • Python: uv run python -m pytest tests/e2e/ -q --deselect tests/e2e/glob_speed_one_level_test.py, 40 passed.
    • The deselected test is the known-red speed test. The receipt records the deselect.
  • TypeScript: just e2e-attest-ts (pnpm test:e2e) passed.
  • Rust: cargo test -p dirsql --features cli --no-fail-fast passed.
  • By hand: the 12 identical-output queries above.

Changelog / Migrations

  • packages/rust/changelog.d/2026-10-05-path-table-walk-readdir.md
  • packages/rust/migrations.d/2026-10-05-path-table-walk-readdir.md (nothing breaks)

@thekevinbot thekevinbot mentioned this pull request Oct 5, 2026
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants