Repository navigation
Path-table walk: read directory names into one buffer - #1324
Merged
Merged
Conversation
This was referenced Oct 5, 2026
# Conflicts: # testing-conventions.toml
2 tasks done
…ude/1322-walk-readdir
Base automatically changed from
claude/1320-walk-speed
to
claude/1318-lazy-stat
October 6, 2026 00:07
Base automatically changed from
claude/1318-lazy-stat
to
claude/1305-glob-speed
October 6, 2026 00:08
This was referenced Oct 6, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Read each directory's names into one shared buffer instead of one allocation per entry. Stacked on #1321.
Fixes #1322
rustix::fs::Dirreads each directory, a safe API with no newunsafe. Each name is copied into one buffer per directory. Elsewhere,std::fs::read_dirfills the same structure.d_typesays what an entry is. An entry whose type the filesystem does not report islstated, asDirEntry::file_typedid. A symlink is followed only where it was followed before..gitignorechecks.Red commit first:
a_listing_sorts_its_names_into_byte_order. CI on e55027b failed on exactly that assertion: names tied on their first 8 bytes stayed unsorted.Measurements (release build, warm cache, 32 cores)
findSELECT count(*) FROM './*', 1M flat filesSELECT path FROM './*', same treeSELECT count(*) FROM './**/*.md', 1.6M files in 2.3k dirsSELECT path FROM './**/*.md', same treeA microbenchmark of 1M names gives the floor:
std::fs::read_dirplus sort takes 0.26s,rustixDirplus a shared buffer takes 0.17s, and rawgetdents64with a 1MB buffer takes 0.15s.Output is byte-identical to the base for 12 queries:
./**/*.rs,./**/*,./.github/**/*.yml,./*/*and./packages/**/*.md(withsize) on this repo./*,./**/*.mdand./docs/**on the 1M tree./**/*.md,./a/b/**/*.md,./**/.cache/*and./*/*/*/*/*on the recursive treeglob_speed_one_level_test(#1305) still fails, now at 0.69s after startup against a 0.40s budget (native 0.36s). On the 1M tree,SELECT pathbreaks down as:HashMapinrun_statementabout 0.2sresult_to_json0.13sThe output pipeline is next.
E2E Verification
uv run python -m pytest tests/e2e/ -q --deselect tests/e2e/glob_speed_one_level_test.py, 40 passed.just e2e-attest-ts(pnpm test:e2e) passed.cargo test -p dirsql --features cli --no-fail-fastpassed.Changelog / Migrations
packages/rust/changelog.d/2026-10-05-path-table-walk-readdir.mdpackages/rust/migrations.d/2026-10-05-path-table-walk-readdir.md(nothing breaks)