Repository navigation
perf: walk drops unmatchable file names early and reads ahead in walk order - #1521
Merged
Merged
Conversation
…lk's reads The walk rejects a regular file whose name no glob word's last component matches before building its path, checks for .git and .gitignore in the listing it already read, opens directories without duplicating the fd, and has workers claim the open directory the walk reaches first, each directory behind its own lock.
This was referenced Oct 15, 2026
…d-1015 # Conflicts: # packages/rust/src/scanner.rs # packages/rust/src/tree_walk.rs
…king a worker for it
8 tasks done
8 tasks done
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1480
Closes the matcher-glob gap to fd on the walk side.
PathGlob::may_match_name) before building its path or running the full-path match..git/.gitignoreare looked up in the listing already read, so a directory without them costs no failingstatx.tree_walk: each directory has its own lock and condvar; the open directories sit in a heap keyed by DFS position, so workers read just ahead of the consuming thread instead of deep in far subtrees. Keepsbusy_workersfrom main: a worker is not busy while it waits for a directory.nsubdirectories wakesn - 1waiting workers, because it takes one next itself. On a chain of nested directories no worker is woken, so the walk no longer passes each level from one thread to another. That handoff stopped two walks in one statement from overlapping on 4 cores, andpath_table_parallel_scanfailed on Windows. Locally on 4 cores, the overlap went from 0.6-0.7 of the serial time to about 0.53.Diragain, notRawDir(see below).Glob semantics unchanged.
The
RawDirfindingAfter merging main, this PR measured the same as main: the gain disappeared. The cause is the
RawDirread on Linux. WithDir, the same walk runs about twice as fast on a deep tree of small directories. Buffer size is not the cause: 768 B, 4 KB, 32 KB and 1 MB buffers, thread-local or on the stack, are all slow. glibc malloc tunables changed nothing. The slow build makes about 140k voluntary context switches against about 20k, waiting on glibc arena mutexes. The root cause is still unknown.On a flat directory of 1M files,
DirandRawDirare within noise of each other (./*0.366s on main, 0.385s here;./*.md0.380s on main, 0.367s here).Measurements
Paired medians of dirsql/fd (21 interleaved rounds). Tree: 20k directories × 80 files, the
glob_speed_matcherslayout, on ZFS. Host load was 15-37 from other agents.Final A/B at head, one run of 15 interleaved rounds:
./**/{a,b}*.md./**/file?.md./**/[a-c]*.md./**/[[:digit:]]*.mdAn earlier run isolated the two parts:
DirRawDir./**/{a,b}*.md./**/file?.md./**/[a-c]*.md./**/[[:digit:]]*.mdE2E Verification
cargo test -p dirsql --lib)packages/python/e2e-attestations/<branch>.jsonreceipt writtenpackages/ts/e2e-attestations/<branch>.jsonreceipt written (just e2e-attest-ts)cargo test -p dirsql --features cli(all pass);testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_one_level_test.py tests/e2e/glob_speed_recursive_test.py tests/e2e/glob_speed_depth_bound_test.py tests/e2e/glob_speed_gitignore_test.py tests/e2e/glob_speed_dotfiles_test.py tests/e2e/glob_speed_matchers_test.py tests/e2e/glob_speed_node_modules_test.py tests/e2e/glob_speed_symlink_test.py tests/e2e/stat_speed_test.py tests/e2e/query_contract_test.py -q';just e2e-attest-ts