Skip to content

perf: walk drops unmatchable file names early and reads ahead in walk order - #1521

Merged
thekevinbot merged 12 commits into
mainfrom
bot/1480-matcher-speed-1015
Oct 19, 2026
Merged

thekevinbot merged 12 commits into
mainfrom
bot/1480-matcher-speed-1015

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1480

Closes the matcher-glob gap to fd on the walk side.

  • The walk drops a regular file whose name no glob word's last component can match (PathGlob::may_match_name) before building its path or running the full-path match.
  • .git / .gitignore are looked up in the listing already read, so a directory without them costs no failing statx.
  • tree_walk: each directory has its own lock and condvar; the open directories sit in a heap keyed by DFS position, so workers read just ahead of the consuming thread instead of deep in far subtrees. Keeps busy_workers from main: a worker is not busy while it waits for a directory.
  • A thread that explores a directory and finds n subdirectories wakes n - 1 waiting workers, because it takes one next itself. On a chain of nested directories no worker is woken, so the walk no longer passes each level from one thread to another. That handoff stopped two walks in one statement from overlapping on 4 cores, and path_table_parallel_scan failed on Windows. Locally on 4 cores, the overlap went from 0.6-0.7 of the serial time to about 0.53.
  • Linux reads directory listings through rustix Dir again, not RawDir (see below).

Glob semantics unchanged.

The RawDir finding

After merging main, this PR measured the same as main: the gain disappeared. The cause is the RawDir read on Linux. With Dir, the same walk runs about twice as fast on a deep tree of small directories. Buffer size is not the cause: 768 B, 4 KB, 32 KB and 1 MB buffers, thread-local or on the stack, are all slow. glibc malloc tunables changed nothing. The slow build makes about 140k voluntary context switches against about 20k, waiting on glibc arena mutexes. The root cause is still unknown.

On a flat directory of 1M files, Dir and RawDir are within noise of each other (./* 0.366s on main, 0.385s here; ./*.md 0.380s on main, 0.367s here).

Measurements

Paired medians of dirsql/fd (21 interleaved rounds). Tree: 20k directories × 80 files, the glob_speed_matchers layout, on ZFS. Host load was 15-37 from other agents.

Final A/B at head, one run of 15 interleaved rounds:

glob main this PR
./**/{a,b}*.md 1.78 0.84
./**/file?.md 2.02 0.95
./**/[a-c]*.md 1.66 0.92
./**/[[:digit:]]*.md 1.72 0.78

An earlier run isolated the two parts:

glob main main + Dir this PR, still RawDir this PR
./**/{a,b}*.md 1.57 1.22 1.52 0.87
./**/file?.md 1.82 1.22 1.85 0.93
./**/[a-c]*.md 1.38 1.17 1.33 0.87
./**/[[:digit:]]*.md 1.52 1.04 1.64 0.81

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass, 26 passed (impacted speed + contract subset); after the wake change, re-attested on matchers/recursive/gitignore/depth_bound/one_level/query_contract: 18 passed
  • TypeScript SDK e2e: pass, 18 passed (re-attested at head)
  • Rust core e2e (if applicable): pass (cargo test -p dirsql --lib)
  • packages/python/e2e-attestations/<branch>.json receipt written
  • packages/ts/e2e-attestations/<branch>.json receipt written (just e2e-attest-ts)
  • Command(s) run: cargo test -p dirsql --features cli (all pass); testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_one_level_test.py tests/e2e/glob_speed_recursive_test.py tests/e2e/glob_speed_depth_bound_test.py tests/e2e/glob_speed_gitignore_test.py tests/e2e/glob_speed_dotfiles_test.py tests/e2e/glob_speed_matchers_test.py tests/e2e/glob_speed_node_modules_test.py tests/e2e/glob_speed_symlink_test.py tests/e2e/stat_speed_test.py tests/e2e/query_contract_test.py -q'; just e2e-attest-ts

…lk's reads

The walk rejects a regular file whose name no glob word's last component
matches before building its path, checks for .git and .gitignore in the
listing it already read, opens directories without duplicating the fd,
and has workers claim the open directory the walk reaches first, each
directory behind its own lock.
@thekevinbot
thekevinbot merged commit 5493d78 into main Oct 19, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1480-matcher-speed-1015 branch October 19, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: matcher globs under ** are 3.4-7.6x the fastest native (fd)

1 participant