Skip to content

perf: keep walk workers off the shared heap lock - #1549

Merged
thekevinbot merged 3 commits into
mainfrom
bot/1478-perdir-walk-1020
Nov 2, 2026
Merged

thekevinbot merged 3 commits into
mainfrom
bot/1478-perdir-walk-1020

Conversation

@thekevinbot

Copy link
Copy Markdown
Collaborator

Fixes #1478

What

Walk workers spent nearly all their idle time acquiring the shared open mutex, not waiting on the condvar. Instrumented on a 100k-file tree (18,579 dirs, 32 threads): about 200ms of about 230ms of total worker idle time was lock acquisition. Each heap comparison dereferenced Arc<Slot>.key, memory another core had written, so every sift cache-missed while all the workers queued behind it.

tree_walk.rs only:

  • Pending carries an inline u128 rank, an order-preserving encoding of the key's leading bytes. Most comparisons stay in the heap's own memory, and full keys break rank ties.
  • A worker registers the directories it found and takes back the earliest in one lock acquisition, not two.

listing.rs and scanner.rs are untouched.

Timings

Paired and alternating against fdfind, median per-pair ratio of (dirsql - startup) / fd, with the machine shared with other agents (load about 10). The query is ./**/*.js against fdfind --glob '*.js'.

tree main this PR
100k files, 18.6k dirs (node_modules layout) 1.02-1.07x 0.82-0.86x
1M files, 279k dirs (fanout 16) 1.15x 0.99x
800k files, 18.6k dirs (file-heavy) 0.87-0.89x 0.87-0.92x

./**/node_modules/pkg/*.js against fd --full-path on 100k: 0.89x on main, 0.67x with this PR.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (32 impacted tests)
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): pass
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_*_test.py tests/e2e/query_contract_test.py tests/e2e/fanout_test.py tests/e2e/use_case_join_test.py tests/e2e/stat_speed_test.py tests/e2e/release_speed_test.py -q' (after maturin develop --release and a release CLI build); just e2e-attest-ts; cargo test --workspace --features cli
  • Result summary: Python 32 passed; TS 18 passed; Rust all passed.

Changelog: skip-changelog trailer (internal scheduling, same output).

Order pending directories by an inline rank so heap comparisons stay in
the heap's memory, and let a worker register the directories it found and
take back the earliest in one lock acquisition.

skip-changelog: internal walk scheduling, same output
@thekevinbot
thekevinbot merged commit 07ee000 into main Nov 2, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1478-perdir-walk-1020 branch November 2, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: ./**/*.js is 2.8x the fastest native (fd)

1 participant