Skip to content

perf: wide directories make leaves on the judging threads and read entries without a copy - #1515

Merged
thekevinbot merged 5 commits into
mainfrom
bot/1461-wide-read-1015
Oct 16, 2026
Merged

thekevinbot merged 5 commits into
mainfrom
bot/1461-wide-read-1015

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1461

Two changes for wide directories (hundreds of thousands of entries):

  • Leaves are made on the judging threads. explore used to build each file's leaf, which is its path-table row, serially after judge_all returned. It now builds it inside the judging closure, so in a directory of at least PARALLEL_ENTRIES entries row-building runs on every core. Taken is generic over what a file carries: a path, or the leaf made of it. PR perf: build path-table rows on the walk threads #1510 moved row-building into explore, but serially per directory, so this part was still open.
  • Linux reads entries with rustix::fs::RawDir. It reads into a thread-local 1 MiB buffer, so a 2M-entry directory takes few getdents calls and needs no CString per name. Other Unix systems keep Dir, and Windows keeps std::fs::read_dir. Both are unchanged.

Output is byte-identical: same md5 for path, basename, dir over ./*, ./ and ./docs on a 1.6M-item tree.

Timings

These ran on the glob_speed_one_level tree (1.6M items per level). Other agents were running at the same time, load 5 to 18.

Direct A/B, 11 interleaved runs each, median after/before:

glob after/before
./* 0.882
./ 0.887
./docs 0.840

On the 2M flat corpus, ./* went from 0.923s to 0.834s (0.904x). A deep tree (20k dirs × 10 files, ./**/*.md) was neutral at 1.02x, within noise.

The paired harness ran the .//./docs cases from #1471, untracked and not committed. Before and after ran on the same tree, two rounds each, and ratios are against the fastest native. Load made these noisy:

glob before after
./* 1.03 (fd), 1.28 (rg) 0.99 (fd), 0.82 (rg)
./docs 1.48, 1.65 (rg) 1.30 (rg), 1.10 (fd)
./ 0.61, 0.83 (fd) 0.66, 1.05 (rg)

This PR does not restore the #1471 cases. Getting ./ and ./docs reliably to 1.0x needs the wide-directory redesign: judging, sorting and moves hidden behind the read.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (targeted)
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): pass
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: uv run maturin develop --release, cargo build --release -p dirsql --features cli, testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_one_level_test.py tests/e2e/glob_speed_recursive_test.py tests/e2e/glob_speed_depth_bound_test.py tests/e2e/glob_speed_gitignore_test.py tests/e2e/glob_speed_dotfiles_test.py tests/e2e/glob_speed_matchers_test.py tests/e2e/glob_speed_node_modules_test.py tests/e2e/glob_speed_symlink_test.py tests/e2e/stat_speed_test.py tests/e2e/query_contract_test.py -q', just e2e-attest-ts, cargo test --workspace --features cli
  • Result summary: python 26 passed; ts passed; cargo workspace all passed.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/2026-10-15-wide-directory-listing.md
  • Migration fragment with the same slug added under packages/rust/migrations.d/, every subsection but Summary _None._

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: one-level './*' listing is ~1.1x native

1 participant