Skip to content

perf: match .gitignore without contention between walk threads - #1523

Merged
thekevinbot merged 5 commits into
mainfrom
bot/1481-gitrepo-speed-1015
Oct 16, 2026
Merged

thekevinbot merged 5 commits into
mainfrom
bot/1481-gitrepo-speed-1015

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #1481

./** in a git repo: each walk thread now matches a .gitignore through its own compiled copy.

  • Why. Gitignore::matched borrows its match buffer from a pool. Only the pool's first caller gets it without a lock; the walk's other threads contend on the rest. In a microbenchmark of 1.6M paths, matching on 32 threads took 30.7ms through one shared Gitignore and 7.7ms through one copy per thread.
  • How. A new Frame (gitignore_frame.rs) wraps the Gitignore and keeps a lazily built copy per thread slot, compiled from the same file by the same GitignoreBuilder calls. Matching semantics stay with the ignore crate. If a copy no longer compiles, matching falls back to the shared Gitignore.

Measured on a 3.2M-item corpus built by glob_speed_gitignore_test.build_tree (1.84M visible files), hyperfine, 40 runs, host load ~17, 32 cores. Medians:

Query main this PR fd
SELECT count(*) FROM './**' (walk only) 0.156s, 2.61s user 0.152s, 2.26s user
SELECT path FROM './**' 0.295s 0.273s 0.306s

The walk uses 13% less CPU. The wall-clock gain is small on an idle 32-core host and larger under load.

This PR alone does not close #1481. Most of the gap is in the output path, which PR #1513 fixes. In glob_speed_gitignore_test:

Once #1513 merges, the test's bar can be pinned to FASTEST.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/ for each changed package
  • Migration fragment with the same slug added under packages/rust/migrations.d/ for each changed package (every subsection but Summary _None._)

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (16 passed, impacted subset)
  • TypeScript SDK e2e: pass (18 passed)
  • Rust core e2e: pass (cargo test --workspace --features cli)
  • packages/python/e2e-attestations/<branch>.json receipt written
  • packages/ts/e2e-attestations/<branch>.json receipt written
  • Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_gitignore_test.py tests/e2e/glob_speed_matchers_test.py tests/e2e/glob_speed_recursive_test.py tests/e2e/query_contract_test.py -x -q -s' (in packages/python, after maturin develop --release and cargo build --release -p dirsql --features cli), testing-conventions e2e attest 'pnpm test:e2e' (in packages/ts), cargo test --workspace --features cli
  • Result summary: all pass. SELECT path FROM './**' output is byte-identical to main (md5 over 1.84M rows).

@thekevinbot
thekevinbot merged commit 860d7ef into main Oct 16, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1481-gitrepo-speed-1015 branch October 16, 2026 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: ./** in a git repo is 1.6x the fastest native (fd)

1 participant