[thekevinbot]

Refs #1477

./*/*.md over the depth-bound tree: 32 directories, each tens of thousands of entries wide. Two kinds of contention were inflating the judge phase (matching each entry against the glob):

1. Shared regex scratch. regex::Regex keeps its match scratch in a pool shared by every thread. With 32 walk workers matching at once, threads other than the owner fight over sharded mutexes, and a thread that loses builds fresh scratch for a single match. PerThread<T> (per_thread.rs) gives each thread its own clone, and matcher::Pattern matches through it.
2. Oversubscription. judge_all splits a directory of 65536+ entries across every core, and it runs inside each of the walk's 32 workers, for up to about 1024 threads. The walk now counts its busy workers (tree_walk::busy_workers, which leaves out a worker waiting for a directory). A wide directory splits only across idle_cores(cores, busy), never fewer than one.

Output is unchanged: byte-identical SELECT path FROM './*/*.md' against main, md5 checked on the 3.2M-item tree.

Measurements

These are paired medians, net of startup, as a ratio to fdfind. The host ran at load 15-37 from concurrent agents, so expect a few percent of noise. The issue's 4.2x was not reproduced; main measures ~1.6x here.

case	main	this PR
depth-bound ./*/*.md (2.7M rows)	1.57-1.71	1.135-1.17
recursive ./**/*.md, same tree (PerThread only)	1.577	1.354
one-level ./* (single wide root)	1.089	0.983
mixed (one 1M-file dir + 12 small dirs, ./*/*.md)	1.262	1.111

Never splitting judge_all measures 1.07-1.12 on the depth-bound tree, but regresses the mixed tree to 1.505. Splitting only across idle cores handles both shapes.

Remaining levers (to reach <=1.0x)

• Serial output tail after the walk: ~0.12-0.15s for 2.7M rows. Speed: ./**/*.md at or under fd #1513 (row streaming, SQLITE_STATIC) covers this.
• Streaming rows from wide directories early: a few wide directories all finish together, so rows can't start before the walk ends. One option is to split a wide listing into chunked continuation steps, judged in order. This only pays after Speed: ./**/*.md at or under fd #1513, and it overlaps Speed: one-level './*' listing is ~1.1x native #1461's wide-directory read.
• Memory: ~300K page faults, RSS 650-760MB.

Expected conflicts: #1480 rewrites tree_walk::work(), and #1513 changes the signature of scan_glob_rows.

E2E Verification

• Ran e2e suites locally for every affected SDK
• Python SDK e2e: pass
• TypeScript SDK e2e: pass
• Rust core e2e (if applicable): pass
• packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
• packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
• Command(s) run:
  • cargo test --workspace --features cli
  • cd packages/python && uv run maturin develop --release
  • testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_{depth_bound,one_level,recursive,matchers,dotfiles,gitignore,node_modules,symlink}_test.py tests/e2e/query_contract_test.py tests/e2e/fanout_test.py tests/e2e/use_case_join_test.py -q'
  • cd packages/ts && testing-conventions e2e attest 'pnpm test:e2e'
• Result summary:
  • Rust: all pass.
  • Python e2e: 28 passed.
  • TS e2e: 18 passed.

Changelog / Migrations

• Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (or: skip-changelog trailer on a commit with reason)
• Migration fragment with the same slug added under <root>/<pkg>/migrations.d/ for each changed package (every subsection but Summary _None._ when nothing breaks)