Skip to content

Speed: ./a/b/**/*.md closer to fd - #1531

Open
thekevinbot wants to merge 5 commits into
mainfrom
bot/1475-prefix-speed-1016
Open

thekevinbot wants to merge 5 commits into
mainfrom
bot/1475-prefix-speed-1016

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #1475

SELECT path FROM './a/b/**/*.md' runs faster. It is not yet held to fd, so this does not close the issue.

  • Current-thread runtime for one-shot commands. dirsql query (and bare dirsql "<sql>", init, context) builds a current-thread tokio runtime. Only server and the bare REPL keep the multi-thread runtime. Before, a one-shot query started and stopped one tokio worker per core.
  • Tree spawn. Each walk worker starts two more before it explores, so the walking thread starts one thread instead of all of them. The walk still explores on every core.

Rebased onto main, whose walk no longer tracks busy workers; the earlier half-the-cores worker count is dropped.

Measured with hyperfine (30 runs, 800k-file tree, 32 logical cores), main vs this PR:

Query main This PR
count(*) over ./a/b/**/*.md 11.2 ms 9.9 ms
count(*) over ./**/*.md 33.1 ms 29.8 ms

The ./a/b/**/*.md e2e case flakes at its fastest-native bar on main too (main 2/3, this PR 3/3 locally); tracked in #1570.

Perf-only change, no red test. New unit tests cover serves and the worker numbering.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/ for each changed package
  • Migration fragment with the same slug added under packages/rust/migrations.d/ for each changed package (every subsection but Summary _None._)

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (37 passed, impacted subset)
  • TypeScript SDK e2e: pass (18 passed)
  • Rust core e2e: pass (cargo test --workspace --features cli)
  • packages/python/e2e-attestations/<branch>.json receipt written
  • packages/ts/e2e-attestations/<branch>.json receipt written
  • Command(s) run: in packages/python, after maturin develop --release and cargo build --release -p dirsql --features cli: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_*_test.py tests/e2e/stat_speed_test.py tests/e2e/frontmatter_parse_speed_test.py tests/e2e/jsonl_parse_speed_test.py tests/e2e/query_contract_test.py tests/e2e/fanout_test.py tests/e2e/repl_interactive_test.py tests/e2e/server_speed_test.py -q'; in packages/ts: testing-conventions e2e attest 'pnpm test:e2e'; cargo test --workspace --features cli
  • Result summary: the walk speed cases, one-shot query path, REPL and server runtime choice all pass.

@thekevinbot
thekevinbot force-pushed the bot/1475-prefix-speed-1016 branch from 9c539e8 to 6e7d75d Compare November 3, 2026 13:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant