Skip to content

Speed: ./**/*.md at or under fd - #1513

Merged
thekevinbot merged 9 commits into
mainfrom
bot/1474-vtab-speed-1014
Oct 16, 2026
Merged

thekevinbot merged 9 commits into
mainfrom
bot/1474-vtab-speed-1014

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1474

SELECT path FROM './**/*.md' now runs at or under fd.

  • Streaming rows. The scan hands rows to the cursor in chunks of 4096 while the walk is still running (row_stream.rs). Before, the whole walk finished before the row loop began.
  • Static text. Path, basename and ext are stored NUL-terminated and handed to SQLite as SQLITE_STATIC text. That saves one copy per cell.

Measured with paired A/B on a 1.36M-file tree (median of 11 pairs, host load ~9):

Build Ratio vs fd
Base 1.27
This PR 0.90

On 4 CPUs (taskset, 2 cores with SMT), this PR runs at 0.80x the base build.

An earlier commit capped the walk at physical cores. It is reverted here: on a 4-vCPU runner the cap halved the walk workers, and the walk ran 1.8x slower than the previous release. Locally, on 4 CPUs, 2 workers took 1.94x the base build's time and 4 workers took 0.85x. With streaming rows, the cap is not needed to beat fd.

Two other levers were measured and dropped because they did not pay off:

  • Fewer get_ref FFI calls: ~12ms, and not reachable through rusqlite.
  • Pre-sizing the output buffer: no gain.

SELECT path output matches fd byte for byte (md5 over the sorted paths).

set_terminated_text gets a line-scoped mutation exemption in testing-conventions.toml. It writes through a raw sqlite3_context, and only a real Connection supplies one. tests/vtab.rs (stat_columns_carry_real_values) kills both of its mutants.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/ for each changed package
  • Migration fragment with the same slug added under packages/rust/migrations.d/ for each changed package (every subsection but Summary _None._)

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (61 passed, every e2e test except watch_speed_test)
  • TypeScript SDK e2e: pass (18 passed)
  • Rust core e2e: pass (cargo test --workspace --features cli)
  • packages/python/e2e-attestations/<branch>.json receipt written
  • packages/ts/e2e-attestations/<branch>.json receipt written (just e2e-attest-ts)
  • Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/ --ignore=tests/e2e/watch_speed_test.py -q' (in packages/python, after maturin develop --release and cargo build --release -p dirsql --features cli), just e2e-attest-ts, cargo test --workspace --features cli
  • Result summary: watch_speed_test was left out on purpose. It measures write-to-event latency on a one-file tree through dirsql server, a path this change does not touch, and it is currently red on main for an unrelated reason.

Rows reach the cursor in chunks of 4096 while the walk is still running,
so the row loop overlaps the walk instead of following it. Path, basename
and ext are handed to SQLite as static NUL-terminated text, saving a copy
per cell.
A walker on the sibling hyperthread of the thread consuming the streamed
rows slowed that consumer more than it sped the walk; walk-bound queries
are unchanged at one walker per physical core.
Capping the walk at physical cores halved the workers on a 4-vCPU runner and made the walk 1.8x slower than the previous release. With streaming rows the cap is not needed to beat fd.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: ./**/*.md is 2.1x the fastest native (fd)

1 participant