Repository navigation
Speed: ./**/*.md at or under fd - #1513
Merged
Merged
Conversation
Rows reach the cursor in chunks of 4096 while the walk is still running, so the row loop overlaps the walk instead of following it. Path, basename and ext are handed to SQLite as static NUL-terminated text, saving a copy per cell.
A walker on the sibling hyperthread of the thread consuming the streamed rows slowed that consumer more than it sped the walk; walk-bound queries are unchanged at one walker per physical core.
Capping the walk at physical cores halved the workers on a 4-vCPU runner and made the walk 1.8x slower than the previous release. With streaming rows the cap is not needed to beat fd.
8 tasks done
This was referenced Oct 15, 2026
thekevinbot
marked this pull request as ready for review
October 15, 2026 15:41
This was referenced Oct 15, 2026
This was referenced Oct 16, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1474
SELECT path FROM './**/*.md'now runs at or under fd.row_stream.rs). Before, the whole walk finished before the row loop began.SQLITE_STATICtext. That saves one copy per cell.Measured with paired A/B on a 1.36M-file tree (median of 11 pairs, host load ~9):
On 4 CPUs (
taskset, 2 cores with SMT), this PR runs at 0.80x the base build.An earlier commit capped the walk at physical cores. It is reverted here: on a 4-vCPU runner the cap halved the walk workers, and the walk ran 1.8x slower than the previous release. Locally, on 4 CPUs, 2 workers took 1.94x the base build's time and 4 workers took 0.85x. With streaming rows, the cap is not needed to beat fd.
Two other levers were measured and dropped because they did not pay off:
get_refFFI calls: ~12ms, and not reachable through rusqlite.SELECT pathoutput matches fd byte for byte (md5 over the sorted paths).set_terminated_textgets a line-scoped mutation exemption intesting-conventions.toml. It writes through a rawsqlite3_context, and only a realConnectionsupplies one.tests/vtab.rs(stat_columns_carry_real_values) kills both of its mutants.Changelog / Migrations
packages/rust/changelog.d/for each changed packagepackages/rust/migrations.d/for each changed package (every subsection but Summary_None._)E2E Verification
watch_speed_test)cargo test --workspace --features cli)packages/python/e2e-attestations/<branch>.jsonreceipt writtenpackages/ts/e2e-attestations/<branch>.jsonreceipt written (just e2e-attest-ts)testing-conventions e2e attest 'uv run python -m pytest tests/e2e/ --ignore=tests/e2e/watch_speed_test.py -q'(inpackages/python, aftermaturin develop --releaseandcargo build --release -p dirsql --features cli),just e2e-attest-ts,cargo test --workspace --features cliwatch_speed_testwas left out on purpose. It measures write-to-event latency on a one-file tree throughdirsql server, a path this change does not touch, and it is currently red onmainfor an unrelated reason.