Skip to content

perf: build path-table rows on the walk threads - #1510

Merged
thekevinbot merged 4 commits into
mainfrom
bot/speed-walk-rows-1013
Oct 14, 2026
Merged

thekevinbot merged 4 commits into
mainfrom
bot/speed-walk-rows-1013

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #1474

Path-table rows are now built on the walk worker threads, so building them overlaps the walk instead of running serially afterwards. Each row is also smaller: 80 bytes instead of 104. The path is a Box<str>, the scanned path a Box<Path>, and the basename/dir/ext spans are niche-packed NonZeroU32. Output is byte-identical (same md5 over all 341k rows).

Timings

SELECT path FROM './**/*.md' on the 400k-file glob_speed_recursive tree (341k matches). Runs are paired and interleaved, 40 each, medians shown. Other agents were running, load 3 to 8.5.

binary median RSS
main (462b3c6) 72.1 ms 107 MB
this PR 61.6 ms 77 MB
fd --no-ignore -t f -g '*.md' 40.7 ms 20 MB

That takes the gap from 1.77x fd to 1.51x. Phase breakdown (stderr stamps):

phase main this PR
walk ~17 ms ~21 ms
row build ~13 ms 0 ms (inside the walk)
SQL loop + JSON ~28 ms ~27 ms
exit ~6.5 ms ~4.4 ms

I measured both parts separately. Rows-in-walk alone gave 71.5 to 60.8 ms (hyperfine). The row shrink gave a further 69 to 63 ms against rows-in-walk (hyperfine), and fewer page faults (22.9k to 19.4k).

Remaining gap (not in this PR):

  • SQL loop (~27 ms). Profiling shows fill_cell at 28% self time (from_utf8_lossy plus a copy into Value::Text), sqlite3Step at 20%, value_ref at 15% and serde string escaping at 13%. Writing JSON straight from ValueRef text would skip the copy and the validation.
  • Exit (~4-5 ms). This scales with RSS.

E2E Verification

  • Python, targeted at the changed path table and walk: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_recursive_test.py tests/e2e/glob_speed_one_level_test.py tests/e2e/glob_speed_depth_bound_test.py tests/e2e/glob_speed_gitignore_test.py tests/e2e/glob_speed_dotfiles_test.py tests/e2e/glob_speed_matchers_test.py tests/e2e/glob_speed_node_modules_test.py tests/e2e/glob_speed_symlink_test.py tests/e2e/stat_speed_test.py tests/e2e/query_contract_test.py -q'. Result: 26 passed. The module was built with uv run maturin develop --release and cargo build --release -p dirsql --features cli.
  • TypeScript: just e2e-attest-ts. Result: 18 passed.
  • cargo test --workspace --features cli: pass.
  • cargo bench -p dirsql: ran clean.

Manual run: dirsql query with path, basename, dir, ext, size and content, over root-relative, ../ and absolute globs. Every column reads correctly, and the full ./**/*.md output matches main byte for byte.

Changelog / Migrations

  • packages/rust/changelog.d/2026-10-13-path-rows-built-on-walk-threads.md
  • packages/rust/migrations.d/2026-10-13-path-rows-built-on-walk-threads.md: not breaking, all subsections _None._

@thekevinbot
thekevinbot merged commit 6807a17 into main Oct 14, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/speed-walk-rows-1013 branch October 14, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant