Skip to content

perf: config build does less serial per-file work - #1519

Merged
thekevinbot merged 4 commits into
mainfrom
bot/1483-fm-speed-1015
Oct 16, 2026
Merged

thekevinbot merged 4 commits into
mainfrom
bot/1483-fm-speed-1015

Conversation

@thekevinbot

Copy link
Copy Markdown
Collaborator

Fixes #1483

Cuts the serial bookkeeping in the config-table build. Before this change, dirsql's CPU time on the frontmatter case was already about the same as find + xargs -P. The gap came from work done one file at a time on a single thread.

  • The scanner returns Found { path, rel_path, table_name }. Each walk worker builds the relative path for the files it matches (found_in), so the calling thread no longer recomputes it for every file. relative_path is removed, and reconcile_scan no longer needs the anchors.
  • finish_build_with_fs groups the scanned files on a scoped thread (group_scanned) while the main thread inserts rows from the per-table hooks that started early (stream_early). Before, grouping ran first and the inserts waited behind it. Grouping also checks the last batch before searching the list.

Not on the vtab/output path that #1513 changes.

Measurements

Corpus: 131072 notes. Paired, alternating runs; median ratio vs find+xargs-P. The machine was shared with other agents, load 11-27.

before after
ab.py, 21 pairs, wall time 204ms, ratio 2.10 169ms, ratio 1.81
frontmatter_parse_speed_test (startup subtracted), ratio 2.4-3.2 (issue) 1.91

The test's bar native (find+awk) is unchanged, and the test still passes against it (ratio 0.185). It is not switched to bar=FASTEST, because that would leave the test red at 1.9x.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/
  • Migration fragment with the same slug added under packages/rust/migrations.d/

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (impacted subset, 22 passed)
  • TypeScript SDK e2e: pass (18 passed)
  • Rust core e2e (if applicable): pass (cargo test --workspace --features cli)
  • packages/python/e2e-attestations/bot-1483-fm-speed-1015.json receipt written
  • packages/ts/e2e-attestations/bot-1483-fm-speed-1015.json receipt written
  • Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/frontmatter_parse_speed_test.py tests/e2e/jsonl_parse_speed_test.py tests/e2e/on_file_query_test.py tests/e2e/fanout_test.py tests/e2e/declared_table_name_test.py tests/e2e/query_contract_test.py tests/e2e/use_case_notebook_test.py tests/e2e/use_case_transcripts_test.py tests/e2e/sigint_interrupt_test.py -x -q -s'; just e2e-attest-ts
  • Result summary: all pass. Speed ratios against the fastest native: frontmatter 1.91, jsonl 1.31, notebook 0.91, transcripts 0.63.

…ouping

The config build's startup scan now carries each match's relative path from the walk worker that found it, instead of recomputing it serially, and streams early per-table hook rows into SQLite while the scanned files are grouped on a second thread.
@thekevinbot
thekevinbot merged commit db458b1 into main Oct 16, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1483-fm-speed-1015 branch October 16, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: frontmatter parse is 3.2x the fastest native (find + xargs -P)

1 participant