Skip to content

perf: shape per-table hook rows beside their insert - #1536

Merged
thekevinbot merged 5 commits into
mainfrom
bot/1534-fm-native-1016
Oct 19, 2026
Merged

thekevinbot merged 5 commits into
mainfrom
bot/1534-fm-native-1016

Conversation

@thekevinbot

Copy link
Copy Markdown
Collaborator

Refs #1534

Takes single-threaded work off the critical path of a config-table build whose per-table hook starts during the walk:

  • stream_table_rows shapes each chunk of hook rows on its own thread (db::shape_json_rows, split out of Db::shape_rows). Shaping one chunk now overlaps inserting the previous one.
  • The grouping thread also settles the early tables' file lists (split_streamed). The batches loop no longer walks those files after the inserts.
  • Prepare no longer copies each scan result (Found) into a ScannedFile. The grouping thread makes the copy (Scanned::into_files).
  • Db::get_table_columns quotes the table name. The new code reads columns before any rows arrive, which broke a table named order that matched no files.

Avoids tree_walk.rs, scanner.rs and listing.rs (#1521, #1528).

Measurements

Corpus: 131072 notes. Paired, alternating runs; medians. The machine was shared, load 18-30.

ratio
this PR vs main, release binary, A/B (31, 61, 61 pairs) 0.957, 0.969, 0.980
on top of #1521, A/B (41 pairs) 0.952
vs fd+xargs -P (fastest native), main / this PR, 31 pairs 2.34 / 2.20
frontmatter_parse_speed_test, fastest (15 pairs) 1.70

The goal (at most 1.0x the fastest native) is not met, so the test stays on the find+awk bar (it passes at 0.18). Timing a build of this PR on top of #1521 (131k files, ~125ms total):

  • walk: ~33-48ms
  • last hook invocation ends: ~77-93ms (about native wall time)
  • inserts end: ~20ms after that
  • query (GROUP BY over 131k rows): ~21-25ms

What is left needs the walk files (#1521's sys CPU cut, then more) plus a way around the ~20ms SQLite GROUP BY, which the natives do not pay.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/
  • Migration fragment with the same slug added under packages/rust/migrations.d/

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (impacted subset, 22 passed)
  • TypeScript SDK e2e: pass (18 passed)
  • Rust core e2e: pass (cargo test --workspace --features cli, 2068 passed)
  • packages/python/e2e-attestations/bot-1534-fm-native-1016.json receipt written
  • packages/ts/e2e-attestations/bot-1534-fm-native-1016.json receipt written
  • Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/frontmatter_parse_speed_test.py tests/e2e/jsonl_parse_speed_test.py tests/e2e/on_file_query_test.py tests/e2e/fanout_test.py tests/e2e/declared_table_name_test.py tests/e2e/query_contract_test.py tests/e2e/use_case_notebook_test.py tests/e2e/use_case_transcripts_test.py tests/e2e/sigint_interrupt_test.py -q -s'; just e2e-attest-ts; cargo test --workspace --features cli
  • Result summary: all pass. Ratios against the fastest native: frontmatter 1.70, jsonl 1.36, notebook 0.90, transcripts 0.62.

@thekevinbot
thekevinbot merged commit 3bba132 into main Oct 19, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1534-fm-native-1016 branch October 19, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant