Skip to content

perf: scan file entries in one pass - #1556

Merged
thekevinbot merged 7 commits into
mainfrom
bot/1534-fm-native-1102
Nov 2, 2026
Merged

thekevinbot merged 7 commits into
mainfrom
bot/1534-fm-native-1102

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1534

The scanner handles a file-only directory's entries in one pass, with no intermediate list. Frontmatter is about 2-3% faster: 100 alternating pairs of release CLI builds on the same 131,072-note corpus gave a median ratio of 0.971 for this branch against main, and 1.023 for main against this branch.

This is not enough for the fastest-native bar. With the bar set to FASTEST, CI's runner measured 1.132x find+xargs-P before the fix and 1.156x after it, against a 1.1x limit. That difference is noise. Locally the ratio is 1.89x. This PR therefore leaves the frontmatter bar at find+awk and leaves the native assertion out of speed-sanity-ci.yml, which only times against the previous release. The remaining lever, creating the DB and inserting rows during the scan, is L-sized and is tracked in #1560. #1560 also covers switching the bar to FASTEST.

A borrowed-table-name variant of the match step was measured and dropped: 1.064x slower over 100 pairs.

No API change, so there is no parity drift.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (on_file_query_test.py, query_contract_test.py, frontmatter_parse_speed_test.py)
  • TypeScript SDK e2e: pass (18/18)
  • Rust core e2e (if applicable): pass (cargo test --workspace --features cli)
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: uv run python -m pytest tests/e2e/on_file_query_test.py tests/e2e/query_contract_test.py tests/e2e/frontmatter_parse_speed_test.py -x -q; pnpm test:e2e; cargo test --workspace --features cli; cargo bench -p dirsql
  • Result summary: all pass. Frontmatter is 0.180x find+awk (bar) and 1.887x find+xargs-P (fastest, printed only), load 16.7.

Manual CLI: on 131,072 notes the query returned four status rows of 32,768 notes each, with the expected word sums.

Changelog / Migrations

  • Rust changelog fragment under packages/rust/changelog.d/
  • Rust migration fragment under packages/rust/migrations.d/

@thekevinbot thekevinbot changed the title Speed up frontmatter against fastest native perf: scan file entries in one pass Nov 2, 2026
@thekevinbot
thekevinbot merged commit aa71bc0 into main Nov 2, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1534-fm-native-1102 branch November 2, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: frontmatter is 1.6x the fastest native (fd + xargs -P)

1 participant