Skip to content

perf: let SQLite's sorter use worker threads - #1500

Merged
thekevinbot merged 4 commits into
mainfrom
bot/speed-parse-pragmas-1011
Oct 13, 2026
Merged

thekevinbot merged 4 commits into
mainfrom
bot/speed-parse-pragmas-1011

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #1484

Sets PRAGMA threads = 8 on the ephemeral database so SQLite's sorter (GROUP BY / ORDER BY over a large table) runs on background threads.

Measurements (load 11-16, interleaved runs, 1M-row jsonl case)

config min median peak RSS
base 0.642s 0.711s ~20MB
threads=8 0.579s 0.645s ~28MB
cache_size=64MB 0.604s 0.661s ~58MB
threads=8 + cache_size=64MB 0.601s 0.674s ~57MB

Decisions:

  • Kept threads=8 only: about -9%, small RSS cost.
  • Dropped cache_size and temp_store=MEMORY: no win beyond threads, 2-4x the RSS.
  • Frontmatter (65k rows, tiny sort) and a plain path-table query: no change within noise (path query 21.1ms vs 21.6ms median).
  • The on-file table is not an inserted table dirsql shapes for the user's GROUP BY, so no insert-order or layout lever was found; SQLite has no hash aggregate to ask for.
  • Speed: jsonl parse is 1.4x the fastest native (jq per file under xargs -P) #1484 stays at about 1.34x the fastest native (jq+xargs-P) in the harness, so its bar is not flipped to FASTEST here.

Changelog / Migrations

  • Changelog fragment added under packages/rust/changelog.d/
  • Migration fragment with the same slug added under packages/rust/migrations.d/

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass
  • TypeScript SDK e2e: pass
  • Rust core e2e: covered by the two above
  • packages/python/e2e-attestations/bot-speed-parse-pragmas-1011.json receipt written
  • packages/ts/e2e-attestations/bot-speed-parse-pragmas-1011.json receipt written
  • Command(s) run: pytest tests/e2e/{jsonl_parse_speed,frontmatter_parse_speed,on_file_query,query_contract}_test.py -x -q (14 passed); pnpm test:e2e (18 passed)
  • Result summary: pass

Red run: 38187945310 (Unit tests failed on new_lets_the_sorter_use_worker_threads, the pragma reading its default).

@thekevinbot
thekevinbot merged commit aa0f3aa into main Oct 13, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/speed-parse-pragmas-1011 branch October 13, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant