Skip to content

perf: rows from one on-file block share their key strings - #1520

Merged
thekevinbot merged 4 commits into
mainfrom
bot/1484-jsonl-speed-1015
Oct 16, 2026
Merged

thekevinbot merged 4 commits into
mainfrom
bot/1484-jsonl-speed-1015

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1484

Rows parsed from one block of on-file output now share one Arc<str> per key instead of each row owning its own key Strings. Before this change, the thread that inserts rows freed two small strings per row. Those strings were allocated on the parse threads, and the frees cost about 120ns per row. On the jsonl case that made the insert thread fall behind jq.

Profile (4 files, 4M events; the native jq+xargs-P takes about 2.0s):

phase before after
last jq exits ~2.0s ~2.0s
shape rows (insert thread) 0.78-1.0s 0.19-0.23s
insert (insert thread) 0.9s 0.9s
inserting finishes after the last jq exits +0.15-0.25s +0.00s
query (GROUP BY scan + sort) 0.4-0.5s 0.4-0.5s

Paired wall-clock medians against jq+xargs-P on the same machine (load 8-37 from other agents):

  • 4M events: 1.38x before, 1.26x after.
  • 1M events (the e2e case's size): 1.38x before, 1.31x after. The e2e test (15 pairs, net of startup) reports 1.325x.

What remains is the query itself. At both sizes, inserting now ends when the last jq exits, and SQLite's GROUP BY scan and sort (about 0.1s at 1M rows) is the whole gap. The cache_size, temp_store, threads and mmap_size pragmas did not move it.

infer::JsonRow is public, so Rust callers that build one by hand must change their code (see the migration fragment).

Red commit first: parse_rows_shares_one_allocation_per_key_across_a_blocks_rows failed in CI on unequal key pointers (run 38751432473).

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (impacted subset)
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): N/A (unit suite cargo test -p dirsql --features cli --lib pass)
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/jsonl_parse_speed_test.py tests/e2e/frontmatter_parse_speed_test.py tests/e2e/on_file_query_test.py tests/e2e/fanout_test.py tests/e2e/declared_table_name_test.py tests/e2e/query_contract_test.py tests/e2e/use_case_transcripts_test.py -x -q -s'; just e2e-attest-ts
  • Result summary: 20 passed (python), 18 passed (ts).

Changelog / Migrations

  • Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (or: skip-changelog trailer on a commit with reason)
  • Migration fragment with the same slug added under <root>/<pkg>/migrations.d/ for each changed package (every subsection but Summary _None._ when nothing breaks)

@thekevinbot
thekevinbot merged commit a2269ba into main Oct 16, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1484-jsonl-speed-1015 branch October 16, 2026 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: jsonl parse is 1.4x the fastest native (jq per file under xargs -P)

1 participant