Skip to content

perf: write JSON text cells from SQLite's buffer - #1511

Merged
thekevinbot merged 5 commits into
mainfrom
bot/speed-sql-valueref-1013
Oct 14, 2026
Merged

thekevinbot merged 5 commits into
mainfrom
bot/speed-sql-valueref-1013

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #1474

The CLI's JSON rows now read each cell as a rusqlite ValueRef (new pub(crate) query_each_ref on Db/DirSQL) and write text straight from SQLite's buffer, skipping the owned Value::Text copy. ASCII text with no control character, quote or backslash is copied verbatim (a branch-free, vectorizable check), skipping UTF-8 validation and serde's escape loop; everything else still goes through serde_json on the lossy string. The per-row cell Vec keeps its allocation across rows.

Side fix: text SQLite holds as invalid UTF-8 (e.g. CAST(x'ff41' AS TEXT)) panicked in fill_cell on main; it now prints with U+FFFD.

Timings

400k-file fixture, SELECT path FROM './**/*.md' (341,333 rows), --format json. Paired, interleaved hyperfine (-N -w1 -r5 per round, order alternating), medians:

base before after after/before fd --glob '*.md' load (32 cores)
main 67.7 ms 64.3 ms 0.949 47.7 ms (1.42x -> 1.35x fd) 5.7
main + #1510 53.2 ms 50.8 ms 0.956 47.1 ms (1.13x -> 1.08x fd) 9.2

Re-measured on the final head (Range::contains form of the check), n=50, load 5.7: main 68.9 ms -> 65.2 ms (0.946), fd 47.6 ms; same as the pre-clippy build (0.985 head-to-head, within noise).

Output is byte-identical to main: md5 of 7 queries matched (the 400k tree, plus a fixture of escape-heavy names with quotes, backslashes, control chars, U+2028, emoji and non-UTF-8 bytes). The only difference is the 2 queries where main panics on non-UTF-8 text. Variants dropped because they measured neutral or worse: a hand-rolled escape loop, and a vectorized check that kept from_utf8. Allocating a fresh Vec per row instead of reusing one cost about 1.7 ms.

Manual run

$ dirsql query "SELECT CAST(x'ff41' AS TEXT) AS t" --format json
[{"t":"�A"}]

On main this panics with invalid UTF-8.

E2E Verification

  • cargo test --workspace --features cli: 2028 passed, 0 failed
  • Python e2e (impacted: query_contract, on_file_query, jsonl_parse_speed, all glob_speed_*), via testing-conventions e2e attest: 29 passed in 1498.65s; receipt committed
  • TypeScript: just e2e-attest-ts: 18 passed; receipt committed
  • New Rust CLI e2e packages/rust/tests/json_text_e2e.rs (non-UTF-8 text, escape-heavy text)

Changelog / Migrations

  • packages/rust/changelog.d/2026-10-13-json-text-from-sqlite.md
  • packages/rust/migrations.d/2026-10-13-json-text-from-sqlite.md
  • No public SDK API change (query_each_ref is pub(crate)); PARITY.md untouched

Failing tests for the ValueRef row path: query_each_ref and its helpers
(stubbed so the tests compile), the plain-ASCII check for JSON text, and
text SQLite holds as invalid UTF-8, which panics in fill_cell today.
The CLI's JSON rows read each cell as a rusqlite ValueRef and write text
straight from it; ASCII text needing no escape is copied verbatim. Text
that is not UTF-8 is replaced lossily rather than panicking.
@thekevinbot
thekevinbot merged commit 44139e1 into main Oct 14, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/speed-sql-valueref-1013 branch October 14, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant