Repository navigation
perf: write JSON text cells from SQLite's buffer - #1511
Merged
Merged
Conversation
Failing tests for the ValueRef row path: query_each_ref and its helpers (stubbed so the tests compile), the plain-ASCII check for JSON text, and text SQLite holds as invalid UTF-8, which panics in fill_cell today.
The CLI's JSON rows read each cell as a rusqlite ValueRef and write text straight from it; ASCII text needing no escape is copied verbatim. Text that is not UTF-8 is replaced lossily rather than panicking.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1474
The CLI's JSON rows now read each cell as a rusqlite
ValueRef(newpub(crate)query_each_refonDb/DirSQL) and write text straight from SQLite's buffer, skipping the ownedValue::Textcopy. ASCII text with no control character, quote or backslash is copied verbatim (a branch-free, vectorizable check), skipping UTF-8 validation and serde's escape loop; everything else still goes throughserde_jsonon the lossy string. The per-row cellVeckeeps its allocation across rows.Side fix: text SQLite holds as invalid UTF-8 (e.g.
CAST(x'ff41' AS TEXT)) panicked infill_cellon main; it now prints with U+FFFD.Timings
400k-file fixture,
SELECT path FROM './**/*.md'(341,333 rows),--format json. Paired, interleaved hyperfine (-N -w1 -r5per round, order alternating), medians:--glob '*.md'Re-measured on the final head (
Range::containsform of the check), n=50, load 5.7: main 68.9 ms -> 65.2 ms (0.946), fd 47.6 ms; same as the pre-clippy build (0.985 head-to-head, within noise).Output is byte-identical to main: md5 of 7 queries matched (the 400k tree, plus a fixture of escape-heavy names with quotes, backslashes, control chars, U+2028, emoji and non-UTF-8 bytes). The only difference is the 2 queries where main panics on non-UTF-8 text. Variants dropped because they measured neutral or worse: a hand-rolled escape loop, and a vectorized check that kept
from_utf8. Allocating a freshVecper row instead of reusing one cost about 1.7 ms.Manual run
On main this panics with
invalid UTF-8.E2E Verification
cargo test --workspace --features cli: 2028 passed, 0 failedquery_contract,on_file_query,jsonl_parse_speed, allglob_speed_*), viatesting-conventions e2e attest: 29 passed in 1498.65s; receipt committedjust e2e-attest-ts: 18 passed; receipt committedpackages/rust/tests/json_text_e2e.rs(non-UTF-8 text, escape-heavy text)Changelog / Migrations
packages/rust/changelog.d/2026-10-13-json-text-from-sqlite.mdpackages/rust/migrations.d/2026-10-13-json-text-from-sqlite.mdquery_each_refispub(crate)); PARITY.md untouched