Repository navigation
Read text that is not UTF-8 with replacement characters, not a panic - #1612
Merged
Merged
Conversation
… panic Failing tests for the collected-row path (--format table, the SDK query): text SQLite holds as invalid UTF-8 panics inside rusqlite and poisons the SDK lock today.
Per the maintainer's decision: an error, never replacement characters, never a panic or a poisoned SDK lock. Replaces the lossy assertions in the JSON path's tests.
rusqlite panics converting such text to an owned value; in the SDK the panic poisoned the shared-state lock. Check each text cell on the collected and streamed row paths and return DbError::NotUtf8, so every format and SDK fails the same way and the instance stays usable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1593
A TEXT cell SQLite holds as invalid UTF-8 (
CAST(x'ff' AS TEXT),char(55296)) panicked inside rusqlite on the collected-row path (--format table, every SDKquery). In the SDK the panic poisoned the shared-state lock, so every later query on that instance failed.--format jsontook a different path and printed replacement characters.Decision
The maintainer decided: such text fails the query with a clear error. It is never read lossily as U+FFFD, and it never panics or poisons the lock; the SDK instance stays usable after the error. The query now fails with
column "<name>" holds text that is not valid UTF-8(newDbError::NotUtf8) in every CLI format and SDK. This reverses the lossy--format jsonbehavior from2026-10-13-json-text-from-sqlite, so its e2e assertion is replaced. The migration fragment covers it, anddocs/reference/cli.mddocuments the error and thehex()/CAST(… AS BLOB)workaround.The error names the column. A computed value has no source file, so it cannot name one.
Red
91e2175f19d27f7a: Binding tests (3.10) failstext_not_utf8_test.py::it_raises_an_exception_naming_the_columnwithpyo3_runtime.PanicException: invalid UTF-8. The Rust CLI e2e tests (text_not_utf8_e2e.rs) are local-only: before the fix they failed on the panic.E2E Verification
packages/python/e2e-attestations/<branch>.jsonreceipt written ifpackages/pythonchanged (just e2e-attest-python)packages/ts/e2e-attestations/<branch>.jsonreceipt written ifpackages/tschanged (just e2e-attest-ts)cargo test -p dirsql --features cli(full);cd packages/python && testing-conventions e2e attest 'uv run python -m pytest tests/e2e/query_contract_test.py tests/e2e/on_file_query_test.py tests/e2e/cli_version_test.py -x -q';cd packages/ts && testing-conventions e2e attest 'pnpm exec vitest run --dir tests/e2e query-contract query-params'dirsql query: column "v" holds text that is not valid UTF-8. The issue's Python repro now raisesRuntimeError(anException), and the next query on the same instance returns rows.Changelog / Migrations
<root>/<pkg>/changelog.d/for each changed package (or:skip-changelogtrailer on a commit with reason)<root>/<pkg>/migrations.d/for each changed package (every subsection but Summary_None._when nothing breaks)