Skip to content

Read text that is not UTF-8 with replacement characters, not a panic - #1612

Merged
thekevinbot merged 6 commits into
mainfrom
bot/1593-non-utf8-text-1104
Nov 4, 2026
Merged

thekevinbot merged 6 commits into
mainfrom
bot/1593-non-utf8-text-1104

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1593

A TEXT cell SQLite holds as invalid UTF-8 (CAST(x'ff' AS TEXT), char(55296)) panicked inside rusqlite on the collected-row path (--format table, every SDK query). In the SDK the panic poisoned the shared-state lock, so every later query on that instance failed. --format json took a different path and printed replacement characters.

Decision

The maintainer decided: such text fails the query with a clear error. It is never read lossily as U+FFFD, and it never panics or poisons the lock; the SDK instance stays usable after the error. The query now fails with column "<name>" holds text that is not valid UTF-8 (new DbError::NotUtf8) in every CLI format and SDK. This reverses the lossy --format json behavior from 2026-10-13-json-text-from-sqlite, so its e2e assertion is replaced. The migration fragment covers it, and docs/reference/cli.md documents the error and the hex() / CAST(… AS BLOB) workaround.

The error names the column. A computed value has no source file, so it cannot name one.

Red

  • First red commit (asserted lossy output, superseded by the decision): 91e2175f
  • Red commit for the error contract, 19d27f7a: Binding tests (3.10) fails text_not_utf8_test.py::it_raises_an_exception_naming_the_column with pyo3_runtime.PanicException: invalid UTF-8. The Rust CLI e2e tests (text_not_utf8_e2e.rs) are local-only: before the fix they failed on the panic.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): pass
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: cargo test -p dirsql --features cli (full); cd packages/python && testing-conventions e2e attest 'uv run python -m pytest tests/e2e/query_contract_test.py tests/e2e/on_file_query_test.py tests/e2e/cli_version_test.py -x -q'; cd packages/ts && testing-conventions e2e attest 'pnpm exec vitest run --dir tests/e2e query-contract query-params'
  • Result summary: all passed. Checked by hand with the release binary: table and json both exit 1 with dirsql query: column "v" holds text that is not valid UTF-8. The issue's Python repro now raises RuntimeError (an Exception), and the next query on the same instance returns rows.

Changelog / Migrations

  • Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (or: skip-changelog trailer on a commit with reason)
  • Migration fragment with the same slug added under <root>/<pkg>/migrations.d/ for each changed package (every subsection but Summary _None._ when nothing breaks)

… panic

Failing tests for the collected-row path (--format table, the SDK query): text SQLite holds as invalid UTF-8 panics inside rusqlite and poisons the SDK lock today.
Per the maintainer's decision: an error, never replacement characters, never a panic or a poisoned SDK lock. Replaces the lossy assertions in the JSON path's tests.
rusqlite panics converting such text to an owned value; in the SDK the panic poisoned the shared-state lock. Check each text cell on the collected and streamed row paths and return DbError::NotUtf8, so every format and SDK fails the same way and the instance stays usable.
@thekevinbot
thekevinbot merged commit 5bbd47f into main Nov 4, 2026
90 checks passed
@thekevinbot
thekevinbot deleted the bot/1593-non-utf8-text-1104 branch November 4, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Return an error, not a panic, for a TEXT value that is not valid UTF-8

1 participant