Skip to content

Commit 0fcd385

Browse files
committed
sqlitedb: recover a poisoned handle inside a write operation by replaying it
Closes the last deviation recorded for DID-matlab da83fa3. MATLAB reopens the connection before retrying a READONLY statement, because a poisoned handle (SQLITE_READONLY_DBMOVED) is cleared by reopening and never by waiting. mksqlite autocommits, so there the failed statement is all there is to re-run. In SQLiteDB it is part of a transaction whose earlier statements die with the handle. The four write operations (_do_add_branch, _do_add_doc, _do_remove_doc, _do_delete_branch) now run through a journaling cursor. It records each statement's SQL, parameters, rowcount, an INSERT's lastrowid and every row read. On a READONLY at a statement or at COMMIT, _recover_transaction rolls back the dead transaction, waits on the same backoff schedule, reopens and replays the journal. Each statement is checked against what it did the first time before the failed statement runs or the transaction commits. Only SQL is replayed: files already copied, or deleted with delete_original, are not needed again. A replay that differs (another writer took the next doc_idx, a different file is at the path, a statement fails) is refused, nothing is written, and the original error is raised. The tests reproduce a real DBMOVED by replacing the database file under the open connection. Doing that showed SQLite reports it at the first write of a transaction rather than at COMMIT, so a COMMIT-only recovery would have missed the real case. It also showed that a swap after the first write is not detected at all, which no retry can see. Adding documents is about 4% slower (1,500 documents: 6.1 s -> 6.4 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Dn4gCNCU2jUGdmfPVCAGK
1 parent 881a897 commit 0fcd385

3 files changed

Lines changed: 623 additions & 34 deletions

File tree

‎src/did/did_matlab_python_bridge_implementations.yaml‎

Lines changed: 32 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -153,27 +153,38 @@ classes:
153153
UserWrite), and a temp-file write probe of the directory -- but print
154154
the time as ISO-8601 UTC rather than MATLAB's local datestr.
155155
156-
DEVIATION -- where the retry may run. mksqlite autocommits every
157-
statement, so in MATLAB a failed statement is the whole unit of work
158-
and re-running it (even after a reopen) is safe. SQLiteDB groups each
159-
add/remove/branch operation into one sqlite3 transaction. Rolling back
160-
or reopening mid-transaction would silently discard the statements
161-
already pending, and a retried tail would then commit a partial
162-
operation. So do_run_sql_query retries only when the connection had no
163-
uncommitted work before the statement; inside a transaction the error
164-
is reported (with diagnostics) and raised at once, and the caller's
165-
existing rollback runs.
166-
167-
DEVIATION -- the commit is retried too. Because Python's writes reach
168-
the disk at COMMIT rather than at each statement, that is where a
169-
slow-drive failure lands here. Every commit now goes through
170-
_commit(), which retries a BUSY / READONLY COMMIT on the same schedule
171-
while the transaction is still open. It never reopens -- that would
172-
lose the transaction -- so a genuinely poisoned handle (DBMOVED) is not
173-
recovered at COMMIT: the error is raised and the operation rolled back.
174-
Replaying a whole operation after a reopen would cover that case, but
175-
_do_add_doc also copies ingested files, so it is not obviously
176-
idempotent; not attempted.
156+
DEVIATION -- what is re-run after a reopen. mksqlite autocommits
157+
every statement, so in MATLAB a failed statement is the whole unit of
158+
work and re-running it after a reopen is safe. SQLiteDB groups each
159+
add/remove/branch operation into one sqlite3 transaction, and a
160+
poisoned handle takes the transaction's earlier statements with it.
161+
Since 2026-10-23 (da83fa3's reopen, carried into the transaction
162+
case) the four write operations -- _do_add_branch,
163+
_do_add_doc, _do_remove_doc, _do_delete_branch -- run through a
164+
journaling cursor (_JournalCursor, @_journaled) that records each
165+
statement's SQL, parameters, rowcount, an INSERT's lastrowid and every
166+
row the operation read. On a READONLY inside one -- at a statement or
167+
at COMMIT -- _recover_transaction rolls the dead transaction back,
168+
waits on the same backoff schedule, reopens, replays the journal and
169+
checks each statement against what it did the first time, then runs
170+
the failed statement (or commits). Only SQL is replayed: files the
171+
operation copied, or deleted with delete_original, before the failure
172+
are not touched again, which is why the operation is not simply re-run.
173+
A replay that differs -- another writer took the next doc_idx, a
174+
different file is now at the path, a statement fails -- is refused,
175+
nothing is written, and the original error is raised; it is not
176+
retried. Two things found while porting: SQLite reports DBMOVED at a
177+
transaction's FIRST WRITE, not at COMMIT, so that is where recovery
178+
usually happens; and a file swapped out after that first write is not
179+
detected at all -- COMMIT "succeeds" into the unlinked file -- which no
180+
retry in either language can see.
181+
do_run_sql_query, outside a write operation, keeps its own retry: only
182+
when no transaction is open, reopening first for READONLY; inside an
183+
unjournaled transaction the error is raised at once.
184+
185+
BUSY at COMMIT is still waited out on the same connection while the
186+
transaction stays open, never replayed: a lock clears with time, and
187+
the transaction is intact.
177188
178189
DEVIATION -- classification. Like MATLAB, BUSY vs READONLY is judged
179190
from the error message (sqlite3.Error.sqlite_errorcode would be exact

0 commit comments

Comments
 (0)