Skip to content

Commit 121b98d

Browse files
Merge pull request #93 from VH-Lab/claude/port-sqlite-reopen-recovery
sqlitedb: recover a poisoned handle inside a write operation by replaying it
2 parents 881a897 + 0fcd385 commit 121b98d

3 files changed

Lines changed: 623 additions & 34 deletions

File tree

‎src/did/did_matlab_python_bridge_implementations.yaml‎

Lines changed: 32 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -153,27 +153,38 @@ classes:
153153
UserWrite), and a temp-file write probe of the directory -- but print
154154
the time as ISO-8601 UTC rather than MATLAB's local datestr.
155155
156-
DEVIATION -- where the retry may run. mksqlite autocommits every
157-
statement, so in MATLAB a failed statement is the whole unit of work
158-
and re-running it (even after a reopen) is safe. SQLiteDB groups each
159-
add/remove/branch operation into one sqlite3 transaction. Rolling back
160-
or reopening mid-transaction would silently discard the statements
161-
already pending, and a retried tail would then commit a partial
162-
operation. So do_run_sql_query retries only when the connection had no
163-
uncommitted work before the statement; inside a transaction the error
164-
is reported (with diagnostics) and raised at once, and the caller's
165-
existing rollback runs.
166-
167-
DEVIATION -- the commit is retried too. Because Python's writes reach
168-
the disk at COMMIT rather than at each statement, that is where a
169-
slow-drive failure lands here. Every commit now goes through
170-
_commit(), which retries a BUSY / READONLY COMMIT on the same schedule
171-
while the transaction is still open. It never reopens -- that would
172-
lose the transaction -- so a genuinely poisoned handle (DBMOVED) is not
173-
recovered at COMMIT: the error is raised and the operation rolled back.
174-
Replaying a whole operation after a reopen would cover that case, but
175-
_do_add_doc also copies ingested files, so it is not obviously
176-
idempotent; not attempted.
156+
DEVIATION -- what is re-run after a reopen. mksqlite autocommits
157+
every statement, so in MATLAB a failed statement is the whole unit of
158+
work and re-running it after a reopen is safe. SQLiteDB groups each
159+
add/remove/branch operation into one sqlite3 transaction, and a
160+
poisoned handle takes the transaction's earlier statements with it.
161+
Since 2026-10-23 (da83fa3's reopen, carried into the transaction
162+
case) the four write operations -- _do_add_branch,
163+
_do_add_doc, _do_remove_doc, _do_delete_branch -- run through a
164+
journaling cursor (_JournalCursor, @_journaled) that records each
165+
statement's SQL, parameters, rowcount, an INSERT's lastrowid and every
166+
row the operation read. On a READONLY inside one -- at a statement or
167+
at COMMIT -- _recover_transaction rolls the dead transaction back,
168+
waits on the same backoff schedule, reopens, replays the journal and
169+
checks each statement against what it did the first time, then runs
170+
the failed statement (or commits). Only SQL is replayed: files the
171+
operation copied, or deleted with delete_original, before the failure
172+
are not touched again, which is why the operation is not simply re-run.
173+
A replay that differs -- another writer took the next doc_idx, a
174+
different file is now at the path, a statement fails -- is refused,
175+
nothing is written, and the original error is raised; it is not
176+
retried. Two things found while porting: SQLite reports DBMOVED at a
177+
transaction's FIRST WRITE, not at COMMIT, so that is where recovery
178+
usually happens; and a file swapped out after that first write is not
179+
detected at all -- COMMIT "succeeds" into the unlinked file -- which no
180+
retry in either language can see.
181+
do_run_sql_query, outside a write operation, keeps its own retry: only
182+
when no transaction is open, reopening first for READONLY; inside an
183+
unjournaled transaction the error is raised at once.
184+
185+
BUSY at COMMIT is still waited out on the same connection while the
186+
transaction stays open, never replayed: a lock clears with time, and
187+
the transaction is intact.
177188
178189
DEVIATION -- classification. Like MATLAB, BUSY vs READONLY is judged
179190
from the error message (sqlite3.Error.sqlite_errorcode would be exact

0 commit comments

Comments
 (0)