[stevevanhooser]

Not run locally: no MATLAB was available where this was written. Per AGENTS.md, it needs this PR's CI run and a review by someone with MATLAB.

The problem

sqlitedb.m never starts a transaction (there is no BEGIN/COMMIT/ROLLBACK anywhere in sqlitedb.m or database.m), and mksqlite doesn't start one either. So every statement commits on its own, and a document that fails part-way through do_add_doc is left half-written. That covers an error, Ctrl-C, a full disk or a poisoned handle. The upfront validation in do_add_doc catches bad input, but not a failure after the inserts have started.

da83fa3 made one case worse. On a poisoned handle (SQLITE_READONLY_DBMOVED), it reopens and carries on, writing the rest of the document to whatever file is at the path by then.

The change

Each write operation is one transaction. do_add_doc, do_remove_doc, do_add_branch and do_delete_branch call beginWrite (BEGIN IMMEDIATE) and commitWrite. The handle beginWrite returns is an onCleanup, so any error or early return rolls the operation back. fields_cache is restored with the rollback; otherwise a field added in a rolled-back transaction would stay cached, and the next document's doc_data would point at a field that doesn't exist.

Files are deleted only after the commit (deleteAfterCommit). This covers delete_original in do_add_doc and the reclaimed cached copies in reclaim_unreferenced_doc. A rollback must not leave the only copy of a file unreferenced, or leave rows pointing at files already deleted.

da83fa3's reopen still works inside a transaction, by replaying it:

• every statement of the operation is journaled with what it returned and, for a write, how many rows it changed (SELECT changes());
• on a READONLY, recoverTransaction rolls back the dead transaction first. That releases its locks, and keeps its rollback journal from being played back into the new file.
• it then waits on the same backoff schedule, reopens, replays the journal, and checks each statement against what it did the first time;
• only then does it run the failed statement, or commit.

Only SQL is replayed. Files already copied stay where they are, which is why the operation isn't simply run again.

• A replay that differs is refused. Examples: another writer took the next doc_idx, a different file is at the path, or a statement fails. Nothing is written, and the original error is raised with DID:SQLITEDB:ReplayDiverged as its cause.
• A database file that no longer exists is not reopened. That would create a new, empty database at the path. The error carries DID:SQLITEDB:DatabaseFileGone as its cause.

Replaced-file check before COMMIT. SQLite notices a replaced file only when a transaction's first write opens its journal. If a sync client, a restore or a copy-then-rename swaps the file after that, COMMIT "succeeds" into the old file, which no longer has a name, and the write is lost without an error. commitWrite (and beginWrite) compare the path's unix dev:inode (read through Java) with the one recorded at open. If they differ, the transaction is recovered as above. On Windows, or with -nojvm, there's no inode, so the check is skipped.

add_docs(..., 'Validate', false) uses journal_mode=MEMORY instead of OFF. With OFF, SQLite documents that ROLLBACK "behaves in an undefined way". MEMORY still writes no journal file, but rollback works.

This mirrors DID-python's SQLiteDB (VH-Lab/DID-python#93 and #94).

Behaviour changes to review

• The write lock is held for a whole document, including its file copies and any customFileHandler downloads, rather than per statement. Another process writing the same database waits, up to the 30 s busy_timeout, for one document's ingest. DID-python behaves the same way.
• delete_original deletes the source after the document commits, not during the file loop.
• A deleted database file mid-operation is now an error carrying DID:SQLITEDB:DatabaseFileGone, instead of a write into a new, empty file.
• There's one extra SELECT changes() per write statement inside an operation. In exchange there's one commit per document instead of one per statement, so bulk adds should not be slower; I haven't measured it.

Tests

tests/+did/+unittest/TestWriteTransactions.m (8 tests) uses did.test.helper.WriteFaultSqlitedb, a subclass that can refuse a commit, or run an action (replace or delete the file) just before the first INSERT into a chosen table:

• a failed commit leaves no docs or branch_docs rows;
• fields_cache is restored, so no doc_data row ends up pointing at a missing field;
• originals are deleted only once the add commits;
• cached copies survive a rolled-back remove;
• a file replaced between operations is recovered;
• a file replaced mid-write is caught before COMMIT, and the document lands in the file now at the path;
• a deleted file is an error, and no new database is created;
• a replay against an older file is refused with ReplayDiverged.

The file-replacing tests are skipped on Windows.

Follow-up in DID-python

Once this merges, DID-python's "Bridge coverage vs DID-matlab" check will report drift on sqlitedb.m and database.m until it's synced. The Python side already has the transactions, replay and replaced-file check. It still needs:

• originals and cached copies deleted only after commit (it deletes delete_original sources before commit today);
• a refusal to reopen a database file that's gone (it currently creates an empty database there);
• the bridge hash bump.

The journal_mode change doesn't apply, since DID-python never changes it.

---

Generated by Claude Code

[github-actions]

Test Results

437 tests  +9   437 ✅ +9   2m 5s ⏱️ -9s
 38 suites +1     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 453f381. ± Comparison against base commit 057f641.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

❌ Patch coverage is 68.00000% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.71%. Comparing base (057f641) to head (453f381).

Files with missing lines	Patch %	Lines
src/did/+did/+implementations/sqlitedb.m	67.83%	64 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #227      +/-   ##
==========================================
+ Coverage   65.63%   65.71%   +0.07%     
==========================================
  Files          66       66              
  Lines        4391     4579     +188     
==========================================
+ Hits         2882     3009     +127     
- Misses       1509     1570      +61     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.