Skip to content

sqlitedb: catch a database file replaced after a transaction's first write - #94

Merged
stevevanhooser merged 1 commit into
mainfrom
claude/sqlite-detect-replaced-file
Oct 24, 2026
Merged

stevevanhooser merged 1 commit into
mainfrom
claude/sqlite-detect-replaced-file

Conversation

@stevevanhooser

Copy link
Copy Markdown
Contributor

Closes the gap #93 recorded as undetectable.

The problem

SQLite checks whether its open file is still the one at the path only when a write transaction starts (SQLITE_READONLY_DBMOVED, which #93 recovers from).

If the file is replaced after that first write, the rest of the transaction and its COMMIT go to the old file. Typical causes are a sync client (Dropbox, OneDrive, iCloud), a backup restore, or a copy-then-rename script. That old file no longer has a name, and it's deleted when the connection closes. COMMIT reports success and the write is lost. test_sqlite_alone_loses_the_write shows this on a bare sqlite3 connection.

The change

  • _open_db records the (st_dev, st_ino) of the file it opened.
  • Inside a journaled write operation, _commit checks that the path still names that file before committing (_file_was_replaced). A deleted file counts as replaced.
  • If the file has changed, the handle isn't committed. It's treated as poisoned, and the transaction goes through sqlitedb: recover a poisoned handle inside a write operation by replaying it #93's recovery: reopen, replay against the file now at the path, verify, commit.
  • Recovery repeats the check before its own commit, in case the file was replaced again.

The data-loss window shrinks from "first write to commit" (the whole operation) to the microseconds between the check and COMMIT. This is the same comparison SQLite's unix VFS makes for DBMOVED, applied at the one moment SQLite doesn't make it.

Cost

One os.stat per commit: about 0.9 µs locally, against about 1,800 µs for an insert and commit. On network drives a stat costs more, but so does the commit.

Edge cases

  • False alarm (a volume that reports unstable inode numbers): costs one replay onto the same file. Nothing was committed before the check, so the document still lands exactly once; there's a test for this.
  • File deleted mid-operation: an error, not a silent loss. Reopening creates a new, empty database at the path. The replay finds no branch there and is refused, so nothing is written to it.
  • Writes outside a journaled write operation, such as a write sent straight through do_run_sql_query, aren't covered. They're rare.

This is a Python-only addition, recorded in the sqlitedb bridge entry. MATLAB has the same gap, narrower because each mksqlite statement commits on its own.

Tests (tests/test_sqlitedb_transient_errors.py, 6 new)

  • the loss on a bare connection;
  • a file replaced mid-add_docs is caught and the document lands;
  • a deleted file is an error;
  • a false alarm lands the document exactly once;
  • an unchanged file is never replayed;
  • the recorded identity follows a reopen.

The replaced-file and deleted-file tests fail without the change.

Locally: black --check, ruff check (0.16.5), pytest (800 passed), and check_bridge_coverage.py (0 problems).

It doesn't remove the advice not to keep a DID database that's in use inside a sync folder: sync clients can also overwrite in place or make conflicted copies, which no check inside DID can see.


Generated by Claude Code

…write

SQLite checks that its open file is still the one at the path
(SQLITE_READONLY_DBMOVED) only when a write transaction starts. A file
replaced after that, by a sync client, a restore or a copy-then-rename,
isn't noticed. COMMIT succeeds into the old, now nameless file and the
write is lost without an error.

_open_db now records the (st_dev, st_ino) of the file it opened. Inside a
journaled write operation, _commit checks that the path still names that
file before committing; a deleted file counts as replaced. If it doesn't,
the handle is treated as poisoned and the transaction goes through #93's
recovery: reopen, replay against the file now at the path, verify, commit.
Only a window of microseconds between the check and the COMMIT remains.

The check is one stat per commit (about 1 us locally, against about
1.8 ms for the commit). A volume with unstable inode numbers can trip it
falsely, which costs one replay onto the same file, since nothing had
been committed yet. This is a Python-only addition, recorded in the
bridge entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Dn4gCNCU2jUGdmfPVCAGK
@stevevanhooser
stevevanhooser merged commit 5d591ac into main Oct 24, 2026
7 checks passed
@stevevanhooser
stevevanhooser deleted the claude/sqlite-detect-replaced-file branch October 24, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants