livesync-bridge: stop restart-time vault deletion + make bulk replay resilient - #174
Merged
Merged
Conversation
…ne-scan) A restart of the bridge mass-deleted the materialized notes/ vault. Root cause: the storage peer ran scanOfflineChanges against the notes/ 9p bind BEFORE Docker Desktop finished populating it. The initial scan read the partially/not-yet- mounted folder as thousands of offline deletions and propagated them (once saved only by an unrelated 9p I/O error; a later restart actually removed ~1.5k files). CouchDB was never at risk of loss here (it is the authoritative superset), but the disk mirror the AI stack reads was destroyed. Two root-cause safeguards: - scanOfflineChanges=false: the offline reconciliation is the destructive path and is unnecessary for a 24/7 bridge — useChokidar (CHOKIDAR_USEPOLLING) carries host/AI edits to CouchDB and device->CouchDB->file is network-driven. Only bridge-downtime changes are missed, and they re-sync on next touch (never a delete). CouchDB->disk full materialization still happens via the couchdb peer's fetch-from-first on a fresh index, which safely rebuilds the mirror. - Stable-populated mount gate: when CouchDB is populated, refuse to start the sync daemon until the on-disk file count has stabilized (unchanged across 3x5s), i.e. the 9p mount finished enumerating — so no startup scan ever sees a partial tree. Validated: on the fixed image the gate waits for the mount to stabilize, boots with zero Unlink/delete events, and the couchdb peer's fetch-from-first replay rebuilds notes/ from CouchDB with doc_del_count held at 1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…terialize escape hatch
Recovering the notes/ mirror after the deletion incident surfaced a second class of
bug: the bridge's bulk fetch-from-first replay (rebuilding the whole vault from
CouchDB on a fresh index) would silently stall partway, leaving the mirror short.
Two root causes, both in the vendored LiveSync core's watch handlers:
- A per-document fetch (getByMeta) ran OUTSIDE the try/catch guarding the sync
callback, so a corrupted doc ("Corrupted document", a 451 MB .rtb) or an oversized
one ("RangeError: string too long", a 34 MB PDF) rejected the changes handler and
stalled the feed — every document after it was skipped.
patch-watch-resilience.ts (applied at build time, self-verifying — the build fails
if the upstream pattern is gone) now, in both beginWatch() and followUpdates():
* skips documents over 25 MiB BEFORE the expensive getByMeta() — such files can't
reliably materialize to the disk mirror anyway and stay safe in CouchDB;
* moves getByMeta() INSIDE the try/catch so any remaining bad doc is logged and
skipped instead of halting the feed.
entrypoint.sh: LSB_ALLOW_EMPTY_STORAGE=1 escape hatch to skip the mount-readiness
gate for a deliberate fresh-device re-materialization (intentionally-emptied notes/).
Validated: with polling contention removed, a full CouchDB->disk rebuild now
completes to the full ~5,260 files; the 451 MB and 34 MB monsters are cleanly
skipped (kept in CouchDB), not stalling the sync.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
A
livesync-bridgerestart wiped the materializednotes/mirror. Root cause: the storage peer'sscanOfflineChangesran its offline diff against thenotes/9p bind mount before Docker Desktop finished populating it — it read the not-yet-mounted (empty) tree as thousands of deletions and propagated them toward CouchDB and disk. Every bridge restart was a coin-flip that could mass-delete the vault.CouchDB is the authoritative store (every Obsidian device syncs to it);
notes/is a materialized mirror the 24/7 AI stack reads. Recovery from this incident surfaced a second bug: the bridge's bulk fetch-from-first replay (rebuilding the whole vault from CouchDB on a fresh index) would silently stall partway on a single bad document, leaving the mirror short.Fixes
1. Stop the restart-time deletion (
20d8b15)scanOfflineChanges: false— the offline reconciliation is the destructive path and is unnecessary for a 24/7 bridge:useChokidar(withCHOKIDAR_USEPOLLING) carries host/AI edits → CouchDB, and device → CouchDB → file is network-driven. Only changes made while the bridge was down are missed, and they re-sync on next touch — a benign staleness, never a deletion. CouchDB→disk full materialization still happens via the couchdb peer's fetch-from-first.entrypoint.sh: when CouchDB is populated, refuse to start the sync daemon until the on-disk file count has been unchanged across 3×5s (i.e. the 9p mount finished enumerating), so no startup scan ever sees a partial tree.2. Make the bulk replay resilient (
5452b9a)patch-watch-resilience.ts(applied at build time, self-verifying — the build fails if the upstream pattern is gone) patches the vendored LiveSync core'sbeginWatch/followUpdates:getByMeta()(a 451 MB corrupted.rtband a 34 MB PDF were stalling the feed) — such files can't reliably materialize to the disk mirror anyway and stay safe in CouchDB;getByMeta()inside the try/catch, so a corrupted doc ("Corrupted document") or oversized one (RangeError: string too long) is logged and skipped instead of halting every document sequenced after it.LSB_ALLOW_EMPTY_STORAGE=1escape hatch to skip the mount-readiness gate for a deliberate fresh-device re-materialization (intentionally-emptiednotes/).Validation
Unlink/delete events;doc_del_countheld throughout.Notes / follow-ups (not in this PR)
sincecheckpoint lives in ephemeral in-container Deno localStorage, so every recreate re-runs the full replay. Persisting/deno-dir/location_dataon a named volume would make restarts resume instead of rebuild — a candidate follow-up.deleted:truefield) are invisible todoc_del_count— audit deletions via_find {"selector":{"deleted":true}}, and un-delete via_bulk_docs(content chunks are retained).🤖 Generated with Claude Code