Skip to content

livesync-bridge: stop restart-time vault deletion + make bulk replay resilient - #174

Merged
AlienWalker1995 merged 2 commits into
mainfrom
fix/livesync-bridge-mount-guard
Aug 12, 2026
Merged

livesync-bridge: stop restart-time vault deletion + make bulk replay resilient#174
AlienWalker1995 merged 2 commits into
mainfrom
fix/livesync-bridge-mount-guard

Conversation

@AlienWalker1995

Copy link
Copy Markdown
Owner

What broke

A livesync-bridge restart wiped the materialized notes/ mirror. Root cause: the storage peer's scanOfflineChanges ran its offline diff against the notes/ 9p bind mount before Docker Desktop finished populating it — it read the not-yet-mounted (empty) tree as thousands of deletions and propagated them toward CouchDB and disk. Every bridge restart was a coin-flip that could mass-delete the vault.

CouchDB is the authoritative store (every Obsidian device syncs to it); notes/ is a materialized mirror the 24/7 AI stack reads. Recovery from this incident surfaced a second bug: the bridge's bulk fetch-from-first replay (rebuilding the whole vault from CouchDB on a fresh index) would silently stall partway on a single bad document, leaving the mirror short.

Fixes

1. Stop the restart-time deletion (20d8b15)

  • scanOfflineChanges: false — the offline reconciliation is the destructive path and is unnecessary for a 24/7 bridge: useChokidar (with CHOKIDAR_USEPOLLING) carries host/AI edits → CouchDB, and device → CouchDB → file is network-driven. Only changes made while the bridge was down are missed, and they re-sync on next touch — a benign staleness, never a deletion. CouchDB→disk full materialization still happens via the couchdb peer's fetch-from-first.
  • Stable-populated mount gate in entrypoint.sh: when CouchDB is populated, refuse to start the sync daemon until the on-disk file count has been unchanged across 3×5s (i.e. the 9p mount finished enumerating), so no startup scan ever sees a partial tree.

2. Make the bulk replay resilient (5452b9a)

  • patch-watch-resilience.ts (applied at build time, self-verifying — the build fails if the upstream pattern is gone) patches the vendored LiveSync core's beginWatch/followUpdates:
    • skips documents over 25 MiB before the expensive getByMeta() (a 451 MB corrupted .rtb and a 34 MB PDF were stalling the feed) — such files can't reliably materialize to the disk mirror anyway and stay safe in CouchDB;
    • moves getByMeta() inside the try/catch, so a corrupted doc ("Corrupted document") or oversized one (RangeError: string too long) is logged and skipped instead of halting every document sequenced after it.
  • LSB_ALLOW_EMPTY_STORAGE=1 escape hatch to skip the mount-readiness gate for a deliberate fresh-device re-materialization (intentionally-emptied notes/).

Validation

  • Fixed image boots with the gate waiting for the mount to stabilize, then zero Unlink/delete events; doc_del_count held throughout.
  • With polling contention removed for the one-shot rebuild, a full CouchDB→disk materialization completes to the full ~5,260 files; the 451 MB / 34 MB monsters are cleanly skipped (kept in CouchDB).
  • Steady state (polling on) holds the complete mirror with no regression.

Notes / follow-ups (not in this PR)

  • The bridge's since checkpoint lives in ephemeral in-container Deno localStorage, so every recreate re-runs the full replay. Persisting /deno-dir/location_data on a named volume would make restarts resume instead of rebuild — a candidate follow-up.
  • Operational lesson recorded separately: LiveSync soft-deletes (deleted:true field) are invisible to doc_del_count — audit deletions via _find {"selector":{"deleted":true}}, and un-delete via _bulk_docs (content chunks are retained).

🤖 Generated with Claude Code

Hermes Bot and others added 2 commits August 11, 2026 17:46
…ne-scan)

A restart of the bridge mass-deleted the materialized notes/ vault. Root cause:
the storage peer ran scanOfflineChanges against the notes/ 9p bind BEFORE Docker
Desktop finished populating it. The initial scan read the partially/not-yet-
mounted folder as thousands of offline deletions and propagated them (once saved
only by an unrelated 9p I/O error; a later restart actually removed ~1.5k files).

CouchDB was never at risk of loss here (it is the authoritative superset), but the
disk mirror the AI stack reads was destroyed. Two root-cause safeguards:

- scanOfflineChanges=false: the offline reconciliation is the destructive path and
  is unnecessary for a 24/7 bridge — useChokidar (CHOKIDAR_USEPOLLING) carries
  host/AI edits to CouchDB and device->CouchDB->file is network-driven. Only
  bridge-downtime changes are missed, and they re-sync on next touch (never a
  delete). CouchDB->disk full materialization still happens via the couchdb peer's
  fetch-from-first on a fresh index, which safely rebuilds the mirror.
- Stable-populated mount gate: when CouchDB is populated, refuse to start the sync
  daemon until the on-disk file count has stabilized (unchanged across 3x5s), i.e.
  the 9p mount finished enumerating — so no startup scan ever sees a partial tree.

Validated: on the fixed image the gate waits for the mount to stabilize, boots with
zero Unlink/delete events, and the couchdb peer's fetch-from-first replay rebuilds
notes/ from CouchDB with doc_del_count held at 1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…terialize escape hatch

Recovering the notes/ mirror after the deletion incident surfaced a second class of
bug: the bridge's bulk fetch-from-first replay (rebuilding the whole vault from
CouchDB on a fresh index) would silently stall partway, leaving the mirror short.
Two root causes, both in the vendored LiveSync core's watch handlers:

- A per-document fetch (getByMeta) ran OUTSIDE the try/catch guarding the sync
  callback, so a corrupted doc ("Corrupted document", a 451 MB .rtb) or an oversized
  one ("RangeError: string too long", a 34 MB PDF) rejected the changes handler and
  stalled the feed — every document after it was skipped.

patch-watch-resilience.ts (applied at build time, self-verifying — the build fails
if the upstream pattern is gone) now, in both beginWatch() and followUpdates():
  * skips documents over 25 MiB BEFORE the expensive getByMeta() — such files can't
    reliably materialize to the disk mirror anyway and stay safe in CouchDB;
  * moves getByMeta() INSIDE the try/catch so any remaining bad doc is logged and
    skipped instead of halting the feed.

entrypoint.sh: LSB_ALLOW_EMPTY_STORAGE=1 escape hatch to skip the mount-readiness
gate for a deliberate fresh-device re-materialization (intentionally-emptied notes/).

Validated: with polling contention removed, a full CouchDB->disk rebuild now
completes to the full ~5,260 files; the 451 MB and 34 MB monsters are cleanly
skipped (kept in CouchDB), not stalling the sync.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlienWalker1995
AlienWalker1995 merged commit 112fb5a into main Aug 12, 2026
3 checks passed
@AlienWalker1995
AlienWalker1995 deleted the fix/livesync-bridge-mount-guard branch August 12, 2026 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant