Repository navigation
Verify a WAL catch-up after the Wedding backup cutover #3778
Description
🤖 Generated by the Agentic Engineer
Evidence
The Wedding backup mirror (platform#3776) proves the dedicated bucket holds the shared catalogue at the moment it runs. The shared archive keeps receiving WAL until the Cluster's archive reference changes, so any segment archived between the last CONVERGED run and that change exists only in the old bucket. Codex raised this on platform#3776, and it holds: the mirror cannot quiesce archiving, and it refuses to run once the Cluster has switched.
A post-switch pass is not possible with today's evaluator either. After the switch the Cluster writes new WAL and base backups straight into the dedicated bucket. Those objects were never in the source listing, so the evaluator refuses them as unexpected destination objects.
Audience and problem
Wedding guests and the operator. Without a catch-up, the dedicated catalogue can carry a WAL gap right at the cutover. That breaks point-in-time recovery across the switch, which is exactly the window in which a restore is most likely to be needed.
Hypothesis
A catch-up mode that runs after the reference change, copies the source (which is now quiescent) once more, and accepts only destination objects that are either copied source objects or written by the Cluster after the switch, closes the gap with evidence.
Acceptance criteria
- A mode that runs only when the Cluster archives through
wedding-db-dedicated, copies the shared catalogue again, and proves every source object is present with a matching size and checksum. - Destination objects that are not in the source are accepted only when they were written after the recorded switch time. Anything older is still refused.
- The WAL sequence across the switch is continuous: the newest source segment and the oldest post-switch segment leave no gap.
- RED/GREEN tests cover a segment archived just before the switch that is missing, a stale destination object, and a gap in WAL continuity.
- Seed the dedicated wedding backup credential and repoint the ExternalSecret and ObjectStore #3252's cutover steps name this pass as mandatory after the reference change.
Rough size: M. Blocks treating #3252's cutover as complete.
Metadata
Metadata
Assignees
Labels
Type
Fields
Projects
- Status
Activity
fix(dr): keep mc credentials in memory and stop overclaiming convergence
chore: agent-claim #3778 nonce=1388870ac79d5af66b3b6ad427455171
fix(wedding-backup): judge catch-up WAL continuity within the Cluster…
fix(wedding-backup): refuse catch-up rewrites and misplaced WAL segments
devantler commentedon Sep 14, 2026
Shipped in #3798, now in 📊 Verifying
#3798 merged at 08:02:46Z at its evaluated head
c8d6844, with merge commit95de29e4. The merge-queue CI run succeeded,🚀 Deploy to Prodpassed, and🩹 Heal Prodwas skipped.What is now in place.
Mirror Wedding Backup Catalogueworkflow has acatch-upmode that takes the recorded switch time.wedding-db-dedicated, and works within the Cluster's server directory.What still has to happen before this is Done. This change can only be proven for real during the Wedding cutover (#3768), which has not happened yet:
catch-upmode with the switch time.CAUGHT UPcloses the acceptance criteria against live data. A refusal is a finding to fix before the cutover counts as complete.devantler commentedon Oct 11, 2026
Verification read 2026-10-11 — the live proof is in, card moved to Done.
The step this card was waiting for has happened. The Wedding cutover (#3768) merged on 2026-10-01, and the catch-up pass was run on
mainright after it: run 36832028928 (Mirror Wedding Backup Catalogue, started 07:43:56Z, concludedsuccess). As recorded on #3768, it found the fresh base backup in the dedicated archive, proved the next WAL segment after the shared archive, and confirmed the shared catalogue had not grown after the switch. I re-read only the run conclusion today, not its log. The remaining cutover work (isolated restore, access denial, retirement) is tracked on #3769, #3253 and #3252.