Repository navigation
Mirror the shared Wedding backup catalogue into the dedicated bucket and prove parity #3767
Description
🤖 Generated by the Agentic Engineer
Evidence
The dedicated Wedding backup path is staged but empty of history. #3690 seeded the dedicated credential and #3727 added the inactive wedding-db-dedicated ObjectStore pointing at s3://wedding-db-backups/cnpg/wedding-db. The live Cluster still archives to the shared catalogue through wedding-db (s3://${r2_bucket}/cnpg/wedding-db, credential wedding-db-backup-r2). The 2026-09-13 recovery (#3738) proved how much that history matters: it restored guest data from the shared archive's September 8 base backup and WAL.
Switching the Cluster reference before copying that history would start the dedicated archive with no recoverable base backup, which is exactly what #3727's recovery guide forbids.
Audience and problem
Wedding guests and the operator. Until the catalogue exists in the dedicated bucket, #3252's cutover cannot start safely, and the shared credential (#3253) cannot be retired.
Hypothesis
A repeatable, production-locked mirror that copies the shared cnpg/wedding-db catalogue into the dedicated bucket, and proves parity afterwards, makes the reference switch a low-risk step.
Constraints the design must respect
- Two identities at once. The dedicated credential is bucket-scoped and was observed denied on the shared bucket (Provision a dedicated R2 bucket and identity for the wedding database backup #3251), so a single credential cannot do the copy. Reading uses the shared identity; writing uses the dedicated one.
- The source keeps moving. WAL continues to archive to the shared store until the switch, so the mirror must be safe to re-run, and the final pass happens immediately before Seed the dedicated wedding backup credential and repoint the ExternalSecret and ObjectStore #3252's reference change.
- Read-only source. The shared catalogue must be byte-for-byte unchanged by the mirror.
- Follow the existing production-lock pattern used by the staging verifier (
cd.yamlverify-wedding-backup-staging) and the recovery workflow: dispatch-only, bound tomain, no secret values in logs. Identify the egress path the mirror job needs to reach R2 rather than widening any network policy by default.
Acceptance criteria
- A dispatch-only, production-locked mirror copies every object under the shared
cnpg/wedding-dbprefix intos3://wedding-db-backups/cnpg/wedding-db, and can be re-run without harm. - A parity check proves that every source object at the time of the run exists in the destination with a matching size and checksum. Its output is non-secret: counts, the newest base backup ID, and the newest WAL segment on each side.
- The run proves the source objects are unchanged, and the live Cluster still references
wedding-db. - RED/GREEN tests cover refusal on a partial copy, a source that changed during the run, the wrong destination, and credential reuse.
- One production dispatch has succeeded, with its parity output recorded on this issue.
Success signal
The dedicated bucket holds the full shared catalogue, including the September 8 base backup and its WAL, confirmed by a recorded parity run.
Rough size: M. First step of #3252's cutover; blocks the reference switch.
Completion
The real production mirror reported CONVERGED at 2026-10-01T02:53:45Z in run 36807688668, from merged main 86e7fbf22e200e947a049e4393c35b872294830d.
| Parity result | Source | Destination |
|---|---|---|
| Objects with matching size and checksum | 808 | 808 |
| Pending objects | 0 | 0 |
| Newest base backup | 20260930T030001 |
20260930T030001 |
| Newest WAL | 000000A40000000200000005 |
000000A40000000200000005 |
The evaluator passed its source-stability and destination guards, with zero late objects requiring separate verification. Independent live readback confirms the database is healthy, continuous archiving and the last backup are successful, and its archive reference remains wedding-db. The temporary mirror pod and configuration map are gone. A fresh independent Platform check returns PLATFORM-HEALTH=OK.
Repair #4296 is merged; its protected deployment and successor CI succeeded. The actual pinned runtime passed all 38 copy/evaluator cases in CI 36784229893, and the merged revision's Validate Main run passed.
This completes the catalogue-mirror step. Parity covers the recorded run; WAL archived afterwards needs a final pre-switch pass and the post-switch catch-up implemented in #3778. The archive-reference switch and its operational proof remain a separate step in #3768.
Metadata
Metadata
Assignees
Type
Fields
Projects
- Status
Activity
chore: agent-claim #3767 nonce=f9a5b6419157f15a3819ae8c56b03134
fix(dr): refuse incomplete listings and misrooted keys in the mirror …
fix(dr): require listing completion proof and valid digests in the mi…
devantler commentedon Sep 13, 2026
Progress: the parity evaluator from #3770 is merged (17:13:52Z, merge commit
04ef2e95601e93849b65fbbcb5d138a196ac1a5e, at the reviewed head502c76fc). The merge-group🚀 Deploy to Prodjob for that revision succeeded, and the heal job was skipped.The evaluator is now the gate the mirror run has to pass. It accepts a copy only when both listings are bound to this pass, the source did not change during the run, every object arrived with a matching size and checksum, the newest base backup includes its data, and nothing unexpected sits in the destination. The remaining step for this issue is running the real copy into the dedicated bucket and recording the evaluator's verdict.
5 remaining items
fix(dr): make the Wedding backup mirror verifiable on real catalogues
fix(dr): keep mc credentials in memory and stop overclaiming convergence
fix(dr): pin the Wedding backup mirror to the committed R2 endpoint
devantler commentedon Sep 13, 2026
Progress: the mirror workflow is merged
platform#3776 merged at 2026-09-13T20:18:43Z, at the reviewed head
3cec8772.mainnow has the dispatch-only Mirror Wedding Backup Catalogue workflow. It copies the sharedcnpg/wedding-dbcatalogue intowedding-db-backupsfrom insidewedding-app, then runs the merged parity evaluator on the result.What it guarantees before a verdict counts:
r2_endpoint.Remaining step for this issue: dispatch the workflow once from
mainwith the confirmation phrasemirror-wedding-backup-catalogue, then record its parity output here: counts, the newest base backup and the newest WAL on each side. ANOT CONVERGEDresult (exit 3) only means the run should be repeated.A
CONVERGEDrun means the switch may start. It does not cover WAL archived after that run, so the cutover in #3252 must end with the verified catch-up tracked in #3778.chore: agent-claim #3767 nonce=ca55f523117583932b7f215d1dba1f12
chore: agent-claim #3767 nonce=ba68e930e32c638d3c11d0f9ae28bfdd
chore: agent-claim #3767 nonce=a7782d48e7864ecea1f4487d74d3ba05
devantler commentedon Oct 1, 2026
The real production mirror reported CONVERGED at 2026-10-01T02:53:45Z in run 36807688668, from merged main
86e7fbf22e200e947a049e4393c35b872294830d.20260930T03000120260930T030001000000A40000000200000005000000A40000000200000005The evaluator passed its source-stability and destination guards, with zero late objects requiring separate verification. Independent live readback confirms the database is healthy, continuous archiving and the last backup are successful, and its archive reference remains
wedding-db. The temporary mirror pod and configuration map are gone. A fresh independent Platform check returnsPLATFORM-HEALTH=OK.Repair #4296 is merged; its protected deployment and successor CI succeeded. The actual pinned runtime passed all 38 copy/evaluator cases in CI 36784229893, and the merged revision's Validate Main run passed.
This completes the catalogue-mirror step. Parity covers the recorded run; WAL archived afterwards needs a final pre-switch pass and the post-switch catch-up implemented in #3778. The archive-reference switch and its operational proof remain a separate step in #3768.