Prove a Wedding database restore from the dedicated backup bucket alone #3769

🤖 Generated by the Agentic Engineer

Evidence

A backup that has never been restored is not proven. #3727's recovery guide requires an isolated restore proof from the dedicated bucket before the shared credential can be retired, and #3253 cannot start until that exists. The 2026-09-09 data loss (#3738) showed the restore path is what matters in practice.

Audience and problem

Wedding guests and the operator. After the reference switch, the dedicated bucket is the only live Wedding archive. If it cannot restore, revoking the shared credential would remove the last known-good path.

Hypothesis

Restoring into an isolated, temporary Cluster that can read only the dedicated bucket, then checking core guest data, proves the dedicated archive is complete and usable on its own.

Acceptance criteria

• A dispatch-only, production-locked job restores the Wedding database from wedding-db-dedicated into an isolated one-instance Cluster that cannot reach the production database.
• The restore uses only the dedicated credential. The job refuses to run if the shared credential is available to it.
• Validation compares non-secret core guest counts and the newest recovered timestamp with the live database, within an agreed tolerance for recent writes.
• The temporary Cluster, volume and any temporary network policy are removed afterwards, including when the run fails.
• One production run has succeeded, with its non-secret proof recorded on this issue. The restore prerequisite of Verify the dedicated wedding backup identity, retire shared access, and update the DR runbooks #3253 is cleared; its other acceptance gates remain separate.

Success signal

A recorded restore from the dedicated bucket alone that matches live guest data.

Rough size: M. The reference-switch dependency is complete. This issue owns the isolated restore implementation and production proof; its success unblocks #3253.

Delivery: #4316 merged on 2026-10-01. Protected main run 36856764796 succeeded on its first attempt at merged commit 822dd9f206df40d9feb7d4d8013ba5d4f8391766, restoring fresh dedicated backup 20261001T114013. The proof recovered 13 guest pairs, 24 guests, 17 answered guest records and 7 room bookings, passing comparison with live data and the documented 300-second timestamp tolerance. The production-connectivity refusal probe passed. Both normal and always-step cleanup verified removal of temporary resources and backing storage. The live database retained its identity and generation with all three replicas ready and healthy archiving. Failure cleanup is covered by regression tests; the real successful cleanup is recorded in the production run. Shared recovery access remains active pending #3252 and #3253.