Find why two database replicas rejected their recovery checkpoint on 8 October #4955

🤖 Generated by the Agentic Engineer

Description

On 8 October two production database replicas, managed by the same CloudNativePG version, failed in the same way within hours: each fetched a segment from the write-ahead-log archive at start-up and PostgreSQL rejected the record there as not a valid checkpoint, so neither replica could ever catch up by waiting. One was the Coroot metadata database (#4662), the other the wedding database (#4673). Both recovered by being rebuilt from their primary, and both applications are healthy.

Why it happened is still unknown, so nothing yet stops it from happening again. A recurrence costs more than a replica: with one member down, the disruption budget blocks node drains, and on 8–9 October that failed production deploys until a person replaced the replica by hand.

Evidence

• Coroot's own evidence did not survive. Its recovery wrote a fresh base backup into the affected storage, the volume snapshots that exist were all taken after the incident, and no retained copy of the failed data exists (read-only check, 16 October). Coroot database replica cannot recover its PostgreSQL checkpoint #4662 records this limit.
• The wedding replica's original volume is still retained: it was released, not deleted, when the replica was rebuilt. It is the only surviving pre-replacement copy of this failure class.
• The upstream report [Bug]: Detached PVCs may be reused after destroy --keep-pvc cloudnative-pg/cloudnative-pg#11632 explains why a retained claim is not reused cleanly after a replica is destroyed. It does not explain the rejected checkpoint itself.

Hypothesis

Reading the retained wedding volume offline (its last checkpoint location and timeline, compared with the archived segment the replica rejected) will show whether the archive, the volume or the replacement sequence was wrong. That decides whether the durable fix is a configuration change, an upstream report, or evidence-preservation only.

Acceptance criteria

• The retained volume is inspected read-only from a copy, never attached to the running database, and the production database is not changed.
• The record states which of: archive content, local data, or the node-replacement sequence caused the rejection, or states plainly that the copy cannot decide it.
• If a cause is established, its repair ships through the reviewed pipeline with a regression check. If not, a recurrence keeps its failed storage for diagnosis instead of having it overwritten (Coroot lost its evidence that way).
• The retained volume is released for deletion only after the record is written.

Notes

Attaching or copying a retained production volume is a write the engineer's read-only cluster access cannot perform; the first step is a reviewed, bounded procedure for whoever holds that access. Raw identifiers, positions and logs stay in the private operator store.

Part of #2043.