Pair reconciliation retries errors with bounded exponential backoff. If PostgreSQL stays unavailable beyond the retry budget while an AgentTemplate or Harness pair disappears, retirement is never persisted. The pending-preparation poll cannot see removed pairs, and restarting the controller loses the deletion event. The active database pair then retains runtime revisions indefinitely. See #2742.
Add a system doctor that compares persisted pair identities with the fully synced Kubernetes graph on startup and periodically. It should retire identities that no longer belong to the current graph and requeue incomplete preparation using the pair control plane's existing operations. Recovery must derive from PostgreSQL and Kubernetes, so it also works after replica replacement.
Keep pair lifecycle ownership in the pair control plane; runtime revision GC should continue to depend only on PostgreSQL and Substrate. Distinguish an absent identity from an unavailable or unsynced Kubernetes source before retiring anything.
Acceptance coverage:
- An outage exceeds the retry budget during template deletion; after controller restart and database recovery, the stale pair is retired and its unreferenced revision is collected.
- Template/harness replacement and removal from the admitted graph receive the same repair.
- Current identities and revisions retained by instances or checkpoints remain intact.
- Repeated repair is idempotent, and an unsynced graph cannot retire persisted pairs.
Pair reconciliation retries errors with bounded exponential backoff. If PostgreSQL stays unavailable beyond the retry budget while an AgentTemplate or Harness pair disappears, retirement is never persisted. The pending-preparation poll cannot see removed pairs, and restarting the controller loses the deletion event. The active database pair then retains runtime revisions indefinitely. See #2742.
Add a system doctor that compares persisted pair identities with the fully synced Kubernetes graph on startup and periodically. It should retire identities that no longer belong to the current graph and requeue incomplete preparation using the pair control plane's existing operations. Recovery must derive from PostgreSQL and Kubernetes, so it also works after replica replacement.
Keep pair lifecycle ownership in the pair control plane; runtime revision GC should continue to depend only on PostgreSQL and Substrate. Distinguish an absent identity from an unavailable or unsynced Kubernetes source before retiring anything.
Acceptance coverage: