Adoption deletes the legacy cluster-state ConfigMap but leaves Pods referencing it, and neither self-heal path detects the result #373

What happens

In-place adoption deletes the legacy cluster-state ConfigMap while leaving the running Pods untouched. Those Pods reference that ConfigMap through envFrom, and a Pod spec cannot be edited in place. The cluster therefore keeps running normally until something recreates the container, at which point the Pod fails with CreateContainerConfigError: configmap "etcd-cluster-state" not found and never starts again.

The gap between adoption and the failure can be weeks. On the cluster where I hit this it was 21 days, and it surfaced as two tenant control planes going down at once with no change having been made that day.

internal/migrate/adopt.go:

// ConfigMapName is the legacy cluster-state ConfigMap to delete.
ConfigMapName string
...
plan.Adoption.ConfigMapName = LegacyStateConfigMapName(name)

and the note a few lines further down:

in-place adoption: the etcd pods and their PVCs stay exactly as they are;
only ownership, labels and member annotations change

Both halves are true, and together they leave every adopted Pod holding a reference to an object that adoption itself removed.

Why neither self-heal path catches it

This is the part that turns a restart into a permanent outage.

etcdContainerStuck requires restarts:

if cs.Ready || cs.RestartCount < dataLossRestartThreshold {   // threshold = 5
    return false
}

CreateContainerConfigError means the container was never created, so RestartCount stays at 0 forever. The counter never reaches the threshold.

The terminal-Pod path added in #370 does not fire either: the Pod phase is Running, only the container is stuck in waiting. So the affected Pods sit in 0/1 CreateContainerConfigError with RESTARTS = 0 indefinitely, and ensurePod treats them as healthy because the Pod object exists:

err := r.Get(ctx, ..., pod)
if err == nil {
    // records status, reconciles the role label, returns
    return nil
}

Observed state on three tenants:

NAME     READY   STATUS                       RESTARTS   AGE
etcd-0   0/1     CreateContainerConfigError   0          21d
etcd-1   0/1     CreateContainerConfigError   0          21d
etcd-2   0/1     CreateContainerConfigError   0          21d

Meanwhile the EtcdCluster CR reported Available=True QuorumAvailable: 2/3 members ready, which did not match reality.

Impact

Every cluster that went through adoption carries this. It is invisible while the containers keep running, and it detonates on the next node reboot, eviction, or kubelet restart. On a platform where each tenant control plane has its own etcd, one node reboot can take out several tenants at once, and none of them recover on their own.

Half the tenants on my cluster had already failed this way; the rest were healthy only because their containers had not restarted since adoption.

Workaround

Delete the Pod. ensurePod then takes the IsNotFound branch and rebuilds it from buildPod, which carries no envFrom, resumes the existing data directory via the data-dir-subpath annotation, and passes --initial-cluster-state=existing. Data is preserved: the replacement came back reading raft index 8743018.

On a healthy cluster this has to be done one member at a time, checking quorum between each.

Suggested fix

Either of these closes the hole; the first is the real fix and the second is worth having anyway.

1. Do not leave Pods referencing a deleted object. Adoption could drop the envFrom reference as part of the ownership rewrite, or recreate the Pods itself, or keep the ConfigMap until every Pod referencing it has been replaced.
2. Make self-heal see a container that cannot be created. A container stuck in waiting with a config-related reason and RestartCount == 0 is currently invisible to both paths, even though it is more clearly unrecoverable than a crash loop.

Related, same area, different trigger: #368, #370, #372.