You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(deploy): an unrelated Coroot fault must not refuse every production deploy #4659
The pre-publish Coroot baseline check (scripts/guard-coroot-baseline-context.sh before-publish) refuses a deploy whenever any of the six operator-owned Coroot workloads is not fully ready. It cannot tell why a workload is unready, so a Coroot fault that has nothing to do with the admission label it guards stops every deploy, including the deploy that would repair production and the automatic heal.
Evidence
2026-10-08: one Coroot server was unready because its metadata database had no free connection slots (platform#4652). All 10 merge-queue CI runs between 02:43Z and 14:35Z failed at this check and evicted their pull request.
One of them was platform#4653, the fix for a missing OpenBao server. Production stayed on 2 of 3 OpenBao servers for the whole period.
The heal job in the same runs stopped at the same check, so there was no automatic way back.
Cluster access for the engineer is read-only, so the only remaining recovery was a manual restart by the maintainer.
Who is affected
Anyone relying on production deploys: an observability fault becomes a full delivery freeze, and an unrelated production incident cannot be fixed through the normal path while it lasts.
Hypothesis
The check exists to catch one thing: the Coroot operator rewriting its workloads against the admission mutation. Before publishing, only signals that point at that loop need to stop a deploy (unexpected population, lost ownership, a generation the controller has not observed, or repeated rewrites). A workload that was already unready before the deploy, with stable generations and the admitted fields present, is not evidence of the loop. If the check distinguishes the two, an unrelated Coroot fault no longer freezes delivery, and the loop is still caught.
Acceptance criteria
A deploy that does not change Coroot is not refused only because a Coroot workload was already unready, when the stored templates are stable and carry the admitted fields.
A rewrite loop, a changed population, lost ownership or a missing admitted field is still refused, before and after reconcile.
The after-reconcile phase does not report a pre-existing, unchanged unready state as a failure of the deploy that found it, and never reports it as healthy either: it is surfaced as its own named result.
A deploy that itself makes Coroot unready is still refused.
Regression tests cover each of the cases above.
Out of scope
Fixing the connection exhaustion itself (platform#4652) and any change to what the admission label does.
Activity
chore: agent-claim #4659 nonce=08a5a5b1c084fe8e6d45a34718dfc2b3