Skip to content

fix(deploy): an unrelated Coroot fault must not refuse every production deploy #4659

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Problem

The pre-publish Coroot baseline check (scripts/guard-coroot-baseline-context.sh before-publish) refuses a deploy whenever any of the six operator-owned Coroot workloads is not fully ready. It cannot tell why a workload is unready, so a Coroot fault that has nothing to do with the admission label it guards stops every deploy, including the deploy that would repair production and the automatic heal.

Evidence

  • 2026-10-08: one Coroot server was unready because its metadata database had no free connection slots (platform#4652). All 10 merge-queue CI runs between 02:43Z and 14:35Z failed at this check and evicted their pull request.
  • One of them was platform#4653, the fix for a missing OpenBao server. Production stayed on 2 of 3 OpenBao servers for the whole period.
  • The heal job in the same runs stopped at the same check, so there was no automatic way back.
  • Cluster access for the engineer is read-only, so the only remaining recovery was a manual restart by the maintainer.

Who is affected

Anyone relying on production deploys: an observability fault becomes a full delivery freeze, and an unrelated production incident cannot be fixed through the normal path while it lasts.

Hypothesis

The check exists to catch one thing: the Coroot operator rewriting its workloads against the admission mutation. Before publishing, only signals that point at that loop need to stop a deploy (unexpected population, lost ownership, a generation the controller has not observed, or repeated rewrites). A workload that was already unready before the deploy, with stable generations and the admitted fields present, is not evidence of the loop. If the check distinguishes the two, an unrelated Coroot fault no longer freezes delivery, and the loop is still caught.

Acceptance criteria

  • A deploy that does not change Coroot is not refused only because a Coroot workload was already unready, when the stored templates are stable and carry the admitted fields.
  • A rewrite loop, a changed population, lost ownership or a missing admitted field is still refused, before and after reconcile.
  • The after-reconcile phase does not report a pre-existing, unchanged unready state as a failure of the deploy that found it, and never reports it as healthy either: it is surfaced as its own named result.
  • A deploy that itself makes Coroot unready is still refused.
  • Regression tests cover each of the cases above.

Out of scope

Fixing the connection exhaustion itself (platform#4652) and any change to what the admission label does.

Blocker: none

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions