Repository navigation
Production deploys fail only for the change they deploy #4979
Description
🤖 Generated by the Agentic Engineer (interactive Claude Code session)
Evidence
- Every platform change reaches
mainthrough the merge queue, and each queue candidate deploys to production before it merges. A candidate whose deploy fails is evicted. - In the 30 days to 2026-10-17,
maintook 689 commits. Leaving out dependency updates, the two largest commit scopes werefix(ci)(48) andfix(deploy)(22): about one delivery-pipeline repair a day. - Open bugs where a deploy fails, stalls or is judged wrongly for reasons outside the change being deployed:
- Production deploys fail intermittently at the data product rollout check and evict unrelated pull requests #4545 — deploys fail intermittently at the data product rollout check and evict unrelated pull requests
- fix(deploy): an unrelated Coroot fault must not refuse every production deploy #4659 — an unrelated Coroot fault refuses every production deploy
- Avoid redeploying production after an evicted candidate was refused before deployment #4937 — production is redeployed after an evicted candidate was refused before it deployed
- Deploy GHCR step can time out on the Flux policy pause without saying which condition never held #4873 — the GHCR step times out on the Flux policy pause without naming the condition that never held
- Talos node upgrades can exceed the 10-minute drain timeout, evicting the bump from the merge queue #3388 — Talos node upgrades can exceed the 10-minute drain timeout, which evicts the bump
- The deploy bridge loses its own sync lease mid-run, and each failure fences a node that fails the next deploy #3071 — the deploy bridge loses its own sync lease mid-run
- A deploy passes its Flux step when Flux could not delete a removed resource #4734 — a deploy passes its Flux step when Flux could not delete a removed resource (the opposite failure: a broken deploy judged healthy)
- Complete the Platform release left after a rejected notes push #4839 — a Platform release is left incomplete after a rejected notes push
- No current epic owns this class: the earlier delivery-correctness theme on Roadmap: a resilient, current, well-abstracted GitOps platform #2043 has closed all its children.
Audience and problem
Everyone who changes the platform (the maintainer and both agent lanes) and every tenant waiting on a platform change. When a deploy gate fails for a reason the candidate did not cause, the candidate is evicted, the next candidate inherits the same fault, and a working change can take hours to land. When a gate passes something broken, production drifts from main unnoticed.
Hypothesis
If every deploy gate judges only the effect of the candidate it is deploying, and names the condition that failed, then evictions caused by something outside the evicted pull request become rare, and a gate never reports success for a step it could not complete.
Success signal
- Signal: the share of
failed_checksmerge-queue evictions whose root cause lies outside the evicted pull request. - Baseline: not measured yet. The first child classifies the evictions from the last 30 days, so the baseline is a number, not an estimate.
- Target: under 10% over a 30-day window after the fixes, with every eviction's cause named by the failing step's own output.
- Guardrail: no gate is removed or made non-blocking, and a candidate that does break production is still evicted and healed. Fixes narrow what a gate looks at; they never stop it looking.
Smallest useful change
Measure first: classify the last 30 days of failed_checks evictions by root cause (candidate's own change, unrelated platform fault, infrastructure transient, timeout) and rank the unrelated causes by how many evictions each caused. Then fix the top cause.
Acceptance criteria
- An eviction-cause baseline for the last 30 days is recorded on this issue.
- The children above are fixed or closed with a reason, highest-ranked cause first.
- A 30-day re-measurement meets the target without breaching the guardrail.
Size: large — about 8 child issues, each small to medium. Decompose further as the baseline names new causes.
Activity
1 remaining item
devantler commentedon Oct 17, 2026
Eviction-cause baseline, 2026-09-17 to 2026-10-17 (from #4993)
Source. Every
CI(ci.yaml) run withevent=merge_groupcreated in the window: 871 runs fetched, matching the run list's total of 871. Conclusions: 699 success, 160 failure, 12 cancelled. All 160 failures are classified below; the 12 cancelled runs are not.Method. Each failed run's failing job and step were read from the Actions API. The deploy job runs one composite step,
🚀 Deploy to Production, so for deploy failures the cause comes from the script's own error annotation (43 runs) or the job log (43 logs read). About 42 runs were not read one by one and are grouped by time with runs that were: 39 in cause 1 and 3 in cause 5. A run is "own change" only when its log shows a failure in what the pull request changed and a fix was pushed before it merged (or it was closed). Where the run cannot show the cause, the class is unknown.Baseline for this issue's signal
failed_checks, 4checks_timed_out): 121 of 145 (83.4%) outside the candidate, 4 unknown, 20 own change.Causes, ranked by failed runs
Run-minutes run from start to last update, so they include the heal job.
mainfailed the same wayThe single-run causes: the data product rollout check changing mid-check (#4545); #4873's own message (#4873); an ARC transport test harness failure (unrelated, no issue); a KSail download HTTP 500 (transient); a Coroot admission rollout over 15 minutes (timeout); a Coroot check failing on a Coroot change (own change); a data product UI expected-state error (own change); a leftover route after a refused prune (own change, the case #4734 describes); the GHCR lease that could not be inspected (unknown, closest #3071); and required checks cancelled with no failing step (unknown).
#4839 explains none of these runs: it is about the Release workflow, not merge-group CI.
Decision: what to fix first
The remaining new issues are each small, and the transients (#5012, #5013) are cheap wins once the top three are done.