Count why merge-queue candidates failed over the last 30 days, to rank the deploy-eviction causes #4993

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence

• Production deploys fail only for the change they deploy #4979 asks for an eviction-cause baseline before any fix is chosen, and names this as its first child. No issue held it yet.
• In the 30 days to 2026-10-17, the CI workflow ran 871 times for merge-queue candidates: 698 succeeded, 160 failed and 12 were cancelled (read from the Actions run list, event=merge_group, ci.yaml). A failed required run removes its candidate from the queue; part of the spike is confirming how many of the 160 did.
• Eight open bugs under Production deploys fail only for the change they deploy #4979 each describe one cause, but nobody has counted how many of the 160 failures each one explains, or how many have a cause no issue names.

Problem

Without the count, the next fix is chosen by whichever failure was seen last, not by which one costs the most queue time.

Spike: what to decide

1. For every failed CI merge-group run in the window 2026-09-17 to 2026-10-17, record the failing job and step and put the cause in one class: candidate's own change, unrelated platform fault, infrastructure transient (runner, registry, network, GitHub), timeout, or unknown. Read the failing step's own output; where the run cannot show the cause, the class is unknown, never a guess.
2. Map each failure to the open child of Production deploys fail only for the change they deploy #4979 that explains it (Production deploys fail intermittently at the data product rollout check and evict unrelated pull requests #4545, fix(deploy): an unrelated Coroot fault must not refuse every production deploy #4659, Avoid redeploying production after an evicted candidate was refused before deployment #4937, Deploy GHCR step can time out on the Flux policy pause without saying which condition never held #4873, Talos node upgrades can exceed the 10-minute drain timeout, evicting the bump from the merge queue #3388, The deploy bridge loses its own sync lease mid-run, and each failure fences a node that fails the next deploy #3071, A deploy passes its Flux step when Flux could not delete a removed resource #4734, Complete the Platform release left after a rejected notes push #4839), or to "no issue".
3. Rank the causes outside the candidate by the number of runs they failed, and by queue time lost where the run shows it.

Acceptance criteria

• A table on Production deploys fail only for the change they deploy #4979 with one row per cause: class, matching issue or "no issue", runs failed, share of the 160.
• The baseline number for Production deploys fail only for the change they deploy #4979's signal — the share of failures whose cause lies outside the candidate — recorded on Production deploys fail only for the change they deploy #4979.
• A follow-up issue filed under Production deploys fail only for the change they deploy #4979 for each cause that failed at least three runs and has no issue yet.
• The decision on which child to fix first, with the reason, recorded on Production deploys fail only for the change they deploy #4979.
• The public record carries no secret values, cluster addresses or pod IPs; it quotes step names and summarised messages only.

Notes

Read-only: GitHub run, job and log reads only, no cluster access and no re-runs. The run list returns at most 1,000 rows per query, so query ci.yaml alone (871 rows fits) and check the row count against the run list total.

Size: small to medium — about 160 failed runs to classify; scriptable from the job and step conclusions, with log reads only where the step name does not settle the class.

Priority rationale

Priority High, Effort Medium, Service class Standard. It is the measure-first slice of #4979 and decides the order of its eight bug children; each failed candidate costs a queue slot and often a restore deploy, at about five failures a day. Read-only, no prerequisites, no deploy or review risk beyond one decision record.