💡 Canary Gate Failure Classification Taxonomy & Structured Verdict Standard

[github-actions[bot]]

Summary

Formalize a failure classification taxonomy for the canary-rollout gate that distinguishes infrastructure failures (evicted runs, quota exhaustion, runner timeouts) from genuine candidate regressions. Emit structured verdicts with a failure_class field so auto-triage, dashboards, and promotion decisions can filter out noise. The canary gate is the project's #1 bug source by volume — 15+ issues filed in the past 30 days, each a different failure mode misclassified as a regression.

Market Signal

Industry canary systems (Argo Rollouts, Flagger, LaunchDarkly) all expose structured failure reasons with enumerated states. GitHub's own deployment status API uses enumerated conclusion types. The org's canary gate currently outputs a single pass/fail verdict without classifying why a run failed, forcing human investigation of every false positive. Recent platform improvements (Sept 2026 job.workflow_ref context, runner deprecation API) show the industry trend toward structured metadata in CI systems.

User Signal

Issues #1047 (concurrency-evicted runs counted as failures), #1054 (eviction exclusion follow-up), #1057 (Claude 429 quota failures counted as regressions), #1066 (autocut/promote timing causing dwell resets), #1086 (gate counting non-candidate-executing runs), #1090 (unoccupiable ring tier), #1102 (frontier resets making outer rings unreachable), #1118 (newest candidate cancelling pending confirmations), #1119 (bash 3.2 compatibility in cut script), #1120 (referenced_workflows attribution), #1121 (dwell/sample coverage), #1123 (timeout step name mismatch) — each a distinct false-positive class. Dev-lead stable was frozen for weeks because every new cut cancelled pending ring1→stable human confirmations (#1118).

Technical Opportunity

The canary gate decision logic lives in scripts/lib/canary-rollout.sh — a pure, side-effect-free, unit-tested script (per AGENTS.md "Decision Logic Lives in a Pure, Tested Script" pattern). The evaluate function already receives run metadata (conclusion, steps, referenced_workflows). Adding a failure_class enum (CANDIDATE_REGRESSION, INFRASTRUCTURE, QUOTA_EXHAUSTION, CONCURRENCY_EVICTION, NO_COVERAGE, TIMEOUT_INFRASTRUCTURE) is a natural extension of the existing data model. The bats test suite (tests/canary_rollout.bats) can cover every classification branch pre-merge with zero canary-time cost.

Proposed enum values and detection heuristics:

failure_class	Heuristic	Example issues
CANDIDATE_REGRESSION	Run executed the candidate workflow, completed, and failed in candidate-specific steps	(genuine regression)
INFRASTRUCTURE	Run failed in infrastructure steps (runner setup, checkout, dependency install)	#1119
QUOTA_EXHAUSTION	Run failed with engine-quota errors (429, rate limit, session cap)	#1057
CONCURRENCY_EVICTION	Run was cancelled with 0 steps executed (GitHub concurrency group eviction)	#1047
NO_COVERAGE	Run completed but did not execute the candidate workflow (referenced_workflows check)	#1086, #1120
TIMEOUT_INFRASTRUCTURE	Run timed out in a non-candidate step	#1123

Assessment

Dimension	Score	Rationale
Feasibility	high	Pure function extension in an already-tested script; enum + heuristics
Impact	high	Eliminates the #1 bug class by volume; unblocks canary promotions
Urgency	high	15+ bugs in 30 days; dev-lead stable frozen; active pain

Adversarial Review

Strongest objection: The individual bugs are being fixed tactically — eviction exclusion (#1047), quota exclusion (#1057), attribution fix (#1086). Each gets its own PR. Why add a taxonomy on top?

Rebuttal: Tactical fixes address known symptoms; the taxonomy addresses the root cause — the gate has no vocabulary for "this failure is not the candidate's fault." Each new infrastructure failure mode (a new runner image, a new rate limit, a new concurrency policy) will produce the same false-regression pattern until the gate can classify structurally. The 15-bug cluster in 30 days proves the current symptom-by-symptom approach doesn't scale. The taxonomy is the difference between whack-a-mole and a systematic safety net.

Suggested Next Step

Define the failure_class enum values and their detection heuristics in an ADR. Implement the classification in scripts/lib/canary-rollout.sh with bats test coverage for each class. Update the canary gate evaluate output to include the failure_class field. Modify the promotion logic to exclude non-CANDIDATE_REGRESSION failures from cum_fail by default.