[agent-job-health] Agent Job Health: 15.1% fleet failure rate (2026-10-21) - 1 novel MCP Gateway cluster

[github-actions[bot]]

Summary

Window evaluated: 2026-10-20 23:18 UTC → 2026-10-21 23:18 UTC (last 24 full hours)

• total_runs analyzed: 384 (398 agentic-workflow runs were identified in the window; 14 runs / 3.5% were pruned from the local log cache by the storage budget during continuation fetches, so coverage is ~96.5%, not 100%)
• distinct_workflows: 173
• agent_failures: 58
• Run-weighted fleet failure rate: 58 / 384 = 15.10%
• Workflow-weighted rate: median 0.0%, mean 11.75% (173 workflows, 146 of them had zero agent-job failures)
• Tracked vs. novel: 30 failures (51.7%) match a previously-identified, explicitly-accepted pattern (not re-flagged); 5 failures (8.6%) are a genuinely new cluster; 23 failures (39.7%) are a fragmented long tail of isolated single-workflow incidents below the clustering bar.
• Trend: -2.07pp vs 2026-10-20 (17.17%), +0.67pp vs 2026-10-19 (14.43%) — within normal day-to-day fluctuation, not a >10pp regression.

Warning

One novel cluster was found this window (MCP Gateway startup readiness timeout, below), affecting 5 distinct workflows simultaneously. An issue has been opened. The large gap between the run-weighted rate (15.10%) and the workflow-weighted median (0.0%) confirms the fleet number is dominated by a handful of high-frequency PR-reviewer workflows rather than reflecting broad fleet degradation.

Failure Rate by Step

All 16 distinct failing steps (58 total agent-job failures)

Failing step	Runs	Share of agent_failures
Checkout PR branch	30	51.7%
Start MCP Gateway	5	8.6%
Check Serena Go semantic analysis	4	6.9%
Execute GitHub Copilot CLI	4	6.9%
Execute Pi CLI	3	5.2%
Checkout github/gh-aw-side-repo into gh-aw-side-repo	2	3.4%
Assert Goose smoke evidence	1	1.7%
Assert declared sub-agent ran on its resolved model	1	1.7%
Assert sub-agent model and wire API evidence	1	1.7%
Build gh-aw Docker image	1	1.7%
Checkout drive-memory file share (default)	1	1.7%
Enforce vendored vulnerability and license gates	1	1.7%
Execute Aider CLI	1	1.7%
Execute Codex CLI	1	1.7%
Execute DeepSeek Harness	1	1.7%
Execute Gemini CLI	1	1.7%

Tracked Failures

Signature	Workflows	Failing runs	Status
Checkout PR branch failure — git fetch fails with fatal: couldn't find remote ref <branch> because the PR's head branch was deleted (merge + auto-delete) between when the run was queued and when the checkout step executed	Design Decision Gate 🏗️, Matt Pocock Skills Reviewer, PR Code Quality Reviewer, Ponytail Reviewer, Test Quality Sentinel	30	No open GitHub issue tracks this signature by title/text search, but this exact signature/workflow-set was already identified and explicitly decided to be expected behavior, not a bug in this report's own trend history on 2026-10-19 and 2026-10-20 (both noted "no issue opened"). Treated as tracked/accepted again this run for consistency — see Novel Clusters below for the confirmed root cause (verified via job logs this run: PR #70676, branch pelikhan-special-robot, merged 2026-10-21T00:04:04Z, workflow run started 85s later at 00:05:29Z).

Novel Failure Clusters

Cluster: Start MCP Gateway readiness timeout — issue opened

• Count: 5 runs across 5 distinct workflows: Design Decision Gate 🏗️, Matt Pocock Skills Reviewer, PR Code Quality Reviewer, Ponytail Reviewer, Test Quality Sentinel
• Failing step: Start MCP Gateway
• Engines: pi, codex, copilot (mixed engines rules out an engine-specific cause)
• Representative run: 39529635821 (Ponytail Reviewer), all 5 fired on the same pull_request event at 2026-10-21T01:50:41Z
• Error excerpt:

  [error] FATAL: MCP gateway safeoutputs readiness failed after 137 attempt(s): no tools registered or backend unavailable. The agent will not start. Check MCP gateway startup logs for connection resets (ECONNRESET) or delayed backend registration. Last probe: insufficient startup budget for a readiness probe (requires at least 750ms). (waited ~137s before giving up)

• Suspected root cause: these 5 PR-reviewer workflows all trigger on the same pull_request event and start concurrently. In this instance all 5 timed out waiting for the safeoutputs MCP gateway backend to register its tools — consistent with resource contention or a shared-backend hiccup when 5 gateway instances start at once for the same commit.
• Blast radius: same 5-workflow reviewer set that also exhibits the (tracked/accepted) checkout-branch race, but this is a different failure point and, per search, not covered by an existing open issue.

Long tail (not actionable as a cluster): 23 of 58 failures are isolated single-run, single-workflow incidents spread across 22 different workflows — mostly Smoke * canary workflows exercising individual engine configurations (Pi, Codex, Aider, Copilot variants, DeepSeek, Gemini, Goose, Claude sub-agents) plus a handful of one-off production workflow failures (Daily Community Attribution Updater, The Great Escapi, Daily Container Image Security Scan, Package Specification Extractor, Blog Auditor, one gh-aw work-queue job). None meet the 2-distinct-workflow or 10%-of-runs bar individually, so none are clustered or issued separately this run.

Schedule Heartbeat

Workflow	.lock.yml	Last observed run	Expected cadence	Gap	State
Daily Geo Optimizer	daily-geo-optimizer.lock.yml	2026-08-18	daily	~64d	disabled_manually (explains gap)
Daily Hippo Learn	daily-hippo-learn.lock.yml	2026-07-27	daily	~86d	disabled_manually (explains gap)
Slide Deck Maintainer	slide-deck-maintainer.lock.yml	2026-09-23	daily	~28d	disabled_manually (explains gap)
Smoke CI	smoke-ci.lock.yml	2026-07-18	daily	~95d	disabled_manually (explains gap)

No new blind spots detected among the workflows checked. These same 4 workflows have been found disabled_manually in every report since 2026-08-23 — stable, explained state, not a silent misfire.

Coverage caveat: 61 of ~249 schedule-triggered workflows were checked this run (prioritizing the daily/sub-daily tier). ~30 additional daily-named workflows and the entire weekly/monthly-cadence tier (~160+ workflows) were not checked this run due to budget — relying on 10 consecutive prior reports (since 2026-08-23) finding no blind spot beyond the 4 above as supporting context, not as a substitute for direct verification.

Out-of-scope observation (flagged for awareness, not counted above): daily-report-dispatcher.lock.yml ("Daily Discussion Report Dispatcher") is still active and firing daily, but its work-queue dispatch job (not the agent job measured by this report) has failed 4 of its last 5 overall runs. As a result, 6 downstream report workflows that are invoked only via workflow_dispatch from this dispatcher — daily-compiler-quality, daily-evals-report, daily-firewall-report, daily-issues-report, daily-observability-report, daily-regulatory — have gone silent for 13-14 days. These workers have no schedule: trigger of their own, so they don't meet this report's Phase 6 definition of a schedule blind spot, and the dispatch job isn't an "agent job" in this report's scope — but the pattern is real and worth routing to whoever owns daily-report-dispatcher.

View per-workflow breakdown (27 of 173 workflows had ≥1 agent-job failure; 146 had zero)

Workflow	Runs	Agent failures	Failure rate
Test Quality Sentinel	26	8	30.8%
Matt Pocock Skills Reviewer	25	7	28.0%
PR Code Quality Reviewer	24	7	29.2%
Ponytail Reviewer	23	7	30.4%
Design Decision Gate 🏗️	24	6	25.0%
Smoke Copilot Sub Agents	5	2	40.0%
Agent Container Smoke Test	1	1	100%
Blog Auditor	1	1	100%
Daily Community Attribution Updater	1	1	100%
Daily Container Image Security Scan	1	1	100%
Package Specification Extractor	1	1	100%
Smoke Aider	2	1	50.0%
Smoke Azure Codex	1	1	100%
Smoke Azure PI	1	1	100%
Smoke Claude Sub Agents	3	1	33.3%
Smoke Copilot - AOAI (Entra)	1	1	100%
Smoke Copilot - AOAI (apikey)	1	1	100%
Smoke Copilot ARM64	1	1	100%
Smoke Copilot SDK Routed Claude	3	1	33.3%
Smoke Create Cross-Repo PR	1	1	100%
Smoke DeepSeek Harness	1	1	100%
Smoke Drive	1	1	100%
Smoke Goose	1	1	100%
Smoke Pi Routed	3	1	33.3%
Smoke Update Cross-Repo PR	1	1	100%
The Great Escapi	1	1	100%
gh-aw work-queue (worker job)	1	1	100%

Recommendations

1. Investigate concurrent MCP Gateway startup contention (new issue opened) — when the 5 PR-reviewer workflows (Design Decision Gate, Matt Pocock Skills Reviewer, PR Code Quality Reviewer, Ponytail Reviewer, Test Quality Sentinel) all fire on the same pull_request event, consider staggering gateway starts or increasing the safeoutputs readiness budget/timeout to avoid the simultaneous 137s-timeout failure seen this window.
2. No action needed on the Checkout PR branch / obsolete-PR race — consistent with the explicit "not a bug, no issue" decision recorded in this report's history on 2026-10-19 and 2026-10-20; root cause reconfirmed this run (PR merged + branch auto-deleted before checkout ran).
3. Route the daily-report-dispatcher work-queue failure to its owner — 6 downstream daily reports have been silently stuck for 13-14 days because the dispatcher's dispatch job (not its agent job) is failing 4 of 5 recent runs. This falls outside this report's agent-job/schedule-trigger scope, but is a real production gap worth separate follow-up.
4. No fleet-wide regression action needed — the run-weighted rate decreased from 17.17% to 15.10%, workflow-weighted median remains 0%, and the headline rate continues to be explained by a small set of high-frequency, already-understood clusters rather than broad degradation.

Generated by 🩺 Agent Job Health Monitor · claude · agent · 797.1 AIC · ⌖ 0.94 AIC · ⊞ 6.9K · ◷

• expires on Oct 24, 2026, 3:54 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-24T23:54:59.706Z.

Closed by Workflow