[safe-output-health] Safe Output Health Report - 2026-10-29

[github-actions[bot]]

🏥 Safe Output Health Report — 2026-10-29

Scope: safe-output job failures only (create_discussion, create_issue, add_comment, create_pull_request, etc.). Agent/detection job failures are out of scope (covered by other monitors). Window: 2026-10-28T04:19Z → 2026-10-29T04:19Z (last 24h).

Overview

• Monitoring cadence: Healthy — 24.0h since the previous successful audit (§40631618037, started 2026-10-28T04:18:52Z).
• Runs analyzed: 476 (full 24h window, collected across 3 paginated fetches; 458 executed, 15 non-execution, remainder in-progress).
• Workflows active: 206 distinct workflows.
• Safe-output (safe_outputs) job failures confirmed: 48, all with the identical signature: the Process Safe Outputs step fails while the rest of the job (download artifact, upload items) succeeds.
• Error clusters: 3 (one fully resolved multi-hour incident, one fleet-wide blip within the same incident, one unrelated isolated flake).
• Headline finding: this is a known, previously-escalated recurring issue (create-issue-fails-no-result-pr-sous-chef, first seen 2026-09-22, escalated to High/Critical in the 2026-10-28 report after a 12-occurrence spike) that got dramatically worse today (46 occurrences, ~3.8x the prior-day count) before self-resolving around 2026-10-28T15:49-16:00Z. It has stayed clean for the ~12.5h since. Root cause is still unconfirmed — raw step logs have never been successfully retrieved across any of the now 59 total occurrences (13 historical + 46 today).

Safe Output Job Statistics

Workflow (job: safe_outputs)	Runs in window	Process Safe Outputs failures	Failure rate
PR Sous Chef	100 (48 overall-failure, 51 success, 1 in-progress)	46	46%
Daily Container Image Security Scan	1	1	100% (single daily run)
Smoke Project	-	1	n/a (smoke/conformance test)
All other 203 workflows	~375	0	0%

All 476 run directories were scanned for job_details[].name == "safe_outputs" with a non-success, non-skipped conclusion — no other workflow safe-outputs job failed in this window, and no legacy per-type job names (create_issue, add_comment, etc.) exist anymore; every workflow routes through the single consolidated safe_outputs job.

Error Clusters

Cluster 1 (dominant): PR Sous Chef extended outage, 2026-10-28T07:25Z-15:49Z (38 consecutive failures)

• Count: 38 consecutive failed runs, one every ~10-13 minutes for ~8.4 hours straight.
• Job / step: safe_outputs -> Process Safe Outputs (step 8), consistently ~4s duration before failing.
• Pattern: in every single failure, exactly one create_issue safe-output item was still recorded as successfully created (full type/provider/timestamp, same shape previously flagged as a possible "missing result fields" symptom) — but add-comment, approve-workflow-run, and dismiss-pull-request-review (all configured for this workflow) never appear as succeeded items in any of the 46 failures, and Upload Safe Outputs Items still succeeds immediately after.
• Representative runs: §40719833571 (16:01Z, last failure before recovery — only one with a downloaded audit.json), §40683971641 (middle of the streak), §40649991672 (near the start of this sub-window).
• Recovery: §40721426251 (first clean run after) and every subsequent PR Sous Chef run through end-of-window are green except one unrelated failure (§40747618019, 18:42Z — overall run failed, but the safe_outputs job itself was not the cause; out of scope here).

Cluster 2: Fleet-wide blip, 2026-10-28T04:31Z-05:38Z (9 failures across 2 workflows)

• Count: 8 PR Sous Chef failures + 1 new workflow never seen in this pattern before: Daily Container Image Security Scan (§40638214806, 05:28:36Z).
• Same exact signature as Cluster 1 (Process Safe Outputs fails, create_issue item still recorded).
• This is the first time a workflow other than PR Sous Chef has shown this failure, which weakens the "PR-Sous-Chef-specific config" theory (its close-older-issues/group-by-day create-issue options) and strengthens a shared safe-outputs runtime/dependency regression theory for at least this sub-window — the Container Image Security Scan create-issue config has no close-older-issues/group-by-day at all.
• This sub-window is contiguous with the end of the prior-day already-escalated spike (last occurrence reported in the 2026-10-28 report: 2026-10-28T04:00:04Z) and the start of Cluster 1 — i.e., the incident was effectively continuous from ~2026-10-27T20:13Z through 2026-10-28T15:49Z (~19.6 hours), with only a ~1h45m quiet gap (05:38-07:25Z) where PR Sous Chef apparently had no eligible PRs to process.

Cluster 3 (minor, likely unrelated): Smoke Project, 2026-10-29T00:09:54Z (1 occurrence)

• §40792230830 — the safe_outputs job Process Safe Outputs step failed, but zero safe-output items were recorded (unlike Clusters 1-2, which always have exactly one create_issue).
• Different failure shape; smoke-project.md is a conformance/smoke-test workflow. No prior history of this pattern for this workflow. Treat as a low-priority one-off pending a second occurrence.

Root Cause Analysis

Still unconfirmed after 59 total occurrences (13 historical + 46 today). As flagged in the 2026-10-28 report, this compact log-monitoring pipeline has access to structured run_summary.json/audit.json (job/step conclusions, timing, recorded safe-output items) but no raw stderr/stdout text for the Process Safe Outputs step — confirmed again today via agenticworkflows audit with every available artifacts set (outcomes, all); none return step console output. The only way to get the actual exception/stack trace is a direct gh run view --log --job <safe_outputs-job-id> against a live run, which this read-only monitor cannot do.

Two competing hypotheses remain open, now with more evidence for each:

1. Benign no-op treated as a hard failure: close-older-issues/group-by-day processing (or an analogous duplicate-detection step) finds an existing issue, does not populate a full result object, and the handler throws instead of treating it as success. This fits all 46 Cluster-1 occurrences (PR Sous Chef has close-older-issues: true, group-by-day: true).
2. Shared safe-outputs runtime regression: Cluster 2 new victim (Container Image Security Scan, which has no close-older-issues/group-by-day config) was hit with the identical signature in the same time window, which theory (1) alone does not explain.

Both can be true simultaneously (a shared regression that happens to manifest most visibly through PR Sous Chef aggressive 5-minute schedule). The incident clean self-resolution by ~16:00Z, with no corresponding create-issue handler code change found in git log for that window (the only merged commit in 14:00-17:00Z PT changed work-queue footer logic, #72632, not safe-outputs code), suggests this may be an infra-side transient (e.g. a backing API dependency, MCP gateway, or token/rate issue) rather than a code regression that was "fixed" by a merge.

Recommendations

Critical / Immediate

1. Retrieve raw logs for the safe_outputs job of §40719833571 (last failure in the streak, 2026-10-28T16:01Z) via gh run view 40719833571 --log --job <safe_outputs-job-id> from an authenticated environment. This is the single highest-value action to finally confirm root cause after 59 occurrences across 5+ weeks.
2. Re-escalate and broaden the existing tracked issue (investigate-create-issue-failure-pr-sous-chef, proposed 2026-09-22, escalated 2026-10-28) to cover the new Container Image Security Scan occurrence — this is no longer a single-workflow issue.

Bug Fixes

1. Create-issue handler (close-older-issues/group-by-day path or equivalent dedup logic): if a duplicate/dedup case is found, it should populate the created_items entry with the existing issue url/number and return success, not fail the step after a partial no-op. File/location: the safe-outputs create-issue processing script referenced by pkg/workflow/safe_outputs_jobs.go / safe_outputs_runtime.go (exact function needs confirmation once raw logs are available).

Process Improvements

1. Capture step-level stderr for Process Safe Outputs failures automatically (e.g., persist a safe-output-errors.json or workflow-logs/safe_outputs.txt artifact on failure) so this daily monitor can self-diagnose without a manual gh run view --log. This exact gap has now blocked root-causing the same issue across two consecutive daily reports and 59 occurrences.
2. Add an explicit alert threshold for the safe_outputs job failure rate per workflow (e.g., >20% of runs in a rolling 6h window) so a recurrence of this incident triggers a same-day page rather than waiting for the next daily audit.

Work Item Plans

Work Item 1: Confirm root cause of Process Safe Outputs step failures (PR Sous Chef + Container Image Security Scan)

• Type: Investigation -> Bug Fix
• Priority: Critical
• Description: 59 occurrences (13 over 2026-09-22 to 2026-10-28, 46 on 2026-10-28) of Process Safe Outputs failing after a create_issue item is still recorded as created. Incident self-resolved ~2026-10-28T16:00Z; no corresponding code fix identified in git history for that window.
• Acceptance Criteria:
  • Raw log text retrieved for at least one occurrence (recommend §40719833571)
  • Root cause classified as handler bug vs. infra transient
  • If handler bug: fix shipped and verified against a reproduction
  • If infra transient: dependency identified and either monitored or hardened with retry
• Estimated Effort: Medium
• Dependencies: Authenticated access to run logs (not available to this monitor)

Work Item 2: Add raw-log capture for safe-outputs job failures

• Type: Enhancement
• Priority: High
• Description: This monitor has now been blocked twice in a row by the absence of step console text in the structured audit data it can access.
• Acceptance Criteria:
  • Process Safe Outputs step persists stderr/stdout to a downloadable artifact on non-zero exit
  • This monitor log collector picks it up automatically
• Estimated Effort: Small
• Dependencies: None

Historical Context

• 2026-09-22: First occurrence of this exact pattern (1 run), then dormant 5+ weeks.
• 2026-10-28 (prior-day audit): Spiked to 12-13 occurrences in an ~8h window, escalated to High/Critical, confirmed PR-Sous-Chef-only at the time.
• 2026-10-29 (today): Spiked again to 46 occurrences in PR Sous Chef (3.8x prior-day count) plus a first-ever occurrence in a second workflow (Container Image Security Scan), spanning a combined ~19.6h near-continuous incident from 2026-10-27T20:13Z through 2026-10-28T15:49Z — then fully recovered and has stayed clean for ~12.5h since.
• Trend: Escalating in frequency and breadth through 2026-10-28, then abrupt resolution. Root cause confidence remains low-medium (frequency/recurrence well-established; exact mechanism still unconfirmed for the third audit in a row).
• The prior-day report also flagged a since-resolved resolve-review-thread/dismiss-pull-request-review pattern and noted the previously-dominant approve_workflow_run failure mode had gone quiet — neither reappeared in today window.

Metrics and KPIs

• Safe-output job failure count (in scope): 48 / ~458 executed runs in window with a safe_outputs job present
• Most reliable safe-outputs job type: all configured types other than the create_issue edge case (dedup/no-op path) — 0 failures across 203 other active workflows
• Most problematic workflow: PR Sous Chef (46 of 48 failures, 46% of its runs in-window)
• Time to recovery: ~16h after the prior-day escalation, ~19.6h after incident onset — self-resolved, not attributable to a tracked code fix

Next Steps

• Pull raw gh run view --log output for the §40719833571 safe_outputs job
• Re-escalate the tracked issue to include Container Image Security Scan
• Add stderr-capture artifact for Process Safe Outputs failures (Work Item 2)
• Monitor next 24h for recurrence now that the incident appears resolved

References:

• §40719833571 — last failure before recovery, PR Sous Chef
• §40638214806 — Container Image Security Scan, fleet-wide blip
• §40792230830 — Smoke Project, isolated unrelated failure

Generated by 🔒 Safe Output Health Monitor · claude · agent · ~596 AIC · ⌖ ~1 AIC · ⊞ 4.6K · ◷

• expires on Oct 29, 2026, 8:59 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #73106.