[safe-output-health] Safe Output Health Report - 2026-10-30

[github-actions[bot]]

🏥 Safe Output Health Report - 2026-10-30

Executive Summary

• Monitoring cadence: ✅ Healthy — 24.0 hours since the previous successful audit (2026-10-29T04:18:57Z, run 40817051215).
• Period: Last 24 hours (2026-10-29T04:18:31Z – 2026-10-30T04:18:31Z). Initial log collection hit the storage/count limit at 2026-10-29T10:45:32Z; a continuation fetch following the returned before_run_id/date-range parameters closed the gap, so this report covers the full 24h window (530 runs combined across both fetches).
• Runs analyzed: 530 (374 + 156 across two fetches)
• Workflows active: 70+ distinct workflows (exact combined unique count not recoverable — see note below)
• Safe Output Jobs Executed: 521 completed job executions across the window
• Safe Output Jobs Failed (in scope): 5, all under a single step signature (safe_outputs / "Process Safe Outputs"), spanning 3 workflows
• Error Clusters Identified: 1 step-level symptom cluster, resolved today into 3 distinct root causes (see below) — a significant investigative advance over prior audits

Note on workflow count: a diagnostic query against the first log batch's summary.json was run after two follow-up agenticworkflows logs calls for deep-dive root-causing; those calls refreshed the cached summary.json in place, so the batch-1 unique-workflow count could not be recomputed afterward. This does not affect the failure analysis below, which was captured before the file was overwritten.

Safe Output Job Statistics

This repository's current compiled workflows process all safe-output types through a single unified safe_outputs job (step name "Process Safe Outputs"), not separate per-type jobs as older architectures used. A full type-by-type success/failure breakdown across all 521 executions would require per-run inspection beyond this audit's scope; the table below reflects the 5 confirmed failures only.

Safe-output type involved in a failure	Failures	Workflow(s)
add_comment	2	PR Sous Chef
create_pull_request_review_comment	2	Matt Pocock Skills Reviewer, PR Code Quality Reviewer
submit_pull_request_review	2	Matt Pocock Skills Reviewer, PR Code Quality Reviewer
comment_memory	1	PR Code Quality Reviewer

Overall job-step success rate: (521 − 5) / 521 ≈ 99.0%.

Error Clusters

This is the long-tracked recurring cluster failure-f3527e2bef0656a6 (job=safe_outputs, step="Process Safe Outputs"), first logged 2026-08-22 and present in nearly every audit since under the working label create-issue-fails-no-result-pr-sous-chef. Today, for the first time in several audit cycles, raw error text was successfully retrieved for 4 of the 5 occurrences (via agenticworkflows logs with artifacts:["all"], filtered by workflow_name + a narrow start_date/end_date window — the audit tool returned only structured, byte-identical summaries with no raw text regardless of the artifacts value tried). This resolves the evidence gap flagged in roughly half of all prior audits and reveals the cluster is not one root cause — it shares a step-level symptom across at least three independent failure modes.

Cluster 1: PR Sous Chef add_comment — "Bad credentials" (401)

• Count: 2 occurrences, 17.5 hours apart
• Affected Jobs: safe_outputs / "Process Safe Outputs"
• Affected Workflows: PR Sous Chef (.github/workflows/pr-sous-chef.lock.yml), both schedule-triggered, engine pi, model copilot/claude-haiku-4.5
• Runs: 40859228926 (2026-10-29T10:46:06Z, 1/1 add_comment failed), 41000452261 (2026-10-30T04:13:55Z, 4/4 add_comment failed)
• Sample Error:

  errorCode: E099
  4 safe output(s) failed:
    - add_comment: Bad credentials - https://docs.github.com/rest
  

• Root Cause: A direct GitHub REST authentication failure (HTTP 401), not a validation/schema issue and not rate limiting. This is new evidence — prior audits (e.g. 2026-10-28/29) attributed this workflow's failures to a create_issue field-shape defect with no url/number in created_items; today's two occurrences are a genuinely different failure mode (confirmed errorCode: E099 with explicit "Bad credentials" text) affecting a different safe-output type (add_comment).
• Impact: Both runs lost 100% of their configured add_comment output. Recurring (2x in ~18h) rather than a one-off blip.

Cluster 2: Installation-wide GitHub API rate limit — concurrent PR-review bots

• Count: 3 occurrences within a single 4-minute window (2026-10-29T10:30:21Z–10:34:19Z)
• Affected Jobs: safe_outputs / "Process Safe Outputs"
• Affected Workflows: Matt Pocock Skills Reviewer (.github/workflows/mattpocock-skills-reviewer.md) and PR Code Quality Reviewer (.github/workflows/pr-code-quality-reviewer.md), both pull_request-triggered
• Runs: 40856105754 (Matt Pocock, PR [duplicate-code] refactor: reuse bounded work-queue content reader #72757), 40856106234 (PR Code Quality Reviewer, same PR [duplicate-code] refactor: reuse bounded work-queue content reader #72757), 40857098425 (PR Code Quality Reviewer, separate PR fix: explain scoped exclusions in protected-file warnings #72920)
• Sample Error:

  errorCode: E099
  create_pull_request_review_comment: Review finalization failed: ERR_API: pulls.createReview github/gh-aw#72757
  failed after 1 retry attempts
  Original error: API rate limit exceeded for installation. ...
  Retryable: true
  Suggestion: All retry attempts exhausted. This may be a persistent issue.
  

• Root Cause: Two independent review workflows (Matt Pocock Skills Reviewer + PR Code Quality Reviewer) both fired on the same PR ([duplicate-code] refactor: reuse bounded work-queue content reader #72757) within seconds of each other, and a third occurrence hit a different PR (fix: explain scoped exclusions in protected-file warnings #72920) in the same narrow window — consistent with multiple workflows sharing and exhausting one GitHub App installation's secondary rate limit on pulls.createReview. A single retry attempt was insufficient to recover.
• Impact: All 3 occurrences lost their entire review output (create_pull_request_review_comment, submit_pull_request_review, and in one case comment_memory too) — not partial failures.

(Known, dormant today) Create_issue missing-result-fields shape

The historically dominant sub-case of this cluster (an agent-side create_issue call that returns with only type/provider/timestamp and no url/number/id) — responsible for the 2026-10-28/29 incident spike (48 occurrences in ~19.6h) — had zero occurrences today. Its root cause remains unconfirmed from raw text (none of today's occurrences matched this shape); it should continue to be tracked separately from the two new causes above.

Root Cause Analysis

API-Related Issues

Both clusters above are, at bottom, API-related: one is an authentication failure (401 Bad credentials), the other is a secondary rate limit (429-class, surfaced as a GitHub REST abuse-detection response) triggered by concurrent installation-wide usage. These are mechanically distinct and should not be conflated — a credential fix will not address the rate-limit cluster or vice versa.

Data Validation Issues

None observed in today's window for in-scope safe-output jobs.

Permission Issues

None observed today beyond the credentials/rate-limit issues above (the previously-tracked approve_workflow_run fork-PR PAT-scope gap and protected-files-decline-miscategorization patterns did not recur today).

Other Issues

None.

Recommendations

Critical Issues (Immediate Action Required)

1. PR Sous Chef add_comment authentication failures ("Bad credentials")
   • Priority: High
   • Root Cause: GitHub REST API returns 401 for add_comment calls from this workflow's safe_outputs job, recurring twice 17.5 hours apart on schedule-triggered runs.
   • Recommended Action: Verify the token provisioning path for PR Sous Chef's safe-outputs job (installation token refresh timing, token scope/expiry, or a race specific to schedule-triggered dispatch). Compare against the still-open 2026-08-31 "Smoke Project Bad credentials" finding (a different workflow, same symptom class) to determine if this is one shared credential-provisioning bug or two independent ones.
   • Affected: PR Sous Chef (schedule-triggered runs specifically)

Bug Fixes Required

1. Add backoff/retry hardening for pulls.createReview under installation rate limits
   • File/Location: Safe-outputs processor's review-finalization path (the component issuing pulls.createReview for create_pull_request_review_comment / submit_pull_request_review)
   • Problem: A single retry attempt is exhausted almost immediately when the installation's secondary rate limit is hit, causing a hard job failure instead of a delayed success.
   • Fix: Increase retry budget and backoff duration (GitHub's secondary rate limits typically clear within 60–120s) specifically for this call path.
   • Affected Jobs: create_pull_request_review_comment, submit_pull_request_review, comment_memory

Configuration Changes

1. Review-workflow trigger concurrency on the same PR
   • Current: Matt Pocock Skills Reviewer and PR Code Quality Reviewer both trigger independently on pull_request and can fire within seconds of each other on the same PR, each making unthrottled Reviews-API calls.
   • Recommended: Stagger/debounce concurrently-triggered review workflows on the same PR event, or coordinate their GitHub API usage, so they do not simultaneously exhaust one installation's rate limit.
   • Reason: 2 of today's 3 rate-limit failures were two different workflows reviewing the exact same PR ([duplicate-code] refactor: reuse bounded work-queue content reader #72757) in the same few seconds.

Process Improvements

1. Root-cause retrieval technique for this long-tracked cluster — confirmed and should be standardized
   • Current State: Most prior audits (roughly half, per cache-memory history) reported "no raw error text retrievable" for this exact cluster and resorted to grader-inferred guesses.
   • Proposed: This audit confirms agenticworkflows logs (not agenticworkflows audit, which returns only structured summaries regardless of the artifacts parameter) with artifacts:["all"], filtered by workflow_name plus a narrow start_date/end_date window bracketing the known failing run, reliably downloads safe-output-errors.json with full errorCode/message/failures[] text. This should be the default first move for any future "Process Safe Outputs" / safe_outputs-job investigation, replacing the structured-summary-only approach.
   • Benefits: Converts a step-level symptom cluster that has been tracked (and periodically misattributed to a single cause) since 2026-08-22 into concretely falsifiable, independently fixable root causes.

Work Item Plans

Work Item 1: Investigate PR Sous Chef add_comment "Bad credentials" (401)

• Type: Investigation / Bug Fix
• Priority: High
• Description: Two schedule-triggered PR Sous Chef runs (2026-10-29T10:46Z and 2026-10-30T04:13Z) failed 100% of their add_comment safe-output items with a direct GitHub REST 401 "Bad credentials" error. Determine whether the credential used by this workflow's safe_outputs job is expiring, being incorrectly rotated, or affected by a schedule-trigger-specific race.
• Acceptance Criteria:
  • Root cause of the 401 identified (token expiry, scope, rotation race, or other)
  • Fix applied or credential provisioning corrected
  • No recurrence across the next 3 audit cycles
• Technical Approach: Inspect the token/credential path feeding PR Sous Chef's safe_outputs job specifically for schedule-triggered dispatches; compare with other schedule-triggered workflows that did not fail in this window.
• Estimated Effort: Medium
• Dependencies: None known

Work Item 2: Harden review-finalization retries against installation rate limits

• Type: Bug Fix
• Priority: Medium
• Description: pulls.createReview (backing create_pull_request_review_comment and submit_pull_request_review) exhausts its single retry attempt almost immediately when an installation-wide secondary rate limit is hit, turning a transient, short-lived condition into a hard job failure that loses 100% of the review's output.
• Acceptance Criteria:
  • Retry budget/backoff duration increased for this call path
  • Verified recovery from a simulated/observed rate-limit condition without a full job failure
• Technical Approach: Add exponential backoff with a longer ceiling (60–120s+) specifically for secondary-rate-limit responses on the Reviews API, consistent with GitHub's documented clearing time.
• Estimated Effort: Small–Medium
• Dependencies: None known

Historical Context

This failure signature (failure-f3527e2bef0656a6, job=safe_outputs, step="Process Safe Outputs") is the single longest-running recurring pattern tracked by this monitor, first logged 2026-08-22 and appearing in the majority of audits since, almost always on PR Sous Chef. Its working label (create-issue-fails-no-result-pr-sous-chef) reflected only its most frequently-observed sub-case. Yesterday's audit (2026-10-29) reported a 48-occurrence incident spike in that sub-case, self-resolved by the time of reporting, with the evidence gap for raw error text still open after 59 cumulative occurrences across 3 reports.

Today's audit materially advances this investigation: for the first time, raw error text was retrieved for 4 of 5 occurrences, and none of them match the previously-dominant missing-result-fields shape. Instead, two entirely distinct and independently actionable root causes were confirmed — an authentication failure and an installation rate limit — demonstrating that this long-tracked cluster is actually an umbrella over multiple unrelated underlying bugs that happen to share the same step-level symptom (safe_outputs job / "Process Safe Outputs" step failing). Future audits should track these as separate named issues rather than continuing to roll them into one "PR Sous Chef Process Safe Outputs" bucket.

Trends

• Error rate trend: Volatile — 13 failures (2026-10-28), 48 failures/incident spike (2026-10-29), 5 failures today (2026-10-30, back near the ~1-13/day baseline typical of most of the last 10 weeks).
• Most common recurring issue: The failure-f3527e2bef0656a6 step-level symptom remains the single most persistent pattern in this monitor's 10-week history, but is now understood to be multi-causal rather than one bug.
• Improvement since last audit: The long-standing "no raw error text retrievable" evidence gap — present in roughly half of all prior audits for this exact cluster — did not recur today; root causes for 4 of 5 occurrences are now concretely known rather than inferred.

Metrics and KPIs

• Overall Safe Output Job Success Rate: ≈99.0% (516/521 completed safe_outputs job executions succeeded)
• Most Reliable Job Type: All safe-output types other than add_comment, create_pull_request_review_comment, submit_pull_request_review, and comment_memory had zero observed failures in this window
• Most Problematic Job Type (today): add_comment (PR Sous Chef) and the Reviews-API-backed types (create_pull_request_review_comment, submit_pull_request_review) tied, 2 failing runs each
• Average Time to Failure: The "Process Safe Outputs" step itself fails fast (consistently ~4–47 seconds in; most under 10s) in all 5 occurrences — the job is not timing out, it is receiving hard errors quickly

Next Steps

• Investigate and resolve the PR Sous Chef add_comment 401 "Bad credentials" authentication failure (Work Item 1)
• Harden pulls.createReview retry/backoff against installation-wide rate limits (Work Item 2)
• Continue monitoring for recurrence of the historically-dominant create_issue missing-result-fields shape (dormant today, root cause still unconfirmed after 59+ cumulative occurrences)
• Standardize the agenticworkflows logs ... artifacts:["all"] + narrow workflow/date-window technique as the default first step for future "Process Safe Outputs" investigations

Generated by 🔒 Safe Output Health Monitor · claude · agent · ~333 AIC · ⌖ ~1 AIC · ⊞ 4.6K · ◷

• expires on Oct 30, 2026, 8:49 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-31T04:49:25.791Z.

Closed by Workflow