Skip to content

Treat Claude usage caps and expired logins as fatal - #156

Merged
AbirAbbas merged 6 commits into
Agent-Field:mainfrom
i2020github:fix/claude-subscription-fatal-errors
Oct 20, 2026
Merged

AbirAbbas merged 6 commits into
Agent-Field:mainfrom
i2020github:fix/claude-subscription-fatal-errors

Conversation

@i2020github

@i2020github i2020github commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Final design (maintainer takeover, 2026-10-20). This replaces the "What changed" bullets below where they disagree.

  • _FATAL_PATTERNS / is_fatal_error are unchanged from main. No new substring patterns, so schema diagnoses, transient auth/network errors and failed envelopes that quote cap text stay recoverable.
  • Claude caps, extra usage, session limits and auth failures are fatal only when the session contains the CLI's synthetic provider reply (model="<synthetic>", which survives the pinned SDK's parser and AgentField's message copy). Raw-only results count only if the whole completion is exactly one known CLI reply, generated from the pinned claude-agent-sdk 0.1.20 / CLI 2.1.9 formatters (34 fixtures, including the API Error: 401/403 … · Please run /login forms).
  • A failed execution envelope propagates FatalHarnessError across processes only when its error starts with Fatal API error (non-retryable):, which is how AgentField serializes the raised exception. The marker quoted anywhere else is ordinary output.

What changed and why

When the claude_code runtime runs on a Claude subscription and hits a usage cap, every call fails identically until the cap resets. The same happens when its OAuth login expires and can't be refreshed. The Claude Code CLI reports this only as a synthetic assistant reply, for example:

  • You've hit your session limit · resets 9:50pm (UTC)
  • Failed to authenticate: OAuth session expired and could not be refreshed

The harness then reports a generic Schema validation failed … The output file was NOT created, so check_fatal_harness_error never fires. Observed during multi-hour builds:

  • coders return no changes, reviewers "fail — not blocking", advisors fail and issues are "accepted with debt";
  • whole DAG levels (8 and later 28 issues) were recorded as completed with zero files changed within minutes, and the build reported success.

This PR makes these conditions fatal so the build stops cleanly and can be resumed:

  • check_fatal_harness_error also scans the raw completion and HarnessResult.messages for the CLI's exact replies. During planning the raw completion is empty and the reply survives only in the messages. The raw-text check is anchored to the start of the reply, and the messages check requires the CLI's … · resets / Failed to authenticate: OAuth phrasing, so model prose that mentions limits or authentication does not trip it.
  • _FATAL_PATTERNS gains hit your … limit, usage limit reached, out of extra usage, failed to authenticate and oauth session expired.

Behaviour changes to orchestration

The changes only take effect once a result is already is_error. In that case these replies now raise FatalHarnessError instead of being retried or failing open.

A related issue, not addressed here: when a fatal error fails some issues mid-level, their dependents are skipped, yet build can still report succeeded. Callers chaining builds need to check failed_issues / skipped_issues in the checkpoint.

Test evidence

  • New TestClaudeUnavailableReplies covers replies in the raw completion and in session messages, model prose that must not raise, and successful results not being inspected. The new phrases are also added to the is_fatal_error parametrisation.
  • Full suite on this branch (rebased on main @ 4d1f11c): 1346 passed, 1 skipped.
  • Verified live: a session-limit hit now stops the build with Fatal API error (non-retryable): Claude unavailable: You've hit your session limit · resets 7pm (UTC), and a later resume_build continues from the checkpoint.

The Go node has no equivalent patterns yet and likely needs the same handling.

🤖 Generated with Claude Code

When a Claude subscription hits a usage cap or its OAuth login expires,
every call fails identically. The Claude Code CLI reports this only as a
synthetic reply ("You've hit your session limit · resets …", "Failed to
authenticate: OAuth session expired …"), so the harness error just says
the output file was never written. Retries then burn through, reviewers
and advisors fail open, and issues are recorded as completed with no
changes.

check_fatal_harness_error now also scans the raw completion and the
session messages for these exact replies (anchored, so model prose can't
trip it), and the error-message patterns cover session limits and auth.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Oct 10, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@ZeroPoint95 ZeroPoint95 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@i2020github, this addresses a useful failure case. Two medium-severity issues remain:

  1. Quoted content can incorrectly stop the build. fatal_error.py:205 searches str(message) across all messages. During an ordinary schema-validation failure, a tool result or assistant response quoting You've hit your session limit · resets ... therefore raises FatalHarnessError. Actual SDK-message probes reproduce the planning retry being aborted; those probes pass on the base. Please inspect structured content and provider-error provenance rather than serialized user/tool messages, and test that quoted examples remain recoverable.

  2. The messages-only path misses genuine cap replies. The message pattern:62 misses You're out of extra usage · resets ... and session-limit replies without a reset suffix when raw completion text is empty. The pinned CLI emits these forms. This gap also exists on the base, so it is incomplete coverage of the proposed fix rather than a new regression. Please cover these synthetic replies while retaining the provenance checks above.

The Python suite passed: 1,346 tests passed and 1 skipped. No live provider or CLI execution was used. Separately, the unchanged Go round-trip test failed in CI but passed locally on both head and base; that CI failure remains unexplained.

AbirAbbas and others added 2 commits October 20, 2026 09:04
…lies

Require synthetic assistant provenance before inspecting session text, so
quoted usage and auth examples remain eligible for schema retries. Cover
the cap and login reply forms emitted by the pinned Claude Code CLI,
including messages-only extra usage and limits without reset times.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AbirAbbas

Copy link
Copy Markdown
Collaborator

@i2020github thanks for this. The failure mode you describe (whole DAG levels "completing" with zero changes after a cap) is a nasty one. Since the review had been waiting a while, I've taken it over and pushed on top of your branch.

@ZeroPoint95, both points:

Quoted content no longer stops the build. Detection now inspects the structured reply rather than str(message). I checked the pinned claude-agent-sdk 0.1.20, which bundles Claude Code 2.1.9. The CLI's synthetic cap/auth replies are an assistant message with model="<synthetic>" and a single text block, and the SDK's parser drops the CLI's top-level error field, so the synthetic model is the provenance that survives. Quoted cap/auth text in user messages, tool results or ordinary assistant prose stays recoverable, even when raw result repeats it. A planning test checks that the next schema attempt then succeeds, and that a genuine cap aborts before a second call.

Missing reply forms. These come from the bundled CLI's own formatter: You're out of extra usage, and session/weekly/Opus/Sonnet limits with or without · resets …. The 401/403 forms are covered too (Invalid API key · Please run /login, OAuth token revoked · Please run /login, API Error: … · Please run /login). Both the messages-only and raw-only paths are tested. With only fatal_error.py reverted, 90 of the 182 cases fail against the original PR source.

The Go CI failure is real but pre-existing. furrowd's runChild calls cmd.Wait() before the stdout copy finishes, so a slow write drops the tail of the child's output. That gives exactly the "MARKER:" vs "MARKER:ciphertext" failure, and it reproduces deterministically on main. This PR doesn't touch Go; it's filed as #.

I've merged current main (including #155, which routes the QA synthesizer through check_fatal_harness_error). make check on 3.12 passed with 1,457 passed before the merge, and the fatal-error, synthesizer and planner suites pass after it.

@AbirAbbas
AbirAbbas requested a review from ZeroPoint95 October 20, 2026 14:00
AbirAbbas
AbirAbbas previously approved these changes Oct 20, 2026

@AbirAbbas AbirAbbas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at b4bfb4f: genuine Claude CLI cap/auth replies (synthetic-model provenance from the pinned SDK 0.1.20 / CLI 2.1.9, including out-of-extra-usage and no-reset-suffix forms) are fatal; quoted examples in tool results, user messages and model prose stay recoverable and planning retries through them. make check + Go job green.

Keep schema diagnoses and transient authentication failures recoverable by replacing the new global substrings with complete known Claude replies. Preserve synthetic-assistant detection and propagate serialized FatalHarnessError markers across execution envelopes.

Add public HarnessRunner, QA fallback, envelope, and anchored-reply regression coverage.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AbirAbbas and others added 2 commits October 20, 2026 14:18
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Keep the generic fatal classifier identical to main. Match complete Claude CLI login replies only in raw completions, and propagate serialized fatal exceptions only at the start of failed envelope errors. Add the independent review suites, pinned formatter fixtures, and an offline SDK subprocess round trip.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AbirAbbas AbirAbbas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging after two more independent review rounds. Each one found the classifier could still turn recoverable output into a build-killing error. First it was the new global substrings; then it was the exception marker I'd added to get errors across processes. Both came from matching text without knowing where it came from, so the final version stops doing that (see the note at the top of the description):

  • Main's rules are untouched. _FATAL_PATTERNS and is_fatal_error are byte-identical to main.
  • Genuine Claude caps and auth failures are recognised from the CLI's synthetic provider reply (model="<synthetic>"). For raw-only results, the whole completion must be exactly one known reply generated from the pinned CLI 2.1.9 formatters. This also covers the API Error: 401 status code (no body) · Please run /login forms the earlier version missed.
  • Across processes, an envelope error only counts when it starts with the FatalHarnessError marker. That's checked end to end through the real SDK callback path.

Both reviews' adversarial suites are now in tests/ and pass. They cover schema diagnoses quoting cap text on all three adapters, transient auth errors, quoted markers in pytest and tool output, the QA synthesizer fallback, and all 34 CLI-generated replies on the raw, wire and SDK-parsed paths. make check (2,174 passed) and the Go job are green. Thanks @i2020github!

@AbirAbbas
AbirAbbas merged commit b7bec03 into Agent-Field:main Oct 20, 2026
3 checks passed
AbirAbbas added a commit that referenced this pull request Oct 20, 2026
Include the newly merged #156 provider-provenance and remote error propagation fixes unchanged. The AForge and Command Code landing does not modify these files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants