Repository navigation
Conversation
When a Claude subscription hits a usage cap or its OAuth login expires,
every call fails identically. The Claude Code CLI reports this only as a
synthetic reply ("You've hit your session limit · resets …", "Failed to
authenticate: OAuth session expired …"), so the harness error just says
the output file was never written. Retries then burn through, reviewers
and advisors fail open, and issues are recorded as completed with no
changes.
check_fatal_harness_error now also scans the raw completion and the
session messages for these exact replies (anchored, so model prose can't
trip it), and the error-message patterns cover session limits and auth.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ZeroPoint95
left a comment
There was a problem hiding this comment.
@i2020github, this addresses a useful failure case. Two medium-severity issues remain:
-
Quoted content can incorrectly stop the build. fatal_error.py:205 searches
str(message)across all messages. During an ordinary schema-validation failure, a tool result or assistant response quotingYou've hit your session limit · resets ...therefore raisesFatalHarnessError. Actual SDK-message probes reproduce the planning retry being aborted; those probes pass on the base. Please inspect structured content and provider-error provenance rather than serialized user/tool messages, and test that quoted examples remain recoverable. -
The messages-only path misses genuine cap replies. The message pattern:62 misses
You're out of extra usage · resets ...and session-limit replies without a reset suffix when raw completion text is empty. The pinned CLI emits these forms. This gap also exists on the base, so it is incomplete coverage of the proposed fix rather than a new regression. Please cover these synthetic replies while retaining the provenance checks above.
The Python suite passed: 1,346 tests passed and 1 skipped. No live provider or CLI execution was used. Separately, the unchanged Go round-trip test failed in CI but passed locally on both head and base; that CI failure remains unexplained.
…lies Require synthetic assistant provenance before inspecting session text, so quoted usage and auth examples remain eligible for schema retries. Cover the cap and login reply forms emitted by the pinned Claude Code CLI, including messages-only extra usage and limits without reset times. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@i2020github thanks for this. The failure mode you describe (whole DAG levels "completing" with zero changes after a cap) is a nasty one. Since the review had been waiting a while, I've taken it over and pushed on top of your branch. @ZeroPoint95, both points: Quoted content no longer stops the build. Detection now inspects the structured reply rather than Missing reply forms. These come from the bundled CLI's own formatter: The Go CI failure is real but pre-existing. I've merged current main (including #155, which routes the QA synthesizer through |
AbirAbbas
left a comment
There was a problem hiding this comment.
Verified at b4bfb4f: genuine Claude CLI cap/auth replies (synthetic-model provenance from the pinned SDK 0.1.20 / CLI 2.1.9, including out-of-extra-usage and no-reset-suffix forms) are fatal; quoted examples in tool results, user messages and model prose stay recoverable and planning retries through them. make check + Go job green.
Keep schema diagnoses and transient authentication failures recoverable by replacing the new global substrings with complete known Claude replies. Preserve synthetic-assistant detection and propagate serialized FatalHarnessError markers across execution envelopes. Add public HarnessRunner, QA fallback, envelope, and anchored-reply regression coverage. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Keep the generic fatal classifier identical to main. Match complete Claude CLI login replies only in raw completions, and propagate serialized fatal exceptions only at the start of failed envelope errors. Add the independent review suites, pinned formatter fixtures, and an offline SDK subprocess round trip. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AbirAbbas
left a comment
There was a problem hiding this comment.
Merging after two more independent review rounds. Each one found the classifier could still turn recoverable output into a build-killing error. First it was the new global substrings; then it was the exception marker I'd added to get errors across processes. Both came from matching text without knowing where it came from, so the final version stops doing that (see the note at the top of the description):
- Main's rules are untouched.
_FATAL_PATTERNSandis_fatal_errorare byte-identical to main. - Genuine Claude caps and auth failures are recognised from the CLI's synthetic provider reply (
model="<synthetic>"). For raw-only results, the whole completion must be exactly one known reply generated from the pinned CLI 2.1.9 formatters. This also covers theAPI Error: 401 status code (no body) · Please run /loginforms the earlier version missed. - Across processes, an envelope error only counts when it starts with the
FatalHarnessErrormarker. That's checked end to end through the real SDK callback path.
Both reviews' adversarial suites are now in tests/ and pass. They cover schema diagnoses quoting cap text on all three adapters, transient auth errors, quoted markers in pytest and tool output, the QA synthesizer fallback, and all 34 CLI-generated replies on the raw, wire and SDK-parsed paths. make check (2,174 passed) and the Go job are green. Thanks @i2020github!
Include the newly merged #156 provider-provenance and remote error propagation fixes unchanged. The AForge and Command Code landing does not modify these files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
What changed and why
When the
claude_coderuntime runs on a Claude subscription and hits a usage cap, every call fails identically until the cap resets. The same happens when its OAuth login expires and can't be refreshed. The Claude Code CLI reports this only as a synthetic assistant reply, for example:You've hit your session limit · resets 9:50pm (UTC)Failed to authenticate: OAuth session expired and could not be refreshedThe harness then reports a generic
Schema validation failed … The output file was NOT created, socheck_fatal_harness_errornever fires. Observed during multi-hour builds:This PR makes these conditions fatal so the build stops cleanly and can be resumed:
check_fatal_harness_erroralso scans the raw completion andHarnessResult.messagesfor the CLI's exact replies. During planning the raw completion is empty and the reply survives only in the messages. The raw-text check is anchored to the start of the reply, and the messages check requires the CLI's… · resets/Failed to authenticate: OAuthphrasing, so model prose that mentions limits or authentication does not trip it._FATAL_PATTERNSgainshit your … limit,usage limit reached,out of extra usage,failed to authenticateandoauth session expired.Behaviour changes to orchestration
The changes only take effect once a result is already
is_error. In that case these replies now raiseFatalHarnessErrorinstead of being retried or failing open.A related issue, not addressed here: when a fatal error fails some issues mid-level, their dependents are skipped, yet
buildcan still reportsucceeded. Callers chaining builds need to checkfailed_issues/skipped_issuesin the checkpoint.Test evidence
TestClaudeUnavailableRepliescovers replies in the raw completion and in session messages, model prose that must not raise, and successful results not being inspected. The new phrases are also added to theis_fatal_errorparametrisation.main@ 4d1f11c):1346 passed, 1 skipped.Fatal API error (non-retryable): Claude unavailable: You've hit your session limit · resets 7pm (UTC), and a laterresume_buildcontinues from the checkpoint.The Go node has no equivalent patterns yet and likely needs the same handling.
🤖 Generated with Claude Code