Skip to content

fix(retry): treat provider usage-limit errors as terminal, not transient - #1174

Open
samirhvbr wants to merge 6 commits into
usestrix:mainfrom
samirhvbr:fix/usage-limit-terminal-not-retried
Open

samirhvbr wants to merge 6 commits into
usestrix:mainfrom
samirhvbr:fix/usage-limit-terminal-not-retried

Conversation

@samirhvbr

@samirhvbr samirhvbr commented Aug 26, 2026 •

Copy link
Copy Markdown

A subscription usage-limit rejection, such as a provider's exhausted hourly or weekly allowance reported as HTTP 429 with usage_limit_reached, cannot recover through ordinary transient retries. Retrying it in both the SDK and Strix turn loop delays a resumable stop and repeatedly hits the exhausted quota.

Classify usage-limit errors as terminal in both retry layers while retaining retries for ordinary transient 429 throttles. OpenAI RateLimitError follows the existing resumable stop path; a non-OpenAI exception carrying the same usage-limit marker now follows that path too, preserving the scan for resume after the limit resets.

The branch includes upstream 6b7ae2eb. The shared runner test fixture now supplies telemetry.run_record_timings=False, which the current runner requires before entering the agent loop. That missing test setting was inherited from upstream; it was not a retry-logic failure. The added exception handler also uses the already validated root ID without introducing redundant None-check diagnostics.

Validation on macOS/Apple Silicon, Python 3.12.14, frozen dependencies and isolated HOME/environment:

  • 108 passed across transient-retry, SDK retry, runner rate-limit, execution and request-log tests.
  • Coverage includes ordinary 429 retry behavior, terminal usage limits, and resumable stops for OpenAI and non-OpenAI exceptions.
  • Repository-wide Ruff lint/format, Mypy and Bandit passed; git diff --check passed.
  • Pyright reports 941 errors on both this branch and unchanged upstream, with no new diagnostic signatures after normalizing checkout paths.

Exceptions and model behavior were simulated. No real provider request, exhausted account, live scan or full repository test suite was exercised. Remote CI still requires maintainer approval.

Prepared with AI assistance.

A plan/subscription usage-limit rejection (e.g. ChatGPT's 5-hour or weekly cap,
HTTP 429 `usage_limit_reached`) was classified as an ordinary transient throttle
and retried by both retry layers — the SDK ModelSettings retry
(`DEFAULT_MODEL_RETRY`, up to 5x) and Strix's turn replay
(`_is_transient_model_error` -> `_run_cycle`, up to 5x). Retrying cannot succeed
until the window resets (often hours away); it only burns more of the same
exhausted quota and spends minutes thrashing exponential backoff, flooding the
log with `openai.agents: Error streaming response` before the run finally stops.

Classify usage-limit errors as terminal so the run fails fast and lands on the
existing resumable stop path (`run_strix_scan` catches the RateLimitError, logs
the `strix --resume <run>` hint, and stops the root):

- add `codex.is_usage_limit_error()` (text-marker predicate, mirroring
  `is_content_guardrail_error`);
- exclude it from `_is_transient_model_error` (Strix replay layer);
- exclude it from the SDK retry policy via a small wrapper
  (`_default_model_retry_policy`).

Ordinary 429 throttles are unchanged and still retried. Adds regression tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 26, 2026 12:33
samirhvbr added a commit to samirhvbr/strix that referenced this pull request Aug 26, 2026
- Bump .fork-version to 1.5.3+shvia.2; FORK.md changelog + PR usestrix#1174.
- The usage-limit terminal fast-fail (cherry-picked from the upstream PR branch)
  now lands in the fork's master, so `strix-run --auto` fails over promptly
  instead of after minutes of doomed retries.
- Lower the default --auto threshold to 2 (more eager failover).
- escopo: F3 marked done.

Note: API keys were migrated out of ~/.bashrc into .env (gitignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJDMregdMDmfDAYotXAHjh
@greptile-apps

greptile-apps Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds text-based detection of provider usage-limit errors and excludes matching failures from both model retry layers so exhausted subscription windows fail quickly.

  • Adds usage-limit markers and a shared classifier.
  • Composes the classifier into the SDK retry policy.
  • Excludes matching failures from turn replay.
  • Adds regression coverage for usage-limit and ordinary 429 behavior.

Confidence Score: 4/5

The PR needs adjustment before merging because usage-limit failures from LiteLLM-routed providers can now fail the scan instead of producing the promised resumable stop.

Retry suppression is provider-agnostic, while terminal rate-limit recovery is restricted to OpenAI's exception class, leaving supported non-OpenAI routes on the generic failure path.

Files Needing Attention: strix/core/execution.py, strix/config/models.py, and strix/core/runner.py

Important Files Changed

Filename Overview
strix/config/codex.py Adds the shared text-marker predicate used to identify usage-limit failures.
strix/config/models.py Prevents SDK retries for matching errors, but applies the classification across all provider routes.
strix/core/execution.py Suppresses turn replay globally, exposing non-OpenAI provider exceptions to a runner that cannot convert them into resumable stops.
tests/test_execution_transient_retry.py Covers OpenAI usage-limit and ordinary-rate-limit classification but not a LiteLLM-routed exception.
tests/test_model_retry.py Verifies the composed retry policy declines matching text while retaining ordinary retry behavior.
Prompt To Fix All With AI
### Issue 1
strix/core/execution.py:135-136
**Non-OpenAI limits fail**

When a LiteLLM-routed provider raises a marker-bearing usage-limit error, this classification suppresses both retry layers, but `run_strix_scan` only handles `openai.RateLimitError` as resumable. The provider exception therefore reaches the generic failure path, marking the root failed and re-raising instead of showing the resume hint.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "fix(retry): treat provider usage-limit e..." | Re-trigger Greptile

Comment thread strix/core/execution.py
Comment on lines +135 to +136
if codex.is_usage_limit_error(exc):
return False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Non-OpenAI limits fail

When a LiteLLM-routed provider raises a marker-bearing usage-limit error, this classification suppresses both retry layers, but run_strix_scan only handles openai.RateLimitError as resumable. The provider exception therefore reaches the generic failure path, marking the root failed and re-raising instead of showing the resume hint.

Knowledge Base Used:

Prompt To Fix With AI
This is a comment left during a code review.
Path: strix/core/execution.py
Line: 135-136

Comment:
**Non-OpenAI limits fail**

When a LiteLLM-routed provider raises a marker-bearing usage-limit error, this classification suppresses both retry layers, but `run_strix_scan` only handles `openai.RateLimitError` as resumable. The provider exception therefore reaches the generic failure path, marking the root failed and re-raising instead of showing the resume hint.

**Knowledge Base Used:**
- [Scan execution lifecycle](https://app.greptile.com/strix-org-3/-/custom-context/knowledge-base/usestrix/strix/-/docs/scan-execution.md)
- [Configuration and provider settings](https://app.greptile.com/strix-org-3/-/custom-context/knowledge-base/usestrix/strix/-/docs/configuration-and-provider-settings.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Strix’s retry behavior to treat provider “usage-limit reached” (plan/subscription window exhausted) errors as terminal rather than transient, preventing repeated retries that cannot succeed until the provider’s window resets.

Changes:

  • Add codex.is_usage_limit_error(exc) predicate based on provider error text markers.
  • Exclude usage-limit errors from Strix’s execution replay retry (execution._is_transient_model_error).
  • Exclude usage-limit errors from the SDK-level model retry policy by composing the existing retry policy with an additional “not usage limit” guard.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
strix/config/codex.py Adds is_usage_limit_error and marker definitions for detecting subscription usage-limit rejections.
strix/core/execution.py Treats usage-limit errors as non-transient to stop Strix turn replay retries.
strix/config/models.py Wraps the default SDK retry policy with an additional predicate to prevent retrying usage-limit errors.
tests/test_model_retry.py Adds regression test ensuring usage-limit errors are not retried by the SDK retry policy.
tests/test_execution_transient_retry.py Adds regression tests ensuring usage-limit errors are terminal in Strix replay logic and ordinary 429s remain transient.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Follow-up to the usage-limit terminal classification (review feedback): retry
suppression is provider-agnostic (text-marker based), but run_strix_scan only
mapped openai.RateLimitError to the resumable stop. A LiteLLM-routed provider
that surfaces a usage-limit as a different exception type therefore reached the
generic failure path (root marked failed, re-raised) instead of the promised
resumable stop with the `strix --resume` hint.

Route any error where codex.is_usage_limit_error(exc) is true to the same
resumable stop as RateLimitError. Adds a regression test covering a non-OpenAI
(LiteLLM-routed) usage-limit exception.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
samirhvbr added a commit to samirhvbr/strix that referenced this pull request Aug 26, 2026
Resolves the conflicts with the retry and reporting changes that landed on main since
this branch was opened. The resulting tree is the PR rebased onto main, tested there.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TavwVMH7PvrD68JTDfd6Hb
The current upstream runner reads run_record_timings while constructing usage hooks. Its rate-limit test fixture predates that setting and raises AttributeError before reaching the mocked agent loop. Complete the shared fixture so both the existing rate-limit test and the non-OpenAI usage-limit regression exercise their intended paths.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants