Skip to content

fix(cli): Don't auto-retry non-idempotent requests; honor rate-limit reset on 429 - #1677

Merged
BYK merged 4 commits into
mainfrom
fix/cli-retry-idempotent-429
Oct 27, 2026
Merged

BYK merged 4 commits into
mainfrom
fix/cli-retry-idempotent-429

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Problem

packages/cli/src/lib/sentry-client.ts retried every HTTP method, including POST/PUT/DELETE, on 408/429/500/502/503/504 and on network errors and timeouts. That can replay writes. A 429 also only got the generic 1s/2s backoff: the client never read Sentry's X-Sentry-Rate-Limit-Reset header, and Sentry doesn't send Retry-After on API rate limits.

Behavior changes

  • 5xx, timeouts and network errors now retry only for GET, HEAD and OPTIONS, or when a request sets the new idempotent: true opt-in on getSdkConfig / ApiRequestOptions.
  • 429 retries for any method. Sentry's rate-limit middleware rejects the request in process_view, before the endpoint runs, so nothing was applied.
  • Retried statuses stay 408/429/500/502/503/504. Deterministic 5xx like 501 and 505 are not retried.
  • PUT and DELETE are deliberately not retried. Sentry PUTs have side effects (merge, activity), and a replayed DELETE returns 404.
  • The 401-refresh retry still covers every method, and no longer waits 1s first.
  • A timed-out write now gets a "may still have been applied" hint.
  • The 429 wait is the longer of X-Sentry-Rate-Limit-Reset (read only when Remaining is 0) and Retry-After (seconds or HTTP date), both measured against the response Date to guard against clock skew. Concurrency-limit 429s, or 429s with no hint, use jittered backoff: the upper half of min(1s·2^n, 10s).
  • Caps are unchanged: MAX_RETRIES is 2 and each wait is at most 10s.
  • New RateLimitError (extends ApiError, status 429). It's thrown when the wait would exceed the cap or retries run out. The message is agent-friendly: server detail, then "Retry in N seconds; do not repeat the request before then. Make fewer requests in parallel."
  • The policy is pure, with no CLI imports, in packages/cli/src/lib/retry-policy.ts. That way the CLI can switch to the shared @sentry/toolkit-core/api-retry from fix(mcp): Retry rate-limited GETs using Sentry rate-limit reset headers #1673 without a rewrite.

POST callers given the idempotent opt-in

Each was checked against its getsentry/sentry handler:

  • Chunk uploads (chunk-upload.ts): blobs are stored by SHA-1 (FileBlob.from_files).
  • Artifact-bundle assemble and pollAssembly (sourcemaps.ts).
  • DIF assemble in debug-files.ts, proguard.ts and dart-symbols.ts: the server checks the cached assemble status by checksum.

Not opted in: preprod assemble (each call creates a new PreprodArtifact), Seer /autofix/, snapshot create and archive trigger, and SDK and sentry api writes.

Tests

  • The POST-retry body-cloning tests now go through the 429 and opt-in paths.
  • New cases:
    • a rate-limited POST is resent with the same body
    • a 501 is not retried
    • a 503 on a non-idempotent method is not retried
    • the reset-header and Retry-After wait, clock skew, and the cap leading to RateLimitError
    • the exact rate-limit message text
  • api-client.coverage used 429 as a sample error and now uses 400.

Checks

  • tsc --noEmit and pnpm run lint pass.
  • The sentry-client, retry-policy, api-client and api-client.coverage tests pass: 204/204.
  • Full CLI suite on bf3b710: 11,482 passed and 6 failed. The same 6 fail on main (37c7193) in this sandbox. They're chmod-based permission tests (completions ×2, code scanner, init file-changes, telemetry readonly, upgrade lock), and the sandbox runs with full capabilities, so they can't fail the way they expect.
  • After the 429-any-method change (7099e2b), all other CLI test files that mention 429 pass (1,195), apart from the 2 known sandbox permission failures.

Relation to #1669

This matches the retry policy in #1669 (task 5.0 and the sentry-api-transport spec). Task 5.1 will move this behavior into toolkit-core's shared transport unchanged.

via BYK.

--

View Junior Session [Sentry]

sentry-junior Bot and others added 2 commits October 26, 2026 23:42
…reset on 429

The authenticated fetch retried every method on 408/429/5xx, network
errors, and timeouts, so a POST that reached the server could run up to
three times (for example, three Seer autofix runs). On 429 it ignored
X-Sentry-Rate-Limit-Reset and Retry-After and backed off blindly.

Automatic retries now apply only to GET/HEAD/OPTIONS, or to requests
that opt in with `idempotent: true`. Chunk uploads (stored by SHA-1) and
artifact-bundle/DIF assemble calls (state keyed by checksum) opt in. The
401 token-refresh retry still applies to every method.

On 429 the client waits for the server-requested delay (rate-limit reset
measured against the response Date header, or Retry-After), capped at
10s. Concurrency-limit 429s and responses without a hint use jittered
exponential backoff. When the wait exceeds the cap, retries run out, or
the method can't be retried, it throws RateLimitError with the time to
wait.

The policy lives in a pure retry-policy.ts so it can move to
toolkit-core unchanged.

Co-Authored-By: Burak Yigit Kaya <126780+BYK@users.noreply.github.com>
Sentry's rate-limit middleware rejects a request in process_view, before
the endpoint runs, so a 429 never applied the write. Retry 429 for every
method. 5xx, timeouts, and network errors stay limited to GET/HEAD/OPTIONS
and the `idempotent` opt-in. Retried statuses stay 408/429/500/502/503/504;
501, 505 and other deterministic 5xx are not retried.

RateLimitError now tells callers (including agents) to retry in N seconds,
or a few seconds without a hint, not to repeat the request before then,
and to make fewer requests in parallel.
@vercel

vercel Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
sentry-local Skipped Skipped Oct 27, 2026 3:23pm UTC

Request Review

@github-actions github-actions Bot added the risk: medium PR risk score: medium label Oct 27, 2026
Comment thread packages/cli/src/lib/sentry-client.ts
The assemble loop already re-sends the identical body to poll, and the
server reuses the in-progress PreprodArtifact row via get_or_create, so
retrying a transient 5xx/timeout is no riskier than the next poll.
Comment thread packages/cli/src/lib/api/proguard.ts
Comment thread packages/cli/src/lib/errors.ts
Comment thread packages/cli/src/lib/retry-policy.ts Outdated
…try methods

- RateLimitError now exits with EXIT.RATE_LIMITED (32) so scripts and
  agents can back off without parsing stderr. Documented in exit-codes.md.
- Rename IDEMPOTENT_METHODS to SAFE_METHODS and reference RFC 9110
  9.2.1/9.2.2; note the retryable status list mirrors common client
  defaults since HTTP defines none.
@BYK
BYK merged commit dd15d91 into main Oct 27, 2026
26 checks passed
@BYK
BYK deleted the fix/cli-retry-idempotent-429 branch October 27, 2026 17:23
BYK added a commit to getsentry/sentry that referenced this pull request Oct 27, 2026
…128930)

Lets endpoints declare which non-safe methods are idempotent, and
publishes that as an OpenAPI operation extension. Generated clients such
as `@sentry/api`, and through it the CLI and MCP, can then retry those
operations on timeouts and 5xx responses instead of hard-coding per-call
opt-ins.

Refs #128927. Context: getsentry/toolkit#1677.

**What changes**

- `Endpoint.idempotent_methods: frozenset[HTTP_METHOD_NAME]` (default
empty). Listing a method there means resending the identical request
after a timeout or 5xx cannot change the outcome.
- `custom_preprocessing_hook` collects declarations for every endpoint,
like silos, and resets them on every build. `custom_postprocessing_hook`
stamps `"x-sentry-idempotent": true` on matching operations. Operations
without a declaration get no marker.
- The build fails if an endpoint lists a safe method
(`GET`/`HEAD`/`OPTIONS`/`TRACE`, RFC 9110 §9.2.1) or a method it does
not declare in `publish_status`.
- `PUT`/`DELETE` are not marked automatically. Not every handler lives
up to RFC 9110 §9.2.2, so the absence of the marker means "not known to
be idempotent".
- Documented in `src/sentry/apidocs/README.md`.

**Declared on**

- `ChunkUploadEndpoint` `POST`: blobs are unique per checksum.
- `DifAssembleEndpoint` `POST` and
`OrganizationArtifactBundleAssembleEndpoint` `POST`: assemble state is
cached per checksum, so a resend reports it instead of starting a second
job.

`ProjectPreprodArtifactAssembleEndpoint` is deliberately **not** marked.
`create_preprod_artifact` only reuses a row while it is still
`UPLOADING`, so a resend after that creates a second artifact and
enqueues another assemble task.

**Open questions for review**

- All three declared endpoints are `PRIVATE`, so the public schema
doesn't change yet. The declarations are inert until those endpoints are
published, or until `@sentry/api` builds from a schema that includes
them. Which route do we want?
- Name and shape: a boolean `x-sentry-idempotent`, or a richer
`x-sentry-retry` object?

**Verification**

- `ruff check` and `ruff format --check` are clean on the changed files.
- New `IdempotentOperationsTest` cases in `tests/apidocs/test_hooks.py`
cover: marking only declared methods, collecting unpublished methods,
resetting between builds, and both validation errors.
- I couldn't run the Sentry pytest/mypy environment in my sandbox, so I
ran the same scenarios against the real `hooks.py` with stubbed imports,
and they pass. CI is the authoritative run.

<!-- junior-request-attribution:start -->
via **BYK**.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC08J1NSPU6S%3A1792772794.175999 -->

--

[View Junior
Session](https://junior-prod.sentry.dev/conversations/slack%3AC08J1NSPU6S%3A1792772794.175999)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC08J1NSPU6S%3A1792772794.175999/?project=4510944073809921)

<!-- junior-session-footer:end -->

---------

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: BYK <burak.kaya@sentry.io>

This branch was previously deployed

1 inactive deployment
Preview — 8e1c9a9c Deployed Oct 27, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant