Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 72 additions & 2 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -572,7 +572,7 @@ Kotlin, and Ruby.
- **TypeScript** implements the three-gate algorithm with the retry loop beneath the openapi-fetch middleware chain (the client's custom `fetch`), so attempts run to each operation's declared `retry.max` and network errors retry under the same idempotency gate; caller aborts and request timeouts are terminal. **Kotlin** implements the three-gate algorithm for both HTTP status and network-error retries (POST retries only when `idempotent: true`, full exponential backoff): one eligibility gate covers both failure shapes, with the whole-request timeout (Ktor's `HttpRequestTimeoutException`) deliberately not retried and auth headers re-attached per attempt.
- **Go** implements the three-gate on its generated operation path: it retries operations classified idempotent at generation time — GET/HEAD by method, plus any operation carrying `x-basecamp-idempotent` (the naturally-idempotent PUT/DELETE mutations like `UpdateProject`/`TrashProject`, and the flagged-idempotent POSTs like `CompleteTodo`) — with exponential backoff; non-idempotent operations (e.g. `CreateTodo`) are single-attempt. The separate hand-written `doRequestURL` helper remains GET-only for ordinary retries, with a mutation-specific single re-attempt after successful 401 token refresh.
- **Ruby** is stricter: only GET retries; all non-GET methods do not retry. Governed GETs (those carrying their canonical operation ID) are bounded by the per-op ceiling and status-gated on the declared `retryOn`; ungoverned GETs (`get_absolute`, OAuth discovery) keep the taxonomy-driven pre-metadata contract. Ruby is acceptably conservative.
- **Swift** implements the three-gate algorithm: the transport retries only when the method is naturally idempotent (GET/HEAD/PUT/DELETE) **or** the operation is marked `idempotent: true`, so non-idempotent POSTs like `CreateProject` are attempted exactly once while the seven idempotent POSTs (`CompleteTodo`, `CreateBookmark`, `EnableCardColumnOnHold`, `PauseQuestion`, `PrioritizeAssignment`, `Subscribe`, `SubscribeToCardColumn`) keep retrying. The gate covers both retry paths — HTTP status (`429`/`503`) and network errors — so Swift retries network errors but only for retry-eligible operations. `BaseService` threads the per-operation flag from generated `Metadata` into the transport; the naturally-idempotent method set is allowlisted so PATCH/OPTIONS and future methods stay fail-closed.
- **Swift** implements the three-gate algorithm: the transport retries only when the method is naturally idempotent (GET/HEAD/PUT/DELETE) **or** the operation is marked `idempotent: true`, so non-idempotent POSTs like `CreateProject` are attempted exactly once while the seven idempotent POSTs (`CompleteTodo`, `CreateBookmark`, `EnableCardColumnOnHold`, `PauseQuestion`, `PrioritizeAssignment`, `Subscribe`, `SubscribeToCardColumn`) keep retrying. The gate covers both retry paths — HTTP status (`429`/`503`) and network errors — so Swift retries network errors but only for retry-eligible operations. A network error is classified by *meaning*, not by type: a `Transport` that reports connectivity failure as the SDK's own `BasecampError.network` reaches the retry branch exactly as a raw `URLError` does (#567). `Transport` is `public`, so that normalization is the natural implementation and must not be the one that disables retry. Any other `BasecampError` out of the transport (`.auth`, `.usage`, `.api`, …) stays terminal on sight, and the non-HTTP-response guard raises a distinct internal error so a deterministic programming fault is never mistaken for a transport blip. `BaseService` threads the per-operation flag from generated `Metadata` into the transport; the naturally-idempotent method set is allowlisted so PATCH/OPTIONS and future methods stay fail-closed.
- The spec prescribes the three-gate algorithm.

### Retry Algorithm
Expand Down Expand Up @@ -635,15 +635,84 @@ The loop always terminates via step 3e (raise on network error), 3f (return non-
### Backoff Formula

```
delay = base_delay_ms * 2^(retry_index) + random(0, max_jitter)
delay = min(base_delay_ms * 2^(retry_index), MAX_BACKOFF_DELAY_MS) + random(0, max_jitter)
```

Where `retry_index` is the 0-indexed retry count (first retry = 0, second retry = 1, etc.). In the `executeWithRetry` loop, `retry_index = attempt` — when the initial request (attempt=0) fails and reaches step 3h, it computes the delay for the first retry using `2^0 = 1×base_delay_ms`. Default constants (from `retry_config` or Config):
- `base_delay_ms` = 1000 (from `retry_config.base_delay_ms`)
- `max_jitter` = 100ms (from Config; not part of `retry_config` — sourced from the client's Config RECORD)
- `MAX_BACKOFF_DELAY_MS` = 30,000 (30s) — the ceiling on the backoff term, below

Retry-After header value takes precedence when present and valid.

### Backoff Ceiling `[CONFLICT]`

The backoff term is **saturating**: it grows exponentially up to `MAX_BACKOFF_DELAY_MS`
and then stops. This is a correctness requirement, not a politeness one — an unbounded
`2^n` is not merely a long sleep, it is a different failure in every host language, and
each of the six SDKs demonstrated one before #577:

| Overflow shape | SDKs | What actually happens |
|---|---|---|
| Signed 64-bit wrap | Kotlin (`1L shl`), Go (`1<<`) | The product goes negative and the sleep primitive treats a non-positive delay as "no delay" — the client tight-loops against a server that is already answering 429/503, which is the exact traffic pattern backoff exists to prevent |
| Trap on overflow | Swift (`UInt64` multiply) | The process **crashes**. Swift's `<<` is a smart shift, so an over-shift silently yields `0` (tight loop), but at `1 << 63` the multiply overflows `UInt64` and traps |
| Saturate to infinity | TypeScript (`Math.pow`) | `Infinity` reaches `setTimeout`, which clamps an out-of-range delay to **1ms** — a tight loop again |
| Unbounded integer | Python (`2 **`), Ruby (`2**`) | No wrap: the multiplier becomes an arbitrary-precision bignum. Python raises `OverflowError` converting it to a float; Ruby coerces to `Float::INFINITY` and `sleep` never returns |

Requirements:

1. **Saturate, never wrap, never diverge.** An implementation must not evaluate an
unbounded `2^retry_index`. Compare against `MAX_BACKOFF_DELAY_MS / base_delay_ms`
*before* multiplying — either the multiplier itself, or, where the power is the
thing that overflows, the exponent against the point at which the term first
reaches the ceiling. That keeps every intermediate inside the host's numeric range
**and** lands the term on the ceiling.

A **fixed** exponent cap followed by `min(base × 2^capped, MAX_BACKOFF_DELAY_MS)`
is not an acceptable substitute, however generous the cap looks. It bounds the
intermediate but not the *outcome*: for a base delay small enough that
`base × 2^cap < MAX_BACKOFF_DELAY_MS`, the term plateaus below the ceiling for
every subsequent attempt instead of saturating at it. At `base_delay_ms = 1e-30`
a cap of 64 pins every attempt from the 65th onward at ~1.84e-11s — which is
requirement 1's tight loop, not a fix for it.

The bound must be derived from the base **without overflowing its own
arithmetic**. `MAX_BACKOFF_DELAY_MS / base_delay_ms` is itself infinite once
`base_delay_ms` drops below `MAX_BACKOFF_DELAY_MS / MAX_FLOAT`, and falling back
to a fixed exponent there fails in the mirror direction: the term saturates
**early**, returning the ceiling at an attempt whose specified value is still far
below it. Compute the crossing in the log domain (`log2(ceiling) - log2(base)`)
and scale the term directly — `ldexp`, or repeated bounded multiplication where
the host has no `ldexp` — so no fixed exponent cap is needed at all. Saturating
early is as much a deviation as plateauing: the term must track
`base × 2^retry_index` at every attempt below the ceiling and equal the ceiling
at every attempt at or above it.
2. **The ceiling bounds the backoff term, not the total sleep.** Jitter is added after
clamping, so the longest single backoff sleep is `MAX_BACKOFF_DELAY_MS + max_jitter`.
This matches Go's generated client, which has capped at `RetryConfig.MaxDelay = 30s`
since it was first templated; 30s is adopted as the cross-SDK constant because it is
the one value already shipping.
3. **The ceiling applies to `linear` and `constant` backoff too**, and therefore to
`base_delay_ms` itself: a caller configuring a base delay above the ceiling gets the
ceiling. The rule is "no single computed backoff sleep exceeds `MAX_BACKOFF_DELAY_MS`",
with no carve-out for the first one. No shipped configuration approaches it —
`behavior-model.json` tops out at `base_delay_ms: 2000`, and the default three
attempts never compute past `2000 × 2 = 4000ms`, so the ceiling is unreachable on
default paths and changes no shipped behavior.
4. **`Retry-After` is exempt.** It is server-directed and takes precedence per step 3h;
the ceiling governs the locally-computed formula only. Implementations may still
bound it against host limits — Swift clamps its seconds→nanoseconds conversion to
86,400s because `UInt64(_:)` on an out-of-range `Double` is a trap.

**Reachability.** Every SDK exposes a path to a high attempt count: Kotlin's builder
validates `maxRetries >= 0` with no upper bound, Go's `WithMaxRetries` only rejects
`n < 1`, and Python/Ruby take a caller cap that is intersected with — never raised
above — the per-operation max, so a caller who *lowers* the cap is fine but the
operation ceiling itself is whatever `behavior-model.json` says. Reaching the overflow
needs a long genuine failure streak, so this is a robustness gap rather than a live
incident; the consequences (crash, tight loop, infinite sleep) are severe enough that
the gate belongs in the spec rather than in six independent judgment calls.

### Default and No-Retry Configs

```
Expand Down Expand Up @@ -1904,6 +1973,7 @@ All magic numbers in one place, derived from shipping SDK code (not `rubric-audi
| `DEFAULT_MAX_RETRIES` | 3 | — | All six SDKs |
| `DEFAULT_BASE_DELAY` | 1000 | milliseconds | All six SDKs |
| `DEFAULT_MAX_JITTER` | 100 | milliseconds | All six SDKs |
| `MAX_BACKOFF_DELAY` | 30,000 (30s) | milliseconds | All six SDKs; ceiling on the §7 backoff term, jitter added on top. Was Go's generated `RetryConfig.MaxDelay` before #577 generalized it |
| `DEFAULT_MAX_PAGES` | 10,000 | — | All six SDKs |
| `MAX_CACHE_ENTRIES` | 1000 | entries | `typescript/src/client.ts` |
| `MAX_TOKEN_HASH_ENTRIES` | 100 | entries | `typescript/src/client.ts` |
Expand Down
61 changes: 61 additions & 0 deletions go/pkg/basecamp/backoff_ceiling_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
package basecamp

import (
"testing"
"time"
)

// TestBackoffDelaySaturates pins SPEC §7's backoff ceiling (#577).
//
// Before the fix, backoffDelay computed base * time.Duration(1<<(attempt-1)).
// The shift is evaluated in int: at attempt 64 it is 1<<63, which is negative,
// and past 64 Go defines an over-wide shift as 0. Multiplying a 1s base by
// either produces a delay that is negative or zero, and time.After / time.Sleep
// treat a non-positive duration as "no wait" — so a long failure streak stopped
// backing off entirely and hammered a server already answering 429/503.
//
// WithMaxRetries only rejects n < 1, so a caller can set the attempt count that
// reaches this. The bound is two-sided on purpose: a one-sided "never too long"
// check passes on exactly the attempts that tight-loop.
func TestBackoffDelaySaturates(t *testing.T) {
client := &Client{httpOpts: HTTPOptions{
BaseDelay: DefaultBaseDelay,
MaxJitter: DefaultMaxJitter,
}}

// With a 1s base, attempt 6 is the first whose unclamped term (32s) exceeds
// the 30s ceiling, so every attempt from there on must sit at the ceiling.
for _, attempt := range []int{6, 10, 32, 33, 62, 63, 64, 65, 128, 1 << 30} {
delay := client.backoffDelay(attempt)
if delay < MaxBackoffDelay || delay > MaxBackoffDelay+DefaultMaxJitter {
t.Errorf("backoffDelay(%d) = %v, want within [%v, %v]",
attempt, delay, MaxBackoffDelay, MaxBackoffDelay+DefaultMaxJitter)
}
}

// The unsaturated attempts keep their exact exponential values, so the
// ceiling changes nothing on any path a shipped configuration reaches.
for attempt, want := range map[int]time.Duration{1: time.Second, 2: 2 * time.Second, 3: 4 * time.Second} {
delay := client.backoffDelay(attempt)
if delay < want || delay > want+DefaultMaxJitter {
t.Errorf("backoffDelay(%d) = %v, want within [%v, %v]", attempt, delay, want, want+DefaultMaxJitter)
}
}
}

// TestBackoffDelayEdgeConfigurations covers the two configurations that make
// the clamp arithmetic itself interesting: a base delay above the ceiling
// (SPEC §7 requirement 3 — it is clamped, with no carve-out for the first
// sleep) and a zero base delay (must stay at zero rather than saturating,
// since MaxBackoffDelay/0 would otherwise divide by zero).
func TestBackoffDelayEdgeConfigurations(t *testing.T) {
above := &Client{httpOpts: HTTPOptions{BaseDelay: 10 * time.Minute, MaxJitter: DefaultMaxJitter}}
if delay := above.backoffDelay(1); delay < MaxBackoffDelay || delay > MaxBackoffDelay+DefaultMaxJitter {
t.Errorf("base delay above the ceiling: backoffDelay(1) = %v, want the ceiling %v", delay, MaxBackoffDelay)
}

zero := &Client{httpOpts: HTTPOptions{BaseDelay: 0, MaxJitter: DefaultMaxJitter}}
if delay := zero.backoffDelay(90); delay > DefaultMaxJitter {
t.Errorf("zero base delay: backoffDelay(90) = %v, want at most the jitter %v", delay, DefaultMaxJitter)
}
}
38 changes: 36 additions & 2 deletions go/pkg/basecamp/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -906,15 +906,49 @@ func (c *Client) assertCredentialOrigin(req *http.Request) error {
}

func (c *Client) backoffDelay(attempt int) time.Duration {
// Exponential backoff: base * 2^(attempt-1)
delay := c.httpOpts.BaseDelay * time.Duration(1<<(attempt-1))
delay := saturatingBackoff(c.httpOpts.BaseDelay, attempt)

// Add jitter
jitter := time.Duration(rand.Int63n(int64(c.httpOpts.MaxJitter))) // #nosec G404 -- jitter doesn't need cryptographic randomness

return delay + jitter
}

// saturatingBackoff computes base * 2^(attempt-1) for a 1-based attempt,
// saturating at MaxBackoffDelay (SPEC §7, "Backoff Ceiling").
//
// The clamp is load-bearing. `1 << (attempt-1)` is evaluated in int: at attempt
// 64 that is 1<<63, which is negative, and Go defines a shift at or past the
// operand width as 0. Either way the product stops being a delay — and both
// time.Sleep and time.After treat a non-positive duration as "no wait", so an
// unclamped formula turned a long failure streak into a retry loop with no
// backoff at all, against a server already answering 429/503.
//
// The multiplier is compared against MaxBackoffDelay/base before multiplying,
// so no intermediate can overflow int64.
func saturatingBackoff(base time.Duration, attempt int) time.Duration {
if base <= 0 {
return 0
}
shift := attempt - 1
if shift < 0 {
shift = 0
}
// 62 is the largest shift that stays positive in int64; anything at or past
// it is far above any ceiling worth computing.
if shift >= 62 {
return MaxBackoffDelay
}
// The multiplier is a plain count, not a duration — multiplying two
// time.Durations would be a unit error (and golangci's durationcheck says
// so), so the arithmetic happens in int64 nanoseconds.
multiplier := int64(1) << uint(shift)
if multiplier > int64(MaxBackoffDelay)/int64(base) {
return MaxBackoffDelay
}
return time.Duration(int64(base) * multiplier)
}

// parseNextLink extracts the next URL from a Link header.
func parseNextLink(linkHeader string) string {
if linkHeader == "" {
Expand Down
9 changes: 9 additions & 0 deletions go/pkg/basecamp/http.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,15 @@ const (
DefaultMaxPages = 10000
)

// MaxBackoffDelay is the ceiling on the exponential backoff term (SPEC §7,
// "Backoff Ceiling"). Jitter is added after the clamp, so the longest single
// backoff sleep is MaxBackoffDelay + MaxJitter.
//
// It matches the generated client's RetryConfig.MaxDelay, which has capped at
// 30s since that retry loop was first templated; #577 generalized the value to
// every backoff site in every SDK.
const MaxBackoffDelay = 30 * time.Second

// HTTPOptions configures the HTTP client behavior.
type HTTPOptions struct {
// Timeout is the request timeout (default: 30s).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -334,9 +334,36 @@ internal class BasecampHttpClient(

private const val MAX_JITTER_MS = 100L

/** Exponential backoff: base * 2^(attempt-1) + jitter. */
/**
* Ceiling on the backoff term (SPEC §7 "Backoff Ceiling"). Jitter is
* added on top, so the longest single backoff sleep is this plus
* [MAX_JITTER_MS].
*/
internal const val MAX_BACKOFF_DELAY_MS = 30_000L

/**
* Exponential backoff: `min(base * 2^(attempt-1), MAX_BACKOFF_DELAY_MS) + jitter`.
*
* The clamp is load-bearing, not defensive. `1L shl 63` is
* [Long.MIN_VALUE] and `1L shl 64` wraps the shift back to 1, so an
* unclamped product goes negative at attempt 54, lands on exactly 0 at
* 63, and collapses to the base value past 64 — and `delay()` returns
* immediately for any non-positive argument. The result is a client
* hammering a server that is already answering 429/503, which is the
* precise traffic pattern backoff exists to prevent.
*
* The multiplier is compared against `MAX_BACKOFF_DELAY_MS / base`
* before multiplying, so no intermediate ever leaves `Long` range. The
* shift itself is bounded at 62 for the same reason.
*/
internal fun calculateBackoffDelay(baseDelayMs: Long, attempt: Int): Long {
val delay = baseDelayMs * (1L shl (attempt - 1))
val base = baseDelayMs.coerceAtLeast(0L)
val multiplier = 1L shl (attempt - 1).coerceIn(0, 62)
val delay = when {
base == 0L -> 0L
multiplier > MAX_BACKOFF_DELAY_MS / base -> MAX_BACKOFF_DELAY_MS
else -> base * multiplier
}
val jitter = (kotlin.random.Random.nextLong(MAX_JITTER_MS))
return delay + jitter
}
Expand Down
Loading
Loading