feat: Back off big segment synchronization on a rejected SDK key

[keelerm84]

Summary

The big segment synchronizer retried on a flat 10s interval and intended to stop on an unrecoverable HTTP status. It never stopped. poll and connectStream both return &httpStatusError{...}, while syncSupervisor type-asserted for a value-typed httpStatusError, so isHTTPErrorRecoverable was never consulted. A rejected SDK key produced an unbounded 10s hot loop against a service refusing every request, with no error-level log to say so.

Stopping would have been the wrong fix. An operator can make a key valid again without Relay knowing, so the synchronizer now keeps retrying in every case and sorts each failure into one of two classes instead:

• A transient failure stays on a short curve: 10s base, 30s ceiling.
• A failure unlikely to correct itself soon raises the ceiling to an hour, starting from a 5m base. That bounds how much load a fleet of Relay Proxy instances puts on a service that is rejecting all of its requests. Big segment data is reported as potentially stale long before that, so a long wait stays visible rather than silent.

Only an HTTP status can be unexpected, and among 4xx only 400, 408 and 429 are treated as transient. Every transport-level failure, certificate validation included, is normal: it either resolves without the synchronizer's involvement, as a network problem does, or it resolves the moment an operator fixes it, and waiting minutes helps in neither case.

Marking the store as synchronized is the health signal, not a successful HTTP response, because the store is what readers depend on. Enough consecutive marks return the retry state to the short curve, so a key that gets fixed does not leave the process stuck on the hour-long ceiling.

internal/retry

The classification and the curve live in a new internal/retry package rather than in bigsegments, because the auto-configuration stream needs the same behavior bound to its own timings. That is the next ticket, and it is why this is a package and not a couple of methods on the synchronizer.

Also in here

A timer leak. The old defer timer.Stop() sat inside the retry loop, so it accumulated one undrained timer per retry for the life of the goroutine.

Relationship to v8

This brings v9 to parity with v8 #857 and #873. internal/retry is copied across unmodified -- it imports only math, math/rand and time.

TestSyncKeepsRetryingAfterUnauthorized is the regression test for the dead type assertion, and its comment records that history so the next reader does not have to rediscover it.

---

Note

Overview
Replaces the big segment synchronizer’s fixed 10s retry with a shared internal/retry strategy: transient failures use a short exponential curve (10s base, 30s cap), while “unexpected” HTTP failures—mainly rejected credentials like 401—switch to a much slower curve (5m base, 1h cap) while still retrying (operators can fix keys without restarting Relay).

Failure handling now uses errors.As on *httpStatusError (fixing a bug where a value-type assertion meant unrecoverable statuses never triggered the old stop/recover logic) and routes statuses through retry.ClassifyHTTPStatus. Transport/TLS errors stay on the normal curve; unexpected failures log at error and emit an info line when extended backoff engages. setSynced calls OnHealthy() so sustained successful store marks reset backoff after retryResetThreshold.

Also fixes a timer leak in syncSupervisor (removed defer timer.Stop() inside the retry loop) and adds broad unit/integration tests for 401 persistence, 503 vs extended mode, health reset, stream-end doubling, and cert failures.

^{Reviewed by Cursor Bugbot for commit 2370b36. Bugbot is set up for automated code reviews on this repo. Configure here.}