feat: Keep retrying a rejected auto-configuration key

[keelerm84]

Summary

A 401 or 403 on the auto-configuration stream stopped the stream and exited the process. A whole fleet did that at once, recovery took an operator restarting every instance, and a persistent store holding a usable configuration was thrown away.

The stream now keeps retrying. That is what Relay already did for every other failure to reach LaunchDarkly: a 404 from a misconfigured stream URI, any 5xx, a network error, a certificate problem -- all of them left Relay running and trying indefinitely. A rejected key was the single failure treated as permanent, and it was the one an operator was most likely to fix.

A failure that is unlikely to correct itself soon activates a second eventsource retry profile, 5 minutes growing to a one-hour ceiling, so a fleet does not add load to a service that is rejecting every request. Failures that were already retried keep the delays they had, so an outage takes a byte-for-byte identical path.

What happens with no configuration

Relay stays up and answers 503 until the key becomes valid, exactly as it already did for a 404 or a 5xx on the same stream. If a persistent store holds a configuration, Relay serves those environments while it retries -- which is the case that previously exited and discarded the cache.

Detecting a bad key is now the status resource's job rather than the process exiting. #870 reports the stream state and the status code, and the log carries Invalid auto-configuration key; will keep retrying in case it becomes valid. docs/proxy-mode.md says so, for anyone who relied on the exit.

Two consequences in the code

The status is INTERRUPTED rather than OFF for a rejected key, because the stream is still trying. OFF now means only that Close was called.

The stream runs on a cancellable context. Close waits on the subscribe goroutine, and that goroutine cannot exit during a backoff wait -- eventsource's retry loop selects on the request context and the delay timer and nothing else. Without the cancel, closing a Relay whose key was rejected would block for the remainder of a delay that now reaches an hour. This was already a latent bug at the old 30-second ceiling; the longer delays made it matter.

On the size of this change

An earlier revision of this PR was twice as large. It kept the old "exit when Relay cannot serve" behavior and made it conditional on the cache being reachable and empty, which needed a notification channel, three pieces of wait-loop state, a give-up predicate, a distinction between an unreadable cache and an empty one, a bounded cache read, and two new constructor parameters -- about 113 lines implementing when to exit rather than how to retry.

Once it was clear that Relay already stayed up forever for every other failure, all of that became unnecessary. NewStreamManager's signature is unchanged, no configuration option changes meaning, and stream_manager.go is 124 added lines instead of 249.

Tests

Both behaviors are pinned against their own mutation. Removing the profile activation fails TestExtendedDelaysDoubleFromTheExtendedBase; restoring CloseNow: true for a 401 fails the recovery and status tests.

stream_manager_known_limits_test.go records one thing deliberately left alone: a server-directed retry: hint replaces the active profile's base delay and is never cleared, so it would flatten the extended delays. LaunchDarkly does not send one, so reaching it needs an on-path intermediary.

Deliberately not addressed

The cache has no write timestamp, so lastConfigReceived reports when Relay read it rather than how stale it is. A partial DynamoDB snapshot is still read as authoritative; #874 makes the write report the loss, but detecting it on read needs a completeness marker. Event forwarding still latches off permanently on a 401 (SDK-3067). Relay's SDK-facing auth gate still keys on the SDK client's construction error rather than on data availability.

docs/proxy-mode.md:33 claims that in standard mode Relay "will give up and shut down completely" when initTimeout elapses. It does not: waitForAllClients is the only thing that reports that, and relay.go:256 is its only caller, inside if c.Main.ExitAlways. That predates this change, so it needs its own fix.

Release note

Nothing here has shipped. The last release is 8.21.0, and the SDK bump (#856), big segment backoff (#857), status reporting (#870), transport classification (#873) and the DynamoDB write fix (#874) all sit above it. The whole retry story lands as one new thing, so the note should describe the end state rather than the increments.

[tanderson-ld]

Is this a temporary option for backwards compatibility in the current major version?

[tanderson-ld]

This is concluding the error is an auth failure when classifyAndLogStreamError only says it is Unexpected. Does the authFailureCh chan need to be generalized to unexpectedErrorChan? Does the other logic using the authFailureCh remain valid?

An unexpected error that is not an auth failure includes various TLS errors as well as any future errors that get added to the category.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 18bcac3. Configure here.}