Skip to content

feat: replace permanent event-delivery disable with a bounded cooldown (SDK-3375) - #256

Draft
tanderson-ld wants to merge 3 commits into
mainfrom
ta/sdk-3375/evtproc-cooldown
Draft

tanderson-ld wants to merge 3 commits into
mainfrom
ta/sdk-3375/evtproc-cooldown

Conversation

@tanderson-ld

Copy link
Copy Markdown
Contributor

What

Implements the EVTPROC event processor spec's Section 1.8 delivery cooldown (launchdarkly/sdk-specs#278) in the shared internal events code: on an unexpected delivery failure (a non-recoverable HTTP status such as 401 or 403), the event processor enters a bounded, self-recovering cooldown window instead of permanently disabling event delivery for the lifetime of the SDK instance.

  • Windows run 5, 10, 20, 40, then 60 minutes (capped at the 1 hour ceiling), stepping once per failed probe. They are deterministic: RETRY 1.4.3 permits zero jitter for a component whose attempts are internally scheduled, and the flush schedule itself paces attempts, which is also why the windows are not floored at the flush interval.
  • While a window is open, every flush (scheduled, flushAsync()/flushBlocking(), or the shutdown flush) is a no-op: it makes no network attempt and disposes of the batch exactly as a sent-and-refused payload would be disposed of. Two deliberate departures from pure refusal emulation: the diagnostic events-in-last-batch stat reports 0 (no batch was serialized), and the context deduplicator is flushed when the discarded batch contained an index or identify event, so a later payload can introduce those contexts again.
  • Recording is never gated (1.8.8): evaluations keep updating summary counters and intake continues during a window.
  • A successful probe returns delivery to the flush cadence immediately (1.8.3); retry state resets only after 2 consecutive successful deliveries (1.8.4). No cooldown state persists across restarts (1.8.10). The recoverable-failure retry inside DefaultEventSender (1.7.2) is unchanged.

This is a port of the reviewed Go implementation (launchdarkly/go-sdk-events#68), adapted to this module's threading model: delivery outcomes arrive on flush worker threads and the shared executor rather than on the dispatcher thread, so the cooldown tracker is synchronized pure state, and outcomes are correlated to the window era they were dispatched in via an opaque WindowToken (an outcome from a delivery in flight across a window boundary updates nothing).

API (internal module surface)

  • EventSender.Result gains the unexpected classification with static factories success(Date), failure(), and unexpectedFailure(), and the accessor isUnexpectedFailure(), which also accepts the legacy flag so results from senders written against the old API still get cooldown behavior.
  • The three-argument Result constructor and isMustShutDown() are deprecated. unexpectedFailure() keeps writing the legacy flag for consumers that have not migrated (the server SDK's EventSenderWrapper and the Android SDK).
  • DefaultEventSender migrates off the deprecated isHttpErrorRecoverable/checkIfErrorIsRecoverableAndLog helpers to classifyHttpFailure (first commit, behavior-identical) and then returns the new classification (second commit). Transport-level IOExceptions keep their always-retry behavior.
  • EventsConfiguration gains a package-private cooldownNanoTimeFn test seam (deliberately not part of any constructor); the tracker documents that the source must have System.nanoTime() semantics, and all window arithmetic uses overflow-safe difference comparisons.

Design notes

  • Diagnostic event delivery is not gated by the window (matching the spec, which does not cover diagnostics). Diagnostic outcomes drive the cooldown only the way they drove the old permanent disable: an unexpected failure on a diagnostic payload opens a window, while a diagnostic success or ordinary failure has no effect on delivery state - in particular, a diagnostic success is not the probe and cannot count toward the reset threshold, since only analytics deliveries prove the service is accepting analytics payloads again. (Without that asymmetry, periodic diagnostic successes would hold the ladder at 5 minutes forever while analytics payloads kept failing.) A diagnostic dispatched inside a window carries no live token, so its outcome cannot move delivery state even if it lands after the window elapses. This coupling preserves this module's existing shape and is a deliberate divergence from the Go implementation, which excludes diagnostic results entirely; it is flagged for spec follow-up.
  • Server-time calibration from the Date header remains unconditional, exactly as on main.
  • The disabled flag remains as a shutdown-only latch; nothing sets it from delivery results anymore.
  • A delivery that dies in an exception (in serialization, the sender, or a custom sender returning null) is recorded as an ordinary failure, breaking the consecutive-success streak without opening a window, and can no longer kill the delivery worker: outcome handling is guarded so the busy-worker count always comes back down and close() always returns.
  • The cooldown check and the delivery token issue are one atomic tracker call, so a window opening concurrently on a worker thread either diverts the flush to the discard path or makes the dispatched payload's token stale. The once-per-window Warn is also claimed under the tracker's own lock.
  • The tracker's clock seam is a package-private NanoTimeSource interface rather than java.util.function.LongSupplier, which does not exist on Android below API 24 (this module ships in the Android SDK at minSdk 21); the fix is verified with a d8 --min-api 21 dex sweep showing zero java.util.function references.
  • Logging: one Error when a window opens (with its length), a Warn for the first discarded batch of each window (Debug thereafter), and an Info when delivery resumes.
  • Java has no failed-send metrics surface, so the in-window discard fires no metric; the Warn/Debug discard logs carry the count and remaining window.

Testing

  • DeliveryCooldownTrackerTest (24 tests): window ladder with every rung asserted past the shift-guard bound, reset semantics, boundary-round staleness on both probe success and recoverable probe failure, stale-outcome and in-window-outcome guards, null token, default clock path, boundary instant, the atomic gate, and the once-per-window discard notice.
  • DefaultEventProcessorCooldownTest (18 tests): cooldown entry and resume across a fake clock, legacy shutdown-flag results, contradictory success+shutdown results, in-window disposal with context-introduction recovery through both the index and identify arms, summary-only discards leaving the deduplicator alone, empty in-window flushes doing nothing, shutdown during a window, warn-once-then-debug and resume logging, the diagnostic coupling (including a diagnostic held in flight across the window boundary), a null sender result neither stopping delivery nor wedging close(), a sender exception breaking the success streak, simultaneous failures across the full worker pool stepping the ladder exactly once, and the batch-size stat zeroing on discards.
  • diagnosticEventsStopAfter401Error is rewritten as diagnosticEventsContinueAfter401Error; the permanent-disable test is renamed to describe the cooldown contract; a transport-failure sender test pins the deliberate never-unexpected classification of connection errors.
  • Full module suite: 360 tests green; javadoc and checkstyle clean; javadoc warning count unchanged from main.
  • Four multi-agent review rounds ran against this change before it was opened (two on the initial implementation, two on the fixed state), including adversarial reviews with runnable proofs on a real Android API 23 emulator image, a d8/dexdump Android compatibility sweep, mutation testing of the new tests, and behavioral-parity diffing against the spec and the Go reference.

Out of scope (follow-ups)

  • The 413 carve-out: classifyHttpFailure has no locally-unrecoverable classification, so a 413 opens a window rather than dropping only the oversized payload. That is a pre-existing classifier gap with its own Jira bug (it also requires unsuppressing the corresponding contract test), deliberately not fixed here.
  • The server SDK changes (dependency bump, public EventSender.Result.STOP doc reinterpretation, wrapper migration, event-delivery-cooldown contract-test capability, and the 413 test suppression) ship in a follow-up PR once this module releases.
  • Android SDK adoption: the Android SDK's own DirectEventProcessor reads the dual-written legacy flag and keeps its current permanent-disable behavior byte-identical before and after this release; migrating it to the cooldown is its own coordinated change.
  • Diagnostics RETRY alignment is separate spec work.

Release notes

  • Minor (feat). Behavior change worth calling out: a custom event sender result carrying the legacy shutdown flag now causes a temporary cooldown rather than a permanent stop.

…sifier

DefaultEventSender now classifies HTTP failures with
HttpErrors.classifyHttpFailure instead of the deprecated
isHttpErrorRecoverable/checkIfErrorIsRecoverableAndLog helpers. Behavior
and log output are unchanged: the set of statuses treated as unexpected
is identical, unexpected failures still skip the retry and still log at
error level with the same wording, and transport-level exceptions keep
their warn-and-retry-once handling.
…n (SDK-3375)

On an unexpected delivery failure (a non-recoverable HTTP status such as
401 or 403), the event processor now enters a bounded, self-recovering
cooldown window instead of permanently disabling delivery for the
lifetime of the SDK instance, per the EVTPROC spec's Section 1.8.

Windows double from 5 minutes to a 1 hour ceiling, one step per failed
probe. In-window flushes of any kind are no-ops that dispose of the
batch exactly as a sent-and-refused payload would be, without the
network or serialization cost. Recording is never gated, retry state
resets after two consecutive successful analytics deliveries, and no
cooldown state persists across restarts. Diagnostic sends are not gated
by the window; an unexpected diagnostic failure opens one, as it
previously disabled delivery, but diagnostic successes do not drive the
delivery schedule.

EventSender.Result gains the unexpected classification with factory
methods and the isUnexpectedFailure accessor; the legacy constructor and
isMustShutDown are deprecated and still dual-written for consumers that
have not migrated.
Diagnostics obtain their delivery token through the same atomic
cooldown-check-plus-issue call that analytics flushes use, so a delivery
dispatched inside a window carries no token at all and every pre-window
token goes stale when the window opens. All three in-cooldown guards are
therefore uniformly defensive; one of them previously claimed to be
reachable from in-window diagnostic failures.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant