Skip to content

feat(events): persist events to disk so they survive the process (tier 3) - #538

Open
abelonogov-ld wants to merge 184 commits into
andrey/event-durability-tier2-bounded-flushfrom
andrey/event-durability-tier3-persistence
Open

abelonogov-ld wants to merge 184 commits into
andrey/event-durability-tier2-bounded-flushfrom
andrey/event-durability-tier3-persistence

Conversation

@abelonogov-ld

@abelonogov-ld abelonogov-ld commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Third in the event-durability stack, on top of #536. Recorded events go to an on-disk log rather than an in-memory buffer, so an application that dies moments after recording one still reports it: the next run delivers what the last one left behind.

What changes

EventStore is a new on-disk log, with capacity, framing, interrupted-log recovery, and multi-process safety. NullEventStore stands in where the platform gave the SDK nowhere to write, which is the behaviour the SDK had before.

Recording no longer encodes on the caller's thread. An evaluation appends to a held run and updates a counter; a commit turns the whole run into bytes at once. Recording a custom or identify event is a commit point, because an application reporting something it chose to report is saying it matters more than the microseconds it costs. Evaluations commit once pendingCommitThreshold of them accumulate, when a delivery starts, or on flush.

LDConfig.eventPersistence lets the application choose whether a commit point runs on the caller's thread (.immediate), off it (.deferred), or not at all (.disabled).

Delivery sends batches oldest first and stops at the first retryable failure, so the rest keep their place in the log instead of the SDK spending the session's requests on a service that is refusing them.

Reconciling with tier 2

Tier 2 replaced the delivery gate this branch was built on, so the merge had to pick one design rather than keep both.

Coalescing comes from tier 2: flushes arriving before a delivery starts join it, one already started is never joined, and nothing holds the drain back while a retry runs. What tier 2 did not need is a way to keep an in-flight batch from being sent twice — it empties its buffer before publishing, so a second delivery finds nothing. Here a batch stays on disk until its response arrives, which is what isDelivering guarded.

batchesInFlight takes its place. A delivery claims the batches it lists and releases them once it has an answer, so a later delivery still runs and still sends what it closed for itself; only the batches already on their way are left out. A caller whose own events are all that remain unsent is answered true, which is exactly the contract tier 2 documents: an answer is about the batch the caller's own delivery took, never one an earlier delivery lost.

The loss flag is now taken once per delivery rather than once per caller, so everyone joined to it hears about a loss instead of only whoever was answered first, and a scheduled delivery with no waiters leaves it for the caller who is owed it. This branch had also fixed the duplicate event reporter independently; that now comes from #537, so the local copy is gone.

Verification

  • 824 tests pass on iOS; macOS and tvOS build. No new SwiftLint violations.
  • The in-flight claim is load-bearing: removing the filter fails 5 tests, including does not send a batch again when a delivery is asked for while one is in flight.
  • DiagnosticReporterSpec.testRetries flaked once under load and passed on re-run; it has a 0.1 s timeout and is unrelated to these changes.

Note

Overview
Summary
Introduces a disk-backed event processing pipeline with DirectEventProcessor, EventStore, DiskEventLog, and coordinated components (EventCommitter, EventDeliveryScheduler, AnalyticsEventSender, OutboundEventBuffer, BackgroundEventDelivery) so analytics batches survive process death, recover safely, and respect byte/age budgets.

Configuration & API
Adds EventPersistence modes plus eventCapacityBytes, eventMaxAge, and eventDeliveryDelay, raises default eventCapacity, and exposes matching Objective-C surfaces. LDClient now wires eventProcessor instead of EventReporter, delays diagnostics until the processor’s launch gate opens, and centralizes background commit/delivery handling without blocking the instances queue. Flush APIs, docs, and Obj-C shims were rewritten to commit to disk, bypass backoff when required, and report aggregated outcomes.

Infrastructure
Extends CRC helpers, HTTP response parsing for retry semantics, BackgroundActivity/SystemCapabilities platform hooks, regenerated mocks (EventProcessorMock, manual DiagnosticCachingMock), and comprehensive Quick/Nimble coverage (EventStoreSpec, BackgroundEventDeliverySpec, EventProcessorOutcomesSpec, renamed DirectEventProcessorSpec).

Reviewed by Cursor Bugbot for commit 3f8bab5. Bugbot is set up for automated code reviews on this repo. Configure here.

abelonogov-ld and others added 22 commits September 18, 2026 18:38
Tiers 1 and 2 both need the process to survive. When it does not, everything
recorded since the last delivery goes with it -- including the crash report an
application was recording when it died. Tier 3 is the answer to that: events are
written to a log under Application Support, and a later launch delivers whatever
the previous run left behind.

The write is a plain append into the page cache, never an fsync, so a commit
point costs tens of microseconds rather than milliseconds. track and identify
are commit points: at .immediate the write happens on the calling thread, so a
process that dies the instant after track returns still reports that event.
Evaluations are not -- they are held as objects and encoded in runs, so the
hottest path in the SDK pays a counter bump and nothing else.

This is off by default. Writing events changes what a track costs and leaves a
directory of the application's data on the device, and a minor version cannot
decide either on an application's behalf. LDConfig.eventPersistence has three
levels: disabled, which is what Tiers 1 and 2 describe and is the default;
deferred, which writes off the calling thread; and immediate, which is the whole
of the above.

Delivery is held for the round trip rather than just its start, because a batch
now stays on disk until its response is handled and a second delivery beginning
inside that window would send it again under the same payload ID.

Spec: Event Durability §13, §14; Page-Cache Event Persistence.
Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
Tier 3 stages encoded bytes into the store instead of holding events in
memory, so the assertion that nothing was buffered has to ask the store.
…y/event-durability-tier3-persistence

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence

Also renames handWrittenEncoder to eventJSONWriter, after its type.
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly.xcodeproj/project.pbxproj
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventJSONWriter.swift
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
#	LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>
…atomically; reserve capacity during encode

- Without persistence, nothing but a delivery drains the held run: no threshold commit, and track, identify, flush and backgrounding no longer encode on the caller's thread.
- An evaluation's summary counter and its full/debug events are recorded under pendingLock, and a commit takes both under it.
- A commit reserves capacity for the run it takes, so events being encoded still count against eventCapacity.

Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/LDClient.swift
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
#	LaunchDarkly/LaunchDarklyTests/Mocks/DarklyServiceMock.swift
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/LDClient.swift
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
#	LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/LDClient.swift
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
#	LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
… failed write left in memory

A read can fail on a file that is intact, as it does in a process out of file descriptors, and
every failure was treated as a batch that can never be delivered and deleted. Listing now skips
such a batch until a later listing can read it, delivery retries it, and startup recovery closes
the previous run's log off uncounted rather than deleting it. Only a file that is gone, or one
written in a format this version does not read, is nothing to keep.

closeBatch also closed only the log where a failed write had left events both there and in
memory, so a flush could report success and close() discard the rest.

Co-authored-by: Cursor <cursoragent@cursor.com>
…d not be kept

A batch LaunchDarkly refuses in a way that may pass is kept and retried, so it is not a loss.
What is lost is what the store never received: an event or a summary that could not be serialized
when it was committed, or that serialized too large to store. A delivery that finds nothing to
send cannot tell those apart from events that arrived, so flushAndWait reported success for them.

Co-authored-by: Cursor <cursoragent@cursor.com>
Android pins the failed write at the processor level, where the bug was visible: the flush
reports success and both the written and the held events arrive. It also pins that a body reads
correctly once reads work again, and that a previous run's log moved aside leaves this run the
name to append to.

Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Tier 2 replaced the delivery gate this branch was built on, so the two
designs had to be reconciled rather than taken side by side.

Coalescing comes from tier 2: flushes arriving before a delivery starts join
it, and one that has already started is never joined, so nothing holds the
drain back while a retry runs. What tier 2 did not need is a way to keep an
in-flight batch from being sent twice. Tier 2 empties its buffer before
publishing, so a second delivery finds nothing; here a batch stays on disk
until its response arrives, which is what `isDelivering` used to guard.

`batchesInFlight` takes its place. A delivery claims the batches it lists and
releases them when it has an answer, so a later delivery still runs and still
sends what it closed for itself, and only the batches already on their way
are left out of it. A caller whose own events are all that is left unsent is
answered `true`, which is the contract tier 2 documents: an answer is about
the batch the caller's own delivery took.

The loss flag is now taken once per delivery rather than once per caller, so
everyone joined to it hears about a loss instead of only whichever was
answered first, and a scheduled delivery with no waiters leaves it for the
caller who is owed it. `flush` stays a commit point on both paths.

This branch had also fixed the duplicate event reporter on its own, as
`eventSyncLogger`; that now comes from the precursor branch instead, so the
local copy goes.

Co-authored-by: Cursor <cursoragent@cursor.com>
@abelonogov-ld
abelonogov-ld requested a review from a team as a code owner October 8, 2026 22:44

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@abelonogov-ld
abelonogov-ld marked this pull request as draft October 9, 2026 00:44
abelonogov-ld and others added 2 commits October 9, 2026 05:37
Functions that must run under a lock now have plain names, keep "Requires
`xLock`" in their docs, and check it with UnfairLock.assertOwned(), which
traps in debug builds when the calling thread does not hold the lock.

Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/UnfairLock.swift
…'s thread

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

abelonogov-ld and others added 2 commits November 1, 2026 12:46
…rites everything before it returns

Reverts 93f6ed5: a flush, or entering the background, returned while the
split of an oversized summary was only queued, leaving those evaluations in
memory.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ence

The caller's thread encodes and writes the one event, through encoders of its
own and without waiting for a commit in progress; the held evaluations and the
summaries are left to a commit queued on the commit queue in the same call. The
summarizer's counters are still taken at each track or identify, so its
contexts are cleared as before.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

abelonogov-ld and others added 13 commits November 1, 2026 20:59
… reserved place

A track or identify at immediate persistence committed the store, which wrote
whatever a commit in flight had staged so far, up to megabytes of a run or a
split summary, on the application thread. It now writes just its own frame and
leaves the rest to the commit that staged it.

It is also reserved where it is admitted, so a recording on another thread
cannot take its place while it is encoded and have it refused as a loss. A
commit now releases only its own unused reservations rather than all of them.

Co-authored-by: Cursor <cursoragent@cursor.com>
A summary too large for one log frame is refused whole by the store,
logged, and counted as lost, so the flush that committed it answers
false. Reaching that takes one context evaluating dozens of the largest
JSON flags between two commits, which did not justify the packing and
halving that split it.

Co-authored-by: Cursor <cursoragent@cursor.com>
…e again

Writing only the tracked event left the evaluations before it to a commit
queued on the commit queue, so a kill straight after the track, which is what
immediate persistence is for, could take those exposures with it. With the
summary split gone, the commit costs the caller a normal encode of the held run
and the summaries, so the track commits it again before returning.

This reverts 9173f03 and 5c61258.

Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/LDClientFlush.swift
#	LaunchDarkly/LaunchDarkly/ObjectiveC/ObjcLDClient.swift
Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…only Debug

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… several threads

Co-authored-by: Cursor <cursoragent@cursor.com>
…nit event whether or not events persist

Co-authored-by: Cursor <cursoragent@cursor.com>
…and say where they are

The message said the events remain on disk even where persistence is off, and
was at debug level only. Where the store persists it stays at debug and says
they remain on disk for the next launch; otherwise it is at info and says they
are only in memory and lost if the application is terminated.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ies, and event backgrounding into its own class

Three places each had their own platform list: LDClient delivered under an
assertion on iOS and tvOS, BackgroundActivity took assertions on iOS, tvOS and
watchOS, and SystemCapabilities posted lifecycle notifications on iOS, tvOS and
macOS. SystemCapabilities now says, per platform, whether a backgrounded
process is suspended and how to get an assertion, beside the notifications.
BackgroundEventDelivery owns what events do on backgrounding: commit, then
deliver under an assertion where the platform suspends, and log what did not
get out. LDClient and BackgroundActivity carry no platform conditions.

watchOS is now stated rather than implied: it suspends and has assertions,
but posts no notification the SDK observes, so nothing happens there.

Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarklyTests/Networking/DarklyServiceSpec.swift

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

writingSummaries -= staged.summaries - heldSummaries
writingBytes -= frames.count - held
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failed in-memory close drops events

Low Severity

If assembleBody fails while closing staged frames into in-memory batches, the remaining frames are uncounted and discarded instead of being put back. Those events are not reported as lost, so a later flush can still return success. This is the close path used when persistence is off or a write has already failed.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 7bebd16. Configure here.

abelonogov-ld and others added 5 commits November 5, 2026 20:48
The accessors only tests read -- the delivery counts, pending events, the
summarizer and response-date hooks, the queue occupier and the batch lock --
move into DEBUG extensions. The delivery counter itself stays in every build:
it is one increment under a lock already held.

Co-authored-by: Cursor <cursoragent@cursor.com>
Backgrounding commits every started environment before the notification
returns and, where the process is suspended, delivers them all under one
background activity assertion, released once every environment has
answered. Closed clients the application still holds are no longer
reached. Run mode stays per client.

Co-authored-by: Cursor <cursoragent@cursor.com>
flush(completion:), flushAndWait and the background delivery each started
every environment's delivery and counted answers. They now share
flushReportingOutcomes, which tracks each environment, so a reporter
answering twice can no longer end flushAndWait before every environment
has answered.

Co-authored-by: Cursor <cursoragent@cursor.com>
flushReportingOutcomes takes an optional deadline, after which the
environments that have not answered count as undelivered and a late
answer is ignored. The background delivery uses the assertion's 30s cap,
so a delivery that never answers no longer keeps the activity held and
every later backgrounding from delivering. Pins the once-only contract
with never-answering and double-answering reporters.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@cursor

cursor Bot commented Nov 6, 2026

Copy link
Copy Markdown

Current version of PR was reviewed by /review-bugbot with additional instructions on Nov 5, 23:10 PST. It flagged 0 findings.

Bugbot on commit bcbde8a is skipped.

abelonogov-ld and others added 2 commits November 6, 2026 07:57
It was queued unpaced, so it sent a backlog back to back and ended the
pacing of a delivery already running, competing with flag requests.

Co-authored-by: Cursor <cursoragent@cursor.com>
A block on deliveryQueue cannot run below the queue's own userInitiated QoS,
so asking for utility with .enforceQoS left the recovery at userInitiated.
Run it on a utility global queue instead, with deliveryQueue suspended until
it is done, so deliveries, the report timer and the launch gate still run
after it.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

close() held instancesQueue while each environment's flushAtClose() waited up
to closeCommitBudget for its final commit, so LDClient.get(), flush() and
every other call reading instances waited behind those writes. Take the
clients and clear instances under the queue, then close them outside it.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3f8bab5. Configure here.

defer { LDClient.instances = nil }
return LDClient.instances.map { Array($0.values) } ?? []
}
clients.forEach { $0.internalClose() }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close overlaps a concurrent client start

Medium Severity

close() now clears instances before internalClose() runs, so another thread can observe get() as nil and call start() while the old clients are still committing and delivering. Two EventStores then share one directory: the new client can recover and send a batch the closing client is still posting, duplicating events. Previously start() waited on instancesQueue until teardown finished, so this overlap could not happen.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 3f8bab5. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants