Skip to content

Signalling WebSocket over Network.framework: URLSession keeps every WebSocket session after close - #1173

Open
TimofeyIvanenko wants to merge 11 commits into
livekit:mainfrom
nomix-ai:websocket-network-framework
Open

TimofeyIvanenko wants to merge 11 commits into
livekit:mainfrom
nomix-ai:websocket-network-framework

Conversation

@TimofeyIvanenko

@TimofeyIvanenko TimofeyIvanenko commented Oct 26, 2026 •

Copy link
Copy Markdown

Problem

Every signalling connection leaves its URLSession and URLSessionWebSocketTask alive after WebSocket.close()
(task.cancel + finishTasksAndInvalidate): about 25 KB per connection, kept for the life of the process, so every
full or quick reconnect adds to the footprint for good. We publish from a ReplayKit broadcast extension (50 MB limit,
streams lasting days), where that is roughly 700 reconnects to a jetsam kill.

Measured on 2.18.0 (2.17.0 the same):

  • heap of a macOS publisher kicked from its room over and over: one __NSURLSessionLocal,
    __NSURLSessionWebSocketTask, NSURLSessionConfiguration and URLProtocol stay per connection (7 / 23 / 42 of
    each at 7 / 23 / 42 sessions); no LiveKit object grows.
  • iOS 26.5 simulator, one process, a new session every ~30 s for 120 min (244 sessions): footprint between sessions
    21.8 → 31.8 MB.

It is Foundation's doing, not the SDK's. A small repro without LiveKit (open a WebSocket task on its own URLSession,
then close it) keeps 20 of 20 sessions and tasks alive (weak references) whichever way it closes:
cancel(with:) + finishTasksAndInvalidate, invalidateAndCancel, waiting for didClose, a server-initiated close,
.ephemeral, no delegate, or tasks on one shared session. Empty and data-task sessions are freed. The same repro
over NWConnection + NWProtocolWebSocket frees 20 of 20. Apple forum reports of the same:
https://developer.apple.com/forums/thread/650733, https://developer.apple.com/forums/thread/725039.

Change

Support/Network/WebSocket.swift on NWConnection with NWProtocolWebSocket, same interface for SignalClient and
RustTransport (init(url:headers:timeoutInterval:), AsyncSequence of URLSessionWebSocketTask.Message,
send(data:), close(), makeRequest, WebSocketUpgradeFailure):

  • Headers through setAdditionalHeaders, every entry added as before; timeoutInterval as the TCP connect timeout
    and an overall deadline (URLError(.timedOut)); .responsiveData for .callSignaling; multipath .handover on
    iOS; the iOS 26 simulator TLS workaround kept; autoReplyPing; ping/pong frames are not handed out as messages.
  • A rejected upgrade keeps its HTTP status: Network.framework aborts the connection (ECONNABORTED) without one, so
    on that error the same upgrade request is sent once more over a plain TCP/TLS NWConnection and the status line
    read (WebSocketUpgradeFailure(statusCode:), so .Http(status:) for livekit-net as now). A plain GET would not
    do: livekit-server answers it 404 where it answers the upgrade 401.
  • Network.framework's errors are given the shapes isNetworkError and asTransportError read: POSIX codes as
    NSPOSIXErrorDomain errors (ECONNRESET stays .Closed), a timeout as URLError(.timedOut). Our own close()
    ends the sequence, as a cancelled task did.
  • One serial queue for the connections (Network.framework needs one); the continuations sit in StateSync.

Results

main this branch
macOS publisher, per connection one URLSession + task kept, ~25 KB URLSessions flat, no WebSocket tasks
iOS 26.5 sim, 120 min, ~250 sessions (same change on 2.18.0) +8.3 MB, 0.032 MB/session +0.6 MB, 0.001 MB/session
LiveKitCoreTests against livekit-server --dev 1.13.9 with CI's room config, macOS — 594 tests (c449a5b + 3e9b6cb): 1 issue, reliableDelivery simultaneous reconnect, which passes 3 of 3 reruns here and on main

The earlier runs (588 tests, 2–3 issues on both) used a dev server without CI's room.departure_timeout: 1 and
enable_participant_data_blob: true. With them, defineAndGetSchema passes, and the remaining failure is the
intermittent one above; the audio suite fails on this Mac (no microphone). WebSocketTests, SignalClientTests, RustTransportTests (incl. "A rejected upgrade keeps its
HTTP status"), TransportCloseTests, ReconnectTests and DataTrackLifecycleTests pass. On 2.18.0 the same change
passes our own publisher end-to-end suite (25 cases: reconnects, server restarts, a node that swallows packets,
kicks, storms of sessions) and its 40- and 120-minute session soaks.

Not covered

  • HTTP and PAC proxies: URLSession honoured the system proxy settings, an NWConnection does not by default.
  • Older iOS than 26.5, and visionOS/tvOS, were not run.
  • An upgrade the server rejects on the first try and accepts on the second reports no status (the probe switches
    protocols): it stays a network error and is retried.

🤖 Generated with Claude Code

@CLAassistant

CLAassistant commented Oct 26, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

@TimofeyIvanenko
TimofeyIvanenko force-pushed the websocket-network-framework branch from c9e8989 to 1f1bc06 Compare October 26, 2026 06:26
pblazej and others added 7 commits October 26, 2026 12:04
…uation

The opening wait and the upgrade-status probe each kept a continuation in a
private `Once` class, with a separate `asyncAfter` block as the deadline. The
SDK's `AsyncCompleter` already does both: it resumes once, times out on its own,
and throws `LiveKitError(.cancelled)` when the waiting task is cancelled, so the
`catch` below it closes the connection in every failure case. `resume(with:)`
rather than `resume(throwing:)`, as the latter logs a late `.cancelled` at error.

Co-authored-by: omnigent <noreply@omnigent.ai>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Network.framework reports a bare TCP FIN as a completed receive with no
metadata. Treating that as the end of the stream, like a close frame, made the
signal client wait out its ping timeout before reconnecting. A FIN now fails
the receive with a network error so the reconnect starts at once; only a close
frame (or our own cancel) ends the sequence cleanly.

Co-authored-by: omnigent <noreply@omnigent.ai>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Network.framework transport sends its upgrade headers through
`NWProtocolWebSocket.Options.setAdditionalHeaders`; the `URLRequest` builder
was left over from the URLSession transport and had no caller apart from the
test that asserted its header merging.

Co-authored-by: omnigent <noreply@omnigent.ai>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`NWProtocolWebSocket.Options.maximumMessageSize` already defaults to 0, no
limit; the assignment restated the default.

Co-authored-by: omnigent <noreply@omnigent.ai>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SwiftLint's optional_data_string_conversion flags the lossy
`String(decoding:as:)` on an optional payload. The failable form keeps every
valid frame as it was, an absent payload included, and fails the receive on a
frame that is not UTF-8 instead of handing the signal client replacement
characters.

Co-authored-by: omnigent <noreply@omnigent.ai>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A loopback server that answers the upgrade by hand and then misbehaves on
purpose: a close frame, a bare FIN, a reset, an HTTP status, or no answer. The
suite covers how the socket opens and fails to open, how its message sequence
ends for each way a peer can go away, and the retention the transport change
exists for: twenty sockets opened and closed leave no socket and no
NWConnection behind, where the URLSession transport kept its session and task
for the life of the process.

The socket's connection is internal rather than private so the release test
can hold a weak reference to it.

Co-authored-by: omnigent <noreply@omnigent.ai>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…queue, deterministic peer closure

Review findings on the lifecycle suite. Every test now cancels its fake
server in a `defer`, so a throw no longer leaves the listener, and the server
its handler captures, alive; the awaited `stop()` stays where the port has to
refuse connections. The server delivers on the shared global queue, as the
transport under test does, instead of a serial queue of its own. The
cancellation test waits for the listener to accept the TCP connection before
cancelling, so the cancel lands during the connect rather than before it.

The peer-closure cases were flaky: the server went away straight after the
101, racing the client's handshake, and a close frame that arrives while no
receive is pending is consumed by Network.framework's close handshake, after
which the next receive fails with ENOTCONN. The server now goes away only
after a frame from the client, and the close-frame case accepts either a clean
end or that ENOTCONN. Twenty consecutive runs pass.

Co-authored-by: omnigent <noreply@omnigent.ai>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@pblazej pblazej left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The leak is real and NWConnection is the correct fix. One regression must be fixed before the merge: a bare TCP FIN now ends the message sequence without an error, so the client reconnects only after the ping timeout. The fix, a smaller opening path, four cleanups, and a test suite are on the branch blaze/pr-1173-asynccompleter in this repository, on top of 1f1bc06.

What we measured

A standalone program without LiveKit opened and closed 20 WebSocket tasks on their own URLSession. Every close path kept all 20 sessions and tasks alive after 90 seconds, invalidateAndCancel included. The same program on NWConnection with cancel() freed all 20. In the SDK, 30 Room connect and disconnect cycles leave 30 __NSURLSessionWebSocketTask and 31 __NSURLSessionLocal objects on main, about 29 to 39 KB per connection. This branch leaves none.

The regression

A FIN is the TCP signal that the peer stopped sending. A proxy that drops an idle connection sends a FIN without a close frame. WebSocket.swift:218 maps a frame without metadata to nil, so SignalClient stays .connected and reconnects only after the ping timeout, up to 20 seconds with the server defaults. The URLSession transport threw on a FIN and reconnected at once. The fix is one line: case .close?, nil: becomes case .close?:. The next receive then throws ENODATA after a peer FIN and still ends cleanly after our own close(). Commit c484db1 has the fix, and a new test fails without it.

The other commits

AsyncCompleter, the SDK's existing primitive, replaces the private Once class for the opening wait and the upgrade probe. It resumes once, times out on its own, and throws .cancelled when the task is cancelled. Both asyncAfter blocks go away and the file is 27 lines smaller. A stalled upgrade now fails with LiveKitError(.timedOut), which RustTransport and region failover treat as before.

The cleanups delete the unused makeRequest and its test, drop the default maximumMessageSize = 0, and decode text frames with the failable String(bytes:encoding:) to satisfy SwiftLint. The new WebSocketLifecycleTests suite runs the socket against a loopback server that sends a close frame, a FIN, a reset, an HTTP status, or no answer. A release test opens and closes 20 sockets and expects every NWConnection to be freed. SwiftLint reports 0 violations and the strict macOS build adds no warning. 26 tests in 7 suites pass, and the lifecycle suite passes 20 times plain and 5 times under --sanitize=thread.

Three questions

  1. The raw TCP probe in upgradeStatus sends the upgrade twice, and SignalClient.connect already gets the status and the reason from HTTP.requestValidation. Only RustTransport.swift:144 reads WebSocketUpgradeFailure. Does livekit-net still need it, and if so, is the adapter a better home?
  2. A server close frame ends the sequence with nil, so the reconnect waits for the ping timeout. The JS, Android, Flutter, and Rust SDKs reconnect at once on every server close. Do you want the same here?
  3. NWConnection ignores the system HTTP proxy that URLSession honored. Is that acceptable, and does the changeset need to say so?

How to take the commits

Run these commands in your checkout of websocket-network-framework:

git fetch git@github.com:livekit/client-sdk-swift.git blaze/pr-1173-asynccompleter
git merge --ff-only FETCH_HEAD

pblazej and others added 2 commits October 26, 2026 12:45
A raw TCP receive can return part of a frame. The server's first receive
after the upgrade could then leave the rest of the client's trigger frame in
the socket, and the receive meant for the close reply would take those bytes
for it and cancel too early. The server now parses the client frame header,
the masked 7-bit or 16-bit length and the mask key, and keeps receiving until
exactly that many payload bytes are in, both for the trigger frame and for the
close reply.

Co-authored-by: omnigent <noreply@omnigent.ai>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@TimofeyIvanenko

Copy link
Copy Markdown
Author

Thank you for the thorough review, the repro and the tests. I fast-forwarded the branch to blaze/pr-1173-asynccompleter (c449a5b) and added one commit on top, 3e9b6cb, which notes the proxy change in the changeset. The FIN fix is the one we need most: we ship this socket in a long-lived ReplayKit publisher, so we are taking c484db1 into our own build as well.

On the three questions:

  1. The upgrade probe. I added it only to keep .Http(status:) for livekit-net. We do not use RustTransport ourselves. I agree that sending the rejected upgrade twice is worse than what HTTP.requestValidation already gives. I would remove the probe and WebSocketUpgradeFailure from WebSocket, and let the adapter run HTTP.requestValidation when the connect fails, so it still reports .Http(status:). Should I do that in this PR, or in a follow-up so this one stays a leak fix?
  2. A server close frame as a reconnect trigger. Yes, please. Today the result depends on whether a receive is pending, and a client that waits out the ping timeout loses up to 20 s of signalling. I would throw a network error on a close frame from the peer and keep our own close() ending the sequence, as now. That matches the JS, Android, Flutter and Rust SDKs. I can do it as a follow-up with a case in WebSocketLifecycleTests, unless you want it here.
  3. System proxies. That is acceptable for us. The changeset now says so (3e9b6cb). If you want parity, a follow-up could read the system proxy settings into NWParameters.proxyConfigurations on iOS 17 and macOS 14 and later. I can do that one too.

The iOS 26.5 job failed in BroadcastScreenCapturerTests (activeCount → 1 == 0). The same suite fails on main today (run 40305279162, xcode-27 iOS), so I take it to be unrelated. The Lint failure was the String(decoding:) rule, which 632e35e fixes. The CLA is pending on our side.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment on lines +144 to +145
case .failed, .waiting, .cancelled:
answer.resume(returning: nil)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Rejected upgrade status lost on close

If a server closes after rejecting an upgrade, answer can replace the received HTTP status with nil. AsyncCompleter overwrites cached results, so a waiter starting after both callbacks misclassifies the rejection as a connection error.

Learn more

The status probe stores its answer in an AsyncCompleter, whose resume(with:) replaces _result even when an earlier callback resolved it. The receive callback can store an HTTP rejection, followed by a connection state callback after the peer closes. If the task has not entered answer.wait() yet, the second callback replaces the status with nil. The previous one-shot continuation kept the first answer; the new opening waiter has the same first-result concern for rapid state transitions.

Example: A server responds HTTP/1.1 401 Unauthorized and closes its socket. The receive callback records 401, then the connection fails and records nil before the waiting task runs. The caller reports a generic connection error instead of the 401.

Recommended fix: Use a first-result-wins completion primitive for answer and opening, or guard both sets of callbacks so only their first resolution can update the cached result. Preserve cancellation and timeout behavior when introducing that guard.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: AsyncCompleter.resume(with:) keeps the last result for a waiter that comes late. As agreed with the maintainer, #1178 removes the probe, and this finding goes with it. For the opening, a .ready followed by .failed before the wait is a connection that did fail, so the error is the right result there.

@pblazej

pblazej commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Thank you for the fast-forward and the changeset note. Both follow-ups are fine as separate PRs, so this one stays a leak fix with the FIN change.

Please do the probe removal first. When WebSocket no longer has upgradeStatus and WebSocketUpgradeFailure, and the Rust adapter calls HTTP.requestValidation when the connect fails, the new inline finding about a lost 401 goes away with it. For the opening wait the same timing is not a problem: a .ready followed by .failed before the task waits is a connection that failed, and the error is the correct result.

For the close frame, throw a network error that carries the close code, keep our own close() ending the sequence, and add the case to WebSocketLifecycleTests. The close-frame case can then require the error instead of accepting two outcomes.

The proxy parity change can wait until a user asks for it. The iOS 26.5 failure in BroadcastScreenCapturerTests also fails on main, so it does not block this PR.

@TimofeyIvanenko

Copy link
Copy Markdown
Author

The probe removal is up as #1178, a draft stacked on this PR. It also closes the inline finding about a lost 401. The close-frame change comes next as its own PR.

@TimofeyIvanenko

Copy link
Copy Markdown
Author

One more FIN case, found in our server-restart lab: a peer that closes the connection instead of answering the upgrade. A proxy whose upstream is down does that. Network.framework then keeps the NWConnection in .preparing for 60 s without a callback, and only then reports .waiting(ECONNRESET). A bare NWConnection with NWProtocolWebSocket against a server that reads the request and sends a FIN shows it:

0.009 state preparing
60.085 state waiting(POSIXErrorCode(rawValue: 54): Connection reset by peer)

URLSession failed the same connect at once (-1005). On this branch the opening's AsyncCompleter deadline bounds it at timeoutInterval, 10 s for the signal client by default, so it is not a hang. It is still 10 s where it used to be immediate. Our 2.18.0-based build had no overall deadline, and after a 30 s server restart its next session waited a minute behind it. I found no Network.framework hook that reports the FIN earlier.

If you want it covered, I can add a .finOnUpgrade case to WebSocketLifecycleTests. It would expect .timedOut at the deadline, the way deadlineFailsAnUnansweredUpgrade does for silence. I would do that in #1178 rather than here.

@TimofeyIvanenko

Copy link
Copy Markdown
Author

The close-frame change is up as #1179, a draft stacked on this PR and independent of #1178. A close frame now fails the stream with WebSocketClosedByPeer(code), and livekit-net still reads it as .Closed. A new e2e case, ServerRemovalTests, checks that a RemoveParticipant (a leave, then the close) still disconnects without a reconnect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants