Repository navigation
Signalling WebSocket over Network.framework: URLSession keeps every WebSocket session after close - #1173
Signalling WebSocket over Network.framework: URLSession keeps every WebSocket session after close#1173TimofeyIvanenko wants to merge 11 commits into
Conversation
…ebSocket session after it is closed
…mework callbacks on a global queue instead of a new one
c9e8989 to
1f1bc06
Compare
…uation The opening wait and the upgrade-status probe each kept a continuation in a private `Once` class, with a separate `asyncAfter` block as the deadline. The SDK's `AsyncCompleter` already does both: it resumes once, times out on its own, and throws `LiveKitError(.cancelled)` when the waiting task is cancelled, so the `catch` below it closes the connection in every failure case. `resume(with:)` rather than `resume(throwing:)`, as the latter logs a late `.cancelled` at error. Co-authored-by: omnigent <noreply@omnigent.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Network.framework reports a bare TCP FIN as a completed receive with no metadata. Treating that as the end of the stream, like a close frame, made the signal client wait out its ping timeout before reconnecting. A FIN now fails the receive with a network error so the reconnect starts at once; only a close frame (or our own cancel) ends the sequence cleanly. Co-authored-by: omnigent <noreply@omnigent.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Network.framework transport sends its upgrade headers through `NWProtocolWebSocket.Options.setAdditionalHeaders`; the `URLRequest` builder was left over from the URLSession transport and had no caller apart from the test that asserted its header merging. Co-authored-by: omnigent <noreply@omnigent.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`NWProtocolWebSocket.Options.maximumMessageSize` already defaults to 0, no limit; the assignment restated the default. Co-authored-by: omnigent <noreply@omnigent.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SwiftLint's optional_data_string_conversion flags the lossy `String(decoding:as:)` on an optional payload. The failable form keeps every valid frame as it was, an absent payload included, and fails the receive on a frame that is not UTF-8 instead of handing the signal client replacement characters. Co-authored-by: omnigent <noreply@omnigent.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A loopback server that answers the upgrade by hand and then misbehaves on purpose: a close frame, a bare FIN, a reset, an HTTP status, or no answer. The suite covers how the socket opens and fails to open, how its message sequence ends for each way a peer can go away, and the retention the transport change exists for: twenty sockets opened and closed leave no socket and no NWConnection behind, where the URLSession transport kept its session and task for the life of the process. The socket's connection is internal rather than private so the release test can hold a weak reference to it. Co-authored-by: omnigent <noreply@omnigent.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…queue, deterministic peer closure Review findings on the lifecycle suite. Every test now cancels its fake server in a `defer`, so a throw no longer leaves the listener, and the server its handler captures, alive; the awaited `stop()` stays where the port has to refuse connections. The server delivers on the shared global queue, as the transport under test does, instead of a serial queue of its own. The cancellation test waits for the listener to accept the TCP connection before cancelling, so the cancel lands during the connect rather than before it. The peer-closure cases were flaky: the server went away straight after the 101, racing the client's handshake, and a close frame that arrives while no receive is pending is consumed by Network.framework's close handshake, after which the next receive fails with ENOTCONN. The server now goes away only after a frame from the client, and the close-frame case accepts either a clean end or that ENOTCONN. Twenty consecutive runs pass. Co-authored-by: omnigent <noreply@omnigent.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
Summary
The leak is real and NWConnection is the correct fix. One regression must be fixed before the merge: a bare TCP FIN now ends the message sequence without an error, so the client reconnects only after the ping timeout. The fix, a smaller opening path, four cleanups, and a test suite are on the branch blaze/pr-1173-asynccompleter in this repository, on top of 1f1bc06.
What we measured
A standalone program without LiveKit opened and closed 20 WebSocket tasks on their own URLSession. Every close path kept all 20 sessions and tasks alive after 90 seconds, invalidateAndCancel included. The same program on NWConnection with cancel() freed all 20. In the SDK, 30 Room connect and disconnect cycles leave 30 __NSURLSessionWebSocketTask and 31 __NSURLSessionLocal objects on main, about 29 to 39 KB per connection. This branch leaves none.
The regression
A FIN is the TCP signal that the peer stopped sending. A proxy that drops an idle connection sends a FIN without a close frame. WebSocket.swift:218 maps a frame without metadata to nil, so SignalClient stays .connected and reconnects only after the ping timeout, up to 20 seconds with the server defaults. The URLSession transport threw on a FIN and reconnected at once. The fix is one line: case .close?, nil: becomes case .close?:. The next receive then throws ENODATA after a peer FIN and still ends cleanly after our own close(). Commit c484db1 has the fix, and a new test fails without it.
The other commits
AsyncCompleter, the SDK's existing primitive, replaces the private Once class for the opening wait and the upgrade probe. It resumes once, times out on its own, and throws .cancelled when the task is cancelled. Both asyncAfter blocks go away and the file is 27 lines smaller. A stalled upgrade now fails with LiveKitError(.timedOut), which RustTransport and region failover treat as before.
The cleanups delete the unused makeRequest and its test, drop the default maximumMessageSize = 0, and decode text frames with the failable String(bytes:encoding:) to satisfy SwiftLint. The new WebSocketLifecycleTests suite runs the socket against a loopback server that sends a close frame, a FIN, a reset, an HTTP status, or no answer. A release test opens and closes 20 sockets and expects every NWConnection to be freed. SwiftLint reports 0 violations and the strict macOS build adds no warning. 26 tests in 7 suites pass, and the lifecycle suite passes 20 times plain and 5 times under --sanitize=thread.
Three questions
- The raw TCP probe in
upgradeStatussends the upgrade twice, andSignalClient.connectalready gets the status and the reason fromHTTP.requestValidation. OnlyRustTransport.swift:144readsWebSocketUpgradeFailure. Does livekit-net still need it, and if so, is the adapter a better home? - A server close frame ends the sequence with
nil, so the reconnect waits for the ping timeout. The JS, Android, Flutter, and Rust SDKs reconnect at once on every server close. Do you want the same here? NWConnectionignores the system HTTP proxy thatURLSessionhonored. Is that acceptable, and does the changeset need to say so?
How to take the commits
Run these commands in your checkout of websocket-network-framework:
git fetch git@github.com:livekit/client-sdk-swift.git blaze/pr-1173-asynccompleter
git merge --ff-only FETCH_HEAD
A raw TCP receive can return part of a frame. The server's first receive after the upgrade could then leave the rest of the client's trigger frame in the socket, and the receive meant for the close reply would take those bytes for it and cancel too early. The server now parses the client frame header, the masked 7-bit or 16-bit length and the mask key, and keeps receiving until exactly that many payload bytes are in, both for the trigger frame and for the close reply. Co-authored-by: omnigent <noreply@omnigent.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…HTTP or PAC proxy
|
Thank you for the thorough review, the repro and the tests. I fast-forwarded the branch to On the three questions:
The iOS 26.5 job failed in |
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| case .failed, .waiting, .cancelled: | ||
| answer.resume(returning: nil) |
There was a problem hiding this comment.
🟡 Rejected upgrade status lost on close
If a server closes after rejecting an upgrade, answer can replace the received HTTP status with nil. AsyncCompleter overwrites cached results, so a waiter starting after both callbacks misclassifies the rejection as a connection error.
Learn more
The status probe stores its answer in an AsyncCompleter, whose resume(with:) replaces _result even when an earlier callback resolved it. The receive callback can store an HTTP rejection, followed by a connection state callback after the peer closes. If the task has not entered answer.wait() yet, the second callback replaces the status with nil. The previous one-shot continuation kept the first answer; the new opening waiter has the same first-result concern for rapid state transitions.
Example: A server responds HTTP/1.1 401 Unauthorized and closes its socket. The receive callback records 401, then the connection fails and records nil before the waiting task runs. The caller reports a generic connection error instead of the 401.
Recommended fix: Use a first-result-wins completion primitive for answer and opening, or guard both sets of callbacks so only their first resolution can update the cached result. Preserve cancellation and timeout behavior when introducing that guard.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Confirmed: AsyncCompleter.resume(with:) keeps the last result for a waiter that comes late. As agreed with the maintainer, #1178 removes the probe, and this finding goes with it. For the opening, a .ready followed by .failed before the wait is a connection that did fail, so the error is the right result there.
|
Thank you for the fast-forward and the changeset note. Both follow-ups are fine as separate PRs, so this one stays a leak fix with the FIN change. Please do the probe removal first. When For the close frame, throw a network error that carries the close code, keep our own The proxy parity change can wait until a user asks for it. The iOS 26.5 failure in |
|
The probe removal is up as #1178, a draft stacked on this PR. It also closes the inline finding about a lost 401. The close-frame change comes next as its own PR. |
|
One more FIN case, found in our server-restart lab: a peer that closes the connection instead of answering the upgrade. A proxy whose upstream is down does that. Network.framework then keeps the
If you want it covered, I can add a |
|
The close-frame change is up as #1179, a draft stacked on this PR and independent of #1178. A close frame now fails the stream with |
Problem
Every signalling connection leaves its
URLSessionandURLSessionWebSocketTaskalive afterWebSocket.close()(
task.cancel+finishTasksAndInvalidate): about 25 KB per connection, kept for the life of the process, so everyfull or quick reconnect adds to the footprint for good. We publish from a ReplayKit broadcast extension (50 MB limit,
streams lasting days), where that is roughly 700 reconnects to a jetsam kill.
Measured on 2.18.0 (2.17.0 the same):
heapof a macOS publisher kicked from its room over and over: one__NSURLSessionLocal,__NSURLSessionWebSocketTask,NSURLSessionConfigurationandURLProtocolstay per connection (7 / 23 / 42 ofeach at 7 / 23 / 42 sessions); no LiveKit object grows.
21.8 → 31.8 MB.
It is Foundation's doing, not the SDK's. A small repro without LiveKit (open a WebSocket task on its own URLSession,
then close it) keeps 20 of 20 sessions and tasks alive (weak references) whichever way it closes:
cancel(with:)+finishTasksAndInvalidate,invalidateAndCancel, waiting fordidClose, a server-initiated close,.ephemeral, no delegate, or tasks on one shared session. Empty and data-task sessions are freed. The same reproover
NWConnection+NWProtocolWebSocketfrees 20 of 20. Apple forum reports of the same:https://developer.apple.com/forums/thread/650733, https://developer.apple.com/forums/thread/725039.
Change
Support/Network/WebSocket.swiftonNWConnectionwithNWProtocolWebSocket, same interface forSignalClientandRustTransport(init(url:headers:timeoutInterval:),AsyncSequenceofURLSessionWebSocketTask.Message,send(data:),close(),makeRequest,WebSocketUpgradeFailure):setAdditionalHeaders, every entry added as before;timeoutIntervalas the TCP connect timeoutand an overall deadline (
URLError(.timedOut));.responsiveDatafor.callSignaling; multipath.handoveroniOS; the iOS 26 simulator TLS workaround kept;
autoReplyPing; ping/pong frames are not handed out as messages.ECONNABORTED) without one, soon that error the same upgrade request is sent once more over a plain TCP/TLS
NWConnectionand the status lineread (
WebSocketUpgradeFailure(statusCode:), so.Http(status:)for livekit-net as now). A plain GET would notdo: livekit-server answers it 404 where it answers the upgrade 401.
isNetworkErrorandasTransportErrorread: POSIX codes asNSPOSIXErrorDomainerrors (ECONNRESETstays.Closed), a timeout asURLError(.timedOut). Our ownclose()ends the sequence, as a cancelled task did.
StateSync.Results
LiveKitCoreTestsagainstlivekit-server --dev1.13.9 with CI's room config, macOSreliableDeliverysimultaneous reconnect, which passes 3 of 3 reruns here and on mainThe earlier runs (588 tests, 2–3 issues on both) used a dev server without CI's
room.departure_timeout: 1andenable_participant_data_blob: true. With them,defineAndGetSchemapasses, and the remaining failure is theintermittent one above; the audio suite fails on this Mac (no microphone).
WebSocketTests,SignalClientTests,RustTransportTests(incl. "A rejected upgrade keeps itsHTTP status"),
TransportCloseTests,ReconnectTestsandDataTrackLifecycleTestspass. On 2.18.0 the same changepasses our own publisher end-to-end suite (25 cases: reconnects, server restarts, a node that swallows packets,
kicks, storms of sessions) and its 40- and 120-minute session soaks.
Not covered
NWConnectiondoes not by default.protocols): it stays a network error and is retried.
🤖 Generated with Claude Code