VAPI-3929: republish retained streams after a websocket reconnect - #20
Merged
Conversation
When the gateway closes a device's websocket (deploy drain, instance shutdown, media-server loss, heartbeat death) the SDK auto-reconnects and re-emits "init", which builds a fresh, trackless publishing peer connection. Nothing replayed the already-published local streams onto it, so the session came back fully connected but silent: the gateway never saw RTP, never populated publishedTracks, and every subsequent requestOutboundConnection was rejected as "endpoint not eligible". Retain each published stream's codec preferences and acquisition constraints, and on reconnect re-attach every retained stream to the new peer connection followed by a single renegotiation. Tracks that ended while the websocket was down are re-acquired first: an ended track attaches happily and produces valid-looking SDP, but its sender never emits RTP, which leaves the endpoint stuck in exactly the same way. On a first connect nothing is retained and the replay is a no-op. Failures that leave the session unable to publish are reported through a new onError callback rather than leaving the application believing it is healthy. A fatal handshake error on a reconnect (403/409) is surfaced the same way, since by then the connect() promise it used to reject has long since resolved.
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
…w retry-code policy origin/main (VAPI-3917, #19) narrowed rpc-websockets' auto-reconnect to close code 1001 only, tearing down on 1000/4409/1011/unknown instead of retrying them forever. That is a different layer than this branch touches - it decides whether the underlying client reconnects at all, while republishStreams() only runs once it has and "init" re-fires - and the two coexist without overlap: FATAL_HANDSHAKE_ERRORS/fatalError fires on a rejected upgrade before any socket exists, RETRY_CLOSE_CODES fires on a close after a successful one. pv-gateway sends 1001 (Going Away) for every retryable teardown (drain eviction, media-server loss, instance shutdown), so it stays in RETRY_CLOSE_CODES and this branch's fix keeps firing on exactly the paths it was built for. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…the replay Verified against a real lab drain: the republish offer fired the instant init() finished creating the peer connections, before the publish side's own ICE handshake had reached connected. The gateway requires PeerConnectionState == Connected before it will accept an offer (pkg/device/webrtc_device_handler.go's isPeerReady), so it rejected every attempt with "peer not ready for sdp offers" and the endpoint never became eligible. publish() gets away without this wait because an application always calls it well after connect() resolves - by the time a user clicks a button, ICE has long since finished. A reconnect's republish has no such delay; it runs immediately inside init(). Swift and Kotlin already wait for ICE before offering for the same reason; JS never did, on either path, because the gap only mattered on the one path that never had a natural delay. waitForPublishConnected() polls connectionState for up to 10s, matching the timeout the other two SDKs already use, before touching any retained stream. Re-verified end to end against two more lab drains after this fix: eviction to eligible again in under 2 seconds each time, several calls placed and answered cleanly across both reconnects, no further occurrences of "peer not ready." Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Serialize init() behind a mutex (init() is signaling-driven and fires on every socket open, so two closely-spaced reconnects could otherwise run it concurrently: the second overwrites publishingPeerConnection out from under the first, which is still polling the old one in waitForPublishConnected). Also close the outgoing peer connections instead of leaking them on every reconnect. - Isolate each stream's reacquire/attach in republishStreams so one stream's getUserMedia rejection no longer skips every other stream and the renegotiation entirely - only skip the renegotiation if nothing was actually attached. - Guard the application's onError handler so a handler that throws can't propagate out through the signaling "fatalError" listener and skip the disconnect() that follows it. - reacquireEndedTracks now replaces only the track kinds that actually ended (not healthy siblings in the same stream) and carries over enabled=false onto the replacement track instead of silently un-muting it. - waitForPublishConnected bails immediately on "failed"/"closed" instead of waiting out the full timeout on a peer connection that will never reach "connected".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
smoghe-bw
approved these changes
Sep 11, 2026
Open
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the gateway closes an endpoint's websocket (deploy-drain eviction, instance shutdown, media-server loss, heartbeat death), the SDK auto-reconnects and re-emits
init, which rebuilds a fresh, trackless publishing peer connection. Nothing replayed the already-published local streams onto it, so the reconnected session came up half-alive: websocket connected, both peer connectionsconnected, SIP parking leg re-established and acked, but no published media.The gateway's eligibility predicate requires at least one published track, populated only when it sees RTP from the client. With no republished track the endpoint never became eligible again and every outbound call request was rejected with
-32098 endpoint not eligible.Observed in production on 2026-09-04: a deploy drain evicted 32 idle endpoints and 32 of 32 never became eligible again, some dead for 18 minutes. This restores an unmerged fix (PR #15, closed 2026-08-24, absent from published 0.6.0/0.7.0/0.8.0) rebased onto current
main, and closes four gaps in it - three found in review, one found running it against a real drain (see Verification below).Changes
republishStreams()re-attaches every retained stream to the new publishing peer connection and renegotiates once for all of them. Strict no-op on a first connect.PeerConnectionStateto beConnectedbefore it accepts an offer.republishStreams()runs immediately insideinit(), with no natural delay, and was firing before the peer connection's own ICE handshake finished - the gateway rejected every attempt withpeer not ready for sdp offersand the endpoint never became eligible.publish()never needed this wait because an application always calls it well afterconnect()resolves; Swift and Kotlin already wait for ICE for the same reason.waitForPublishConnected()polls for up to 10s (matching the other two SDKs' timeout) before touching any stream.PublishedStreamnow retainscodecPreferences; previously the replay could only re-attach without them, silently changing negotiated codecs after a reconnect.reacquireEndedTracks()re-acquires viagetUserMediausing the retained constraints and swaps the fresh tracks into the sameMediaStream, so the stream id and the object the application holds stay valid.localDtmfSendersis keyed by stream id and guarded with!has(id). Since the replay reuses the id, the guard short-circuited and the map kept theRTCDTMFSenderfrom the closed peer connection, sosendDtmfafter a reconnect went nowhere. The map is cleared before re-attaching.onErrorcallback. The SDK had no error channel at all (there is a standing// TODO: emit this as an error from an EventEmitter), and these failures happen on a signaling event where a throw becomes an unhandled rejection. Fires on republish failure and on a gateway handshake refusal. With no handler registered it logs, so existing applications are unaffected.Signaling now also emits
fatalErroron the fatal-handshake path: during a reconnect the existingreject()targets a long-resolved promise, so the application was left holding a session that would never come back.Verification
Confirmed against three real drains in lab, not just unit tests.
First drain (pre ICE-wait fix) reproduced exactly the production failure: the reconnect completed every gateway-driven step and then hung on
peer not ready for sdp offers-handleOfferSdpfiring beforepeer connection state connected.Second and third drains, after the fix, both completed the full sequence in under 2 seconds:
The third run also placed and answered five calls across both the original session and the reconnected one, all connecting and ending cleanly, with no further
peer not readyoccurrences and no other anomalies in gateway logs.One unrelated finding from that same run, filed separately, not fixed here: an outbound call that takes longer than 2 minutes to be answered can have its endpoint evicted mid-ring by a drain, because the gateway's
callSetupWindowbusy-check is a timer, not a flag, and expires independently of whether the call is still trying to connect. That's a pv-gateway drain/call-engine coordination gap, orthogonal to this SDK's republish behavior.Code review response
An external review surfaced 10 findings; the 8 that were real correctness issues (2 lower-priority ones - a duplicate fatalError notification on initial connect, and listener re-registration if an application calls connect() twice on the same instance - were left as-is, same call made on the Swift review) were fixed:
init()is signaling-driven and fires on every socket open, so two closely-spaced reconnects could run it concurrently - the second overwritespublishingPeerConnectionwhile the first is still polling the old one inwaitForPublishConnected. Serializedinit()behind a mutex (the existing pattern already used forpublish()/unpublish()), and close the outgoing peer connections instead of leaking them on every reconnect.republishStreams()awaitedreacquireEndedTracks()per stream inside the loop's shared try/catch, so onegetUserMediarejection (unplugged webcam, revoked permission) aborted the whole loop beforeofferPublishSdp()ever ran - reproducing the exact "silent endpoint" symptom this PR fixes, for an unrelated reason. Each stream's reacquire/attach is now isolated; the renegotiation still runs for whatever did attach, and is only skipped if nothing did.onErrorhandler broke the disconnect that follows it.handleError()called the application's handler unguarded. Called from the signaling"fatalError"listener, an app handler that throws propagates synchronously back through that listener and skips the_disconnect(false)call after it - so a fatal handshake failure could leave the ping interval running against a dead socket. Now caught and logged.getUserMediatrack always startsenabled: true, silently un-muting a track the application had turned off withsetMicEnabled/setCameraEnabledbefore it ended. Also, the ended-check was per-stream but the swap replaced every track in the stream, including a healthy one of the other kind. Now only the kinds that actually ended are re-acquired, andenabledis carried over onto the replacement.waitForPublishConnected()polled until "connected" or a 10s timeout, even if the peer connection had already reached "failed" or "closed" - states that will never become "connected". Now bails immediately on either.Added tests for the mutex serialization (via call ordering, not timing), the per-stream isolation, the throwing-handler guard, and the mute-state carryover. All 188 tests pass.
Test plan
npm test- 92 passed, 92 total;prettier --check .clean;tsc --noEmitcleanfatalErroremitNotes
Part of a three-SDK change implemented to a shared contract; the Swift and Kotlin SDKs get the same semantics. All three keep the application's stream handle valid across a reconnect and expose the same
onErrorshape.🤖 Generated with Claude Code