Skip to content

VAPI-3929: republish retained streams after a websocket reconnect - #20

Merged
stampercasey merged 5 commits into
mainfrom
VAPI-3929-republish-on-reconnect
Sep 11, 2026
Merged

VAPI-3929: republish retained streams after a websocket reconnect#20
stampercasey merged 5 commits into
mainfrom
VAPI-3929-republish-on-reconnect

Conversation

@stampercasey

@stampercasey stampercasey commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

When the gateway closes an endpoint's websocket (deploy-drain eviction, instance shutdown, media-server loss, heartbeat death), the SDK auto-reconnects and re-emits init, which rebuilds a fresh, trackless publishing peer connection. Nothing replayed the already-published local streams onto it, so the reconnected session came up half-alive: websocket connected, both peer connections connected, SIP parking leg re-established and acked, but no published media.

The gateway's eligibility predicate requires at least one published track, populated only when it sees RTP from the client. With no republished track the endpoint never became eligible again and every outbound call request was rejected with -32098 endpoint not eligible.

Observed in production on 2026-09-04: a deploy drain evicted 32 idle endpoints and 32 of 32 never became eligible again, some dead for 18 minutes. This restores an unmerged fix (PR #15, closed 2026-08-24, absent from published 0.6.0/0.7.0/0.8.0) rebased onto current main, and closes four gaps in it - three found in review, one found running it against a real drain (see Verification below).

Changes

  • Replay on reconnect. republishStreams() re-attaches every retained stream to the new publishing peer connection and renegotiates once for all of them. Strict no-op on a first connect.
  • Waits for the publish peer connection before offering. The gateway requires the publish PeerConnectionState to be Connected before it accepts an offer. republishStreams() runs immediately inside init(), with no natural delay, and was firing before the peer connection's own ICE handshake finished - the gateway rejected every attempt with peer not ready for sdp offers and the endpoint never became eligible. publish() never needed this wait because an application always calls it well after connect() resolves; Swift and Kotlin already wait for ICE for the same reason. waitForPublishConnected() polls for up to 10s (matching the other two SDKs' timeout) before touching any stream.
  • Codec preferences survive the replay. PublishedStream now retains codecPreferences; previously the replay could only re-attach without them, silently changing negotiated codecs after a reconnect.
  • Ended tracks are re-acquired. A track that ended while the socket was down (device unplugged, OS revoked the mic) can still be attached and produces a valid-looking SDP offer, but its sender never emits RTP - indistinguishable from never republishing. reacquireEndedTracks() re-acquires via getUserMedia using the retained constraints and swaps the fresh tracks into the same MediaStream, so the stream id and the object the application holds stay valid.
  • DTMF senders are refreshed. localDtmfSenders is keyed by stream id and guarded with !has(id). Since the replay reuses the id, the guard short-circuited and the map kept the RTCDTMFSender from the closed peer connection, so sendDtmf after a reconnect went nowhere. The map is cleared before re-attaching.
  • Failures reach the application. New onError callback. The SDK had no error channel at all (there is a standing // TODO: emit this as an error from an EventEmitter), and these failures happen on a signaling event where a throw becomes an unhandled rejection. Fires on republish failure and on a gateway handshake refusal. With no handler registered it logs, so existing applications are unaffected.

Signaling now also emits fatalError on the fatal-handshake path: during a reconnect the existing reject() targets a long-resolved promise, so the application was left holding a session that would never come back.

Verification

Confirmed against three real drains in lab, not just unit tests.

First drain (pre ICE-wait fix) reproduced exactly the production failure: the reconnect completed every gateway-driven step and then hung on peer not ready for sdp offers - handleOfferSdp firing before peer connection state connected.

Second and third drains, after the fix, both completed the full sequence in under 2 seconds:

new websocket connection
peer connection state connected
Sent eager SIP INVITE to media-server → 200 OK → sipAcked=true
handleOfferSdp                          (now after connected, not before)
InboundTrack
Endpoint is now eligible for outbound connections

The third run also placed and answered five calls across both the original session and the reconnected one, all connecting and ending cleanly, with no further peer not ready occurrences and no other anomalies in gateway logs.

One unrelated finding from that same run, filed separately, not fixed here: an outbound call that takes longer than 2 minutes to be answered can have its endpoint evicted mid-ring by a drain, because the gateway's callSetupWindow busy-check is a timer, not a flag, and expires independently of whether the call is still trying to connect. That's a pv-gateway drain/call-engine coordination gap, orthogonal to this SDK's republish behavior.

Code review response

An external review surfaced 10 findings; the 8 that were real correctness issues (2 lower-priority ones - a duplicate fatalError notification on initial connect, and listener re-registration if an application calls connect() twice on the same instance - were left as-is, same call made on the Swift review) were fixed:

  • Concurrent init() could swap the peer connection out from under an in-flight republish. init() is signaling-driven and fires on every socket open, so two closely-spaced reconnects could run it concurrently - the second overwrites publishingPeerConnection while the first is still polling the old one in waitForPublishConnected. Serialized init() behind a mutex (the existing pattern already used for publish()/unpublish()), and close the outgoing peer connections instead of leaking them on every reconnect.
  • One stream's reacquire failure skipped every other stream and the renegotiation entirely. republishStreams() awaited reacquireEndedTracks() per stream inside the loop's shared try/catch, so one getUserMedia rejection (unplugged webcam, revoked permission) aborted the whole loop before offerPublishSdp() ever ran - reproducing the exact "silent endpoint" symptom this PR fixes, for an unrelated reason. Each stream's reacquire/attach is now isolated; the renegotiation still runs for whatever did attach, and is only skipped if nothing did.
  • A throwing onError handler broke the disconnect that follows it. handleError() called the application's handler unguarded. Called from the signaling "fatalError" listener, an app handler that throws propagates synchronously back through that listener and skips the _disconnect(false) call after it - so a fatal handshake failure could leave the ping interval running against a dead socket. Now caught and logged.
  • Reacquiring ended tracks discarded mute state and touched healthy tracks. A fresh getUserMedia track always starts enabled: true, silently un-muting a track the application had turned off with setMicEnabled/setCameraEnabled before it ended. Also, the ended-check was per-stream but the swap replaced every track in the stream, including a healthy one of the other kind. Now only the kinds that actually ended are re-acquired, and enabled is carried over onto the replacement.
  • The ICE wait wasted the full timeout on an unrecoverable peer connection. waitForPublishConnected() polled until "connected" or a 10s timeout, even if the peer connection had already reached "failed" or "closed" - states that will never become "connected". Now bails immediately on either.

Added tests for the mutex serialization (via call ordering, not timing), the per-stream isolation, the throwing-handler guard, and the mute-state carryover. All 188 tests pass.

Test plan

  • npm test - 92 passed, 92 total; prettier --check . clean; tsc --noEmit clean
  • New coverage: no replay on first connect, replay on reconnect, exactly one renegotiation for three streams, waits for publish peer connection before offering, reports an error if it never connects, codec preferences preserved, ended-track re-acquisition, no re-acquisition when live, derived constraints for an application-supplied stream, DTMF sender clearing, republish failure surfaced, re-acquisition failure surfaced, signaling fatalError emit
  • Three real lab drains (see Verification)

Notes

Part of a three-SDK change implemented to a shared contract; the Swift and Kotlin SDKs get the same semantics. All three keep the application's stream handle valid across a reconnect and expose the same onError shape.

🤖 Generated with Claude Code

When the gateway closes a device's websocket (deploy drain, instance
shutdown, media-server loss, heartbeat death) the SDK auto-reconnects and
re-emits "init", which builds a fresh, trackless publishing peer
connection. Nothing replayed the already-published local streams onto it,
so the session came back fully connected but silent: the gateway never
saw RTP, never populated publishedTracks, and every subsequent
requestOutboundConnection was rejected as "endpoint not eligible".

Retain each published stream's codec preferences and acquisition
constraints, and on reconnect re-attach every retained stream to the new
peer connection followed by a single renegotiation. Tracks that ended
while the websocket was down are re-acquired first: an ended track
attaches happily and produces valid-looking SDP, but its sender never
emits RTP, which leaves the endpoint stuck in exactly the same way.

On a first connect nothing is retained and the replay is a no-op.

Failures that leave the session unable to publish are reported through a
new onError callback rather than leaving the application believing it is
healthy. A fatal handshake error on a reconnect (403/409) is surfaced the
same way, since by then the connect() promise it used to reject has long
since resolved.
@stampercasey
stampercasey requested review from a team as code owners September 9, 2026 18:59
@bwappsec

bwappsec commented Sep 9, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

stampercasey and others added 4 commits September 10, 2026 15:30
…w retry-code policy

origin/main (VAPI-3917, #19) narrowed rpc-websockets' auto-reconnect to close
code 1001 only, tearing down on 1000/4409/1011/unknown instead of retrying
them forever. That is a different layer than this branch touches - it decides
whether the underlying client reconnects at all, while republishStreams()
only runs once it has and "init" re-fires - and the two coexist without
overlap: FATAL_HANDSHAKE_ERRORS/fatalError fires on a rejected upgrade before
any socket exists, RETRY_CLOSE_CODES fires on a close after a successful one.

pv-gateway sends 1001 (Going Away) for every retryable teardown (drain
eviction, media-server loss, instance shutdown), so it stays in
RETRY_CLOSE_CODES and this branch's fix keeps firing on exactly the paths it
was built for.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…the replay

Verified against a real lab drain: the republish offer fired the instant
init() finished creating the peer connections, before the publish side's own
ICE handshake had reached connected. The gateway requires
PeerConnectionState == Connected before it will accept an offer
(pkg/device/webrtc_device_handler.go's isPeerReady), so it rejected every
attempt with "peer not ready for sdp offers" and the endpoint never became
eligible.

publish() gets away without this wait because an application always calls it
well after connect() resolves - by the time a user clicks a button, ICE has
long since finished. A reconnect's republish has no such delay; it runs
immediately inside init(). Swift and Kotlin already wait for ICE before
offering for the same reason; JS never did, on either path, because the gap
only mattered on the one path that never had a natural delay.

waitForPublishConnected() polls connectionState for up to 10s, matching the
timeout the other two SDKs already use, before touching any retained stream.

Re-verified end to end against two more lab drains after this fix: eviction
to eligible again in under 2 seconds each time, several calls placed and
answered cleanly across both reconnects, no further occurrences of "peer not
ready."

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Serialize init() behind a mutex (init() is signaling-driven and fires
  on every socket open, so two closely-spaced reconnects could otherwise
  run it concurrently: the second overwrites publishingPeerConnection out
  from under the first, which is still polling the old one in
  waitForPublishConnected). Also close the outgoing peer connections
  instead of leaking them on every reconnect.
- Isolate each stream's reacquire/attach in republishStreams so one
  stream's getUserMedia rejection no longer skips every other stream and
  the renegotiation entirely - only skip the renegotiation if nothing
  was actually attached.
- Guard the application's onError handler so a handler that throws
  can't propagate out through the signaling "fatalError" listener and
  skip the disconnect() that follows it.
- reacquireEndedTracks now replaces only the track kinds that actually
  ended (not healthy siblings in the same stream) and carries over
  enabled=false onto the replacement track instead of silently
  un-muting it.
- waitForPublishConnected bails immediately on "failed"/"closed"
  instead of waiting out the full timeout on a peer connection that
  will never reach "connected".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@stampercasey
stampercasey merged commit 3bf821b into main Sep 11, 2026
5 checks passed
@stampercasey
stampercasey deleted the VAPI-3929-republish-on-reconnect branch September 11, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants