Skip to content

fix(google): restart the realtime session on abnormal WebSocket close - #2366

Open
anzemur wants to merge 1 commit into
livekit:mainfrom
anzemur:fix/google-realtime-restart-on-abnormal-close
Open

fix(google): restart the realtime session on abnormal WebSocket close#2366
anzemur wants to merge 1 commit into
livekit:mainfrom
anzemur:fix/google-realtime-restart-on-abnormal-close

Conversation

@anzemur

@anzemur anzemur commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

What

When the Gemini Live WebSocket dies abnormally, the session never recovers. In our production voice agent this branch fires for two distinct error groups: bare WebSocket closed with code 1006 (a few per day) and closes carrying Google's The operation was aborted. reason (TLS connection dropped server-side) - the latter at ~2,800 events / ~1,800 users over two months, ~30 live sessions dying per day. Every one of them passed our teardown filter, i.e. genuinely mid-conversation:

  • onerror (network-level failures) calls markRestartNeeded(), so #mainTask's reconnect loop tears down, reconnects, and re-seeds the chat context.
  • onclose with a non-normal code only logs, emits an unrecoverable error, and marks the generation done. Nothing sets sessionShouldClose, so the main task stays parked on sessionShouldClose.wait() with a dead socket — the user keeps talking into a session that will never answer. Outbound sends on the closed ws socket don't throw, so the retry path in #mainTask's catch never triggers either.

Fix

Mirror onerror in onclose: on a non-normal close while the session isn't already closing, call markRestartNeeded() so the main task reconnects and re-seeds history. The emitted error is marked recoverable: true in that case — a restart is coming, and apps that treat unrecoverable errors as fatal shouldn't tear down a session that's about to recover. When the session is already closing (sessionShouldClose set), behavior is unchanged (recoverable: false, no restart).

Same as the existing onerror path, these restarts don't consume the retry budget; a repeatedly failing endpoint behaves as it does today for network errors.

Tests

realtime_reconnect.test.ts drives the live-connect callbacks through a mocked @google/genai:

  • abnormal close (1006) → a second connect happens and the error is emitted with recoverable: true (fails without the fix: no reconnect, recoverable: false),
  • normal close (1000) → no reconnect, no error emitted.

Full @livekit/agents-plugin-google suite passes (59 tests).

Live verification

Applied this change to the installed 1.7.0 dist of a production voice agent and killed the real Gemini socket mid-session (ws.terminate() on the underlying ws@8.19.0 socket — the client observes exactly a 1006), with a probe question whose answer (47) existed only in the seeded chat history:

  • Unfixed: WebSocket closed with code 1006 logged, no reconnect ever, and one step worse than the parked loop: the recoverable: false error makes AgentSession close outright (AgentSession is closing due to an unrecoverable errorev.type is undefined so _onError's maxUnrecoverableErrors counting is skipped). The session dies mid-conversation.
  • Fixed: reconnect fires 3 ms after the close (sessionShouldClose:true → new Connecting to Gemini Realtime API...), recoverable: true keeps the AgentSession alive, and the model answers 47. — proving both the reconnect and the chat-context re-seed.
  • Full agent smoke suite passes on the fixed build (opening, barge-in, seeded history, typed text, tool injection, closing TTS).

So beyond un-parking the main task, recoverable: true also prevents the framework-level session teardown on a transient socket drop.

@changeset-bot

changeset-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: b616a87

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages
Name Type
@livekit/agents-plugin-google Patch
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-anthropic Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-azure Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-did Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-krisp Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-perplexity Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-protoface Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-soniox Patch
@livekit/agents-plugin-tavus Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch
@livekit/agents-plugins-test Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@anzemur
anzemur marked this pull request as ready for review August 28, 2026 14:06
@anzemur
anzemur requested a review from a team as a code owner August 28, 2026 14:06

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant