feat(sarvam): add STTRealtime for realtime streaming STT - #2414
Conversation
Ports Sarvam's realtime speech-to-text API (saaras:v3-realtime) to Node.js, following the existing legacy STT plugin. Adds STTRealtime and RealtimeSpeechStream alongside the existing STT/SpeechStream classes, exported separately. Key behavior: this endpoint bills per connection, so stream() forces connOptions.maxRetry to 0 -- the stream never silently reconnects on a socket failure.
🦋 Changeset detectedLatest commit: 49cd76b The changes in this PR will be included in the next version bump. This PR includes changesets to release 38 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 4 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| #handleErrorEvent(data: RealtimeServerEvent): void { | ||
| if (!data.is_fatal) { | ||
| this.#logger.warn( | ||
| { code: data.code, message: data.message }, | ||
| 'non-fatal Sarvam realtime STT error', | ||
| ); | ||
| return; | ||
| } | ||
| const statusCode = typeof data.status_code === 'number' ? data.status_code : -1; | ||
| throw new APIStatusError({ | ||
| message: `Sarvam realtime STT error: ${data.message ?? data.code ?? 'unknown'}`, | ||
| options: { | ||
| statusCode, | ||
| requestId: this.#requestId || null, | ||
| body: data as unknown as object, | ||
| retryable: data.code === 'model_unavailable', | ||
| }, |
There was a problem hiding this comment.
🟨 Provider errors bypass PII redaction
Malformed provider text enters an APIStatusError message unchanged. Error listeners and automatic logs can expose customer or transcript content.
Was this helpful? React with 👍 or 👎 to provide feedback.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 221b9636eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if ((code === 1000 || code === 1001) && !looksLikeErrorText(reason)) { | ||
| resolve(); | ||
| return; |
There was a problem hiding this comment.
If the peer sends a clean 1000/1001 close before the caller has ended input, this resolves #processMessages, but run() is still awaiting Promise.all with #processAudio, which remains blocked in its for await (this.input) loop. The output queue therefore never closes and no connection error is surfaced until another frame is pushed or endInput() is called; an idle server-side close leaves a realtime stream hanging indefinitely.
Useful? React with 👍 / 👎.
- Replace the hand-rolled G.711 mu-law/A-law encoders with a faithful port of the standard ITU-T reference algorithm. The previous A-law encoder had an inverted sign check that corrupted positive-amplitude samples; while fixing it, the rest of the derived implementation turned out to diverge from the reference too, so both encoders are now ported directly from the canonical algorithm instead. - Fix a hang: if the server ends the session or closes the socket before the caller calls endInput()/flush(), the audio pump no longer blocks forever waiting for a frame that may never come -- the two connection tasks now share an abort controller so either side finishing unblocks the other. - Streams that complete naturally (not via close()) now unregister from STTRealtime's tracking set, so updateOptions() stops being forwarded to dead connections. - Tag provider-supplied error text as lk.pii.* in logs, matching this repo's PII-redaction convention, instead of embedding it in the thrown error's own message/body. - Reject non-mono audio frames instead of silently corrupting them (the wire framer is hardcoded to mono). Verified against the live API: all four wire encodings now produce correct transcripts (previously alaw silently distorted audio).
|
Addressed all 5 findings from the automated reviews in c065849:
Added regression tests for all 5 ( |
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 3 new potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| } else if (!this.#eosEmittedForUtterance) { | ||
| this.#emitEndOfSpeech(); | ||
| if (this.#finalReceivedForUtterance) { | ||
| this.#tryCommitUtterance(); |
There was a problem hiding this comment.
🔴 Turn completion precedes final transcript
When #handleSpeechEnd receives a boundary, it emits completion before the buffered final transcript. The voice pipeline can commit stale words before the final text arrives.
Was this helpful? React with 👍 or 👎 to provide feedback.
- Emit a buffered final transcript before END_OF_SPEECH in vad endpointing mode, not after. AudioRecognition's runEOUDetection fires on END_OF_SPEECH, so committing it first left the turn missing its own final text. - Treat encoding as connection-only in updateOptions -- the server's decoder is fixed by the connect-time query param, so switching it mid-stream desynced client/server framing. - Flush any audio still buffered in the framer when input ends without a trailing flush(), instead of dropping the tail.
|
Addressed the second round of Devin findings in 09b7ea7:
Added 4 regression tests total for these two rounds' findings (13 passing now), and reverified live against the real API that audio/transcription still works end-to-end after the encoding-guard change. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The endInput()-without-flush() tail drain added in the previous commit called #sendAudioFrames directly, which starts a manual-mode turn (speech_start) but never ends it -- that only happened inside the main loop's isFlush branch. Send speech_end / #endManualUtterance there too.
1 participant
Motivation
Sarvam's realtime speech-to-text API (
saaras:v3-realtime) has been available in the Python SDK since livekit/agents#6562 (STTRealtime), but the Node.js plugin only exposes the legacySTTclass (REST + a non-realtime WebSocket streaming mode). This PR closes that gap.Change
Adds
STTRealtimeandRealtimeSpeechStreamtoplugins/sarvam/src/stt_realtime.ts, exported alongside the existingSTT/SpeechStream— the legacy class is untouched.stream()forcesconnOptions.maxRetryto0, since this endpoint bills per connection and must never silently reconnect on a socket failure.SpeechEventTypes:session.begin,vad.speech_start/vad.speech_end,transcript.partial/transcript.final,session.end,config.updated,error,pong.endpointing: 'vad'(default) bufferstranscript.finaluntil the matchingvad.speech_endsupplies the turn boundary, emittingEND_OF_SPEECH+FINAL_TRANSCRIPTtogether;endpointing: 'manual'emitsFINAL_TRANSCRIPTimmediately and derives turn boundaries fromflush()/pushFrame()instead of server VAD.RECOGNITION_USAGEreporting (5s), reconciled against the server's authoritativeaudio_duration_sonsession.endto avoid double-billing.config.updatesupport for mid-session option changes (endpointing switches apply only at the next utterance boundary, gated by the server's ack).linear16,linear32,mulaw,alaw.Also adds
plugins/sarvam/src/_utils.ts(PeriodicCollector, matching the existing pattern already duplicated indeepgram/xai/elevenlabs), new types inmodels.ts(STTRealtimeLanguagescorrectly usesor-INfor Odia, unlike the legacyod-IN), and a README section documenting the no-reconnect/per-connection-billing behavior.Usage
Validation
pnpm --filter @livekit/agents-plugin-sarvam build | lint | api:checkandprettier --checkall pass; API report regenerated.pnpm vitest run plugins/sarvam— newstt_realtime.test.ts(vad-mode final-transcript gating, manual-mode immediate finals, no-reconnect guarantee, streaming-only contract) plus existing suite, all green.Notes
Companion Python PR: livekit/agents#6562.