meta: add Muse Voice Transcribe STT plugin - #7096
Open
CHVSAnirudh wants to merge 4 commits into
Open
Conversation
Adds livekit-plugins-meta, a streaming speech-to-text plugin for Meta's Muse Voice Transcribe (muse-voice-transcribe-1.0) over the realtime ASR WebSocket at wss://api.meta.ai/v1/asr/realtime. The plugin authenticates in the first frame rather than a header (as the API requires), streams 80 ms chunks of mono s16le PCM as binary frames, and rotates the connection before Muse's 60-minute session limit. It supports ENDPOINTING mode for model-detected turn boundaries, DIARIZATION for speaker labels, keyword biasing (wired to framework-managed keyterms) and language biasing. One ordering detail is worth calling out: Muse emits `speechEnd` before `speechComplete`, but AudioRecognition begins end-of-turn detection as soon as it sees END_OF_SPEECH, using the transcript accumulated so far. Forwarding the frames in wire order would judge each turn before its final text arrived, so the plugin holds the boundary and releases it after the FINAL_TRANSCRIPT. Capabilities: streaming, interim results, keyterms, and diarization in DIARIZATION mode. Muse timestamps turns rather than words, so aligned_transcript is left False.
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 11 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
Addresses review feedback on the initial plugin commit. - STT.aclose() closed nothing: the inherited implementation is a no-op, so every stream's task and WebSocket outlived the recognizer. It now closes the streams it handed out, and one failing closure no longer strands the rest. The HTTP session is left alone, since it is either the caller's or the job-scoped one from http_context. - update_options(language=...) never reached a running stream, because _apply_options preserved each stream's language unconditionally. Streams now record whether stream(language=...) gave them an explicit override; only those keep their own language, the rest follow the recognizer. - Reopening the socket mid-utterance loses the turn, since its final never arrives. Reconnects requested during a turn are now held until END_OF_SPEECH releases them. The 55-minute rotation uses the same path with a hard deadline, because Muse drops the session at 60 minutes whether or not the turn has ended. - update_options(mode=...) left capabilities.diarization at its constructor value, so adapters could trust speaker labels that were no longer coming (or reject ones that were). - The audio-duration collector was never flushed when input ended, dropping up to one collection period of audio from STT usage metrics.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Class, stream(), model and provider docstrings for pdoc3; the constructor already carried the parameter documentation.
Every plugin here names its credential <PROVIDER>_API_KEY; MODEL_API_KEY was Meta's SDK convention and reads as generic in a multi-provider agent. The missing-key error now follows the assemblyai/baseten wording.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds livekit-plugins-meta, a streaming speech-to-text plugin for Meta's Muse Voice Transcribe (muse-voice-transcribe-1.0) over the realtime ASR WebSocket at wss://api.meta.ai/v1/asr/realtime.
The plugin authenticates in the first frame rather than a header (as the API requires), streams 80 ms chunks of mono s16le PCM as binary frames, and rotates the connection before Muse's 60-minute session limit. It supports ENDPOINTING mode for model-detected turn boundaries, DIARIZATION for speaker labels, keyword biasing (wired to framework-managed keyterms) and language biasing.
One ordering detail is worth calling out: Muse emits
speechEndbeforespeechComplete, but AudioRecognition begins end-of-turn detection as soon as it sees END_OF_SPEECH, using the transcript accumulated so far. Forwarding the frames in wire order would judge each turn before its final text arrived, so the plugin holds the boundary and releases it after the FINAL_TRANSCRIPT.Capabilities: streaming, interim results, keyterms, and diarization in DIARIZATION mode. Muse timestamps turns rather than words, so aligned_transcript is left False.