feat(telnyx): add Telnyx LLM, STT and TTS - #629
Conversation
Telnyx Inference serves an OpenAI-compatible /v2/ai/chat/completions endpoint, so the LLM is a thin ChatCompletionsLLM subclass pointed at the Telnyx base URL with bearer auth. Follows the plugins/sarvam precedent of one vendor plugin covering LLM, STT, and TTS.
Streams text to speech over the Telnyx WebSocket endpoint and decodes the returned MP3 into PcmData as it arrives. Telnyx closes the socket after each stop frame, so the plugin connects per synthesis rather than holding one socket open. Follows the plugins/sarvam precedent of one vendor plugin covering LLM, STT, and TTS.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdded Telnyx LLM, streaming STT, and streaming TTS integrations with API-key configuration and lifecycle handling. Exported the integrations through the package interface. Added unit and opt-in integration tests. Added an inbound Telnyx voice-agent example with webhook verification, media streaming, call setup, cleanup, CLI options, and startup validation. Updated documentation and dependencies. Mergeability Score: ⚪ Minimal · up to This PR adds Telnyx LLM, STT, and TTS support with related tests and documentation; no actionable merge-blocking risk remains beyond normal checks and review. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
plugins/telnyx/tests/test_telnyx_stt.py (1)
120-126: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert the emitted error, not just the absence of an exception.
test_send_failure_does_not_propagatepasses even ifprocess_audiosilently drops the failure without reporting it. Collect the output and assert that the send failure surfaces as an error event, so the test covers the behavior of_emit_error_event.Source: Path instructions
plugins/telnyx/vision_agents/plugins/telnyx/tts.py (1)
143-152: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winClose the WebSocket in
close().
close()releases the aiohttp session but leavesself._wsset and open. Ifclose()runs while a synthesis is in flight, the socket is torn down only as a side effect of the session close, andself._wskeeps a stale reference. Call_close_ws()first.♻️ Proposed change
async def close(self) -> None: """Close the current WebSocket and release the aiohttp session.""" await super().close() + await self._close_ws() session = self._session self._session = None
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 871d2669-480e-4bea-abd9-5c6202ebf214
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (12)
CHANGELOG.mdplugins/telnyx/README.mdplugins/telnyx/examples/README.mdplugins/telnyx/examples/voice_agent_call.pyplugins/telnyx/pyproject.tomlplugins/telnyx/tests/test_telnyx_llm.pyplugins/telnyx/tests/test_telnyx_stt.pyplugins/telnyx/tests/test_telnyx_tts.pyplugins/telnyx/vision_agents/plugins/telnyx/__init__.pyplugins/telnyx/vision_agents/plugins/telnyx/llm.pyplugins/telnyx/vision_agents/plugins/telnyx/stt.pyplugins/telnyx/vision_agents/plugins/telnyx/tts.py
| <<<<<<< HEAD | ||
| - vision-agents-plugins-openai | ||
| ======= | ||
| >>>>>>> 4c8d97ad (feat(telnyx): add Telnyx STT plugin) | ||
| - aiohttp |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔴 Critical | ⚡ Quick win
Unresolved merge conflict markers in the README.
Lines 269-273 still contain <<<<<<< HEAD, =======, and >>>>>>> 4c8d97ad. This README is the PyPI description page for the plugin, so the markers ship to users. Resolve the conflict and keep vision-agents-plugins-openai, which plugins/telnyx/pyproject.toml now declares as a dependency.
🐛 Proposed fix
- vision-agents
-<<<<<<< HEAD
- vision-agents-plugins-openai
-=======
->>>>>>> 4c8d97ad (feat(telnyx): add Telnyx STT plugin)
- aiohttpStreams audio to the Telnyx transcription WebSocket as raw linear16 frames and emits transcripts as they arrive. The configurable sample rate lets telephony audio from TelnyxMediaStream pass through at 8kHz without an upsample. Follows the plugins/sarvam precedent of one vendor plugin covering LLM, STT, and TTS.
LLM - Drop the hand-built AsyncOpenAI client; ChatCompletionsLLM already builds one from api_key/base_url. Follows the minimax plugin. - Export a single LLM name, matching the sarvam plugin. STT - self.model is str, not Optional[str]. Since #618 the STT base class declares `model: str`, so the Optional assignment failed mypy. - Default interim_results to False. Measured against the live API with the same audio: Speechmatics and Soniox stream partials, while the default Telnyx engine and Deepgram accept the parameter and return finals only. The old default advertised partials that never arrived. - Drop the SUPPORTED_ENGINES allowlist. The catalogue is served by Telnyx and grows over time, and the server already rejects unknown engines with an explicit error. - Flatten close() to the same shape as the sarvam plugin. - Do not emit a second error event for the socket close that an error payload causes. - Drop audio instead of blocking forever when start() has not run. - Declare turn_detection explicitly. TTS - Report the voice as `model` so agent metadata is populated. - Give the handshake its own timeout instead of reusing idle_timeout. - Emit connected/disconnected once per plugin rather than once per utterance; this provider opens a socket per synthesis. - Build PcmData with PcmData.from_av_frame. Tests - Fold the TTS test classes into one and replace static helpers with fixtures. - Cover the interim-transcript path end to end against an engine that emits partials; it had no live coverage. - Assert behaviour rather than private client attributes.
- Add `examples/voice_agent_call.py`: an inbound call answered by a pipeline that runs entirely on Telnyx, reusing the existing example helpers. Turn detection is local because the transcription endpoint sends no VAD signals. - Add a CHANGELOG entry, as the speechify plugin did when it landed. - Document what the live API actually does: which engines honour `interim_results`, and why the TTS decodes MP3 for every voice even though `audio_format` exists.
…f sensitive information' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
7608dea to
811f2cd
Compare
stop_audio() set the stop event and closed the WebSocket, but the receive loop only checked the event once per WebSocket frame. Telnyx serves a whole synthesis in very few large frames, so a single payload decodes to hundreds of PcmData chunks and the check never ran again: measured against the live API, stopping after 3 chunks still delivered all 445, the same as an uninterrupted synthesis. The Agent's own barge-in was unaffected, because interrupt() also bumps the epoch and send_iter() checks that per chunk. Direct users of stream_audio() plus stop_audio() got the whole utterance regardless. Check the stop event per decoded chunk as well. Stopping after 3 chunks now yields 3. The integration test pins this against an uninterrupted baseline.
3 participants
Why
The Telnyx plugin is a phone transport today. @a692570 contributed the three AI modalities that make a Telnyx phone agent run end to end, as #620 (LLM), #621 (TTS), and #622 (STT). This branch consolidates all three and applies review fixes on top. The original commits are preserved with their authorship; @a692570 offered to fold the three PRs together, and
plugins/sarvam/— the precedent all three cite — landed its LLM, STT, and TTS in a single PR (#488).Consolidating also settles decisions the three PRs answered inconsistently on their own. #620 argues against pinning a provider catalogue locally ("a hardcoded allowlist would go stale") while #622 shipped a hardcoded
SUPPORTED_ENGINESset; that set is now gone, since Telnyx keeps adding engines and the server already rejects unknown ones with an explicit error.Two fixes came out of measuring the live API rather than reading docs:
telnyx.STTdefaulted tointerim_results=Trueon the defaultTelnyxengine, which accepts the parameter and returns finals only — the same silent-finals-only trap #622 documents forpartial_results, but undetected, because both integration tests asserted on finals. Running the same audio through each engine: Speechmatics streams 21 partials and Soniox 41, while Telnyx and Deepgram return none. The default is nowFalse, the behaviour is documented per engine, and an integration test covers the partial-transcript path against an engine that actually emits them — that path had no live coverage before.The TTS spends roughly 120 lines decoding MP3 and stripping ID3 tags, and the endpoint does take an
audio_formatparameter, so raw PCM looked like a way to delete all of it. It is not:AWS.Polly.*andTelnyx.NaturalHD.*honouraudio_format=linear16, but the defaultTelnyx.KokoroTTS.*returns MP3 regardless, and the PCM sample rate is not reported on the wire and differs per voice. A voice-to-rate table would be exactly the kind of local catalogue this PR is removing elsewhere, so the MP3 path stays and the reasoning is now written down. Measuring also retired a suspicion that the decoder needed flushing: decoding with and without a flush returns an identical sample count on both voices, so no audio is being clipped.Rebasing onto
mainalso surfaced a hard failure. #618 addedmodel: strto theSTTbase class after these PRs branched, soself.model = modelwithOptional[str]no longer type checks.Changes
telnyx.LLM,telnyx.STT, andtelnyx.TTS, plus tests and README sections (@a692570, feat(telnyx): add Telnyx LLM plugin #620/feat(telnyx): add Telnyx TTS plugin #621/feat(telnyx): add Telnyx STT plugin #622)plugins/telnyx/examples/voice_agent_call.py: an inbound call answered by an all-Telnyx pipeline, reusing the existing example helpersmodeltypedstr,interim_resultsdefaults toFalse,SUPPORTED_ENGINESremoved,close()flattened to match sarvam, no duplicate error event on the close that follows an error payload, audio dropped rather than blocking forever whenstart()has not run,turn_detectiondeclared explicitlymodelso agent metadata is populated, a dedicated handshake timeout, connect/disconnect emitted once per plugin instead of once per utterance,PcmData.from_av_frameinstead of hand-builtPcmDataAsyncOpenAIconstruction dropped in favour of the base class's, and a single exported nameOpen questions for the reviewer
Dependencies. This makes
vision-agents-plugins-openaiandaiohttphard dependencies of a plugin most people install for phone transport. The four plugins that depend onplugins-openaitoday (inworld,minimax,openai,sarvam) are all AI-vendor plugins; telnyx would be the first transport plugin to do it. Left as-is to match sarvam, but gating the AI modalities behind an extra is the alternative.Duplication with sarvam.
stt.py's_receive_loop,process_audio, and_build_ws_url, andtts.py's_receive_audio, are near-copies of the sarvam equivalents — roughly 120 lines of aiohttp WebSocket scaffolding now living in two plugins. Extracting a shared base intoagents-corewould rewrite a shipped plugin from inside a plugin-addition PR, so it is left for a follow-up. Two of the smaller fixes here (the duplicate error event, andprocess_audioblocking whenstart()never ran) apply to sarvam as well.Supersedes #620, #621, #622.