[Plugin] Palabra TTS (+ voice cloning)

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## New Features
 
+### `palabra` plugin: Palabra AI TTS and voice cloning (#627)
+
+Adds a new `palabra` plugin exposing `palabra.TTS`, backed by Palabra's realtime text-to-speech WebSocket API. It is a streaming TTS plugin, so the agent speaks each sentence while the LLM is still writing, and it keeps one session open across utterances — `stop_audio()` cancels synthesis server-side instead of reconnecting. Defaults to the `default_low` voice at 24 kHz and reads `PALABRA_API_KEY` from the environment. Install with `vision-agents[palabra]`.
+
+`palabra.Voices` wraps Palabra's cloned-voice API: `clone()` runs the whole create → upload → poll sequence and returns a `voice_id` that drops straight into `TTS(voice_id=...)`, alongside `get()`, `list()`, `delete()` and `limits()`. `TTS` also gained `idle_timeout` (default 5 s), which abandons a generation the server stops answering so a wedged utterance can't stall the TTS pipeline.
+
 ### `speechify` plugin: Speechify TTS
 
 Adds a new `speechify` plugin exposing `speechify.TTS`, backed by Speechify's streaming API. It streams raw PCM audio, defaults to the `simba-3.2` model with the `geffen_32` voice, and reads `SPEECHIFY_API_KEY` from the environment. Install with `vision-agents[speechify]`.

agents-core/pyproject.toml

@@ -89,6 +89,7 @@ tencent = ["vision-agents-plugins-tencent; sys_platform == 'linux'"]
 minimax = ["vision-agents-plugins-minimax"]
 twelvelabs = ["vision-agents-plugins-twelvelabs"]
 speechify = ["vision-agents-plugins-speechify"]
+palabra = ["vision-agents-plugins-palabra"]
 
 
 [tool.hatch.metadata]

plugins/palabra/README.md

@@ -0,0 +1,144 @@
+# Palabra AI
+
+[Palabra AI](https://palabra.ai) provides a realtime Text-to-Speech (TTS) API built for streaming: text is accepted
+incrementally over a WebSocket and audio comes back as raw PCM within a few hundred milliseconds, which makes it a good
+fit for voice AI agents.
+
+The Palabra plugin for Vision Agents lets you give your agent a Palabra voice, in 25 languages and with your own cloned
+voices.
+
+## Features
+
+- Streaming TTS over a single persistent WebSocket – the session is opened once and reused for every utterance
+- Sentence-level streaming (`streaming = True`), so the agent starts speaking while the LLM is still writing
+- Instant barge-in: `stop_audio()` cancels synthesis server-side without dropping the connection
+- Raw PCM output at any sample rate between 8 kHz and 48 kHz
+- 25 languages
+- Voice cloning from an audio sample, via `palabra.Voices`
+
+## Installation
+
+```bash
+uv add "vision-agents[palabra]"
+# or directly
+uv add vision-agents-plugins-palabra
+```
+
+## Usage
+
+```python
+from vision_agents.plugins import palabra
+
+tts = palabra.TTS()
+```
+
+Use it in an agent:
+
+```python
+from vision_agents.core.agents import Agent
+from vision_agents.core.edge.types import User
+from vision_agents.plugins import deepgram, gemini, getstream, palabra
+
+agent = Agent(
+    edge=getstream.Edge(),
+    agent_user=User(name="Palabra Voice Bot", id="agent"),
+    instructions="You're a helpful voice AI assistant.",
+    stt=deepgram.STT(),
+    llm=gemini.LLM(),
+    tts=palabra.TTS(voice_id="default_high", language="en"),
+)
+```
+
+<Warning>
+  To initialise without passing in the API key, make sure `PALABRA_API_KEY` is available as an environment variable.
+  You can do this either by defining it in a `.env` file or exporting it directly in your terminal. Create a key in the
+  [Palabra platform](https://platform.palabra.ai/api-keys).
+</Warning>
+
+## Examples
+
+Check out our [Palabra example](https://github.com/GetStream/Vision-Agents/tree/main/plugins/palabra/example) to see
+working code:
+
+- [main.py](https://github.com/GetStream/Vision-Agents/blob/main/plugins/palabra/example/main.py) – a voice bot that
+  uses Palabra TTS in a Stream call
+- [tts_smoke.py](https://github.com/GetStream/Vision-Agents/blob/main/plugins/palabra/example/tts_smoke.py) – synthesize
+  a few sentences to a WAV file and report the time to first audio chunk
+- [clone_voice.py](https://github.com/GetStream/Vision-Agents/blob/main/plugins/palabra/example/clone_voice.py) – clone a
+  voice from an audio sample and speak with it
+
+## Configuration
+
+| Name                | Type              | Default           | Description                                                                                              |
+|---------------------|-------------------|-------------------|----------------------------------------------------------------------------------------------------------|
+| `api_key`           | `str` or `None`   | `None`            | Your Palabra API key. Falls back to the `PALABRA_API_KEY` environment variable.                          |
+| `voice_id`          | `str`             | `"default_low"`   | Voice to synthesize with: `default_low`, `default_high`, or the id of a [cloned voice](https://platform.palabra.ai/docs/assets/voices). |
+| `language`          | `str`             | `"en"`            | BCP-47 language code of the text, e.g. `en`, `en-gb`, `de`, `pt-eu`, `ko`.                               |
+| `model`             | `str`             | `"auto"`          | TTS model id. `auto` lets Palabra pick the model.                                                        |
+| `sample_rate`       | `int`             | `24000`           | Output sample rate in Hz. Must be between `8000` and `48000`.                                            |
+| `speed`             | `float` or `None` | `None`            | Speech speed multiplier between `0.0` and `2.0`. `None` uses the server default.                         |
+| `deaccent_strength` | `float` or `None` | `None`            | Accent reduction for cloned voices, between `0.0` and `1.0`. `None` uses the server default.             |
+| `ws_url`            | `str`             | `WS_URL_EU`       | Palabra endpoint. Pass `palabra.WS_URL_US` to use the US region.                                          |
+| `idle_timeout`      | `float`           | `5.0`             | Seconds to wait for the next audio frame before abandoning a generation. Guards against a server that stops answering without sending an error. |
+
+## Functionality
+
+### Send text to convert to speech
+
+`send_iter()` sends the text to Palabra and yields `TTSOutputChunk`s carrying the produced PCM audio:
+
+```python
+async for chunk in tts.send_iter("Demo text you want the AI voice to say"):
+    pass
+```
+
+Text longer than Palabra's 1024-character per-message limit is split across several messages automatically, on word
+boundaries.
+
+### Stop speaking
+
+```python
+await tts.stop_audio()
+```
+
+This sends a `cancel` to Palabra and drops any audio still in flight. The WebSocket session stays open, so the next
+utterance does not pay for a new handshake.
+
+## Voice cloning
+
+`palabra.Voices` wraps Palabra's [cloned voice API](https://platform.palabra.ai/docs/assets/voices). `clone()` runs the
+whole sequence — reserve the voice, upload the sample to the presigned target, poll until Palabra reports it `ready` —
+and returns a `voice_id` you pass straight to `TTS`:
+
+```python
+from vision_agents.plugins import palabra
+
+async with palabra.Voices() as voices:
+    voice = await voices.clone("Narrator", "sample.wav", lang_code="en")
+
+tts = palabra.TTS(voice_id=voice.voice_id, deaccent_strength=0.7)
+```
+
+`deaccent_strength` exists specifically for cloned voices: lower it to keep more of the speaker's accent, raise it to
+neutralise it.
+
+Palabra needs **at least 30 seconds** of clean, single-speaker audio, at most **10 MB**, as MP3, WAV, FLAC, WEBM, MP4,
+MPEG or MPG. Cloning usually finishes in under a minute; `clone()` polls until then, or pass `wait=False` to return
+immediately and check `voices.get(voice_id).ready` yourself.
+
+| Method                                    | Description                                                                       |
+|-------------------------------------------|-----------------------------------------------------------------------------------|
+| `clone(name, sample, ...)`                | Clone a voice from an audio or video sample. Returns a `ClonedVoice`.              |
+| `get(voice_id)`                           | Fetch one voice, including `processing_status` and any errors or warnings.         |
+| `list(search=..., lang=..., page_size=...)`| List cloned voices.                                                              |
+| `delete(voice_id)`                        | Permanently delete a cloned voice. Irreversible.                                  |
+| `limits()`                                | Cloned voice quota for the account (`total`, `limit`, `remaining`, …).             |
+
+Cloned voices count against your account quota, so delete the ones you no longer need. Only clone a person's voice with
+their explicit consent.
+
+## Dependencies
+
+- [`vision-agents`](https://pypi.org/project/vision-agents/)
+- [`websockets`](https://pypi.org/project/websockets/)
+- [`httpx`](https://pypi.org/project/httpx/) – for the voice cloning REST API

plugins/palabra/example/.env.example

@@ -0,0 +1,12 @@
+# Stream API credentials
+STREAM_API_KEY=your_stream_api_key_here
+STREAM_API_SECRET=your_stream_api_secret_here
+
+# Palabra TTS
+PALABRA_API_KEY=your_palabra_api_key_here
+
+# Deepgram STT
+DEEPGRAM_API_KEY=your_deepgram_api_key_here
+
+# Gemini LLM
+GOOGLE_API_KEY=your_google_api_key_here

plugins/palabra/example/README.md

@@ -0,0 +1,73 @@
+# Stream + Palabra Voice Bot Example
+
+This example demonstrates how to build a voice bot that joins a Stream video call, transcribes participants with
+Deepgram STT, and speaks responses with Palabra AI TTS.
+
+## What it does
+
+- Creates a voice bot that joins a Stream video call
+- Uses Deepgram for realtime STT and turn detection
+- Uses Palabra for streaming TTS responses
+- Uses Gemini for the LLM response
+
+## Prerequisites
+
+1. **Stream Account**: Get your API credentials from [Stream Dashboard](https://getstream.io/try-for-free/?utm_source=github.com&utm_medium=referral&utm_campaign=vision_agents)
+2. **Palabra Account**: Create an API key at [platform.palabra.ai/api-keys](https://platform.palabra.ai/api-keys)
+3. **Deepgram Account**: Set a `DEEPGRAM_API_KEY` for STT.
+4. **Google AI Account**: Set a `GOOGLE_API_KEY` for the example LLM.
+5. **Python 3.10+**: Required for running the example
+
+## Installation
+
+You can use your preferred package manager, but we recommend [`uv`](https://docs.astral.sh/uv/).
+
+1. **Navigate to this directory:**
+   ```bash
+   cd plugins/palabra/example
+   ```
+
+2. **Install dependencies:**
+   ```bash
+   uv sync
+   ```
+
+3. **Set up environment variables:**
+   Copy `.env.example` to `.env` and fill in `STREAM_API_KEY`, `STREAM_API_SECRET`, `PALABRA_API_KEY`,
+   `DEEPGRAM_API_KEY`, and `GOOGLE_API_KEY`.
+
+## Usage
+
+Run the voice bot:
+
+```bash
+uv run main.py run
+```
+
+Join the generated call, speak into your microphone, and the bot should answer out loud.
+
+## Checking TTS on its own
+
+`tts_smoke.py` drives the plugin without a call: it synthesizes a few sentences over one WebSocket session, prints the
+time to first audio chunk for each, and writes the result to `palabra_smoke.wav`.
+
+```bash
+uv run tts_smoke.py
+uv run tts_smoke.py "Anything else you would like to hear"
+```
+
+Only `PALABRA_API_KEY` is needed for this one.
+
+## Cloning a voice
+
+`clone_voice.py` clones a voice and then speaks with it. Pass your own recording, or pass nothing and it reads a passage
+with Palabra's stock voice to produce the sample itself, so the script works with no extra files:
+
+```bash
+uv run clone_voice.py                    # synthesize a sample, then clone it
+uv run clone_voice.py my_recording.wav   # clone from your own recording
+uv run clone_voice.py --keep             # don't delete the voice afterwards
+```
+
+Palabra needs at least 30 seconds of clean, single-speaker audio. Cloned voices count against your quota, so the script
+deletes the voice on the way out unless you pass `--keep`. Only clone someone's voice with their explicit consent.

plugins/palabra/example/clone_voice.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""
+Clone a voice with Palabra, then speak with it.
+
+Palabra needs at least 30 seconds of clean, single-speaker audio to clone from.
+Pass your own recording, or pass nothing and the script records a sample by
+reading a passage with Palabra's stock voice — which makes the example
+self-contained, at the cost of cloning a synthetic voice rather than a human one.
+
+Usage::
+    uv run clone_voice.py                      # synthesize a sample, then clone it
+    uv run clone_voice.py my_recording.wav     # clone from your own recording
+    uv run clone_voice.py --keep               # don't delete the voice afterwards
+
+Requires ``PALABRA_API_KEY`` (see `.env.example`). Cloned voices count against
+your account quota, so the script deletes the voice on the way out unless you
+pass ``--keep``.
+
+Only clone someone's voice with their explicit consent.
+"""
+
+import asyncio
+import sys
+import wave
+from pathlib import Path
+
+from dotenv import load_dotenv
+from vision_agents.plugins import palabra
+
+load_dotenv()
+
+SAMPLE_PATH = Path("palabra_voice_sample.wav")
+OUTPUT_PATH = Path("palabra_cloned_voice.wav")
+
+# ~40 seconds of speech: comfortably over Palabra's 30 second minimum.
+SAMPLE_SCRIPT = [
+    "Every morning the harbour wakes slowly, one boat at a time.",
+    "The fishermen speak in short sentences, mostly about the weather.",
+    "By seven the market is loud, and the gulls have taken the high walls.",
+    "A woman sells coffee from a cart she has pushed to the same corner for years.",
+    "She knows every regular by the way they hold their cup.",
+    "Later the tide turns and the water goes flat and grey.",
+    "Children run along the pier, daring each other to look over the edge.",
+    "In the evening the boats come back heavier than they left.",
+    "Someone always sings on the way in, badly, and nobody minds.",
+    "The harbour sleeps again before the town does.",
+]
+
+
+async def write_sample(tts: palabra.TTS, path: Path) -> float:
+    """Synthesize the passage into a WAV file and return its duration."""
+    audio = bytearray()
+    for line in SAMPLE_SCRIPT:
+        async for chunk in tts.send_iter(line):
+            if chunk.data is not None:
+                audio += chunk.data.samples.tobytes()
+
+    with wave.open(str(path), "wb") as wav:
+        wav.setnchannels(1)
+        wav.setsampwidth(2)
+        wav.setframerate(tts.sample_rate)
+        wav.writeframes(audio)
+    return len(audio) / 2 / tts.sample_rate
+
+
+async def speak(voice_id: str, path: Path) -> float:
+    """Say a line with the cloned voice and write it to ``path``."""
+    tts = palabra.TTS(voice_id=voice_id, deaccent_strength=0.7)
+    audio = bytearray()
+    try:
+        async for chunk in tts.send_iter(
+            "This is my cloned voice, generated from a short audio sample."
+        ):
+            if chunk.data is not None:
+                audio += chunk.data.samples.tobytes()
+        rate = tts.sample_rate
+    finally:
+        await tts.close()
+
+    with wave.open(str(path), "wb") as wav:
+        wav.setnchannels(1)
+        wav.setsampwidth(2)
+        wav.setframerate(rate)
+        wav.writeframes(audio)
+    return len(audio) / 2 / rate
+
+
+async def main() -> None:
+    args = [a for a in sys.argv[1:] if a != "--keep"]
+    keep = "--keep" in sys.argv
+    sample = Path(args[0]) if args else SAMPLE_PATH
+
+    async with palabra.Voices() as voices:
+        quota = await voices.limits()
+        print(f"voice quota: {quota.total}/{quota.limit} used, {quota.remaining} left")
+        if quota.remaining < 1:
+            print("no quota left — delete a voice first (voices.delete(voice_id))")
+            return
+
+        if not args:
+            tts = palabra.TTS()
+            try:
+                duration = await write_sample(tts, sample)
+            finally:
+                await tts.close()
+            print(f"recorded {duration:.1f}s sample -> {sample}")
+
+        print("cloning (this takes a moment)...")
+        voice = await voices.clone("Vision Agents demo", sample, lang_code="en")
+        print(f"voice {voice.voice_id} is {voice.processing_status}")
+        if voice.warnings:
+            print(f"warnings: {voice.warnings}")
+
+        try:
+            duration = await speak(voice.voice_id, OUTPUT_PATH)
+            print(f"spoke {duration:.1f}s with the cloned voice -> {OUTPUT_PATH}")
+        finally:
+            if keep:
+                print(f"keeping voice {voice.voice_id}")
+            else:
+                await voices.delete(voice.voice_id)
+                print(f"deleted voice {voice.voice_id}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())