From 5c0f77b586fe2b8cc6ebc72f9c230496e1f8c0d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kr=C3=A6n=20Hansen?= Date: Thu, 29 Oct 2026 08:50:49 +0100 Subject: [PATCH 1/2] feat(agents): add LiveKit Agents + Eleven Realtime example (Python) Installs livekit-plugins-elevenlabs from the livekit/agents#8075 branch until its realtime module is released. Co-Authored-By: Claude Opus 5.5 --- README.md | 1 + agents/python/livekit/PROMPT.md | 14 +++++ agents/python/livekit/README.md | 42 +++++++++++++ agents/python/livekit/example/.env.example | 1 + agents/python/livekit/example/README.md | 42 +++++++++++++ agents/python/livekit/example/main.py | 62 +++++++++++++++++++ .../python/livekit/example/requirements.txt | 3 + agents/python/livekit/setup.sh | 35 +++++++++++ 8 files changed, 200 insertions(+) create mode 100644 agents/python/livekit/PROMPT.md create mode 100644 agents/python/livekit/README.md create mode 100644 agents/python/livekit/example/.env.example create mode 100644 agents/python/livekit/example/README.md create mode 100644 agents/python/livekit/example/main.py create mode 100644 agents/python/livekit/example/requirements.txt create mode 100755 agents/python/livekit/setup.sh diff --git a/README.md b/README.md index a56ba175..16ea9519 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ Shared base templates live in `templates/` (Expo, Next.js, Python, TypeScript). - [Real-Time Voice Agent (Next.js)](agents/nextjs/quickstart/example/README.md) — Live voice conversations with the ElevenLabs Agents Platform using the React Agents SDK. - [Voice Agent Guardrails Demo (Next.js)](agents/nextjs/guardrails/example/README.md) — Demonstrate custom guardrails and the `guardrail_triggered` client event in a live voice agent. - [OpenAI Realtime SDK with ElevenLabs (Next.js)](agents/nextjs/openai-realtime/example/README.md) — Talk to ElevenLabs Realtime using OpenAI's Realtime SDK pointed at the ElevenLabs OpenAI-compatible endpoint. +- [LiveKit Agents with Eleven Realtime (Python)](agents/python/livekit/example/README.md) — Talk to a LiveKit Agents console voice agent powered by ElevenLabs Realtime, with a function tool. - [Voice Isolator (Next.js)](voice-isolator/nextjs/quickstart/example/README.md) — Record your voice in the browser and remove background noise with the Voice Isolator API. - [Dubbing Recorder (Next.js)](dubbing/nextjs/quickstart/example/README.md) — Record your voice in the browser, dub it into another language, and play or download the result. diff --git a/agents/python/livekit/PROMPT.md b/agents/python/livekit/PROMPT.md new file mode 100644 index 00000000..34afb9bd --- /dev/null +++ b/agents/python/livekit/PROMPT.md @@ -0,0 +1,14 @@ +Before writing any code, invoke the `/agents` skill to learn the correct ElevenLabs SDK patterns. + +Build a LiveKit Agents console voice agent that uses ElevenLabs Realtime through `livekit-plugins-elevenlabs`. + +## `main.py` + +- Import everything at the top, then call `load_dotenv()`. The plugin reads `ELEVEN_API_KEY` from the environment. +- Create an `AgentServer` and an `@server.rtc_session()` entrypoint that sets `ctx.log_context_fields = {"room": ctx.room.name}`. +- Build an `AgentSession` with `llm=elevenlabs.realtime.RealtimeModel(model="eleven_realtime_v1_mini", voice="brian", language="en")`. +- Subclass `Agent` with short voice-first instructions (no formatting, emojis or symbols) that tell it to use `lookup_weather` for weather questions. +- Greet the user in `on_enter` with `self.session.generate_reply()`. +- Add a `lookup_weather(context: RunContext, location: str)` `@function_tool` with a Google-style docstring (it becomes the tool schema) that logs the lookup and returns a canned sunny forecast. +- Call `await session.start(agent=Assistant(), room=ctx.room)`, then `await ctx.connect()`. +- Run with `cli.run_app(server)` under `if __name__ == "__main__":`. diff --git a/agents/python/livekit/README.md b/agents/python/livekit/README.md new file mode 100644 index 00000000..46cfa444 --- /dev/null +++ b/agents/python/livekit/README.md @@ -0,0 +1,42 @@ +# LiveKit Agents with Eleven Realtime (Python) + +Talk to a [LiveKit Agents](https://docs.livekit.io/agents/) voice agent powered by ElevenLabs Realtime, with a function tool, from your terminal. + +> This example depends on the unreleased `elevenlabs.realtime` module from [livekit/agents#8075](https://github.com/livekit/agents/pull/8075), so `requirements.txt` installs `livekit-plugins-elevenlabs` from that pull request's branch. + +## Setup + +1. Copy the environment file and add your API key: + + ```bash + cp .env.example .env + ``` + + Then edit `.env` and set `ELEVEN_API_KEY` to your [ElevenLabs API key](https://elevenlabs.io/app/settings/api-keys). + +2. Create a virtual environment and install dependencies: + + ```bash + python3 -m venv .venv + .venv/bin/pip install -r requirements.txt + ``` + +## Run + +```bash +.venv/bin/python main.py console +``` + +Console mode uses your local microphone and speakers, so no LiveKit server is needed. + +## Usage + +- The agent greets you when the session starts. +- Talk; the agent replies by voice. +- Ask about the weather somewhere to have the agent call the `lookup_weather` tool. +- Press `Ctrl+C` to end the session. + +## How it works + +- `elevenlabs.realtime.RealtimeModel` from `livekit-plugins-elevenlabs` is the session's `llm`, handling speech in and speech out with `eleven_realtime_v1_mini` and the `brian` voice. +- `lookup_weather` is a `@function_tool`; its docstring becomes the tool description the model sees. diff --git a/agents/python/livekit/example/.env.example b/agents/python/livekit/example/.env.example new file mode 100644 index 00000000..3baa2fd3 --- /dev/null +++ b/agents/python/livekit/example/.env.example @@ -0,0 +1 @@ +ELEVEN_API_KEY= diff --git a/agents/python/livekit/example/README.md b/agents/python/livekit/example/README.md new file mode 100644 index 00000000..46cfa444 --- /dev/null +++ b/agents/python/livekit/example/README.md @@ -0,0 +1,42 @@ +# LiveKit Agents with Eleven Realtime (Python) + +Talk to a [LiveKit Agents](https://docs.livekit.io/agents/) voice agent powered by ElevenLabs Realtime, with a function tool, from your terminal. + +> This example depends on the unreleased `elevenlabs.realtime` module from [livekit/agents#8075](https://github.com/livekit/agents/pull/8075), so `requirements.txt` installs `livekit-plugins-elevenlabs` from that pull request's branch. + +## Setup + +1. Copy the environment file and add your API key: + + ```bash + cp .env.example .env + ``` + + Then edit `.env` and set `ELEVEN_API_KEY` to your [ElevenLabs API key](https://elevenlabs.io/app/settings/api-keys). + +2. Create a virtual environment and install dependencies: + + ```bash + python3 -m venv .venv + .venv/bin/pip install -r requirements.txt + ``` + +## Run + +```bash +.venv/bin/python main.py console +``` + +Console mode uses your local microphone and speakers, so no LiveKit server is needed. + +## Usage + +- The agent greets you when the session starts. +- Talk; the agent replies by voice. +- Ask about the weather somewhere to have the agent call the `lookup_weather` tool. +- Press `Ctrl+C` to end the session. + +## How it works + +- `elevenlabs.realtime.RealtimeModel` from `livekit-plugins-elevenlabs` is the session's `llm`, handling speech in and speech out with `eleven_realtime_v1_mini` and the `brian` voice. +- `lookup_weather` is a `@function_tool`; its docstring becomes the tool description the model sees. diff --git a/agents/python/livekit/example/main.py b/agents/python/livekit/example/main.py new file mode 100644 index 00000000..3cef1288 --- /dev/null +++ b/agents/python/livekit/example/main.py @@ -0,0 +1,62 @@ +import logging + +from dotenv import load_dotenv +from livekit.agents import ( + Agent, + AgentServer, + AgentSession, + JobContext, + RunContext, + cli, + function_tool, +) +from livekit.plugins import elevenlabs + +logger = logging.getLogger("agent") + +load_dotenv() + + +class Assistant(Agent): + def __init__(self) -> None: + super().__init__( + instructions="""You are a helpful voice AI assistant. The user is interacting with you via voice. + Keep your answers short and conversational, without any formatting, emojis or symbols. + When the user asks about the weather, use the lookup_weather tool.""", + ) + + async def on_enter(self) -> None: + self.session.generate_reply() + + @function_tool + async def lookup_weather(self, context: RunContext, location: str) -> str: + """Look up the current weather for a location. + + Args: + location: The city or place to look up. + """ + logger.info(f"looking up weather for {location}") + return f"It is sunny and 21 degrees Celsius in {location}." + + +server = AgentServer() + + +@server.rtc_session() +async def my_agent(ctx: JobContext) -> None: + ctx.log_context_fields = {"room": ctx.room.name} + + session = AgentSession( + llm=elevenlabs.realtime.RealtimeModel( + model="eleven_realtime_v1_mini", + voice="brian", + language="en", + ), + ) + + await session.start(agent=Assistant(), room=ctx.room) + await ctx.connect() + + +if __name__ == "__main__": + cli.run_app(server) diff --git a/agents/python/livekit/example/requirements.txt b/agents/python/livekit/example/requirements.txt new file mode 100644 index 00000000..b242dca7 --- /dev/null +++ b/agents/python/livekit/example/requirements.txt @@ -0,0 +1,3 @@ +livekit-agents>=1.8.10 +livekit-plugins-elevenlabs @ git+https://github.com/kraenhansen/livekit-agents@kraen/elevenlabs-realtime#subdirectory=livekit-plugins/livekit-plugins-elevenlabs +python-dotenv>=1.2.1 diff --git a/agents/python/livekit/setup.sh b/agents/python/livekit/setup.sh new file mode 100755 index 00000000..86e73da9 --- /dev/null +++ b/agents/python/livekit/setup.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +set -euo pipefail + +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$DIR/../../.." && pwd)" +cd "$DIR" + +if [ -d example ]; then + find example -mindepth 1 -maxdepth 1 ! -name .venv -exec rm -rf {} + +fi +mkdir -p example + +rsync -a \ + --exclude .venv \ + --exclude example \ + "$REPO_ROOT/templates/python/" example/ + +cp README.md example/README.md + +# TODO: pin a released livekit-plugins-elevenlabs once livekit/agents#8075 ships +cat > example/requirements.txt <<'EOF' +livekit-agents>=1.8.10 +livekit-plugins-elevenlabs @ git+https://github.com/kraenhansen/livekit-agents@kraen/elevenlabs-realtime#subdirectory=livekit-plugins/livekit-plugins-elevenlabs +python-dotenv>=1.2.1 +EOF +echo "ELEVEN_API_KEY=" > example/.env.example + +if [ -f "$DIR/.env" ]; then + cp "$DIR/.env" example/.env +fi + +cd example +python3 -m venv .venv +.venv/bin/pip install --upgrade pip +.venv/bin/pip install -r requirements.txt From 8e92e7805b4f7c4f1f3279389a7003f432885c1e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kr=C3=A6n=20Hansen?= Date: Thu, 29 Oct 2026 08:54:13 +0100 Subject: [PATCH 2/2] docs(agents): call out the LiveKit example's ELEVEN_API_KEY name livekit-plugins-elevenlabs reads ELEVEN_API_KEY, unlike the ELEVENLABS_API_KEY used across this repo. Co-Authored-By: Claude Opus 5.5 --- agents/python/livekit/README.md | 2 +- agents/python/livekit/example/.env.example | 1 + agents/python/livekit/example/README.md | 2 +- agents/python/livekit/setup.sh | 5 ++++- 4 files changed, 7 insertions(+), 3 deletions(-) diff --git a/agents/python/livekit/README.md b/agents/python/livekit/README.md index 46cfa444..5d73570e 100644 --- a/agents/python/livekit/README.md +++ b/agents/python/livekit/README.md @@ -12,7 +12,7 @@ Talk to a [LiveKit Agents](https://docs.livekit.io/agents/) voice agent powered cp .env.example .env ``` - Then edit `.env` and set `ELEVEN_API_KEY` to your [ElevenLabs API key](https://elevenlabs.io/app/settings/api-keys). + Then edit `.env` and set `ELEVEN_API_KEY` to your [ElevenLabs API key](https://elevenlabs.io/app/settings/api-keys). Note the name: `livekit-plugins-elevenlabs` reads `ELEVEN_API_KEY`, not the `ELEVENLABS_API_KEY` the other examples use. 2. Create a virtual environment and install dependencies: diff --git a/agents/python/livekit/example/.env.example b/agents/python/livekit/example/.env.example index 3baa2fd3..034425ad 100644 --- a/agents/python/livekit/example/.env.example +++ b/agents/python/livekit/example/.env.example @@ -1 +1,2 @@ +# livekit-plugins-elevenlabs reads ELEVEN_API_KEY, not the usual ELEVENLABS_API_KEY. ELEVEN_API_KEY= diff --git a/agents/python/livekit/example/README.md b/agents/python/livekit/example/README.md index 46cfa444..5d73570e 100644 --- a/agents/python/livekit/example/README.md +++ b/agents/python/livekit/example/README.md @@ -12,7 +12,7 @@ Talk to a [LiveKit Agents](https://docs.livekit.io/agents/) voice agent powered cp .env.example .env ``` - Then edit `.env` and set `ELEVEN_API_KEY` to your [ElevenLabs API key](https://elevenlabs.io/app/settings/api-keys). + Then edit `.env` and set `ELEVEN_API_KEY` to your [ElevenLabs API key](https://elevenlabs.io/app/settings/api-keys). Note the name: `livekit-plugins-elevenlabs` reads `ELEVEN_API_KEY`, not the `ELEVENLABS_API_KEY` the other examples use. 2. Create a virtual environment and install dependencies: diff --git a/agents/python/livekit/setup.sh b/agents/python/livekit/setup.sh index 86e73da9..f10b016f 100755 --- a/agents/python/livekit/setup.sh +++ b/agents/python/livekit/setup.sh @@ -23,7 +23,10 @@ livekit-agents>=1.8.10 livekit-plugins-elevenlabs @ git+https://github.com/kraenhansen/livekit-agents@kraen/elevenlabs-realtime#subdirectory=livekit-plugins/livekit-plugins-elevenlabs python-dotenv>=1.2.1 EOF -echo "ELEVEN_API_KEY=" > example/.env.example +cat > example/.env.example <<'EOF' +# livekit-plugins-elevenlabs reads ELEVEN_API_KEY, not the usual ELEVENLABS_API_KEY. +ELEVEN_API_KEY= +EOF if [ -f "$DIR/.env" ]; then cp "$DIR/.env" example/.env