Add OpenAI Realtime SDK example (Next.js)

README.md

@@ -27,6 +27,7 @@ Shared base templates live in `templates/` (Expo, Next.js, Python, TypeScript).
 - [Speech Engine Quickstart (Next.js)](speech-engine/nextjs/quickstart/example/README.md) — Add voice to your own LLM-backed chat agent with a Speech Engine WebSocket server and Next.js client.
 - [Real-Time Voice Agent (Next.js)](agents/nextjs/quickstart/example/README.md) — Live voice conversations with the ElevenLabs Agents Platform using the React Agents SDK.
 - [Voice Agent Guardrails Demo (Next.js)](agents/nextjs/guardrails/example/README.md) — Demonstrate custom guardrails and the `guardrail_triggered` client event in a live voice agent.
+- [OpenAI Realtime SDK with ElevenLabs (Next.js)](agents/nextjs/openai-realtime/example/README.md) — Talk to ElevenLabs Realtime using OpenAI's Realtime SDK pointed at the ElevenLabs OpenAI-compatible endpoint.
 - [Voice Isolator (Next.js)](voice-isolator/nextjs/quickstart/example/README.md) — Record your voice in the browser and remove background noise with the Voice Isolator API.
 - [Dubbing Recorder (Next.js)](dubbing/nextjs/quickstart/example/README.md) — Record your voice in the browser, dub it into another language, and play or download the result.
 

agents/nextjs/openai-realtime/PROMPT.md

@@ -0,0 +1,48 @@
+Before writing any code, invoke the `/agents` skill to learn the correct ElevenLabs SDK patterns.
+
+Build a voice demo that talks to ElevenLabs Realtime through OpenAI's Realtime SDK (`@openai/agents-realtime`), pointed at the ElevenLabs OpenAI-compatible endpoint instead of `api.openai.com`.
+
+## 1. `package.json`
+
+- Add `@elevenlabs/elevenlabs-js`, `@openai/agents-realtime`, and `wavtools` dependencies.
+
+## 2. `lib/realtime-session.ts`
+
+Shared constants used by both the server route and the browser.
+
+- Export `REALTIME_URL` (`wss://api.elevenlabs.io/v1/convai/realtime`), `REALTIME_MODEL` (`eleven_realtime_v1_mini`), `REALTIME_VOICE` (`brian`), `AUDIO_SAMPLE_RATE` (`24000`), and short assistant instructions.
+- Export a `sessionConfig` typed as `ElevenLabs.OpenAiSessionConfigInput`: `type: "realtime"`, the model and instructions, `outputModalities: ["audio"]`, `audio.input` with PCM format at 24kHz and `semantic_vad` turn detection, and `audio.output` with PCM format and the voice.
+- Use the GA session shape (`outputModalities`, nested `audio.input`/`audio.output`). Do not use the deprecated top-level `modalities` or `voice` fields.
+
+## 3. `app/api/token/route.ts`
+
+Secure GET endpoint that mints a short-lived realtime client secret. Never expose `ELEVENLABS_API_KEY` to the client.
+
+- Use `client.conversationalAi.realtime.createClientSecret({ session: sessionConfig, expiresAfter: { anchor: "created_at", seconds: 600 } })`.
+- Return only `{ value }` to the browser.
+- Return a JSON `{ error }` with a sensible status when the key is missing or the request fails.
+
+## 4. `hooks/use-realtime-session.ts`
+
+Client hook that owns the realtime session.
+
+- Fetch `/api/token`, then create a `RealtimeAgent` and a `RealtimeSession` with an `OpenAIRealtimeWebSocket` transport using `REALTIME_URL`, the model, and the GA config (`outputModalities`, `audio.output.voice`). Connect with `session.connect({ apiKey: value })`.
+- The SDK only wires audio for WebRTC. Capture and play PCM16 at 24kHz with `wavtools` (`WavRecorder`, `WavStreamPlayer`).
+- Call `player.connect()` before any other `await` in `start` (including the token fetch), so Safari creates the playback `AudioContext` within the click's user activation.
+- Stream mic audio continuously with `session.sendAudio`; `wavtools` types `data.mono` as `Int16Array` but it is already an `ArrayBuffer` at runtime, so pass it through as-is.
+- Play `audio` events via `add16BitPCM`, and interrupt playback on `audio_interrupted`.
+- Track user speaking (`input_audio_buffer.speech_started/stopped`), agent speaking (`audio_start`/`audio_stopped`), response in flight (`response.created`/`response.done`), and the negotiated session (`session.created`/`session.updated`) from `transport_event`.
+- Build the transcript from raw transport events: stream assistant text from `response.output_audio_transcript.delta`, take user text from `conversation.item.input_audio_transcription.completed`, finalize on `.done` or `response.done`, and order turns by `previous_item_id`.
+- Keep a capped log of transport events, skipping the high-frequency audio and transcript deltas.
+- Tear down recorder, player, and session on stop, on a failed start, and when the transport reports `connection_change` as `disconnected`. Do not tear down on every `error` event; many are non-fatal.
+- `wavtools` never closes its `AudioContext`s; close both the player's (`player.context`) and the recorder's (`recorder.source.context`) on teardown so repeated sessions don't leak them.
+
+## 5. `app/page.tsx`
+
+Minimal voice-first page.
+
+- A Start/Stop toggle and connection status.
+- While connected, show independent indicators for user speaking, agent speaking, and responding, plus the negotiated model, voice, and turn detection.
+- A running transcript with a streaming caret for in-progress assistant turns and a "Transcribing…" placeholder for pending user turns.
+- A collapsible list of recent transport events.
+- Show errors inline and allow reconnecting.

agents/nextjs/openai-realtime/README.md

@@ -0,0 +1,41 @@
+# OpenAI Realtime SDK with ElevenLabs (Next.js)
+
+Hold a voice conversation with ElevenLabs Realtime using OpenAI's Realtime SDK (`@openai/agents-realtime`), pointed at the ElevenLabs OpenAI-compatible endpoint instead of `api.openai.com`. See the [OpenAI Realtime API compatibility guide](https://elevenlabs.io/docs/eleven-agents/guides/openai-realtime-compatibility).
+
+## Setup
+
+1. Copy the environment file and add your API key:
+
+   ```bash
+   cp .env.example .env.local
+   ```
+
+   Then set:
+   - `ELEVENLABS_API_KEY`
+
+2. Install dependencies:
+
+   ```bash
+   pnpm install
+   ```
+
+## Run
+
+```bash
+pnpm run dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) in your browser.
+
+## Usage
+
+- Click **Start** and allow microphone access when prompted.
+- Talk; the agent replies by voice and the transcript fills in as you go.
+- Expand **Transport events** to see the raw OpenAI Realtime events exchanged with ElevenLabs.
+- Click **Stop** to end the session.
+
+## How it works
+
+- `app/api/token/route.ts` mints a short-lived client secret with `client.conversationalAi.realtime.createClientSecret`, so `ELEVENLABS_API_KEY` never reaches the browser.
+- The browser passes that secret to `RealtimeSession.connect()` and connects `OpenAIRealtimeWebSocket` to `wss://api.elevenlabs.io/v1/convai/realtime`.
+- The SDK only wires audio automatically for WebRTC, so the page captures and plays PCM16 audio at 24kHz with [`wavtools`](https://www.npmjs.com/package/wavtools).

agents/nextjs/openai-realtime/example/.env.example

@@ -0,0 +1 @@
+ELEVENLABS_API_KEY=

agents/nextjs/openai-realtime/example/README.md

@@ -0,0 +1,41 @@
+# OpenAI Realtime SDK with ElevenLabs (Next.js)
+
+Hold a voice conversation with ElevenLabs Realtime using OpenAI's Realtime SDK (`@openai/agents-realtime`), pointed at the ElevenLabs OpenAI-compatible endpoint instead of `api.openai.com`. See the [OpenAI Realtime API compatibility guide](https://elevenlabs.io/docs/eleven-agents/guides/openai-realtime-compatibility).
+
+## Setup
+
+1. Copy the environment file and add your API key:
+
+   ```bash
+   cp .env.example .env.local
+   ```
+
+   Then set:
+   - `ELEVENLABS_API_KEY`
+
+2. Install dependencies:
+
+   ```bash
+   pnpm install
+   ```
+
+## Run
+
+```bash
+pnpm run dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) in your browser.
+
+## Usage
+
+- Click **Start** and allow microphone access when prompted.
+- Talk; the agent replies by voice and the transcript fills in as you go.
+- Expand **Transport events** to see the raw OpenAI Realtime events exchanged with ElevenLabs.
+- Click **Stop** to end the session.
+
+## How it works
+
+- `app/api/token/route.ts` mints a short-lived client secret with `client.conversationalAi.realtime.createClientSecret`, so `ELEVENLABS_API_KEY` never reaches the browser.
+- The browser passes that secret to `RealtimeSession.connect()` and connects `OpenAIRealtimeWebSocket` to `wss://api.elevenlabs.io/v1/convai/realtime`.
+- The SDK only wires audio automatically for WebRTC, so the page captures and plays PCM16 audio at 24kHz with [`wavtools`](https://www.npmjs.com/package/wavtools).

agents/nextjs/openai-realtime/example/app/api/token/route.ts

@@ -0,0 +1,33 @@
+import { ElevenLabsClient, ElevenLabsError } from "@elevenlabs/elevenlabs-js";
+import { NextResponse } from "next/server";
+
+import { sessionConfig } from "@/lib/realtime-session";
+
+export async function GET() {
+  const apiKey = process.env.ELEVENLABS_API_KEY?.trim();
+  if (!apiKey) {
+    return NextResponse.json(
+      { error: "Server misconfiguration: ELEVENLABS_API_KEY is not set." },
+      { status: 500 }
+    );
+  }
+
+  try {
+    const client = new ElevenLabsClient({ apiKey });
+    const secret = await client.conversationalAi.realtime.createClientSecret({
+      session: sessionConfig,
+      expiresAfter: { anchor: "created_at", seconds: 600 },
+    });
+    return NextResponse.json({ value: secret.value });
+  } catch (err) {
+    const status =
+      err instanceof ElevenLabsError && err.statusCode ? err.statusCode : 502;
+    return NextResponse.json(
+      {
+        error:
+          err instanceof Error ? err.message : "Failed to mint realtime token.",
+      },
+      { status: status >= 400 && status < 600 ? status : 502 }
+    );
+  }
+}

agents/nextjs/openai-realtime/example/app/globals.css

@@ -0,0 +1,126 @@
+@import "tailwindcss";
+@import "tw-animate-css";
+@import "shadcn/tailwind.css";
+
+@custom-variant dark (&:is(.dark *));
+
+@theme inline {
+  --color-background: var(--background);
+  --color-foreground: var(--foreground);
+  --font-sans: var(--font-geist-sans);
+  --font-mono: var(--font-geist-mono);
+  --color-sidebar-ring: var(--sidebar-ring);
+  --color-sidebar-border: var(--sidebar-border);
+  --color-sidebar-accent-foreground: var(--sidebar-accent-foreground);
+  --color-sidebar-accent: var(--sidebar-accent);
+  --color-sidebar-primary-foreground: var(--sidebar-primary-foreground);
+  --color-sidebar-primary: var(--sidebar-primary);
+  --color-sidebar-foreground: var(--sidebar-foreground);
+  --color-sidebar: var(--sidebar);
+  --color-chart-5: var(--chart-5);
+  --color-chart-4: var(--chart-4);
+  --color-chart-3: var(--chart-3);
+  --color-chart-2: var(--chart-2);
+  --color-chart-1: var(--chart-1);
+  --color-ring: var(--ring);
+  --color-input: var(--input);
+  --color-border: var(--border);
+  --color-destructive: var(--destructive);
+  --color-accent-foreground: var(--accent-foreground);
+  --color-accent: var(--accent);
+  --color-muted-foreground: var(--muted-foreground);
+  --color-muted: var(--muted);
+  --color-secondary-foreground: var(--secondary-foreground);
+  --color-secondary: var(--secondary);
+  --color-primary-foreground: var(--primary-foreground);
+  --color-primary: var(--primary);
+  --color-popover-foreground: var(--popover-foreground);
+  --color-popover: var(--popover);
+  --color-card-foreground: var(--card-foreground);
+  --color-card: var(--card);
+  --radius-sm: calc(var(--radius) - 4px);
+  --radius-md: calc(var(--radius) - 2px);
+  --radius-lg: var(--radius);
+  --radius-xl: calc(var(--radius) + 4px);
+  --radius-2xl: calc(var(--radius) + 8px);
+  --radius-3xl: calc(var(--radius) + 12px);
+  --radius-4xl: calc(var(--radius) + 16px);
+}
+
+:root {
+  --radius: 0.625rem;
+  --background: oklch(1 0 0);
+  --foreground: oklch(0.145 0 0);
+  --card: oklch(1 0 0);
+  --card-foreground: oklch(0.145 0 0);
+  --popover: oklch(1 0 0);
+  --popover-foreground: oklch(0.145 0 0);
+  --primary: oklch(0.205 0 0);
+  --primary-foreground: oklch(0.985 0 0);
+  --secondary: oklch(0.97 0 0);
+  --secondary-foreground: oklch(0.205 0 0);
+  --muted: oklch(0.97 0 0);
+  --muted-foreground: oklch(0.556 0 0);
+  --accent: oklch(0.97 0 0);
+  --accent-foreground: oklch(0.205 0 0);
+  --destructive: oklch(0.577 0.245 27.325);
+  --border: oklch(0.922 0 0);
+  --input: oklch(0.922 0 0);
+  --ring: oklch(0.708 0 0);
+  --chart-1: oklch(0.646 0.222 41.116);
+  --chart-2: oklch(0.6 0.118 184.704);
+  --chart-3: oklch(0.398 0.07 227.392);
+  --chart-4: oklch(0.828 0.189 84.429);
+  --chart-5: oklch(0.769 0.188 70.08);
+  --sidebar: oklch(0.985 0 0);
+  --sidebar-foreground: oklch(0.145 0 0);
+  --sidebar-primary: oklch(0.205 0 0);
+  --sidebar-primary-foreground: oklch(0.985 0 0);
+  --sidebar-accent: oklch(0.97 0 0);
+  --sidebar-accent-foreground: oklch(0.205 0 0);
+  --sidebar-border: oklch(0.922 0 0);
+  --sidebar-ring: oklch(0.708 0 0);
+}
+
+.dark {
+  --background: oklch(0.145 0 0);
+  --foreground: oklch(0.985 0 0);
+  --card: oklch(0.205 0 0);
+  --card-foreground: oklch(0.985 0 0);
+  --popover: oklch(0.205 0 0);
+  --popover-foreground: oklch(0.985 0 0);
+  --primary: oklch(0.922 0 0);
+  --primary-foreground: oklch(0.205 0 0);
+  --secondary: oklch(0.269 0 0);
+  --secondary-foreground: oklch(0.985 0 0);
+  --muted: oklch(0.269 0 0);
+  --muted-foreground: oklch(0.708 0 0);
+  --accent: oklch(0.269 0 0);
+  --accent-foreground: oklch(0.985 0 0);
+  --destructive: oklch(0.704 0.191 22.216);
+  --border: oklch(1 0 0 / 10%);
+  --input: oklch(1 0 0 / 15%);
+  --ring: oklch(0.556 0 0);
+  --chart-1: oklch(0.488 0.243 264.376);
+  --chart-2: oklch(0.696 0.17 162.48);
+  --chart-3: oklch(0.769 0.188 70.08);
+  --chart-4: oklch(0.627 0.265 303.9);
+  --chart-5: oklch(0.645 0.246 16.439);
+  --sidebar: oklch(0.205 0 0);
+  --sidebar-foreground: oklch(0.985 0 0);
+  --sidebar-primary: oklch(0.488 0.243 264.376);
+  --sidebar-primary-foreground: oklch(0.985 0 0);
+  --sidebar-accent: oklch(0.269 0 0);
+  --sidebar-accent-foreground: oklch(0.985 0 0);
+  --sidebar-border: oklch(1 0 0 / 10%);
+  --sidebar-ring: oklch(0.556 0 0);
+}
+
+@layer base {
+  * {
+    @apply border-border outline-ring/50;
+  }
+  body {
+    @apply bg-background text-foreground;
+  }
+}

agents/nextjs/openai-realtime/example/app/layout.tsx

@@ -0,0 +1,34 @@
+import type { Metadata } from "next";
+import { Geist, Geist_Mono } from "next/font/google";
+import "./globals.css";
+
+const geistSans = Geist({
+  variable: "--font-geist-sans",
+  subsets: ["latin"],
+});
+
+const geistMono = Geist_Mono({
+  variable: "--font-geist-mono",
+  subsets: ["latin"],
+});
+
+export const metadata: Metadata = {
+  title: "Create Next App",
+  description: "Generated by create next app",
+};
+
+export default function RootLayout({
+  children,
+}: Readonly<{
+  children: React.ReactNode;
+}>) {
+  return (
+    <html lang="en">
+      <body
+        className={`${geistSans.variable} ${geistMono.variable} antialiased`}
+      >
+        {children}
+      </body>
+    </html>
+  );
+}