Skip to content

Add OpenAI Realtime SDK example (Next.js) - #259

Merged
kraenhansen merged 6 commits into
mainfrom
kh/openai-realtime-example
Oct 20, 2026
Merged

kraenhansen merged 6 commits into
mainfrom
kh/openai-realtime-example

Conversation

@kraenhansen

@kraenhansen kraenhansen commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Developers with an existing OpenAI Realtime client need a runnable example of pointing it at ElevenLabs Realtime, to go with the OpenAI Realtime API compatibility guide.

Merging this PR will:

  • Add agents/nextjs/openai-realtime (PROMPT.md, setup.sh, README.md, generated example/): a Next.js voice demo using @openai/agents-realtime over WebSocket against wss://api.elevenlabs.io/v1/convai/realtime.
  • Mint the short-lived client secret server-side in app/api/token with client.conversationalAi.realtime.createClientSecret.
  • List the example in the root README.md.

Additional context intended for coding agents and bots

Origin

Ported from the Vite demo in elevenlabs/elevenlabs-dx#2577 onto templates/nextjs, following the repo's prompt-driven layout (modelled on agents/nextjs/guardrails).

Changes relative to the source demo

  • Token minting uses the published SDK method (@elevenlabs/elevenlabs-js ≥ 2.74.0) instead of a raw fetch in a Vite dev middleware, and returns only { value } to the browser. The /api/token path matches the guide's browser snippet.
  • Audio lifecycle is hardened: both wavtools AudioContexts are closed on teardown, the session tears down on transport connection_change: "disconnected" (not on every non-fatal error event), and player.connect() runs before the token fetch so Safari creates the playback context within the click's user activation.
  • Removed internal-only knobs: local-backend URL env vars and feature-flag-specific error mapping.
  • UI restyled to DESIGN.md (neutral, flat, template header preserved) with plain buttons instead of a shadcn Button and custom theme.

Known issue (not introduced here)

With pnpm 12, setup.sh exits with ERR_PNPM_IGNORED_BUILDS for template deps (sharp, msw, unrs-resolver) after installing packages; this applies to every Next.js setup.sh in the repo.

Test plan and verification

  • tsc --noEmit (clean)
  • eslint (clean)
  • next build (succeeds; / static, /api/token dynamic)
  • next start smoke test: / server-renders; /api/token without a key returns 500 {"error": "Server misconfiguration: ELEVENLABS_API_KEY is not set."}
  • npx prettier . --check (matches CI)
  • End-to-end voice session with a real ELEVENLABS_API_KEY, including Stop → Start cycles (verified manually)

🤖 Generated with Claude Code

Ported from elevenlabs/elevenlabs-dx#2577.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kraenhansen
kraenhansen marked this pull request as ready for review October 20, 2026 13:17
@cursor

cursor Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

PR Summary

Low Risk
Documentation and a self-contained example app; token route follows the same server-side secret pattern as other demos with no production service changes.

Overview
Adds a new OpenAI Realtime SDK with ElevenLabs Next.js example under agents/nextjs/openai-realtime, and links it from the root README.md.

The demo uses @openai/agents-realtime with OpenAIRealtimeWebSocket aimed at ElevenLabs’ OpenAI-compatible WebSocket (wss://api.elevenlabs.io/v1/convai/realtime). A GET /api/token route mints a short-lived client secret via createClientSecret and shared GA sessionConfig (PCM 24kHz, semantic_vad, audio output only) so ELEVENLABS_API_KEY stays server-side.

The client hook streams mic PCM with wavtools (SDK audio is WebRTC-only), plays agent audio, builds an ordered transcript from transport events, and tears down on disconnect with explicit AudioContext cleanup and Safari-friendly player.connect() ordering. PROMPT.md and setup.sh scaffold example/ from templates/nextjs like other agent samples.

Reviewed by Cursor Bugbot for commit e6006e0. Bugbot is set up for automated code reviews on this repo. Configure here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kraenhansen
kraenhansen requested a review from PaulAsjes October 20, 2026 13:19

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

- Close the AudioContexts wavtools leaves open, so repeated Start/Stop cycles don't accumulate them.
- Tear down when the transport disconnects, so the mic stops and the UI leaves "Connected".
- Connect the player before the token fetch, so Safari creates its AudioContext within the click.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

A disconnect-triggered teardown read sessionRef after awaiting audio cleanup, so a Stop → Start during that window could close the new session and leave the UI showing disconnected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

A disconnect during session.connect or recorder.record let the in-flight start() mark the UI connected, or tear down a newer session from its catch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7649ff0. Configure here.

A socket closed before open rejects session.connect() right after emitting "disconnected", so the torn-down attempt skipped setError. Errors are now hidden only when a newer start() has begun.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kraenhansen
kraenhansen merged commit df57686 into main Oct 20, 2026
3 checks passed
@kraenhansen
kraenhansen deleted the kh/openai-realtime-example branch October 20, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants