Repository navigation
feat(voice): spoken Sparring and Crossfire rounds - #84
Conversation
Students can speak their turns and hear the opponent answer aloud. The student's speech goes through the browser's speech recognition into the same draft a typed turn uses. DebateLab never receives the audio. A new capture hook restarts recognition when the browser ends a session on silence, shows interim words, and times the take. From that the app measures speaking time, words per minute, filler words per language, and long pauses. The scorecard and the Crossfire debrief show those numbers, and the sparring judge may quote them. Nothing claims to have heard tone, confidence, or volume. Typed rounds keep their prompt and wording. The opponent speaks through a new `voice` Edge Function that calls ElevenLabs (eleven_v4 for Sparring, eleven_v4_turbo for Crossfire). The ai function lets the model use a short list of audio tags in a voice round, then splits the reply. The clean text is shown, signed, and saved. The tagged text is cut into segments, each with an HMAC token, so the voice function speaks only text the server wrote for that user. The function checks the session, the token, the account, and a per-user daily cap on the existing quota counter, and logs each request with its cost. Without the key, over the cap, or on any failure, the browser's speechSynthesis reads the clean text and the round continues. vercel.json now allows the microphone for the app's own origin. The old value blocked it for every origin, including dictation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two callers could stop the microphone at once, for example the Stop button and then Submit. The second call replaced the first one's completion handler, so the first caller waited forever and a submitted speech could be lost. Both now wait on the same stop. Some browsers never report the end of a speechSynthesis utterance. The player now moves on after a time limit based on the utterance length, so the sparring clock cannot stay paused behind a reply that has ended. A reply stopped while it waited for a tap to play now ends as well. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A microphone start that was still waiting for the permission prompt stayed alive when the turn was submitted, so recognition could open on the next turn. Submitting, and the end of the student's turn, now withdraw a pending start. An attempt counter keeps a late answer to an old prompt from opening a second recogniser. In Crossfire, words spoken but never sent stayed in the capture when the round ended, and the next round's first turn inherited their measurements. The capture is now dropped whenever the round or the account changes. Sparring keys the same reset on the round and the turn. If ElevenLabs started a successful response and the audio stream then broke, the voice function threw without a usage row, although the cap was spent and the clip may have been billed. It now logs the request with an unknown cost and returns the fallback error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (12)
🚧 Files skipped from review as they are similar to previous changes (5)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThis PR adds spoken Sparring and Crossfire rounds. It adds browser speech capture, delivery measurements, signed opponent audio, and an ElevenLabs endpoint with browser-speech fallback. It also updates session data, interface text, privacy disclosures, deployment guidance, and automated tests. ChangesVoice-round feature
Estimated code review effort: 5 (Critical) | ~90 minutes Sequence Diagram(s)sequenceDiagram
participant Student
participant SparringPage
participant SpeechCapture
participant AI
participant VoiceFunction
participant ElevenLabs
Student->>SparringPage: enable voice and submit spoken turn
SparringPage->>SpeechCapture: capture speech
SpeechCapture-->>SparringPage: transcript and delivery measurements
SparringPage->>AI: submit voice-enabled turn
AI-->>SparringPage: clean reply and signed segments
SparringPage->>VoiceFunction: request signed audio segment
VoiceFunction->>ElevenLabs: request opponent audio
ElevenLabs-->>VoiceFunction: audio stream
VoiceFunction-->>SparringPage: MP3 audio or coded error
Architecture SummaryArchitecture risk: 🔵 Low · up to The change affects 7 systems. Changed systems: Architecture concerns Review detailsSystems and components
Before / after behavior
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing. Comment |
|
@coderabbitai full review |
|
- Give the clip back when ElevenLabs refuses a request with an error status, since nothing was billed. Timeouts, aborts and cut-off audio keep their clip, because the provider may have billed them and a refund would let a caller that aborts early spend without a cap. The refund is a compare-and-swap on daily_usage_counters, so it needs no migration. - Read the request body up to 8,000 bytes and no further, and refuse a larger declared Content-Length before reading anything. - Time out the provider call at 25 s, below the browser's 30 s wait, so a slow clip ends with an error the browser acts on instead of being paid for and discarded. - Sign at most 12 segments per reply as before, but return the rest of the spoken text, untagged and unsigned, as voice.overflow for the browser voice. Replies past 12 segments were cut off. - Accept tokens up to 30 s ahead of the verifier's clock, so a small clock difference between the ai and voice functions does not reject a fresh token. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- After any refusal from the voice function other than an invalid token, including voice_provider_error, use the browser voice for the rest of the round instead of asking again on every reply. A new Sparring or Crossfire round (opponentVoice.beginRound) gives ElevenLabs another chance and clears the allowance notice. - Read voice.overflow with the browser voice after the signed segments. - Split a sentence longer than one browser utterance at clause breaks, then spaces, so Chrome does not cut it off. - Resume the Sparring clock only for the turn the reply paused. A reply still playing when the next one arrived ended under it and restarted the clock while the new reply played. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Giving back a clip when ElevenLabs refuses meant the daily cap stopped bounding calls while the provider failed: a replayed token could call the function, and ElevenLabs, without limit during an outage or with a bad key. The client already switches to the browser voice for the rest of the round after a refusal, so a refusal costs at most one clip per round. Keep every attempt counted and test that it stays so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reply that arrived after the student left the Sparring page still started the opponent's voice, playing audio elsewhere in the app and spending a clip. It is now saved but not spoken. When the browser never reported the end of an utterance, the time limit moved on but left the utterance queued, so it could play after the student's clock restarted. The limit now cancels it. Found by a Codex review (gpt-6.1-sol, high). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
Brings in #79 to #83 and #85. Conflict in .superset/db-gate.sh: both sides added a function to the readiness loop. The loop now waits for both `voice` and `account`. #82 replaced the "Which debate is this for?" dialog with the first-debate offer when no debate is current. e2e/voice-rounds.spec.ts still clicked the old button in two places, so it answers the new offer the way e2e/sparring.spec.ts does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The full-stack sparring voice test turned the microphone on as soon as the opponent's reply appeared. Turning it on speaks over the opponent, which stops its voice and aborts the request to the voice function. The test then asserted that the function had answered 503 voice_unavailable exactly once, so it passed only when one of the two requests beat the click. The CI trace at cc4bad7 shows both requests ending in ERR_ABORTED on mobile Chromium, 170 to 350 ms and at most 160 ms after they started. Desktop passed in the same run because its second request got through. This is not a regression after 3c12919. With the local stack up: - The voice function answers in a median 71 to 77 ms once warm and 210 to 240 ms after a reload, almost all of it the auth lookup that runs before the body is read. Swapping readBoundedBody for the old req.text() changes nothing measurable (30 requests each, twice). - The unchanged test failed 2 of 8 on mobile at this head and 3 of 12 on mobile with the 3c12919 voice client and function restored. The test now waits, after each reply, for the voice function's answer and for the browser voice to read that reply. The final assertions are unchanged except one that is tighter: the browser voice must have read both replies, not just contain one. That also makes the test exercise the round-level fallback (no second request) on every run. 48 of 48 passed over 12 repeats of both voice tests on both Chromium projects, and the test fails when the fallback is removed. The interrupted request the test used to hit by chance is now a unit test: an abandoned request is aborted, nothing is read aloud, and the next reply still asks ElevenLabs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Implements plans/2026-10-01-voice-sparring-crossfire.md: students can speak their Sparring and Crossfire turns and hear the opponent answer aloud.
What changed
The student speaks. A voice switch on both setup screens turns a round into a spoken one. The microphone feeds the browser's speech recognition, and the recognised words go into the same draft a typed turn uses, so the student can still type or correct them. DebateLab never receives the audio. Where the browser has no speech recognition, or the microphone is blocked, the switch is replaced by a one-line reason and the round is typed as before. If the microphone is refused in the middle of a round, the round carries on typed.
The opponent answers aloud. A new
voiceEdge Function calls ElevenLabs:eleven_v4for Sparring andeleven_v4_turbofor Crossfire. I chose Turbo for Crossfire because its replies are one to three sentences and the wait matters more there than polish. If the key is missing, the user is over the cap, or the call fails, the browser'sspeechSynthesisreads the reply and the round continues.Audio tags. In a voice round the opponent prompt allows a short list of tags such as
[sighs],[laughs],[pause]and[sarcastic]. The server then splits the reply in two. The clean text has every tag removed, and it is the only text that is shown, signed, saved, or read by the fallback voice. The spoken text keeps allowed tags and goes to ElevenLabs only. The server also rations tags, so a model that ignores the prompt still gets at most one tag per sentence and never two sentences running.Measured feedback. For spoken turns the app measures speaking time against the limit, words per minute, filler words from a per-language list, and silences of three seconds or more. The sparring scorecard and the Crossfire debrief show those numbers. The sparring judge receives them with a prompt that lets it quote them and comment on structure, and forbids any claim about tone, confidence or volume. Typed rounds keep today's prompt and wording byte for byte (a test pins this).
Privacy. A first-use notice and the privacy page both say that DebateLab never records or stores audio, that Chrome's recognition sends audio to Google, and that ElevenLabs receives the opponent's text. All new text exists in English, Spanish and Hungarian.
Decisions Adam should know about
vercel.jsonnow sendsPermissions-Policy: microphone=(self). It wasmicrophone=(), which blocks the microphone for every origin including ours. Production sends the old value today (checked withcurl -I), which makesgetUserMediafail, so voice rounds could not work there and dictation in Ask and Generate is blocked by the same header. This ships with the merge. The header test is updated to pin the new value.consume_daily_quotacounter under avoicekey, so there is no migration. Arate_limitsrow withlimit_type = 'daily_voice_clips'overrides it per user or globally. These numbers are my guess at a sensible start and are easy to change.cost.tsprices voice rows from a dated snapshot of the API pricing page: $0.08 per 1,000 characters for v4 and $0.04 for v4 Turbo. That page shows a promotion until 12 October, and plan credits are not modelled, so a voice row can overstate the real charge. At list price a full School 4-6-6-3 round is roughly $1.30 of opponent speech.[pause]becomes an ellipsis before it reaches ElevenLabs. The Eleven v4 docs say there is no pause tag and that...is how to pause. The model may still write[pause].eleven_v4. SettingELEVENLABS_VOICE_IDreplaces it.What Adam needs to do after merge
The merge ships the frontend, and voice rounds then work on the browser voice straight away. For ElevenLabs audio:
aiandvoiceEdge Functions together.ELEVENLABS_API_KEYsecret.voice_sparringandvoice_crossfirerows with a cost inai_requests.There is no migration. I did not deploy, touch secrets, or merge.
Changed since the first review at
3c12919Five commits from another session (
055d77ato6cfb585) changed how voice behaves. In short:overflowfor the browser voice, so nothing is cut off.Two commits from this session:
72f6f0dmergesmain(feat(ops): add a read-only backend drift check #79 to feat(account): data export and account deletion #83 and test(live): answer the Current Debate prompt and pin Muse Spark 1.3 Contributor #85)..superset/db-gate.shconflicted because both sides added a function to the readiness loop; it now waits for bothvoiceandaccount. First run starts with a debate #82 replaced the "Which debate is this for?" dialog with the first-debate offer, soe2e/voice-rounds.spec.tsanswers the new offer in two places.cc03de2fixes thedatabase-testsfailure ine2e/full-stack/voice.spec.ts:12on mobile Chromium. See the next section.The mobile failure in
e2e/full-stack/voice.spec.ts:12Cause. The scripted student turned the microphone on as soon as the opponent's reply appeared. That speaks over the opponent, which stops its voice and aborts the request to the voice function. The test then asserted that the function had answered
503 voice_unavailableexactly once, so it passed only when one of the round's two requests beat the click. In the failing CI trace both requests ended inERR_ABORTED: the first 170 to 350 ms after it started, the second within 160 ms. Desktop passed in the same run only because its second request got through.It did not start after
3c12919. With the local stack up I measured both things:readBoundedBodyfor the oldreq.text()made no measurable difference (30 requests each, measured twice).cc4bad7, and 3 of 12 on mobile with the3c12919voice client and function restored. Desktop passed 8 of 8.So the race was in the test from its first version, and the one green mobile run at
3c12919was luck.Fix. After each reply the test now waits for the voice function's answer and for the browser voice to finish reading that reply, the same way
e2e/voice-rounds.spec.tsalready does. Nothing the test checks is looser. One assertion is tighter: the browser voice must have read both replies, where it used to need only one. That makes the test exercise the round-level fallback (no second request) on every run. Removing the fallback from the client makes the test fail.The interrupted request the test used to hit by chance is now a unit test: an abandoned request is aborted, nothing is read aloud, and the next reply still asks ElevenLabs.
Test evidence
The head is
cc03de2.GitHub Actions on
cc03de2:quality: pass.database-tests: pass. pgTAP 451 of 451 in 19 files. Full-stack journeys: 71 passed, 1 skipped, none failed. Both voice journeys passed on desktop and mobile Chromium.browser-tests: pass. 250 passed, 6 skipped.live-e2e: pass. It failed at3c12919and has passed since test(live): answer the Current Debate prompt and pin Muse Spark 1.3 Contributor #85 merged intomain.model-matrix: still running when this was written. It is not a check for this PR and was failing onmainbefore.Local, on
cc03de2:npm run verify:docs:check,lintandtypecheckclean; unit and component tests 1208 of 1208 in 116 files (90.65% statements, 84.14% branches); collaboration 30 of 30; Edge Function tests 325 of 325; build and service-worker smoke pass; browser suite 250 passed, 6 skipped, none failed or flaky. A first attempt lost one unit test (SparringPage.test.tsx, "renders setup…") to its 5-second timeout while the machine's load average was above 9; that test takes 0.9 seconds alone and passed in the full rerun../.superset/db-gate.sh: exit 0. pgTAP 451 of 451; full-stack 71 passed, 1 skipped, none failed; both voice journeys passed on both Chromium projects.Three earlier gate attempts on the same commit did not finish cleanly, none because of voice:
debatelab-fake-openroutercontainer was removed mid-run. A process watcher caught the cause:src/test/scripts/db-gate.test.tsruns the realdb-gate.shwith onlysupabaseandnpmstubbed andHOMEpointed at a temp folder. That bypasses the shared lock, and the script's cleanup calls the realdocker rm -fon the shared container. So a unit-test run in any worktree kills a gate running anywhere on the machine. The test is onmainand is not changed here; another branch (t3code/fix-preview-auth-and-faster-ci) had a fix in progress.Review: Codex (
gpt-6.1-sol, high effort) reviewed the original work at7af7209..f5a1821and3c12919; of the later voice commits,6cfb585records a Codex review at the same settings. I did not run a new Codex pass on those five commits. CodeRabbit shows "Review completed" with no review threads. The two commits from this session change tests and the gate script's readiness loop only.What is not proven
eleven_v4accepts every allowlisted tag gracefully, and the real latency are unverified until the key exists.live-e2epasses now, but it has no voice round.microphone=(self).🤖 Generated with Claude Code
Summary by CodeRabbit