💡 Intelligent Turn-Taking: Breath-and-Pause-Aware Voice Pipeline for Natural Conversation Flow

[github-actions[bot]]

Summary

Replace simple silence-timeout-based turn detection with a conversational turn-taking model that recognizes breath sounds, filler words ("um", "uh"), natural pauses, and conversational cues. This transforms voice interaction from robotic call-and-response into fluid, natural dialogue — essential for TalkTerm's voice-first identity with non-technical users.

Market Signal

ElevenLabs Conversational AI 2.0 and Cartesia Sonic both ship production-grade turn-taking models in 2026 that go far beyond silence detection. ElevenLabs detects conversational cues like "um", pauses, and breath sounds — the industry standard for voice AI interaction has moved past simple silence timeouts. VoiceDash and other voice-first tools are setting user expectations for natural conversation flow. TalkTerm's Web Speech API baseline will feel dated without this capability.

User Signal

TalkTerm targets non-technical knowledge workers (PRD persona: Sarah the PM) whose tolerance for awkward voice interaction is zero. FR37 (barge-in) addresses interruption but not the subtleties of natural conversation flow — when to wait for the user to finish thinking aloud, when to jump in, when to hold back. Poor turn-taking will drive users to the text input fallback (FR8), undermining the voice-first thesis that is TalkTerm's core differentiator.

Technical Opportunity

The architecture already plans a SpeechToText abstraction layer (port interface in src/shared/types/ports/) specifically designed to be swappable without app-level code changes. This abstraction can be extended to support turn-taking events (e.g., user_speaking, user_pausing, user_hesitating, user_yielding_floor). Phase 2 cloud voice provider upgrade is already on the roadmap — turn-taking intelligence ships with that upgrade via ElevenLabs or Cartesia SDKs. The Web Speech API adapter simply returns user_yielding on silence timeout, maintaining backward compatibility.

Assessment

Dimension	Score	Rationale
Feasibility	high	Proven technology from ElevenLabs/Cartesia, pluggable via existing abstraction layer design
Impact	high	Core differentiator for voice-first identity — directly validates product thesis
Urgency	high	Competitors setting expectations; architecture decisions needed before MVP port interfaces are finalized

Adversarial Review

Strongest objection: Web Speech API with barge-in support is "good enough" for MVP. Adding cloud-based turn detection increases cost, latency, and complexity before product-market fit is established.

Rebuttal: TalkTerm's entire value proposition is that voice interaction feels natural enough for non-technical users to prefer it over typing. If voice feels robotic, the product thesis fails — users will default to text input and TalkTerm becomes just another chat UI. The proposal is to architect for it now (turn-taking events in the SpeechToText port) and implement with the Phase 2 voice provider upgrade. Zero MVP scope creep; maximum future readiness.

Suggested Next Step

Extend the planned SpeechToText port interface to include a TurnTakingEvent callback type with states: user_speaking, user_pausing, user_hesitating, user_yielding. Web Speech API adapter ignores these (returns yielding on silence). Cloud provider adapters implement real turn detection. This ensures the abstraction is ready when Phase 2 voice providers ship.

[github-actions[bot]]

Weekly Update

What Changed

The 2026 production bar for voice turn-taking is now defined — industry benchmarks and competitive moves crystallize what TalkTerm's voice pipeline must target:

• Production benchmarks established. The 2026 standard for production voice AI barge-in is a 200-400ms turn-taking gap, a false-barge-in rate under 2%, and a TTS flush time below 60ms (FutureAGI, Tavus implementation guides). These are concrete engineering targets TalkTerm's voice pipeline should be designed against.

• Google Gemini Live raises the bar. Gemini Live added affective dialog — it can decide when to stay quiet rather than cut in, a behavior Google calls "proactive audio." It handles barge-in across many languages. This moves beyond simple silence detection into intent-aware turn management.

• Transition-relevant points (TRPs) as the key abstraction. The AI turn-taking research community has converged on TRPs — specific moments when speakers naturally pause, signaling it's the other person's turn — as the core concept. AI systems must detect TRPs through tone changes, completed thoughts, and brief pauses, not just raw silence duration.

• ChatGPT voice agentic features launched (Sep 23, 2026). OpenAI shipped voice-based agentic workflows on mobile and desktop with seamless text/voice switching. While their turn-taking quality isn't detailed publicly, this competitive entry raises user expectations for voice interaction quality across all AI products.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	med	→ Still requires ML-based VAD or integration with a provider that handles it (Cartesia, ElevenLabs)
Impact	high	high	→ Voice naturalness remains core to TalkTerm's non-technical user value proposition
Urgency	med	high	↑ ChatGPT voice agentic launch raises competitive bar; concrete benchmarks now available to target

Recommendation

Advance with specific targets. Design TalkTerm's voice pipeline against the 2026 production benchmarks: ≤300ms turn-taking gap (targeting the middle of the 200-400ms range), <2% false-barge-in rate, <60ms TTS flush. Evaluate whether to build TRP detection in-house or leverage a provider like Cartesia Sonic-3.6 (sub-90ms TTS latency, SSM-based architecture designed for consistent P99 latency) or ElevenLabs Conversational AI 2.0 as the voice backend behind the TextToSpeech/SpeechToText abstraction interfaces.