💡 Full-Duplex Voice Architecture: Simultaneous Listen-and-Speak Pipeline as GPT-Live Competitive Response

[github-actions[bot]]

Summary

Rebuild TalkTerm's voice pipeline around a full-duplex architecture that can listen and speak simultaneously, matching the interaction quality GPT-Live established in July 2026. Instead of strict turn-taking, the avatar can produce backchannel acknowledgments ("mhmm", "got it") while the user speaks, handle natural interruptions without conversation resets, and overlap speech-end detection with response generation for sub-200ms perceived latency.

Market Signal

OpenAI shipped GPT-Live on July 23, 2026 with full-duplex voice on the ChatGPT desktop app (Mac & Windows). GPT-Live uses a full-duplex architecture that listens and speaks simultaneously, supports backchannel acknowledgments, and treats interruption as native behavior rather than an error. This sets a new competitive bar for conversational AI desktop apps — half-duplex turn-taking now feels dated. GPT-Live-1 shipped at $0.05/minute with ~200ms practical latency on September 10, 2026.

User Signal

Existing ideas #502 (Intelligent Turn-Taking) and #435 (Conversational Backchanneling) identify the need for more natural voice interaction, but neither proposes the full architectural shift to duplex audio that GPT-Live has now normalized. Non-technical users especially notice conversation unnaturalness because they benchmark against human conversation, not against prior AI tools.

Technical Opportunity

TalkTerm's architecture separates STT/TTS behind abstraction interfaces (SpeechToText, TextToSpeech ports in shared/types/ports/), enabling pipeline replacement without UI changes. The Electron main process can manage a full-duplex audio state machine that coordinates microphone capture, STT streaming, TTS playback, and agent response generation as parallel tracks. Rive's multi-layer state machine can drive "listening while speaking" avatar animations concurrently via additive layers. Web Audio API provides the mixing layer in the renderer process.

Assessment

Dimension	Score	Rationale
Feasibility	med	Requires audio pipeline architecture changes, but abstraction layer is already designed for this swap. Phase 1 (backchannel-only) reduces scope significantly.
Impact	high	GPT-Live set a new user expectation for conversational AI. Half-duplex feels broken to non-technical users who benchmark against human conversation.
Urgency	high	Every month TalkTerm ships with half-duplex widens the perception gap against ChatGPT Voice.

Adversarial Review

Strongest objection: Full-duplex is architecturally complex — it requires separating audio transport from agent inference, managing concurrent audio streams, and handling the edge case explosion of both parties speaking simultaneously. A single developer may burn weeks on audio engineering that doesn't ship user-visible features.

Rebuttal: TalkTerm doesn't need to build a full-duplex speech model (like GPT-Live) — it needs a full-duplex audio pipeline that coordinates existing components (STT, TTS, agent) in parallel rather than sequentially. Phase 1 can be backchannel-only (avatar produces "mhmm" during user speech) without full overlap, delivering 80% of the perceived improvement at 20% of the complexity. The abstraction layer is already designed for provider swapping.

Suggested Next Step

Prototype a backchannel-capable audio state machine in the main process that can play short TTS acknowledgment clips while STT capture remains active, using Web Audio API mixing in the renderer.