Repository navigation
fix(realtime): derive the remote-audio subscription from the local stream; drop the remoteAudio option - #220
Merged
tomershlasky merged 1 commit intoOct 8, 2026
Conversation
…ream; drop the remoteAudio option Stacked on #219. The server's audio track is the client's own audio played back delayed to stay in sync with the transformed video, so the only correct subscription rule is "subscribe exactly when we publish audio", which #219 already uses as the default. The explicit `remoteAudio` override had no caller: `true` would serve a model that emits audio the client never sent (none is served), `false` would serve an app that sends its microphone but does not want the synced voice back. Removing it keeps the fix transparent to apps and avoids a public knob for a hypothetical. The connect-time decision is stable for the life of the session: the SDK's only post-connect change to the local stream is replaceVideoTrack, which preserves the audio tracks. Tests: the two option tests go; a null-localStream case pins the video-only default for sessions without a camera stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Stacked on #219 so it can be reviewed on its own. #219 already fixes the iOS microphone prompt with the right default: subscribe to the server's audio track only when the local stream carries an audio track. This PR removes the extra
remoteAudioconnect option and keeps that derived rule as the only behavior.The server's audio track is not a generic audio feed. It is the client's own audio, held back so it plays in sync with the transformed video (the model adds latency and the server delays the audio to match). Given that, "subscribe exactly when we publish audio" is the complete rule:
remoteAudio: truewould only serve a model that emits audio the client never sent. No served realtime model does that. When one exists, the SDK should subscribe from a server-advertised capability, not an app-set flag, so apps never need to know which models emit audio.remoteAudio: falsewould only serve an app that sends its microphone but does not want the synced voice back, which defeats the point of sending it.Shipping the option would commit us to a public API surface for a case that does not exist yet.
What
remoteAudioremoved from the connect options schema,StreamSessionconfig andMediaChannelConfig.wantsRemoteAudio()now reads only the local stream.replaceVideoTrack, which keeps the audio tracks, so there is no event to re-evaluate on.Scenario matrix
Same app code in every column; only what arrives in
onRemoteStreamand what iOS does differ.getUserMedia({ audio: false })), e.g. a virtual try-on appgetUserMedia({ audio: true })), user talks over the transformed videoremoteAudio: falseremoteAudio: trueServer side
The server keeps publishing the track at join on purpose; removing it was A/B'd in api#4007 (API-1837) and dropped the publisher's initial bandwidth estimate from ~5.5 Mbps to ~2 Mbps in 10/10 sessions. Client-side selective subscription leaves the publisher leg untouched.
Tests
remoteAudiooverride tests; added alocalStream: nullcase pinning video-only subscription when there is no camera stream.vitest unit: 381 passed.tsc --noEmitandbiome checkclean.