Skip to content

fix(realtime): derive the remote-audio subscription from the local stream; drop the remoteAudio option - #220

Merged
tomershlasky merged 1 commit into
fix/video-only-remote-subscribefrom
fix/video-only-remote-subscribe-transparent
Oct 8, 2026
Merged

tomershlasky merged 1 commit into
fix/video-only-remote-subscribefrom
fix/video-only-remote-subscribe-transparent

Conversation

@AdirAmsalem

@AdirAmsalem AdirAmsalem commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

Stacked on #219 so it can be reviewed on its own. #219 already fixes the iOS microphone prompt with the right default: subscribe to the server's audio track only when the local stream carries an audio track. This PR removes the extra remoteAudio connect option and keeps that derived rule as the only behavior.

The server's audio track is not a generic audio feed. It is the client's own audio, held back so it plays in sync with the transformed video (the model adds latency and the server delays the audio to match). Given that, "subscribe exactly when we publish audio" is the complete rule:

  • remoteAudio: true would only serve a model that emits audio the client never sent. No served realtime model does that. When one exists, the SDK should subscribe from a server-advertised capability, not an app-set flag, so apps never need to know which models emit audio.
  • remoteAudio: false would only serve an app that sends its microphone but does not want the synced voice back, which defeats the point of sending it.

Shipping the option would commit us to a public API surface for a case that does not exist yet.

What

  • remoteAudio removed from the connect options schema, StreamSession config and MediaChannelConfig. wantsRemoteAudio() now reads only the local stream.
  • README "Remote audio" section rewritten to describe the behavior (and why to play the remote audio rather than the local mic) with no option.
  • The connect-time decision is stable for the session: the SDK's only post-connect change to the local stream is replaceVideoTrack, which keeps the audio tracks, so there is no event to re-evaluate on.

Scenario matrix

Same app code in every column; only what arrives in onRemoteStream and what iOS does differ.

App SDK today (main) #219 This PR
Video only (getUserMedia({ audio: false })), e.g. a virtual try-on app remote stream has a silent audio track; iOS shows the mic prompt video only, no prompt video only, no prompt
Mic on (getUserMedia({ audio: true })), user talks over the transformed video voice comes back in sync same same
Mic on, app wants no voice back voice comes back remoteAudio: false not expressible; no known app needs it
Hypothetical model that emits audio the app never sent audio comes back remoteAudio: true needs an SDK change driven by a server capability flag

Server side

The server keeps publishing the track at join on purpose; removing it was A/B'd in api#4007 (API-1837) and dropped the publisher's initial bandwidth estimate from ~5.5 Mbps to ~2 Mbps in 10/10 sessions. Client-side selective subscription leaves the publisher leg untouched.

Tests

…ream; drop the remoteAudio option

Stacked on #219. The server's audio track is the client's own audio played
back delayed to stay in sync with the transformed video, so the only correct
subscription rule is "subscribe exactly when we publish audio", which #219
already uses as the default. The explicit `remoteAudio` override had no
caller: `true` would serve a model that emits audio the client never sent
(none is served), `false` would serve an app that sends its microphone but
does not want the synced voice back. Removing it keeps the fix transparent to
apps and avoids a public knob for a hypothetical.

The connect-time decision is stable for the life of the session: the SDK's
only post-connect change to the local stream is replaceVideoTrack, which
preserves the audio tracks.

Tests: the two option tests go; a null-localStream case pins the video-only
default for sessions without a camera stream.
@tomershlasky
tomershlasky merged commit e44b601 into fix/video-only-remote-subscribe Oct 8, 2026
1 check passed
@tomershlasky
tomershlasky deleted the fix/video-only-remote-subscribe-transparent branch October 8, 2026 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants