feat(voice): expose GA streamed transcription options - #4616
Open
abhay-codes07 wants to merge 3 commits into
Open
feat(voice): expose GA streamed transcription options#4616abhay-codes07 wants to merge 3 commits into
abhay-codes07 wants to merge 3 commits into
Conversation
Add languages, keywords, and delay to STTModelSettings and forward them in the streamed STT session configuration. These GA transcription options exist on the wire schema but had no SDK surface: languages and keywords guide gpt-transcribe / gpt-live-transcribe, and delay controls transcription latency for gpt-realtime-whisper. An explicit languages list takes precedence over the single language setting, which otherwise keeps its existing per-model spelling. The transcription span's model_config records the new options alongside the existing ones. All three fields are optional and appended after the existing fields, so the released dataclass contract is unchanged.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cf8824ae5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The delay option targets gpt-realtime-whisper, which requires
turn_detection: null in the session configuration, but None in
STTModelSettings.turn_detection falls back to the semantic_vad default
so null was unreachable. Accept {"type": "none"} as an explicit
disable marker, sent as null, mirroring the realtime module's
None-disables contract while keeping the released None-means-default
behavior. Cross-reference the requirement from the delay docstring.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ebdc1010b3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
With turn_detection: null the server never finalizes a turn on its own, so a session that only appends audio ends without ever producing a transcript. Send input_audio_buffer.commit for the appended audio when the input stream ends and turn detection is disabled. Sessions with server-side VAD are unchanged, and nothing is committed when no audio was appended so an empty buffer cannot be finalized.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
STTModelSettingsexposes only a subset of the GA transcription options that the streamed STT session's wire schema (AudioTranscriptionin the Realtime session payload) accepts. This PR adds the three missing options and forwards them on the streamed path:languages: possible languages of the input audio in ISO-639-1 format, supported bygpt-transcribeandgpt-live-transcribe. An explicitlanguageslist takes precedence over the singlelanguagesetting; when onlylanguageis set, the existing per-model spelling (languages: [language]for thegpt-transcribefamily,languageotherwise, per fix(voice): forward streamed STT language and prompt #4574) is unchanged.keywords: words or phrases to guide transcription, supported bygpt-transcribeandgpt-live-transcribe.delay: transcription latency/accuracy trade-off ("minimal"to"xhigh"), only supported withgpt-realtime-whisper.Like
turn_detection, the new fields apply to streamed sessions; each is sent only when set, so existing session payloads are byte-for-byte unchanged. Becausegpt-realtime-whisperrequiresturn_detection: null,turn_detectionnow also accepts{"type": "none"}as an explicit disable marker (sent asnull, mirroring the realtime module'sNone-disables contract);Nonekeeps its released meaning of using thesemantic_vaddefault. The transcription span'smodel_configrecords the new options alongside the existing ones. All three fields are optional and appended after the existing fields, so the released dataclass contract is unchanged (tests/test_released_api_contract.pypasses unmodified).Test plan
tests/voice/test_openai_stt_session_config.py:languagestakes precedence overlanguageacrossgpt-4o-transcribe,gpt-transcribe, andgpt-live-transcribe.keywordsanddelayare forwarded when set.gpt-realtime-whisper+delay+turn_detection={"type": "none"}producesturn_detection: nullin the session payload, and the defaultsemantic_vadpayload is unchanged.input_audio_buffer.commit) so the turn is transcribed; the default-VAD path stays append-only and an audio-less stream commits nothing.mainwithout the source change.uv run pytest tests/voice tests/test_released_api_contract.py: 316 passed.uv run ruff format --check, anduv run mypy srcpass locally (Windows). The verification stack could not complete cleanly becausemake lint(uv run ruff check) fails on currentmainwith a pre-existing E501 in.agents/skills/implementation-final-review/scripts/test_skill_contract.py:48(introduced by fe45b41, not touched by this PR); the files in this PR passruff checkandruff format --check.Issue number
None.
Checks
.agents/skills/code-change-verification/scripts/run.sh/reviewbefore submitting this PR