Repository navigation
openai_agents: align implicit model settings with the model a custom provider resolves - #178
Conversation
An agent that names no model carries the Agents SDK's default settings for the SDK's own default model, which for the gpt-5 family include a reasoning effort and `verbosity`. The SDK swaps them for the settings that fit the model name actually used, but only when that name is known at run time (`Agent(model=...)` or `RunConfig(model=...)`). With a custom `model_provider` on the plugin, the model is only resolved inside the model activity, so the gpt-5 defaults were forwarded as-is and strict OpenAI-compatible endpoints rejected the request (Fireworks AI: 400, "Extra inputs are not permitted, field: 'verbosity'"). Both model activities now apply the SDK's own rule where the model is resolved: when the agent named no model and its settings are exactly the implicit defaults, use the SDK's defaults for the resolved model's name. The default OpenAIProvider is unaffected, explicit settings pass through, and models that do not expose a `model` name keep today's behavior.
brianstrauch
left a comment
brianstrauch
left a comment
There was a problem hiding this comment.
Found two cases the new tests do not cover. The 54 tests in tests/test_openai.py and tests/test_openai_streaming.py pass; both additional workflow regression cases reproduced the issues below. The explicit run-settings regression also passes against the pre-PR Activity code.
| resolved_name = getattr(model, "model", None) | ||
| if not isinstance(resolved_name, str): | ||
| return model_settings | ||
| return get_default_model_settings(resolved_name) |
There was a problem hiding this comment.
[P2] Preserve explicit run-level overrides when aligning defaults
The settings arriving here have already been merged with RunConfig.model_settings, so equality with the SDK defaults does not prove they are implicit. With OPENAI_DEFAULT_MODEL=gpt-5, an unnamed agent, RunConfig(model_settings=ModelSettings(reasoning=Reasoning(effort="low"))), and a provider resolving gpt-5.4, the merged settings equal get_default_model_settings(). This replacement silently changes the explicitly requested reasoning effort from low to none. I reproduced this through a Workflow; the same regression test passes with the pre-PR Activity code. Please preserve run-level overrides separately and apply them after aligning the implicit defaults.
There was a problem hiding this comment.
Fixed in cd5bec1. The run's RunConfig.model_settings now travels separately: capture_scope sets it in a ContextVar for the run (covering the streamed path too), both stub creation paths read it into the activity input, and the activity re-applies it after aligning: get_default_model_settings(resolved_name).resolve(run_model_settings). Your gpt-5 + effort "low" + gpt-5.4 case is test_run_level_model_settings_survive_the_alignment, with a run_streamed twin in test_openai_streaming.py.
| the model, so the same swap happens here. Settings the agent set explicitly, and models | ||
| that do not expose a ``model`` name, pass through unchanged. | ||
| """ | ||
| if model_name is not None or model_settings != get_default_model_settings(): |
There was a problem hiding this comment.
[P2] Compare against the originating workflow's defaults
get_default_model_settings() reads the Activity worker's current OPENAI_DEFAULT_MODEL, but these settings were created on the workflow worker. Model Activities can run on a separate task queue with a different environment. For example, if the workflow defaults to gpt-5, the Activity worker defaults to gpt-5.4, and the provider resolves gpt-oss-120b, the incoming implicit settings have reasoning effort low while the Activity's defaults have effort none. The comparison fails, leaving verbosity="low" in the request and retaining the endpoint rejection this PR addresses. I reproduced this by changing the default-model environment at Activity-side model resolution. Please carry the originating defaults or an implicit-settings flag across the Activity boundary.
There was a problem hiding this comment.
Fixed in cd5bec1. The verdict is now made where the agent was built: the stub records agent.model_settings == get_default_model_settings() in the workflow worker's own environment and sends it as implicit_model_settings; the activity no longer calls the environment-dependent variant at all. test_alignment_uses_the_workflow_workers_defaults reproduces your scenario by switching OPENAI_DEFAULT_MODEL to gpt-5.4 inside the provider's get_model on the activity side. Inputs from older workflow workers carry no flag and pass through unchanged.
…l overrides Review found two gaps in the activity-side alignment. The settings that reach the activity are already merged with RunConfig.model_settings, so equality with the SDK defaults did not prove they were implicit, and a run-level reasoning effort equal to the default was silently replaced. And the activity compared against its own worker's OPENAI_DEFAULT_MODEL, which can differ from the workflow worker's, so the comparison could miss and the rejected field stayed in the request. The stub now records, in the workflow worker's own environment, whether the agent's settings are the SDK's implicit defaults, and sends that flag together with the run's RunConfig.model_settings in the activity input. The run-level settings live in a ContextVar set by capture_scope for the run, read by both stub creation paths, and covering the streamed run as well. The activity aligns the defaults with the resolved model only when the flag is set and re-applies the run-level overrides afterwards. Inputs from older workflow workers carry no flag and pass through unchanged.
Problem
An
Agentthat names no model carries the Agents SDK's defaultModelSettingsfor the SDK's own default model. For the gpt-5 family that is a reasoning effort plusverbosity="low". The SDK replaces them with the settings that fit the model actually used, but only when it knows the name at run time (Agent(model=...)orRunConfig(model=...); seeagents/run_internal/turn_preparation.py). With a custommodel_provideronOpenAIAgentsPlugin, the model is only resolved inside the model Activity, so the workflow side forwards the gpt-5 defaults untouched and a strict OpenAI-compatible endpoint rejects the call. Observed live on Fireworks AI with gpt-oss-120b:400 Extra inputs are not permitted, field: 'verbosity', which the plugin correctly classifies as non-retryable, so the Workflow fails on the first model call.Fix
The decision is made on the workflow side, where the agent was built: when neither the agent nor the run names a model, the stub records whether
agent.model_settingsequals the SDK's implicit defaults in the workflow worker's own environment (implicit_model_settings) and sends the run'sRunConfig.model_settingsalongside (run_model_settings). The run-level settings live in a ContextVar set bycapture_scopefor the duration of the run, read by both stub creation paths, and the streamed run is scoped the same way.Both model Activities then align only when the flag is set:
get_default_model_settings(<resolved model name>).resolve(run_model_settings). That mirrors the SDK's own rule for a named model and re-applies the run-level overrides afterwards. The defaultOpenAIProvideris unaffected (its default model is the SDK default, so the settings are identical), explicit agent settings pass through, a resolved model that does not expose amodelname keeps today's behavior, and inputs from older workflow workers carry no flag and pass through unchanged.Tests
Seven end-to-end tests run a Workflow through the Activity with a provider-chosen model: implicit defaults become empty for a non-GPT-5 model, stay for a GPT-5 model, stay for a model without a name, explicit agent settings arrive unchanged, a run-level reasoning effort equal to the implicit default survives the alignment (on
runandrun_streamed), and the verdict holds when the activity worker'sOPENAI_DEFAULT_MODELdiffers from the workflow worker's.make lintand the fullmake testsuite pass.Related
get_model(None)choosing the model) is reported in Agent with no model keeps GPT-5 default model settings when the model provider chooses the model openai/openai-agents-python#5414, with the fix on a fork branch (openai/openai-agents-python/compare/main...DABH:openai-agents-python:fix/implicit-model-settings-provider-model), since that repo only takes pull requests from collaborators. An upstream fix would not cover the Temporal path anyway: on the workflow side the resolved model is the plugin's stub, and the real model is only known in the Activity.