From cd2e7c8b351746e6e9d04bb0fc5e5658a4fd8771 Mon Sep 17 00:00:00 2001 From: DABH Date: Mon, 26 Oct 2026 01:00:45 -0500 Subject: [PATCH 1/2] Add Baseten, Fireworks AI, and Together AI integration pages (Python) Three pages for open-weight model providers with OpenAI-compatible APIs, built on the OpenAI Agents SDK integration's model provider hook. The pages share their prose and differ only in provider facts: base URL, API key, model naming, dedicated deployments, rate-limit behavior, and the provider's own capabilities over Chat Completions. Snippets come from one shared sample in samples-python (openai_agents/model_providers, run_openai_compatible_worker.py with --provider baseten|fireworks|together), rendered in snipsync's format. Also registers the pages in the sidebar, the integrations grid (new tag "Model provider"), and the Python landing list, where OpenRouter was missing, and adds the provider names to the heading-case exceptions. --- docs/develop/python/index.mdx | 4 + docs/develop/python/integrations/baseten.mdx | 264 +++++++++++++++++ .../python/integrations/fireworks-ai.mdx | 269 ++++++++++++++++++ .../python/integrations/together-ai.mdx | 267 +++++++++++++++++ sidebars.js | 3 + .../IntegrationsGrid/integrations-data.json | 27 ++ vale/styles/Temporal/Headings.yml | 3 + 7 files changed, 837 insertions(+) create mode 100644 docs/develop/python/integrations/baseten.mdx create mode 100644 docs/develop/python/integrations/fireworks-ai.mdx create mode 100644 docs/develop/python/integrations/together-ai.mdx diff --git a/docs/develop/python/index.mdx b/docs/develop/python/index.mdx index c1cee808ed..5e3500576d 100644 --- a/docs/develop/python/index.mdx +++ b/docs/develop/python/index.mdx @@ -85,8 +85,10 @@ From there, you can dive deeper into any of the Temporal primitives to start bui ## [Integrations](/develop/python/integrations) +- [Baseten integration](/develop/python/integrations/baseten) - [Braintrust integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#python) - [Deep Agents integration](/develop/python/integrations/deepagents) +- [Fireworks AI integration](/develop/python/integrations/fireworks-ai) - [Google ADK integration](/develop/python/integrations/google-adk) - [Google GenAI integration](/develop/python/integrations/google-genai) - [Laminar integration](https://laminar.sh/docs/integrations/temporal) @@ -96,10 +98,12 @@ From there, you can dive deeper into any of the Temporal primitives to start bui - [MCP integration](/develop/python/integrations/mcp) - [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) - [OpenBox integration](https://docs.openbox.ai/getting-started/temporal) +- [OpenRouter integration](/develop/python/integrations/openrouter) - [Parseable integration](https://github.com/parseablehq/temporal-plugin-python/blob/main/INTEGRATION.MD) - [Pydantic AI integration](https://pydantic.dev/docs/ai/capabilities/durable_execution/temporal/) - [Strands Agents integration](/develop/python/integrations/strands-agents) - [Tenuo integration](https://tenuo.ai/temporal) +- [Together AI integration](/develop/python/integrations/together-ai) ## Temporal Python technical resources diff --git a/docs/develop/python/integrations/baseten.mdx b/docs/develop/python/integrations/baseten.mdx new file mode 100644 index 0000000000..9145cc1ded --- /dev/null +++ b/docs/develop/python/integrations/baseten.mdx @@ -0,0 +1,264 @@ +--- +id: baseten +title: Baseten integration +sidebar_label: Baseten +toc_max_heading_level: 2 +tags: + - Baseten + - Python SDK + - Temporal SDKs +description: + Run OpenAI Agents SDK agents on open-weight models served by Baseten, with every model call retried and recorded as a + Temporal Activity. +--- +[Baseten](https://www.baseten.co/) serves open-weight models such as gpt-oss, DeepSeek, GLM, and Kimi, both on its +shared Model APIs and on dedicated deployments, through an API compatible with the OpenAI Chat Completions API. + +Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an +Activity. A call to Baseten gets retries with backoff, a timeout, and a durable record in Event History, and the agent +loop around it survives Worker restarts. Because Baseten speaks the OpenAI Chat Completions API, the integration's model +provider hook is all you need: the Worker points an OpenAI client at Baseten's base URL and names a model, and the +Workflow code does not change. + +This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the +[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), +which run the same agent on Baseten, Fireworks AI, or Together AI. + +## Prerequisites + +- This guide assumes you are already familiar with the + [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Baseten's + [OpenAI-compatible API](https://docs.baseten.co/inference/model-apis/overview). +- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking + the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. +- Set up your local development environment by following + [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal + development server running. +- Create a [Baseten API key](https://app.baseten.co/settings/api_keys) and export it as `BASETEN_API_KEY` in the + Worker's environment. The key stays in the Worker process; it is never part of Workflow input or Event History. + +## Install + +```bash +uv add temporalio-openai-agents +``` + +The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. The Baseten CLI and the +`truss` package are for deploying models to Baseten; calling Model APIs needs neither. + +## Configure the model provider + +The sample describes each provider as a base URL, the environment variable that holds its API key, and a default model: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +BASETEN = OpenAICompatibleProvider( + name="baseten", + base_url="https://inference.baseten.co/v1", + api_key_env="BASETEN_API_KEY", + model="openai/gpt-oss-120b", +) +``` + + +Model APIs name models the way Hugging Face does, as `/`, so the default here is +`openai/gpt-oss-120b`: an open-weight model under the Apache 2.0 license that Baseten serves on Model APIs, where tool +calling is supported for every model. Its [model library page](https://www.baseten.co/library/gpt-oss-120b/) and +`GET https://inference.baseten.co/v1/models` describe the catalog. A +[dedicated deployment](https://docs.baseten.co/inference/calling-your-model) has its own base URL, +`https://model-.api.baseten.co/environments/production/sync/v1`, takes the same API key, and expects the model name +that the deployment serves. + +A `ModelProvider` turns that description into the Chat Completions model that the Agents SDK calls: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +class OpenAICompatibleModelProvider(ModelProvider): + """OpenAI Agents SDK model provider for an OpenAI-compatible endpoint. + + Client retries are off: the Temporal plugin runs each model call as an + Activity, and the Activity's Retry Policy owns the retries. An agent that + names no model gets the provider's default; an agent can name any other + model the provider serves. + """ + + def __init__(self, provider: OpenAICompatibleProvider) -> None: + self._provider = provider + self._client = AsyncOpenAI( + base_url=provider.base_url, + api_key=os.environ[provider.api_key_env], + max_retries=0, + ) + + def get_model(self, model_name: str | None) -> Model: + # Chat Completions is the API all of these providers serve. The Agents + # SDK defaults to the Responses API, which not every provider offers. + return OpenAIChatCompletionsModel( + model=model_name or self._provider.model, + openai_client=self._client, + ) +``` + + +Two details matter here. Client retries are off (`max_retries=0`), so every attempt is a visible Activity attempt +governed by the Activity's Retry Policy rather than a hidden retry inside the HTTP client. And the provider builds an +`OpenAIChatCompletionsModel` rather than the Agents SDK's default Responses API model, which Baseten does not serve: +Model APIs expose Chat Completions and, in beta, the Anthropic Messages API. + +## Configure the Worker + +Pass the provider to the plugin. A custom model provider must set a model Activity timeout; the sample uses 60 seconds. +Model APIs need no deployment of your own to warm up; a dedicated deployment that has scaled to zero parks a request +until a replica starts, for up to 1200 seconds by default, so cover that cold start in the timeout or keep `min_replica` +above zero. The Heartbeat Timeout lets Temporal notice a Worker that died mid-call within seconds instead of after the +full Start-To-Close Timeout; the plugin heartbeats on its own. + + +[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) +```py +client = await Client.connect( + "localhost:7233", + plugins=[ + OpenAIAgentsPlugin( + model_params=ModelActivityParameters( + # A custom model provider must set a timeout. Raise it for + # slow models or deployments that scale up from zero. + start_to_close_timeout=timedelta(seconds=60), + # The plugin heartbeats, so a Worker that dies mid-call is + # noticed in seconds rather than after the full timeout. + heartbeat_timeout=timedelta(seconds=10), + ), + model_provider=OpenAICompatibleModelProvider(PROVIDERS[provider_name]), + ), + ], +) + +worker = Worker( + client, + task_queue="openai-agents-model-providers-task-queue", + workflows=[OpenAICompatibleAgentWorkflow], +) +await worker.run() +``` + + +The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider: +only the Worker talks to Baseten. + +## Write the Workflow + +The Workflow is ordinary Agents SDK code. It names no model, so the Worker's provider supplies its default; set `model=` +on the `Agent` to any model Baseten serves to override it. + + +[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) +```py +@workflow.defn +class OpenAICompatibleAgentWorkflow: + @workflow.run + async def run(self, prompt: str) -> str: + # Tools that run inside the Workflow must be async: the Agents SDK runs + # sync tools in a thread, which the Workflow sandbox does not allow. + @function_tool + async def get_weather(city: str) -> str: + workflow.logger.debug(f"Getting weather for {city}") + return f"The weather in {city} is sunny." + + # No model is named here, so the Worker's model provider supplies its + # default. Set model= to any model the provider serves to override it. + # Start from empty model settings: an agent that names no model gets + # the Agents SDK's defaults for OpenAI's default model (a reasoning + # effort and `verbosity`), which other providers reject. + agent = Agent( + name="Assistant", + instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", + model_settings=ModelSettings(), + tools=[get_weather], + ) + + result = await Runner.run(agent, prompt) + return result.final_output +``` + + +The empty `ModelSettings()` matters. An `Agent` that names no model gets the Agents SDK's default settings for OpenAI's +current default model, which include `verbosity` and a reasoning effort, and the plugin forwards them to the provider. +Fireworks AI, for example, rejects `verbosity` with a 400. An agent that names its model explicitly does not get those +defaults. + +Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow +sandbox does not allow. Tools that do I/O belong in Activities; see +[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools). + +## Run the sample + +Start the Worker with Baseten selected: + +```bash +export BASETEN_API_KEY="your_api_key" +uv run openai_agents/model_providers/run_openai_compatible_worker.py --provider baseten +``` + +Then start the Workflow in a second terminal: + +```bash +uv run openai_agents/model_providers/run_openai_compatible_workflow.py +``` + +## Retries, rate limits, and timeouts + +The plugin classifies Baseten's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry Policy, +and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as well. Other +4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying. Each attempt, +its failure, and the final result are in Event History. + +Baseten enforces requests-per-minute and tokens-per-minute limits that depend on your +[account tier](https://docs.baseten.co/inference/model-apis/pricing-and-limits) and answers `429` when either is +exceeded; the `x-ratelimit-remaining-requests` and `x-ratelimit-remaining-tokens` headers show which one hit zero. A +`529` means Baseten had no serving capacity for the request even within your limits. Both retry under the Activity's +Retry Policy. Baseten's [error reference](https://docs.baseten.co/inference/errors) asks for exponential backoff and +mentions a `Retry-After` header only as a possibility on `529`, so the Retry Policy's backoff, not the provider, usually +sets the pace. Sustained `429` responses mean the limit itself needs to go up, through Baseten's +[rate limit increase form](https://www.baseten.co/talk-to-us/increase-rate-limits/). + +## What works through Baseten + +Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the +Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter, +and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat +Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the +[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual. + +## Baseten notes + +- A dedicated deployment is called at `https://model-.api.baseten.co/environments/production/sync/v1` with the same + `Authorization: Bearer` key; the legacy `Api-Key` scheme is still accepted. The model name must match what the + deployment serves. A deployment whose `min_replica` is zero releases its replicas when idle, and the first request + after that waits for a replica to start; see Baseten's + [request lifecycle](https://docs.baseten.co/deployment/autoscaling/request-lifecycle) page. +- `GET https://inference.baseten.co/v1/models` lists the Model APIs catalog with context windows, pricing, and supported + features, as does `baseten model-api list` from the Baseten CLI. Baseten notes that the catalog can report different + limits from the ones the endpoint enforces. +- gpt-oss-120b reasons by default at `medium` effort. The reasoning arrives in `reasoning_content` on the assistant + message, and its tokens count as output tokens. Set `ModelSettings(reasoning=Reasoning(effort="low"))` on the agent to + change the effort; Baseten's [reasoning](https://docs.baseten.co/inference/model-apis/reasoning) page lists the + accepted values per model. +- Tool calling, structured outputs, and JSON mode are available on every Model API through `tools` and + `response_format`; see Baseten's [function calling](https://docs.baseten.co/inference/function-calling) and + [structured outputs](https://docs.baseten.co/inference/structured-outputs) guides. Only one completion per request is + supported, so leave `n` at its default. +- Baseten reuses cached input tokens across requests that share an `x-session-affinity` header. An agent run resends the + growing conversation on every turn, so sending the Workflow ID as the session, through + `ModelSettings(extra_headers=...)`, lets later turns reuse the cache. Baseten recommends at most about 60 requests per + minute per session. +- Two Baseten features sit outside this pattern: async inference, which delivers results to a webhook and works only on + dedicated deployments, and server-side web search tools on Model APIs, which are in preview. + +## Samples + +- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): + the same agent on Baseten, Fireworks AI, or Together AI, selected with `--provider`. +- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, + and tracing with the plugin. diff --git a/docs/develop/python/integrations/fireworks-ai.mdx b/docs/develop/python/integrations/fireworks-ai.mdx new file mode 100644 index 0000000000..8c74badce3 --- /dev/null +++ b/docs/develop/python/integrations/fireworks-ai.mdx @@ -0,0 +1,269 @@ +--- +id: fireworks-ai +title: Fireworks AI integration +sidebar_label: Fireworks AI +toc_max_heading_level: 2 +tags: + - Fireworks AI + - Python SDK + - Temporal SDKs +description: + Run OpenAI Agents SDK agents on open-weight models served by Fireworks AI, with every model call retried and recorded + as a Temporal Activity. +--- +[Fireworks AI](https://fireworks.ai/) serves open-weight models such as gpt-oss, Kimi K2, Qwen, and GLM on serverless +endpoints and on-demand deployments through an API compatible with the OpenAI Chat Completions API. + +Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an +Activity. A call to Fireworks AI gets retries with backoff, a timeout, and a durable record in Event History, and the +agent loop around it survives Worker restarts. Because Fireworks AI speaks the OpenAI Chat Completions API, the +integration's model provider hook is all you need: the Worker points an OpenAI client at Fireworks AI's base URL and +names a model, and the Workflow code does not change. + +This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the +[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), +which run the same agent on Baseten, Fireworks AI, or Together AI. + +## Prerequisites + +- This guide assumes you are already familiar with the + [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Fireworks AI's + [OpenAI-compatible API](https://docs.fireworks.ai/tools-sdks/openai-compatibility). +- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking + the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. +- Set up your local development environment by following + [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal + development server running. +- Create a [Fireworks AI API key](https://app.fireworks.ai/settings/users/api-keys) and export it as `FIREWORKS_API_KEY` + in the Worker's environment. The key stays in the Worker process; it is never part of Workflow input or Event History. + +## Install + +```bash +uv add temporalio-openai-agents +``` + +The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. Fireworks AI's own +[`fireworks-ai` package](https://docs.fireworks.ai/tools-sdks/python-sdk) is not needed: Fireworks AI documents the +`openai` client as a supported way to call its API. + +## Configure the model provider + +The sample describes each provider as a base URL, the environment variable that holds its API key, and a default model: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +FIREWORKS = OpenAICompatibleProvider( + name="fireworks", + base_url="https://api.fireworks.ai/inference/v1", + api_key_env="FIREWORKS_API_KEY", + model="accounts/fireworks/models/gpt-oss-120b", +) +``` + + +Serverless models on Fireworks AI are named `accounts/fireworks/models/`, and the +[Model Library](https://fireworks.ai/models) shows each model's ID and whether it supports tool calling. The default +here is [gpt-oss-120b](https://fireworks.ai/models/fireworks/gpt-oss-120b), an open-weight reasoning model with tool +calling that Fireworks AI serves serverless. An +[on-demand deployment](https://docs.fireworks.ai/guides/ondemand-deployments) uses the same base URL; pass the +deployment's resource name, `accounts//deployments/`, as the model instead. + +A `ModelProvider` turns that description into the Chat Completions model that the Agents SDK calls: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +class OpenAICompatibleModelProvider(ModelProvider): + """OpenAI Agents SDK model provider for an OpenAI-compatible endpoint. + + Client retries are off: the Temporal plugin runs each model call as an + Activity, and the Activity's Retry Policy owns the retries. An agent that + names no model gets the provider's default; an agent can name any other + model the provider serves. + """ + + def __init__(self, provider: OpenAICompatibleProvider) -> None: + self._provider = provider + self._client = AsyncOpenAI( + base_url=provider.base_url, + api_key=os.environ[provider.api_key_env], + max_retries=0, + ) + + def get_model(self, model_name: str | None) -> Model: + # Chat Completions is the API all of these providers serve. The Agents + # SDK defaults to the Responses API, which not every provider offers. + return OpenAIChatCompletionsModel( + model=model_name or self._provider.model, + openai_client=self._client, + ) +``` + + +Two details matter here. Client retries are off (`max_retries=0`), so every attempt is a visible Activity attempt +governed by the Activity's Retry Policy rather than a hidden retry inside the HTTP client. And the provider builds an +`OpenAIChatCompletionsModel` rather than the Agents SDK's default Responses API model, which Fireworks AI also offers, +because Chat Completions is the one API every provider in the sample serves. + +## Configure the Worker + +Pass the provider to the plugin. A custom model provider must set a model Activity timeout; the sample uses 60 seconds. +Fireworks AI answers a scaled-to-zero on-demand deployment with an immediate 503 rather than a slow first response, so +the Retry Policy covers scale-up and the timeout only has to cover your longest generation. The Heartbeat Timeout lets +Temporal notice a Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin +heartbeats on its own. + + +[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) +```py +client = await Client.connect( + "localhost:7233", + plugins=[ + OpenAIAgentsPlugin( + model_params=ModelActivityParameters( + # A custom model provider must set a timeout. Raise it for + # slow models or deployments that scale up from zero. + start_to_close_timeout=timedelta(seconds=60), + # The plugin heartbeats, so a Worker that dies mid-call is + # noticed in seconds rather than after the full timeout. + heartbeat_timeout=timedelta(seconds=10), + ), + model_provider=OpenAICompatibleModelProvider(PROVIDERS[provider_name]), + ), + ], +) + +worker = Worker( + client, + task_queue="openai-agents-model-providers-task-queue", + workflows=[OpenAICompatibleAgentWorkflow], +) +await worker.run() +``` + + +The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider: +only the Worker talks to Fireworks AI. + +## Write the Workflow + +The Workflow is ordinary Agents SDK code. It names no model, so the Worker's provider supplies its default; set `model=` +on the `Agent` to any model Fireworks AI serves to override it. + + +[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) +```py +@workflow.defn +class OpenAICompatibleAgentWorkflow: + @workflow.run + async def run(self, prompt: str) -> str: + # Tools that run inside the Workflow must be async: the Agents SDK runs + # sync tools in a thread, which the Workflow sandbox does not allow. + @function_tool + async def get_weather(city: str) -> str: + workflow.logger.debug(f"Getting weather for {city}") + return f"The weather in {city} is sunny." + + # No model is named here, so the Worker's model provider supplies its + # default. Set model= to any model the provider serves to override it. + # Start from empty model settings: an agent that names no model gets + # the Agents SDK's defaults for OpenAI's default model (a reasoning + # effort and `verbosity`), which other providers reject. + agent = Agent( + name="Assistant", + instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", + model_settings=ModelSettings(), + tools=[get_weather], + ) + + result = await Runner.run(agent, prompt) + return result.final_output +``` + + +The empty `ModelSettings()` matters. An `Agent` that names no model gets the Agents SDK's default settings for OpenAI's +current default model, which include `verbosity` and a reasoning effort, and the plugin forwards them to the provider. +Fireworks AI, for example, rejects `verbosity` with a 400. An agent that names its model explicitly does not get those +defaults. + +Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow +sandbox does not allow. Tools that do I/O belong in Activities; see +[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools). + +## Run the sample + +Start the Worker with Fireworks AI selected: + +```bash +export FIREWORKS_API_KEY="your_api_key" +uv run openai_agents/model_providers/run_openai_compatible_worker.py --provider fireworks +``` + +Then start the Workflow in a second terminal: + +```bash +uv run openai_agents/model_providers/run_openai_compatible_workflow.py +``` + +## Retries, rate limits, and timeouts + +The plugin classifies Fireworks AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry +Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as +well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying. Each +attempt, its failure, and the final result are in Event History. + +On serverless, Fireworks AI enforces +[adaptive token-per-minute limits](https://docs.fireworks.ai/serverless/rate-limits) per model that grow with steady +traffic and shrink when it stops. A burst beyond them returns 429, and Fireworks AI asks clients to honor `Retry-After` +when it is present. Under high load Fireworks AI can also shed traffic with `503 Service Overloaded`, which the plugin +retries like any 5xx; the `priority` service tier lowers that chance. +[On-demand deployments](https://docs.fireworks.ai/guides/inference-error-codes) have no token-per-minute limit, so a 429 +there means the deployment is saturated: add replicas or lower concurrency rather than wait. Both paths count against +the account-wide requests-per-minute cap described on the +[account quotas page](https://docs.fireworks.ai/guides/quotas_usage/account-quotas), which also sets the spending tiers +that raise the serverless ceilings. + +## What works through Fireworks AI + +Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the +Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter, +and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat +Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the +[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual. + +## Fireworks AI notes + +- Create an on-demand deployment with `firectl deployment create accounts/fireworks/models/gpt-oss-120b ...` and pass + the deployment name it returns as the model. By default a deployment scales to zero after an hour without traffic and + answers the next request with an immediate 503 while it scales back up, which the Activity's Retry Policy absorbs; a + deployment with zero minimum replicas is deleted after seven days without traffic. See the + [deployments quickstart](https://docs.fireworks.ai/getting-started/ondemand-quickstart). +- Fireworks AI updates and retires serverless models over time and gives + [at least two weeks' notice](https://docs.fireworks.ai/faq-new/deployment-infrastructure/do-you-provide-notice-before-removing-model-availability) + before removing one. For a version that must not change under a long-running Workflow, pin it on an on-demand + deployment. +- An agent's `output_type` is sent as a JSON schema in `response_format`, which Fireworks AI + [supports](https://docs.fireworks.ai/structured-responses/structured-response-formatting). Fireworks AI turns off + reasoning output while it enforces a schema, and turns JSON mode on by itself for tool calls. +- Reasoning models return their thinking in a `reasoning_content` field, and for models that interleave thinking with + tool calls Fireworks AI + [expects each earlier assistant turn's `reasoning_content` back](https://docs.fireworks.ai/guides/reasoning) in the + next request. The Agents SDK replays it only for DeepSeek models by default; pass `should_replay_reasoning_content` to + `OpenAIChatCompletionsModel` to replay it for the model you use. +- [Prompt caching](https://docs.fireworks.ai/guides/prompt-caching) is on by default, so a retried attempt that re-sends + the same prompt pays the discounted cached rate for the prefix. On serverless the cache lives in one replica, so hits + need a session identifier, which you can send as an `x-session-affinity` header through + `ModelSettings(extra_headers=...)`; dedicated deployments share their cache across requests. +- Fireworks AI also serves the [Responses API](https://docs.fireworks.ai/guides/response-api) at the same base URL, with + stored conversation state and server-executed tools of its own: remote MCP servers, web search, and a Python tool. + Build an `OpenAIResponsesModel` instead of `OpenAIChatCompletionsModel` to use it. The sample does not, and those + tools were not exercised with the Agents SDK's hosted tool types. + +## Samples + +- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): + the same agent on Baseten, Fireworks AI, or Together AI, selected with `--provider`. +- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, + and tracing with the plugin. diff --git a/docs/develop/python/integrations/together-ai.mdx b/docs/develop/python/integrations/together-ai.mdx new file mode 100644 index 0000000000..24adf8231c --- /dev/null +++ b/docs/develop/python/integrations/together-ai.mdx @@ -0,0 +1,267 @@ +--- +id: together-ai +title: Together AI integration +sidebar_label: Together AI +toc_max_heading_level: 2 +tags: + - Together AI + - Python SDK + - Temporal SDKs +description: + Run OpenAI Agents SDK agents on open-weight models served by Together AI, with every model call retried and recorded + as a Temporal Activity. +--- +[Together AI](https://www.together.ai/) serves open-weight models such as gpt-oss, DeepSeek, Qwen, and GLM on a +serverless per-token API and on dedicated endpoints through an API compatible with the OpenAI Chat Completions API. + +Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an +Activity. A call to Together AI gets retries with backoff, a timeout, and a durable record in Event History, and the +agent loop around it survives Worker restarts. Because Together AI speaks the OpenAI Chat Completions API, the +integration's model provider hook is all you need: the Worker points an OpenAI client at Together AI's base URL and +names a model, and the Workflow code does not change. + +This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the +[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), +which run the same agent on Baseten, Fireworks AI, or Together AI. + +## Prerequisites + +- This guide assumes you are already familiar with the + [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Together AI's + [OpenAI-compatible API](https://docs.together.ai/docs/inference/openai-compatibility). +- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking + the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. +- Set up your local development environment by following + [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal + development server running. +- Create a [Together AI API key](https://api.together.ai/settings/projects/~current/api-keys) and export it as + `TOGETHER_API_KEY` in the Worker's environment. The key stays in the Worker process; it is never part of Workflow + input or Event History. + +## Install + +```bash +uv add temporalio-openai-agents +``` + +The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. Together AI's own +[`together`](https://pypi.org/project/together/) package is not needed; the sample talks to Together AI through the +`openai` client. + +## Configure the model provider + +The sample describes each provider as a base URL, the environment variable that holds its API key, and a default model: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +TOGETHER = OpenAICompatibleProvider( + name="together", + base_url="https://api.together.ai/v1", + api_key_env="TOGETHER_API_KEY", + model="openai/gpt-oss-120b", +) +``` + + +Together AI model IDs follow the `/` convention, for example `openai/gpt-oss-120b`; OpenAI's +own model names return a 404. The [serverless models](https://docs.together.ai/docs/serverless/models) catalog lists +each model's ID and whether it supports function calling and structured outputs. The sample's default, +`openai/gpt-oss-120b`, is an open-weight reasoning model that supports both. A +[dedicated endpoint](https://docs.together.ai/docs/dedicated-endpoints/overview) is addressed the same way: its endpoint +string, `/`, goes in the `model` field, and requests go to +`https://api-inference.together.ai/v1` instead of the serverless base URL. + +A `ModelProvider` turns that description into the Chat Completions model that the Agents SDK calls: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +class OpenAICompatibleModelProvider(ModelProvider): + """OpenAI Agents SDK model provider for an OpenAI-compatible endpoint. + + Client retries are off: the Temporal plugin runs each model call as an + Activity, and the Activity's Retry Policy owns the retries. An agent that + names no model gets the provider's default; an agent can name any other + model the provider serves. + """ + + def __init__(self, provider: OpenAICompatibleProvider) -> None: + self._provider = provider + self._client = AsyncOpenAI( + base_url=provider.base_url, + api_key=os.environ[provider.api_key_env], + max_retries=0, + ) + + def get_model(self, model_name: str | None) -> Model: + # Chat Completions is the API all of these providers serve. The Agents + # SDK defaults to the Responses API, which not every provider offers. + return OpenAIChatCompletionsModel( + model=model_name or self._provider.model, + openai_client=self._client, + ) +``` + + +Two details matter here. Client retries are off (`max_retries=0`), so every attempt is a visible Activity attempt +governed by the Activity's Retry Policy rather than a hidden retry inside the HTTP client. And the provider builds an +`OpenAIChatCompletionsModel` rather than the Agents SDK's default Responses API model, which Together AI's OpenAI +compatibility guide does not cover. + +## Configure the Worker + +Pass the provider to the plugin. A custom model provider must set a model Activity timeout; the sample uses 60 seconds. +Serverless performance on Together AI is best-effort, and a dedicated endpoint that was stopped or scaled down pays a +cold start on its next request, so size the timeout for the slowest call you expect rather than the median. The +Heartbeat Timeout lets Temporal notice a Worker that died mid-call within seconds instead of after the full +Start-To-Close Timeout; the plugin heartbeats on its own. + + +[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) +```py +client = await Client.connect( + "localhost:7233", + plugins=[ + OpenAIAgentsPlugin( + model_params=ModelActivityParameters( + # A custom model provider must set a timeout. Raise it for + # slow models or deployments that scale up from zero. + start_to_close_timeout=timedelta(seconds=60), + # The plugin heartbeats, so a Worker that dies mid-call is + # noticed in seconds rather than after the full timeout. + heartbeat_timeout=timedelta(seconds=10), + ), + model_provider=OpenAICompatibleModelProvider(PROVIDERS[provider_name]), + ), + ], +) + +worker = Worker( + client, + task_queue="openai-agents-model-providers-task-queue", + workflows=[OpenAICompatibleAgentWorkflow], +) +await worker.run() +``` + + +The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider: +only the Worker talks to Together AI. + +## Write the Workflow + +The Workflow is ordinary Agents SDK code. It names no model, so the Worker's provider supplies its default; set `model=` +on the `Agent` to any model Together AI serves to override it. + + +[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) +```py +@workflow.defn +class OpenAICompatibleAgentWorkflow: + @workflow.run + async def run(self, prompt: str) -> str: + # Tools that run inside the Workflow must be async: the Agents SDK runs + # sync tools in a thread, which the Workflow sandbox does not allow. + @function_tool + async def get_weather(city: str) -> str: + workflow.logger.debug(f"Getting weather for {city}") + return f"The weather in {city} is sunny." + + # No model is named here, so the Worker's model provider supplies its + # default. Set model= to any model the provider serves to override it. + # Start from empty model settings: an agent that names no model gets + # the Agents SDK's defaults for OpenAI's default model (a reasoning + # effort and `verbosity`), which other providers reject. + agent = Agent( + name="Assistant", + instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", + model_settings=ModelSettings(), + tools=[get_weather], + ) + + result = await Runner.run(agent, prompt) + return result.final_output +``` + + +The empty `ModelSettings()` matters. An `Agent` that names no model gets the Agents SDK's default settings for OpenAI's +current default model, which include `verbosity` and a reasoning effort, and the plugin forwards them to the provider. +Fireworks AI, for example, rejects `verbosity` with a 400. An agent that names its model explicitly does not get those +defaults. + +Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow +sandbox does not allow. Tools that do I/O belong in Activities; see +[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools). + +## Run the sample + +Start the Worker with Together AI selected: + +```bash +export TOGETHER_API_KEY="your_api_key" +uv run openai_agents/model_providers/run_openai_compatible_worker.py --provider together +``` + +Then start the Workflow in a second terminal: + +```bash +uv run openai_agents/model_providers/run_openai_compatible_workflow.py +``` + +## Retries, rate limits, and timeouts + +The plugin classifies Together AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry +Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as +well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying. Each +attempt, its failure, and the final result are in Event History. + +Under high demand, Together AI's serverless API answers `429 Too Many Requests` or `503 Service Unavailable` and asks +clients to retry with exponential backoff, which is what the default Activity Retry Policy does; Together AI's +documentation does not describe a `Retry-After` header. Its [error codes](https://docs.together.ai/docs/error-codes) +page lists 500, 503, 504, 524, and 529 as retryable and the rest as request, credential, or billing problems to fix +first, including 402 when the account reaches its monthly spending limit and 403 when the prompt plus `max_tokens` +exceeds the model's context length. Serverless performance is best-effort; see +[rate limits](https://docs.together.ai/docs/serverless/rate-limits) for the current policy and +[provisioned throughput](https://docs.together.ai/docs/inference/provisioned-throughput) for reserved capacity with a +throughput and reliability commitment. + +## What works through Together AI + +Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the +Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter, +and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat +Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the +[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual. + +## Together AI notes + +- Dedicated endpoints are served at `https://api-inference.together.ai/v1`, not the serverless base URL. Deploy a model, + wait for the deployment to reach `DEPLOYMENT_STATE_READY` (first-time provisioning takes minutes), then use the + endpoint string as the model name. A running deployment keeps at least one replica; stopping it ends the cost and + makes the next request pay a cold start. See [autoscaling](https://docs.together.ai/docs/dedicated-endpoints/scaling) + for the replica bounds. +- Regional inference: `https://us-api.together.ai/v1` keeps serverless processing in the United States and + `https://ca-api.together.ai/v1` in Canada, each with its own model list. See + [geo-specific inference](https://docs.together.ai/docs/serverless/geo-specific-inference). +- `openai/gpt-oss-120b` is a reasoning model. Together AI accepts `reasoning_effort` (`low`, `medium`, or `high`; the + default is `medium`), which the Agents SDK sends when an agent sets + `ModelSettings(reasoning=Reasoning(effort="low"))`. + Together AI returns the model's chain of thought in a `reasoning` field, which the Agents SDK's Chat Completions model + does not read, so prior reasoning is not passed back between tool calls. See + [reasoning](https://docs.together.ai/docs/inference/chat/reasoning). +- Structured outputs: an agent's `output_type` becomes a `response_format` JSON schema, which + [Together AI supports](https://docs.together.ai/docs/inference/chat/structured-outputs) on models marked for it in the + catalog. Together AI recommends also describing the schema in the prompt, and a response that stops with + `finish_reason` of `length` leaves the JSON truncated, so give structured answers enough `max_tokens`. +- Model discovery: the `openai` client's `models.list()` works against Together AI, and the serverless catalog marks + function calling and structured outputs per model. Together AI retires serverless models over time (for example, + `openai/gpt-oss-20b` in September 2026), so check the [changelog](https://docs.together.ai/docs/changelog) before + pinning a model in production. + +## Samples + +- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): + the same agent on Baseten, Fireworks AI, or Together AI, selected with `--provider`. +- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, + and tracing with the plugin. diff --git a/sidebars.js b/sidebars.js index 8ffe251e5b..7ff333bb72 100644 --- a/sidebars.js +++ b/sidebars.js @@ -744,7 +744,9 @@ const developPythonCategory = { id: 'develop/python/integrations/index', }, items: [ + 'develop/python/integrations/baseten', 'develop/python/integrations/deepagents', + 'develop/python/integrations/fireworks-ai', 'develop/python/integrations/google-adk', 'develop/python/integrations/google-genai', 'develop/python/integrations/langgraph', @@ -753,6 +755,7 @@ const developPythonCategory = { 'develop/python/integrations/openai-agents', 'develop/python/integrations/openrouter', 'develop/python/integrations/strands-agents', + 'develop/python/integrations/together-ai', ], }, ], diff --git a/src/components/IntegrationsGrid/integrations-data.json b/src/components/IntegrationsGrid/integrations-data.json index 1678261d77..1f36a3a59e 100644 --- a/src/components/IntegrationsGrid/integrations-data.json +++ b/src/components/IntegrationsGrid/integrations-data.json @@ -395,5 +395,32 @@ ], "sdk": "TypeScript", "href": "https://github.com/temporalio/samples-typescript/tree/main/openrouter" + }, + { + "name": "Baseten", + "description": "Run OpenAI Agents SDK agents on open-weight models served by Baseten, with every model call as a durable Temporal Activity.", + "tags": [ + "Model provider" + ], + "sdk": "Python", + "href": "/develop/python/integrations/baseten" + }, + { + "name": "Fireworks AI", + "description": "Run OpenAI Agents SDK agents on open-weight models served by Fireworks AI, with every model call as a durable Temporal Activity.", + "tags": [ + "Model provider" + ], + "sdk": "Python", + "href": "/develop/python/integrations/fireworks-ai" + }, + { + "name": "Together AI", + "description": "Run OpenAI Agents SDK agents on open-weight models served by Together AI, with every model call as a durable Temporal Activity.", + "tags": [ + "Model provider" + ], + "sdk": "Python", + "href": "/develop/python/integrations/together-ai" } ] diff --git a/vale/styles/Temporal/Headings.yml b/vale/styles/Temporal/Headings.yml index 14bbb61775..9231894011 100644 --- a/vale/styles/Temporal/Headings.yml +++ b/vale/styles/Temporal/Headings.yml @@ -371,6 +371,9 @@ exceptions: - NOT_FOUND # AI vendor and model-platform proper nouns - AI + - Baseten + - Fireworks + - Together - Google - Gemini - Vertex From a751b6098bdce00dea511767036bb7a5d34d524a Mon Sep 17 00:00:00 2001 From: DABH Date: Mon, 26 Oct 2026 12:02:26 -0500 Subject: [PATCH 2/2] Route by model name on the Baseten, Fireworks AI, and Together AI pages Follows the samples change: one Worker serves every provider whose key is set, and each Workflow picks its provider through a prefixed model name such as `together/openai/gpt-oss-120b`. The pages now describe that routing, the per-provider key handling and its non-retryable failure, and the concurrent starter, and they no longer need the explicit `ModelSettings()` workaround because the agent names its model. The note on agents that name no model now cites temporalio-openai-agents 1.1.0, the current release. --- docs/develop/python/integrations/baseten.mdx | 162 +++++++++++------- .../python/integrations/fireworks-ai.mdx | 155 +++++++++++------ .../python/integrations/together-ai.mdx | 159 ++++++++++------- 3 files changed, 299 insertions(+), 177 deletions(-) diff --git a/docs/develop/python/integrations/baseten.mdx b/docs/develop/python/integrations/baseten.mdx index 9145cc1ded..d126e99d91 100644 --- a/docs/develop/python/integrations/baseten.mdx +++ b/docs/develop/python/integrations/baseten.mdx @@ -17,12 +17,13 @@ shared Model APIs and on dedicated deployments, through an API compatible with t Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an Activity. A call to Baseten gets retries with backoff, a timeout, and a durable record in Event History, and the agent loop around it survives Worker restarts. Because Baseten speaks the OpenAI Chat Completions API, the integration's model -provider hook is all you need: the Worker points an OpenAI client at Baseten's base URL and names a model, and the -Workflow code does not change. +provider hook is all you need: a model provider that maps a model name such as `baseten/openai/gpt-oss-120b` to +Baseten's endpoint. One Worker can serve Baseten alongside other providers, and each Workflow picks its provider through +the model name it gives its agent. This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the [OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), -which run the same agent on Baseten, Fireworks AI, or Together AI. +which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker. ## Prerequisites @@ -48,7 +49,8 @@ The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `op ## Configure the model provider -The sample describes each provider as a base URL, the environment variable that holds its API key, and a default model: +The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model +with tool calling: [openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) @@ -57,67 +59,98 @@ BASETEN = OpenAICompatibleProvider( name="baseten", base_url="https://inference.baseten.co/v1", api_key_env="BASETEN_API_KEY", - model="openai/gpt-oss-120b", + default_model="openai/gpt-oss-120b", ) ``` -Model APIs name models the way Hugging Face does, as `/`, so the default here is -`openai/gpt-oss-120b`: an open-weight model under the Apache 2.0 license that Baseten serves on Model APIs, where tool -calling is supported for every model. Its [model library page](https://www.baseten.co/library/gpt-oss-120b/) and -`GET https://inference.baseten.co/v1/models` describe the catalog. A -[dedicated deployment](https://docs.baseten.co/inference/calling-your-model) has its own base URL, -`https://model-.api.baseten.co/environments/production/sync/v1`, takes the same API key, and expects the model name -that the deployment serves. +Model APIs name models the way Hugging Face does, as `/`, so the example here is +`openai/gpt-oss-120b`, named `baseten/openai/gpt-oss-120b` from a Workflow: an open-weight model under the Apache 2.0 +license that Baseten serves on Model APIs, where tool calling is supported for every model. Its +[model library page](https://www.baseten.co/library/gpt-oss-120b/) and `GET https://inference.baseten.co/v1/models` +describe the catalog. A [dedicated deployment](https://docs.baseten.co/inference/calling-your-model) has its own base +URL, `https://model-.api.baseten.co/environments/production/sync/v1`, takes the same API key, and expects the model +name that the deployment serves, so in the sample it is a second entry in `PROVIDERS`. -A `ModelProvider` turns that description into the Chat Completions model that the Agents SDK calls: +A `ModelProvider` routes a `/` name to the matching endpoint and returns the Chat Completions model +that the Agents SDK calls: [openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) ```py class OpenAICompatibleModelProvider(ModelProvider): - """OpenAI Agents SDK model provider for an OpenAI-compatible endpoint. + """Routes ``/`` names to the matching OpenAI-compatible endpoint. - Client retries are off: the Temporal plugin runs each model call as an - Activity, and the Activity's Retry Policy owns the retries. An agent that - names no model gets the provider's default; an agent can name any other - model the provider serves. + One Worker serves every provider whose API key is set, so Workflows on + different providers run side by side on the same Task Queue. Client + retries are off: the plugin runs each model call as a Temporal Activity, + and the Activity's Retry Policy owns the retries. """ - def __init__(self, provider: OpenAICompatibleProvider) -> None: - self._provider = provider - self._client = AsyncOpenAI( - base_url=provider.base_url, - api_key=os.environ[provider.api_key_env], - max_retries=0, - ) + def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None: + self._providers = {provider.name: provider for provider in providers} + self._clients: dict[str, AsyncOpenAI] = {} + for provider in self._providers.values(): + api_key = os.environ.get(provider.api_key_env) + if api_key: + self._clients[provider.name] = AsyncOpenAI( + base_url=provider.base_url, api_key=api_key, max_retries=0 + ) + + @property + def configured(self) -> list[str]: + """Names of the providers whose API key is set.""" + return sorted(self._clients) def get_model(self, model_name: str | None) -> Model: + provider_name, _, model = (model_name or "").partition("/") + provider = self._providers.get(provider_name) + if provider is None or not model: + # A mistake in the Workflow's input: fail the Activity without retrying. + raise ApplicationError( + f"Model name {model_name!r} must be / with a " + f"provider in {sorted(self._providers)}", + non_retryable=True, + ) + client = self._clients.get(provider_name) + if client is None: + raise ApplicationError( + f"Set {provider.api_key_env} and restart the Worker to use {provider_name}", + non_retryable=True, + ) # Chat Completions is the API all of these providers serve. The Agents # SDK defaults to the Responses API, which not every provider offers. - return OpenAIChatCompletionsModel( - model=model_name or self._provider.model, - openai_client=self._client, - ) + return OpenAIChatCompletionsModel(model=model, openai_client=client) ``` -Two details matter here. Client retries are off (`max_retries=0`), so every attempt is a visible Activity attempt -governed by the Activity's Retry Policy rather than a hidden retry inside the HTTP client. And the provider builds an -`OpenAIChatCompletionsModel` rather than the Agents SDK's default Responses API model, which Baseten does not serve: -Model APIs expose Chat Completions and, in beta, the Anthropic Messages API. +Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name +for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off +(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a +hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents +SDK's default Responses API model, which Baseten does not serve: Model APIs expose Chat Completions and, in beta, the +Anthropic Messages API. ## Configure the Worker -Pass the provider to the plugin. A custom model provider must set a model Activity timeout; the sample uses 60 seconds. -Model APIs need no deployment of your own to warm up; a dedicated deployment that has scaled to zero parks a request -until a replica starts, for up to 1200 seconds by default, so cover that cold start in the timeout or keep `min_replica` -above zero. The Heartbeat Timeout lets Temporal notice a Worker that died mid-call within seconds instead of after the -full Start-To-Close Timeout; the plugin heartbeats on its own. +Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A +custom model provider must set a model Activity timeout; the sample uses 60 seconds. Model APIs need no deployment of +your own to warm up; a dedicated deployment that has scaled to zero parks a request until a replica starts, for up to +1200 seconds by default, so cover that cold start in the timeout or keep `min_replica` above zero. The Heartbeat Timeout +lets Temporal notice a Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the +plugin heartbeats on its own. [openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) ```py +# One provider for every endpoint whose API key is set. The model name on +# each Workflow's agent selects the endpoint. +model_provider = OpenAICompatibleModelProvider(PROVIDERS.values()) +if not model_provider.configured: + env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values()) + sys.exit(f"No provider API key is set; export at least one of {env_vars}.") +print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True) + client = await Client.connect( "localhost:7233", plugins=[ @@ -130,7 +163,7 @@ client = await Client.connect( # noticed in seconds rather than after the full timeout. heartbeat_timeout=timedelta(seconds=10), ), - model_provider=OpenAICompatibleModelProvider(PROVIDERS[provider_name]), + model_provider=model_provider, ), ], ) @@ -149,16 +182,25 @@ only the Worker talks to Baseten. ## Write the Workflow -The Workflow is ordinary Agents SDK code. It names no model, so the Worker's provider supplies its default; set `model=` -on the `Agent` to any model Baseten serves to override it. +The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one +Worker runs Workflows on Baseten and on other providers at the same time. Any model Baseten serves works, prefixed with +`baseten/`. [openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) ```py +@dataclass +class OpenAICompatibleAgentInput: + prompt: str + # "/", such as "together/openai/gpt-oss-120b". The Worker's + # model provider routes the agent's model calls to that provider. + model: str + + @workflow.defn class OpenAICompatibleAgentWorkflow: @workflow.run - async def run(self, prompt: str) -> str: + async def run(self, input: OpenAICompatibleAgentInput) -> str: # Tools that run inside the Workflow must be async: the Agents SDK runs # sync tools in a thread, which the Workflow sandbox does not allow. @function_tool @@ -166,27 +208,22 @@ class OpenAICompatibleAgentWorkflow: workflow.logger.debug(f"Getting weather for {city}") return f"The weather in {city} is sunny." - # No model is named here, so the Worker's model provider supplies its - # default. Set model= to any model the provider serves to override it. - # Start from empty model settings: an agent that names no model gets - # the Agents SDK's defaults for OpenAI's default model (a reasoning - # effort and `verbosity`), which other providers reject. agent = Agent( name="Assistant", instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", - model_settings=ModelSettings(), + model=input.model, tools=[get_weather], ) - result = await Runner.run(agent, prompt) + result = await Runner.run(agent, input.prompt) return result.final_output ``` -The empty `ModelSettings()` matters. An `Agent` that names no model gets the Agents SDK's default settings for OpenAI's -current default model, which include `verbosity` and a reasoning effort, and the plugin forwards them to the provider. -Fireworks AI, for example, rejects `verbosity` with a 400. An agent that names its model explicitly does not get those -defaults. +Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings +for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0 +forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release, +give such an agent an explicit `model_settings=ModelSettings()`. Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow sandbox does not allow. Tools that do I/O belong in Activities; see @@ -194,25 +231,28 @@ sandbox does not allow. Tools that do I/O belong in Activities; see ## Run the sample -Start the Worker with Baseten selected: +Export your Baseten key and start the Worker: ```bash export BASETEN_API_KEY="your_api_key" -uv run openai_agents/model_providers/run_openai_compatible_worker.py --provider baseten +uv run openai_agents/model_providers/run_openai_compatible_worker.py ``` -Then start the Workflow in a second terminal: +Then start a Workflow on Baseten in a second terminal: ```bash -uv run openai_agents/model_providers/run_openai_compatible_workflow.py +uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model baseten/openai/gpt-oss-120b ``` +Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on +the same Worker. + ## Retries, rate limits, and timeouts The plugin classifies Baseten's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as well. Other -4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying. Each attempt, -its failure, and the final result are in Event History. +4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as does a model +name whose provider has no key on the Worker. Each attempt, its failure, and the final result are in Event History. Baseten enforces requests-per-minute and tokens-per-minute limits that depend on your [account tier](https://docs.baseten.co/inference/model-apis/pricing-and-limits) and answers `429` when either is @@ -259,6 +299,6 @@ Completions API that the sample uses. The sample disables the Agents SDK's hoste ## Samples - [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): - the same agent on Baseten, Fireworks AI, or Together AI, selected with `--provider`. + the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name. - [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, - and tracing with the plugin. + and tracing with the plugin. \ No newline at end of file diff --git a/docs/develop/python/integrations/fireworks-ai.mdx b/docs/develop/python/integrations/fireworks-ai.mdx index 8c74badce3..51d00c32a5 100644 --- a/docs/develop/python/integrations/fireworks-ai.mdx +++ b/docs/develop/python/integrations/fireworks-ai.mdx @@ -17,12 +17,13 @@ endpoints and on-demand deployments through an API compatible with the OpenAI Ch Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an Activity. A call to Fireworks AI gets retries with backoff, a timeout, and a durable record in Event History, and the agent loop around it survives Worker restarts. Because Fireworks AI speaks the OpenAI Chat Completions API, the -integration's model provider hook is all you need: the Worker points an OpenAI client at Fireworks AI's base URL and -names a model, and the Workflow code does not change. +integration's model provider hook is all you need: a model provider that maps a model name such as +`fireworks/accounts/fireworks/models/gpt-oss-120b` to Fireworks AI's endpoint. One Worker can serve Fireworks AI +alongside other providers, and each Workflow picks its provider through the model name it gives its agent. This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the [OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), -which run the same agent on Baseten, Fireworks AI, or Together AI. +which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker. ## Prerequisites @@ -49,7 +50,8 @@ The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `op ## Configure the model provider -The sample describes each provider as a base URL, the environment variable that holds its API key, and a default model: +The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model +with tool calling: [openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) @@ -58,66 +60,97 @@ FIREWORKS = OpenAICompatibleProvider( name="fireworks", base_url="https://api.fireworks.ai/inference/v1", api_key_env="FIREWORKS_API_KEY", - model="accounts/fireworks/models/gpt-oss-120b", + default_model="accounts/fireworks/models/gpt-oss-120b", ) ``` Serverless models on Fireworks AI are named `accounts/fireworks/models/`, and the -[Model Library](https://fireworks.ai/models) shows each model's ID and whether it supports tool calling. The default +[Model Library](https://fireworks.ai/models) shows each model's ID and whether it supports tool calling. The example here is [gpt-oss-120b](https://fireworks.ai/models/fireworks/gpt-oss-120b), an open-weight reasoning model with tool -calling that Fireworks AI serves serverless. An +calling that Fireworks AI serves serverless; a Workflow names it `fireworks/accounts/fireworks/models/gpt-oss-120b`. An [on-demand deployment](https://docs.fireworks.ai/guides/ondemand-deployments) uses the same base URL; pass the -deployment's resource name, `accounts//deployments/`, as the model instead. +deployment's resource name, `accounts//deployments/`, as the model instead, behind the same +`fireworks/` prefix. -A `ModelProvider` turns that description into the Chat Completions model that the Agents SDK calls: +A `ModelProvider` routes a `/` name to the matching endpoint and returns the Chat Completions model +that the Agents SDK calls: [openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) ```py class OpenAICompatibleModelProvider(ModelProvider): - """OpenAI Agents SDK model provider for an OpenAI-compatible endpoint. + """Routes ``/`` names to the matching OpenAI-compatible endpoint. - Client retries are off: the Temporal plugin runs each model call as an - Activity, and the Activity's Retry Policy owns the retries. An agent that - names no model gets the provider's default; an agent can name any other - model the provider serves. + One Worker serves every provider whose API key is set, so Workflows on + different providers run side by side on the same Task Queue. Client + retries are off: the plugin runs each model call as a Temporal Activity, + and the Activity's Retry Policy owns the retries. """ - def __init__(self, provider: OpenAICompatibleProvider) -> None: - self._provider = provider - self._client = AsyncOpenAI( - base_url=provider.base_url, - api_key=os.environ[provider.api_key_env], - max_retries=0, - ) + def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None: + self._providers = {provider.name: provider for provider in providers} + self._clients: dict[str, AsyncOpenAI] = {} + for provider in self._providers.values(): + api_key = os.environ.get(provider.api_key_env) + if api_key: + self._clients[provider.name] = AsyncOpenAI( + base_url=provider.base_url, api_key=api_key, max_retries=0 + ) + + @property + def configured(self) -> list[str]: + """Names of the providers whose API key is set.""" + return sorted(self._clients) def get_model(self, model_name: str | None) -> Model: + provider_name, _, model = (model_name or "").partition("/") + provider = self._providers.get(provider_name) + if provider is None or not model: + # A mistake in the Workflow's input: fail the Activity without retrying. + raise ApplicationError( + f"Model name {model_name!r} must be / with a " + f"provider in {sorted(self._providers)}", + non_retryable=True, + ) + client = self._clients.get(provider_name) + if client is None: + raise ApplicationError( + f"Set {provider.api_key_env} and restart the Worker to use {provider_name}", + non_retryable=True, + ) # Chat Completions is the API all of these providers serve. The Agents # SDK defaults to the Responses API, which not every provider offers. - return OpenAIChatCompletionsModel( - model=model_name or self._provider.model, - openai_client=self._client, - ) + return OpenAIChatCompletionsModel(model=model, openai_client=client) ``` -Two details matter here. Client retries are off (`max_retries=0`), so every attempt is a visible Activity attempt -governed by the Activity's Retry Policy rather than a hidden retry inside the HTTP client. And the provider builds an -`OpenAIChatCompletionsModel` rather than the Agents SDK's default Responses API model, which Fireworks AI also offers, -because Chat Completions is the one API every provider in the sample serves. +Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name +for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off +(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a +hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents +SDK's default Responses API model, which Fireworks AI also offers, because Chat Completions is the one API every +provider in the sample serves. ## Configure the Worker -Pass the provider to the plugin. A custom model provider must set a model Activity timeout; the sample uses 60 seconds. -Fireworks AI answers a scaled-to-zero on-demand deployment with an immediate 503 rather than a slow first response, so -the Retry Policy covers scale-up and the timeout only has to cover your longest generation. The Heartbeat Timeout lets -Temporal notice a Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin -heartbeats on its own. +Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A +custom model provider must set a model Activity timeout; the sample uses 60 seconds. Fireworks AI answers a +scaled-to-zero on-demand deployment with an immediate 503 rather than a slow first response, so the Retry Policy covers +scale-up and the timeout only has to cover your longest generation. The Heartbeat Timeout lets Temporal notice a Worker +that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin heartbeats on its own. [openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) ```py +# One provider for every endpoint whose API key is set. The model name on +# each Workflow's agent selects the endpoint. +model_provider = OpenAICompatibleModelProvider(PROVIDERS.values()) +if not model_provider.configured: + env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values()) + sys.exit(f"No provider API key is set; export at least one of {env_vars}.") +print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True) + client = await Client.connect( "localhost:7233", plugins=[ @@ -130,7 +163,7 @@ client = await Client.connect( # noticed in seconds rather than after the full timeout. heartbeat_timeout=timedelta(seconds=10), ), - model_provider=OpenAICompatibleModelProvider(PROVIDERS[provider_name]), + model_provider=model_provider, ), ], ) @@ -149,16 +182,25 @@ only the Worker talks to Fireworks AI. ## Write the Workflow -The Workflow is ordinary Agents SDK code. It names no model, so the Worker's provider supplies its default; set `model=` -on the `Agent` to any model Fireworks AI serves to override it. +The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one +Worker runs Workflows on Fireworks AI and on other providers at the same time. Any model Fireworks AI serves works, +prefixed with `fireworks/`. [openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) ```py +@dataclass +class OpenAICompatibleAgentInput: + prompt: str + # "/", such as "together/openai/gpt-oss-120b". The Worker's + # model provider routes the agent's model calls to that provider. + model: str + + @workflow.defn class OpenAICompatibleAgentWorkflow: @workflow.run - async def run(self, prompt: str) -> str: + async def run(self, input: OpenAICompatibleAgentInput) -> str: # Tools that run inside the Workflow must be async: the Agents SDK runs # sync tools in a thread, which the Workflow sandbox does not allow. @function_tool @@ -166,27 +208,22 @@ class OpenAICompatibleAgentWorkflow: workflow.logger.debug(f"Getting weather for {city}") return f"The weather in {city} is sunny." - # No model is named here, so the Worker's model provider supplies its - # default. Set model= to any model the provider serves to override it. - # Start from empty model settings: an agent that names no model gets - # the Agents SDK's defaults for OpenAI's default model (a reasoning - # effort and `verbosity`), which other providers reject. agent = Agent( name="Assistant", instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", - model_settings=ModelSettings(), + model=input.model, tools=[get_weather], ) - result = await Runner.run(agent, prompt) + result = await Runner.run(agent, input.prompt) return result.final_output ``` -The empty `ModelSettings()` matters. An `Agent` that names no model gets the Agents SDK's default settings for OpenAI's -current default model, which include `verbosity` and a reasoning effort, and the plugin forwards them to the provider. -Fireworks AI, for example, rejects `verbosity` with a 400. An agent that names its model explicitly does not get those -defaults. +Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings +for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0 +forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release, +give such an agent an explicit `model_settings=ModelSettings()`. Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow sandbox does not allow. Tools that do I/O belong in Activities; see @@ -194,25 +231,29 @@ sandbox does not allow. Tools that do I/O belong in Activities; see ## Run the sample -Start the Worker with Fireworks AI selected: +Export your Fireworks AI key and start the Worker: ```bash export FIREWORKS_API_KEY="your_api_key" -uv run openai_agents/model_providers/run_openai_compatible_worker.py --provider fireworks +uv run openai_agents/model_providers/run_openai_compatible_worker.py ``` -Then start the Workflow in a second terminal: +Then start a Workflow on Fireworks AI in a second terminal: ```bash -uv run openai_agents/model_providers/run_openai_compatible_workflow.py +uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model fireworks/accounts/fireworks/models/gpt-oss-120b ``` +Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on +the same Worker. + ## Retries, rate limits, and timeouts The plugin classifies Fireworks AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as -well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying. Each -attempt, its failure, and the final result are in Event History. +well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as +does a model name whose provider has no key on the Worker. Each attempt, its failure, and the final result are in Event +History. On serverless, Fireworks AI enforces [adaptive token-per-minute limits](https://docs.fireworks.ai/serverless/rate-limits) per model that grow with steady @@ -264,6 +305,6 @@ Completions API that the sample uses. The sample disables the Agents SDK's hoste ## Samples - [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): - the same agent on Baseten, Fireworks AI, or Together AI, selected with `--provider`. + the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name. - [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, - and tracing with the plugin. + and tracing with the plugin. \ No newline at end of file diff --git a/docs/develop/python/integrations/together-ai.mdx b/docs/develop/python/integrations/together-ai.mdx index 24adf8231c..58fb4e8a9e 100644 --- a/docs/develop/python/integrations/together-ai.mdx +++ b/docs/develop/python/integrations/together-ai.mdx @@ -17,12 +17,13 @@ serverless per-token API and on dedicated endpoints through an API compatible wi Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an Activity. A call to Together AI gets retries with backoff, a timeout, and a durable record in Event History, and the agent loop around it survives Worker restarts. Because Together AI speaks the OpenAI Chat Completions API, the -integration's model provider hook is all you need: the Worker points an OpenAI client at Together AI's base URL and -names a model, and the Workflow code does not change. +integration's model provider hook is all you need: a model provider that maps a model name such as +`together/openai/gpt-oss-120b` to Together AI's endpoint. One Worker can serve Together AI alongside other providers, +and each Workflow picks its provider through the model name it gives its agent. This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the [OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), -which run the same agent on Baseten, Fireworks AI, or Together AI. +which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker. ## Prerequisites @@ -50,7 +51,8 @@ The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `op ## Configure the model provider -The sample describes each provider as a base URL, the environment variable that holds its API key, and a default model: +The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model +with tool calling: [openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) @@ -59,67 +61,98 @@ TOGETHER = OpenAICompatibleProvider( name="together", base_url="https://api.together.ai/v1", api_key_env="TOGETHER_API_KEY", - model="openai/gpt-oss-120b", + default_model="openai/gpt-oss-120b", ) ``` Together AI model IDs follow the `/` convention, for example `openai/gpt-oss-120b`; OpenAI's own model names return a 404. The [serverless models](https://docs.together.ai/docs/serverless/models) catalog lists -each model's ID and whether it supports function calling and structured outputs. The sample's default, -`openai/gpt-oss-120b`, is an open-weight reasoning model that supports both. A -[dedicated endpoint](https://docs.together.ai/docs/dedicated-endpoints/overview) is addressed the same way: its endpoint -string, `/`, goes in the `model` field, and requests go to -`https://api-inference.together.ai/v1` instead of the serverless base URL. +each model's ID and whether it supports function calling and structured outputs. The sample's example, +`openai/gpt-oss-120b`, is an open-weight reasoning model that supports both; a Workflow names it +`together/openai/gpt-oss-120b`. A [dedicated endpoint](https://docs.together.ai/docs/dedicated-endpoints/overview) is +addressed the same way: its endpoint string, `/`, goes in the `model` field, and requests +go to `https://api-inference.together.ai/v1` instead of the serverless base URL, so in the sample it is a second entry +in `PROVIDERS`. -A `ModelProvider` turns that description into the Chat Completions model that the Agents SDK calls: +A `ModelProvider` routes a `/` name to the matching endpoint and returns the Chat Completions model +that the Agents SDK calls: [openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) ```py class OpenAICompatibleModelProvider(ModelProvider): - """OpenAI Agents SDK model provider for an OpenAI-compatible endpoint. + """Routes ``/`` names to the matching OpenAI-compatible endpoint. - Client retries are off: the Temporal plugin runs each model call as an - Activity, and the Activity's Retry Policy owns the retries. An agent that - names no model gets the provider's default; an agent can name any other - model the provider serves. + One Worker serves every provider whose API key is set, so Workflows on + different providers run side by side on the same Task Queue. Client + retries are off: the plugin runs each model call as a Temporal Activity, + and the Activity's Retry Policy owns the retries. """ - def __init__(self, provider: OpenAICompatibleProvider) -> None: - self._provider = provider - self._client = AsyncOpenAI( - base_url=provider.base_url, - api_key=os.environ[provider.api_key_env], - max_retries=0, - ) + def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None: + self._providers = {provider.name: provider for provider in providers} + self._clients: dict[str, AsyncOpenAI] = {} + for provider in self._providers.values(): + api_key = os.environ.get(provider.api_key_env) + if api_key: + self._clients[provider.name] = AsyncOpenAI( + base_url=provider.base_url, api_key=api_key, max_retries=0 + ) + + @property + def configured(self) -> list[str]: + """Names of the providers whose API key is set.""" + return sorted(self._clients) def get_model(self, model_name: str | None) -> Model: + provider_name, _, model = (model_name or "").partition("/") + provider = self._providers.get(provider_name) + if provider is None or not model: + # A mistake in the Workflow's input: fail the Activity without retrying. + raise ApplicationError( + f"Model name {model_name!r} must be / with a " + f"provider in {sorted(self._providers)}", + non_retryable=True, + ) + client = self._clients.get(provider_name) + if client is None: + raise ApplicationError( + f"Set {provider.api_key_env} and restart the Worker to use {provider_name}", + non_retryable=True, + ) # Chat Completions is the API all of these providers serve. The Agents # SDK defaults to the Responses API, which not every provider offers. - return OpenAIChatCompletionsModel( - model=model_name or self._provider.model, - openai_client=self._client, - ) + return OpenAIChatCompletionsModel(model=model, openai_client=client) ``` -Two details matter here. Client retries are off (`max_retries=0`), so every attempt is a visible Activity attempt -governed by the Activity's Retry Policy rather than a hidden retry inside the HTTP client. And the provider builds an -`OpenAIChatCompletionsModel` rather than the Agents SDK's default Responses API model, which Together AI's OpenAI -compatibility guide does not cover. +Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name +for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off +(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a +hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents +SDK's default Responses API model, which Together AI's OpenAI compatibility guide does not cover. ## Configure the Worker -Pass the provider to the plugin. A custom model provider must set a model Activity timeout; the sample uses 60 seconds. -Serverless performance on Together AI is best-effort, and a dedicated endpoint that was stopped or scaled down pays a -cold start on its next request, so size the timeout for the slowest call you expect rather than the median. The -Heartbeat Timeout lets Temporal notice a Worker that died mid-call within seconds instead of after the full -Start-To-Close Timeout; the plugin heartbeats on its own. +Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A +custom model provider must set a model Activity timeout; the sample uses 60 seconds. Serverless performance on Together +AI is best-effort, and a dedicated endpoint that was stopped or scaled down pays a cold start on its next request, so +size the timeout for the slowest call you expect rather than the median. The Heartbeat Timeout lets Temporal notice a +Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin heartbeats on its +own. [openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) ```py +# One provider for every endpoint whose API key is set. The model name on +# each Workflow's agent selects the endpoint. +model_provider = OpenAICompatibleModelProvider(PROVIDERS.values()) +if not model_provider.configured: + env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values()) + sys.exit(f"No provider API key is set; export at least one of {env_vars}.") +print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True) + client = await Client.connect( "localhost:7233", plugins=[ @@ -132,7 +165,7 @@ client = await Client.connect( # noticed in seconds rather than after the full timeout. heartbeat_timeout=timedelta(seconds=10), ), - model_provider=OpenAICompatibleModelProvider(PROVIDERS[provider_name]), + model_provider=model_provider, ), ], ) @@ -151,16 +184,25 @@ only the Worker talks to Together AI. ## Write the Workflow -The Workflow is ordinary Agents SDK code. It names no model, so the Worker's provider supplies its default; set `model=` -on the `Agent` to any model Together AI serves to override it. +The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one +Worker runs Workflows on Together AI and on other providers at the same time. Any model Together AI serves works, +prefixed with `together/`. [openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) ```py +@dataclass +class OpenAICompatibleAgentInput: + prompt: str + # "/", such as "together/openai/gpt-oss-120b". The Worker's + # model provider routes the agent's model calls to that provider. + model: str + + @workflow.defn class OpenAICompatibleAgentWorkflow: @workflow.run - async def run(self, prompt: str) -> str: + async def run(self, input: OpenAICompatibleAgentInput) -> str: # Tools that run inside the Workflow must be async: the Agents SDK runs # sync tools in a thread, which the Workflow sandbox does not allow. @function_tool @@ -168,27 +210,22 @@ class OpenAICompatibleAgentWorkflow: workflow.logger.debug(f"Getting weather for {city}") return f"The weather in {city} is sunny." - # No model is named here, so the Worker's model provider supplies its - # default. Set model= to any model the provider serves to override it. - # Start from empty model settings: an agent that names no model gets - # the Agents SDK's defaults for OpenAI's default model (a reasoning - # effort and `verbosity`), which other providers reject. agent = Agent( name="Assistant", instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", - model_settings=ModelSettings(), + model=input.model, tools=[get_weather], ) - result = await Runner.run(agent, prompt) + result = await Runner.run(agent, input.prompt) return result.final_output ``` -The empty `ModelSettings()` matters. An `Agent` that names no model gets the Agents SDK's default settings for OpenAI's -current default model, which include `verbosity` and a reasoning effort, and the plugin forwards them to the provider. -Fireworks AI, for example, rejects `verbosity` with a 400. An agent that names its model explicitly does not get those -defaults. +Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings +for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0 +forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release, +give such an agent an explicit `model_settings=ModelSettings()`. Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow sandbox does not allow. Tools that do I/O belong in Activities; see @@ -196,25 +233,29 @@ sandbox does not allow. Tools that do I/O belong in Activities; see ## Run the sample -Start the Worker with Together AI selected: +Export your Together AI key and start the Worker: ```bash export TOGETHER_API_KEY="your_api_key" -uv run openai_agents/model_providers/run_openai_compatible_worker.py --provider together +uv run openai_agents/model_providers/run_openai_compatible_worker.py ``` -Then start the Workflow in a second terminal: +Then start a Workflow on Together AI in a second terminal: ```bash -uv run openai_agents/model_providers/run_openai_compatible_workflow.py +uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model together/openai/gpt-oss-120b ``` +Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on +the same Worker. + ## Retries, rate limits, and timeouts The plugin classifies Together AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as -well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying. Each -attempt, its failure, and the final result are in Event History. +well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as +does a model name whose provider has no key on the Worker. Each attempt, its failure, and the final result are in Event +History. Under high demand, Together AI's serverless API answers `429 Too Many Requests` or `503 Service Unavailable` and asks clients to retry with exponential backoff, which is what the default Activity Retry Policy does; Together AI's @@ -262,6 +303,6 @@ Completions API that the sample uses. The sample disables the Agents SDK's hoste ## Samples - [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): - the same agent on Baseten, Fireworks AI, or Together AI, selected with `--provider`. + the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name. - [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, - and tracing with the plugin. + and tracing with the plugin. \ No newline at end of file