diff --git a/docs/develop/python/index.mdx b/docs/develop/python/index.mdx index c1cee808ed..5e3500576d 100644 --- a/docs/develop/python/index.mdx +++ b/docs/develop/python/index.mdx @@ -85,8 +85,10 @@ From there, you can dive deeper into any of the Temporal primitives to start bui ## [Integrations](/develop/python/integrations) +- [Baseten integration](/develop/python/integrations/baseten) - [Braintrust integration](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#python) - [Deep Agents integration](/develop/python/integrations/deepagents) +- [Fireworks AI integration](/develop/python/integrations/fireworks-ai) - [Google ADK integration](/develop/python/integrations/google-adk) - [Google GenAI integration](/develop/python/integrations/google-genai) - [Laminar integration](https://laminar.sh/docs/integrations/temporal) @@ -96,10 +98,12 @@ From there, you can dive deeper into any of the Temporal primitives to start bui - [MCP integration](/develop/python/integrations/mcp) - [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) - [OpenBox integration](https://docs.openbox.ai/getting-started/temporal) +- [OpenRouter integration](/develop/python/integrations/openrouter) - [Parseable integration](https://github.com/parseablehq/temporal-plugin-python/blob/main/INTEGRATION.MD) - [Pydantic AI integration](https://pydantic.dev/docs/ai/capabilities/durable_execution/temporal/) - [Strands Agents integration](/develop/python/integrations/strands-agents) - [Tenuo integration](https://tenuo.ai/temporal) +- [Together AI integration](/develop/python/integrations/together-ai) ## Temporal Python technical resources diff --git a/docs/develop/python/integrations/baseten.mdx b/docs/develop/python/integrations/baseten.mdx new file mode 100644 index 0000000000..d126e99d91 --- /dev/null +++ b/docs/develop/python/integrations/baseten.mdx @@ -0,0 +1,304 @@ +--- +id: baseten +title: Baseten integration +sidebar_label: Baseten +toc_max_heading_level: 2 +tags: + - Baseten + - Python SDK + - Temporal SDKs +description: + Run OpenAI Agents SDK agents on open-weight models served by Baseten, with every model call retried and recorded as a + Temporal Activity. +--- +[Baseten](https://www.baseten.co/) serves open-weight models such as gpt-oss, DeepSeek, GLM, and Kimi, both on its +shared Model APIs and on dedicated deployments, through an API compatible with the OpenAI Chat Completions API. + +Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an +Activity. A call to Baseten gets retries with backoff, a timeout, and a durable record in Event History, and the agent +loop around it survives Worker restarts. Because Baseten speaks the OpenAI Chat Completions API, the integration's model +provider hook is all you need: a model provider that maps a model name such as `baseten/openai/gpt-oss-120b` to +Baseten's endpoint. One Worker can serve Baseten alongside other providers, and each Workflow picks its provider through +the model name it gives its agent. + +This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the +[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), +which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker. + +## Prerequisites + +- This guide assumes you are already familiar with the + [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Baseten's + [OpenAI-compatible API](https://docs.baseten.co/inference/model-apis/overview). +- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking + the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. +- Set up your local development environment by following + [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal + development server running. +- Create a [Baseten API key](https://app.baseten.co/settings/api_keys) and export it as `BASETEN_API_KEY` in the + Worker's environment. The key stays in the Worker process; it is never part of Workflow input or Event History. + +## Install + +```bash +uv add temporalio-openai-agents +``` + +The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. The Baseten CLI and the +`truss` package are for deploying models to Baseten; calling Model APIs needs neither. + +## Configure the model provider + +The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model +with tool calling: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +BASETEN = OpenAICompatibleProvider( + name="baseten", + base_url="https://inference.baseten.co/v1", + api_key_env="BASETEN_API_KEY", + default_model="openai/gpt-oss-120b", +) +``` + + +Model APIs name models the way Hugging Face does, as `/`, so the example here is +`openai/gpt-oss-120b`, named `baseten/openai/gpt-oss-120b` from a Workflow: an open-weight model under the Apache 2.0 +license that Baseten serves on Model APIs, where tool calling is supported for every model. Its +[model library page](https://www.baseten.co/library/gpt-oss-120b/) and `GET https://inference.baseten.co/v1/models` +describe the catalog. A [dedicated deployment](https://docs.baseten.co/inference/calling-your-model) has its own base +URL, `https://model-.api.baseten.co/environments/production/sync/v1`, takes the same API key, and expects the model +name that the deployment serves, so in the sample it is a second entry in `PROVIDERS`. + +A `ModelProvider` routes a `/` name to the matching endpoint and returns the Chat Completions model +that the Agents SDK calls: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +class OpenAICompatibleModelProvider(ModelProvider): + """Routes ``/`` names to the matching OpenAI-compatible endpoint. + + One Worker serves every provider whose API key is set, so Workflows on + different providers run side by side on the same Task Queue. Client + retries are off: the plugin runs each model call as a Temporal Activity, + and the Activity's Retry Policy owns the retries. + """ + + def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None: + self._providers = {provider.name: provider for provider in providers} + self._clients: dict[str, AsyncOpenAI] = {} + for provider in self._providers.values(): + api_key = os.environ.get(provider.api_key_env) + if api_key: + self._clients[provider.name] = AsyncOpenAI( + base_url=provider.base_url, api_key=api_key, max_retries=0 + ) + + @property + def configured(self) -> list[str]: + """Names of the providers whose API key is set.""" + return sorted(self._clients) + + def get_model(self, model_name: str | None) -> Model: + provider_name, _, model = (model_name or "").partition("/") + provider = self._providers.get(provider_name) + if provider is None or not model: + # A mistake in the Workflow's input: fail the Activity without retrying. + raise ApplicationError( + f"Model name {model_name!r} must be / with a " + f"provider in {sorted(self._providers)}", + non_retryable=True, + ) + client = self._clients.get(provider_name) + if client is None: + raise ApplicationError( + f"Set {provider.api_key_env} and restart the Worker to use {provider_name}", + non_retryable=True, + ) + # Chat Completions is the API all of these providers serve. The Agents + # SDK defaults to the Responses API, which not every provider offers. + return OpenAIChatCompletionsModel(model=model, openai_client=client) +``` + + +Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name +for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off +(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a +hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents +SDK's default Responses API model, which Baseten does not serve: Model APIs expose Chat Completions and, in beta, the +Anthropic Messages API. + +## Configure the Worker + +Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A +custom model provider must set a model Activity timeout; the sample uses 60 seconds. Model APIs need no deployment of +your own to warm up; a dedicated deployment that has scaled to zero parks a request until a replica starts, for up to +1200 seconds by default, so cover that cold start in the timeout or keep `min_replica` above zero. The Heartbeat Timeout +lets Temporal notice a Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the +plugin heartbeats on its own. + + +[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) +```py +# One provider for every endpoint whose API key is set. The model name on +# each Workflow's agent selects the endpoint. +model_provider = OpenAICompatibleModelProvider(PROVIDERS.values()) +if not model_provider.configured: + env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values()) + sys.exit(f"No provider API key is set; export at least one of {env_vars}.") +print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True) + +client = await Client.connect( + "localhost:7233", + plugins=[ + OpenAIAgentsPlugin( + model_params=ModelActivityParameters( + # A custom model provider must set a timeout. Raise it for + # slow models or deployments that scale up from zero. + start_to_close_timeout=timedelta(seconds=60), + # The plugin heartbeats, so a Worker that dies mid-call is + # noticed in seconds rather than after the full timeout. + heartbeat_timeout=timedelta(seconds=10), + ), + model_provider=model_provider, + ), + ], +) + +worker = Worker( + client, + task_queue="openai-agents-model-providers-task-queue", + workflows=[OpenAICompatibleAgentWorkflow], +) +await worker.run() +``` + + +The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider: +only the Worker talks to Baseten. + +## Write the Workflow + +The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one +Worker runs Workflows on Baseten and on other providers at the same time. Any model Baseten serves works, prefixed with +`baseten/`. + + +[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) +```py +@dataclass +class OpenAICompatibleAgentInput: + prompt: str + # "/", such as "together/openai/gpt-oss-120b". The Worker's + # model provider routes the agent's model calls to that provider. + model: str + + +@workflow.defn +class OpenAICompatibleAgentWorkflow: + @workflow.run + async def run(self, input: OpenAICompatibleAgentInput) -> str: + # Tools that run inside the Workflow must be async: the Agents SDK runs + # sync tools in a thread, which the Workflow sandbox does not allow. + @function_tool + async def get_weather(city: str) -> str: + workflow.logger.debug(f"Getting weather for {city}") + return f"The weather in {city} is sunny." + + agent = Agent( + name="Assistant", + instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", + model=input.model, + tools=[get_weather], + ) + + result = await Runner.run(agent, input.prompt) + return result.final_output +``` + + +Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings +for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0 +forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release, +give such an agent an explicit `model_settings=ModelSettings()`. + +Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow +sandbox does not allow. Tools that do I/O belong in Activities; see +[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools). + +## Run the sample + +Export your Baseten key and start the Worker: + +```bash +export BASETEN_API_KEY="your_api_key" +uv run openai_agents/model_providers/run_openai_compatible_worker.py +``` + +Then start a Workflow on Baseten in a second terminal: + +```bash +uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model baseten/openai/gpt-oss-120b +``` + +Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on +the same Worker. + +## Retries, rate limits, and timeouts + +The plugin classifies Baseten's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry Policy, +and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as well. Other +4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as does a model +name whose provider has no key on the Worker. Each attempt, its failure, and the final result are in Event History. + +Baseten enforces requests-per-minute and tokens-per-minute limits that depend on your +[account tier](https://docs.baseten.co/inference/model-apis/pricing-and-limits) and answers `429` when either is +exceeded; the `x-ratelimit-remaining-requests` and `x-ratelimit-remaining-tokens` headers show which one hit zero. A +`529` means Baseten had no serving capacity for the request even within your limits. Both retry under the Activity's +Retry Policy. Baseten's [error reference](https://docs.baseten.co/inference/errors) asks for exponential backoff and +mentions a `Retry-After` header only as a possibility on `529`, so the Retry Policy's backoff, not the provider, usually +sets the pace. Sustained `429` responses mean the limit itself needs to go up, through Baseten's +[rate limit increase form](https://www.baseten.co/talk-to-us/increase-rate-limits/). + +## What works through Baseten + +Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the +Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter, +and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat +Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the +[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual. + +## Baseten notes + +- A dedicated deployment is called at `https://model-.api.baseten.co/environments/production/sync/v1` with the same + `Authorization: Bearer` key; the legacy `Api-Key` scheme is still accepted. The model name must match what the + deployment serves. A deployment whose `min_replica` is zero releases its replicas when idle, and the first request + after that waits for a replica to start; see Baseten's + [request lifecycle](https://docs.baseten.co/deployment/autoscaling/request-lifecycle) page. +- `GET https://inference.baseten.co/v1/models` lists the Model APIs catalog with context windows, pricing, and supported + features, as does `baseten model-api list` from the Baseten CLI. Baseten notes that the catalog can report different + limits from the ones the endpoint enforces. +- gpt-oss-120b reasons by default at `medium` effort. The reasoning arrives in `reasoning_content` on the assistant + message, and its tokens count as output tokens. Set `ModelSettings(reasoning=Reasoning(effort="low"))` on the agent to + change the effort; Baseten's [reasoning](https://docs.baseten.co/inference/model-apis/reasoning) page lists the + accepted values per model. +- Tool calling, structured outputs, and JSON mode are available on every Model API through `tools` and + `response_format`; see Baseten's [function calling](https://docs.baseten.co/inference/function-calling) and + [structured outputs](https://docs.baseten.co/inference/structured-outputs) guides. Only one completion per request is + supported, so leave `n` at its default. +- Baseten reuses cached input tokens across requests that share an `x-session-affinity` header. An agent run resends the + growing conversation on every turn, so sending the Workflow ID as the session, through + `ModelSettings(extra_headers=...)`, lets later turns reuse the cache. Baseten recommends at most about 60 requests per + minute per session. +- Two Baseten features sit outside this pattern: async inference, which delivers results to a webhook and works only on + dedicated deployments, and server-side web search tools on Model APIs, which are in preview. + +## Samples + +- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): + the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name. +- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, + and tracing with the plugin. \ No newline at end of file diff --git a/docs/develop/python/integrations/fireworks-ai.mdx b/docs/develop/python/integrations/fireworks-ai.mdx new file mode 100644 index 0000000000..51d00c32a5 --- /dev/null +++ b/docs/develop/python/integrations/fireworks-ai.mdx @@ -0,0 +1,310 @@ +--- +id: fireworks-ai +title: Fireworks AI integration +sidebar_label: Fireworks AI +toc_max_heading_level: 2 +tags: + - Fireworks AI + - Python SDK + - Temporal SDKs +description: + Run OpenAI Agents SDK agents on open-weight models served by Fireworks AI, with every model call retried and recorded + as a Temporal Activity. +--- +[Fireworks AI](https://fireworks.ai/) serves open-weight models such as gpt-oss, Kimi K2, Qwen, and GLM on serverless +endpoints and on-demand deployments through an API compatible with the OpenAI Chat Completions API. + +Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an +Activity. A call to Fireworks AI gets retries with backoff, a timeout, and a durable record in Event History, and the +agent loop around it survives Worker restarts. Because Fireworks AI speaks the OpenAI Chat Completions API, the +integration's model provider hook is all you need: a model provider that maps a model name such as +`fireworks/accounts/fireworks/models/gpt-oss-120b` to Fireworks AI's endpoint. One Worker can serve Fireworks AI +alongside other providers, and each Workflow picks its provider through the model name it gives its agent. + +This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the +[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), +which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker. + +## Prerequisites + +- This guide assumes you are already familiar with the + [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Fireworks AI's + [OpenAI-compatible API](https://docs.fireworks.ai/tools-sdks/openai-compatibility). +- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking + the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. +- Set up your local development environment by following + [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal + development server running. +- Create a [Fireworks AI API key](https://app.fireworks.ai/settings/users/api-keys) and export it as `FIREWORKS_API_KEY` + in the Worker's environment. The key stays in the Worker process; it is never part of Workflow input or Event History. + +## Install + +```bash +uv add temporalio-openai-agents +``` + +The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. Fireworks AI's own +[`fireworks-ai` package](https://docs.fireworks.ai/tools-sdks/python-sdk) is not needed: Fireworks AI documents the +`openai` client as a supported way to call its API. + +## Configure the model provider + +The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model +with tool calling: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +FIREWORKS = OpenAICompatibleProvider( + name="fireworks", + base_url="https://api.fireworks.ai/inference/v1", + api_key_env="FIREWORKS_API_KEY", + default_model="accounts/fireworks/models/gpt-oss-120b", +) +``` + + +Serverless models on Fireworks AI are named `accounts/fireworks/models/`, and the +[Model Library](https://fireworks.ai/models) shows each model's ID and whether it supports tool calling. The example +here is [gpt-oss-120b](https://fireworks.ai/models/fireworks/gpt-oss-120b), an open-weight reasoning model with tool +calling that Fireworks AI serves serverless; a Workflow names it `fireworks/accounts/fireworks/models/gpt-oss-120b`. An +[on-demand deployment](https://docs.fireworks.ai/guides/ondemand-deployments) uses the same base URL; pass the +deployment's resource name, `accounts//deployments/`, as the model instead, behind the same +`fireworks/` prefix. + +A `ModelProvider` routes a `/` name to the matching endpoint and returns the Chat Completions model +that the Agents SDK calls: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +class OpenAICompatibleModelProvider(ModelProvider): + """Routes ``/`` names to the matching OpenAI-compatible endpoint. + + One Worker serves every provider whose API key is set, so Workflows on + different providers run side by side on the same Task Queue. Client + retries are off: the plugin runs each model call as a Temporal Activity, + and the Activity's Retry Policy owns the retries. + """ + + def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None: + self._providers = {provider.name: provider for provider in providers} + self._clients: dict[str, AsyncOpenAI] = {} + for provider in self._providers.values(): + api_key = os.environ.get(provider.api_key_env) + if api_key: + self._clients[provider.name] = AsyncOpenAI( + base_url=provider.base_url, api_key=api_key, max_retries=0 + ) + + @property + def configured(self) -> list[str]: + """Names of the providers whose API key is set.""" + return sorted(self._clients) + + def get_model(self, model_name: str | None) -> Model: + provider_name, _, model = (model_name or "").partition("/") + provider = self._providers.get(provider_name) + if provider is None or not model: + # A mistake in the Workflow's input: fail the Activity without retrying. + raise ApplicationError( + f"Model name {model_name!r} must be / with a " + f"provider in {sorted(self._providers)}", + non_retryable=True, + ) + client = self._clients.get(provider_name) + if client is None: + raise ApplicationError( + f"Set {provider.api_key_env} and restart the Worker to use {provider_name}", + non_retryable=True, + ) + # Chat Completions is the API all of these providers serve. The Agents + # SDK defaults to the Responses API, which not every provider offers. + return OpenAIChatCompletionsModel(model=model, openai_client=client) +``` + + +Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name +for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off +(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a +hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents +SDK's default Responses API model, which Fireworks AI also offers, because Chat Completions is the one API every +provider in the sample serves. + +## Configure the Worker + +Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A +custom model provider must set a model Activity timeout; the sample uses 60 seconds. Fireworks AI answers a +scaled-to-zero on-demand deployment with an immediate 503 rather than a slow first response, so the Retry Policy covers +scale-up and the timeout only has to cover your longest generation. The Heartbeat Timeout lets Temporal notice a Worker +that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin heartbeats on its own. + + +[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) +```py +# One provider for every endpoint whose API key is set. The model name on +# each Workflow's agent selects the endpoint. +model_provider = OpenAICompatibleModelProvider(PROVIDERS.values()) +if not model_provider.configured: + env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values()) + sys.exit(f"No provider API key is set; export at least one of {env_vars}.") +print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True) + +client = await Client.connect( + "localhost:7233", + plugins=[ + OpenAIAgentsPlugin( + model_params=ModelActivityParameters( + # A custom model provider must set a timeout. Raise it for + # slow models or deployments that scale up from zero. + start_to_close_timeout=timedelta(seconds=60), + # The plugin heartbeats, so a Worker that dies mid-call is + # noticed in seconds rather than after the full timeout. + heartbeat_timeout=timedelta(seconds=10), + ), + model_provider=model_provider, + ), + ], +) + +worker = Worker( + client, + task_queue="openai-agents-model-providers-task-queue", + workflows=[OpenAICompatibleAgentWorkflow], +) +await worker.run() +``` + + +The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider: +only the Worker talks to Fireworks AI. + +## Write the Workflow + +The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one +Worker runs Workflows on Fireworks AI and on other providers at the same time. Any model Fireworks AI serves works, +prefixed with `fireworks/`. + + +[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) +```py +@dataclass +class OpenAICompatibleAgentInput: + prompt: str + # "/", such as "together/openai/gpt-oss-120b". The Worker's + # model provider routes the agent's model calls to that provider. + model: str + + +@workflow.defn +class OpenAICompatibleAgentWorkflow: + @workflow.run + async def run(self, input: OpenAICompatibleAgentInput) -> str: + # Tools that run inside the Workflow must be async: the Agents SDK runs + # sync tools in a thread, which the Workflow sandbox does not allow. + @function_tool + async def get_weather(city: str) -> str: + workflow.logger.debug(f"Getting weather for {city}") + return f"The weather in {city} is sunny." + + agent = Agent( + name="Assistant", + instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", + model=input.model, + tools=[get_weather], + ) + + result = await Runner.run(agent, input.prompt) + return result.final_output +``` + + +Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings +for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0 +forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release, +give such an agent an explicit `model_settings=ModelSettings()`. + +Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow +sandbox does not allow. Tools that do I/O belong in Activities; see +[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools). + +## Run the sample + +Export your Fireworks AI key and start the Worker: + +```bash +export FIREWORKS_API_KEY="your_api_key" +uv run openai_agents/model_providers/run_openai_compatible_worker.py +``` + +Then start a Workflow on Fireworks AI in a second terminal: + +```bash +uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model fireworks/accounts/fireworks/models/gpt-oss-120b +``` + +Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on +the same Worker. + +## Retries, rate limits, and timeouts + +The plugin classifies Fireworks AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry +Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as +well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as +does a model name whose provider has no key on the Worker. Each attempt, its failure, and the final result are in Event +History. + +On serverless, Fireworks AI enforces +[adaptive token-per-minute limits](https://docs.fireworks.ai/serverless/rate-limits) per model that grow with steady +traffic and shrink when it stops. A burst beyond them returns 429, and Fireworks AI asks clients to honor `Retry-After` +when it is present. Under high load Fireworks AI can also shed traffic with `503 Service Overloaded`, which the plugin +retries like any 5xx; the `priority` service tier lowers that chance. +[On-demand deployments](https://docs.fireworks.ai/guides/inference-error-codes) have no token-per-minute limit, so a 429 +there means the deployment is saturated: add replicas or lower concurrency rather than wait. Both paths count against +the account-wide requests-per-minute cap described on the +[account quotas page](https://docs.fireworks.ai/guides/quotas_usage/account-quotas), which also sets the spending tiers +that raise the serverless ceilings. + +## What works through Fireworks AI + +Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the +Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter, +and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat +Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the +[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual. + +## Fireworks AI notes + +- Create an on-demand deployment with `firectl deployment create accounts/fireworks/models/gpt-oss-120b ...` and pass + the deployment name it returns as the model. By default a deployment scales to zero after an hour without traffic and + answers the next request with an immediate 503 while it scales back up, which the Activity's Retry Policy absorbs; a + deployment with zero minimum replicas is deleted after seven days without traffic. See the + [deployments quickstart](https://docs.fireworks.ai/getting-started/ondemand-quickstart). +- Fireworks AI updates and retires serverless models over time and gives + [at least two weeks' notice](https://docs.fireworks.ai/faq-new/deployment-infrastructure/do-you-provide-notice-before-removing-model-availability) + before removing one. For a version that must not change under a long-running Workflow, pin it on an on-demand + deployment. +- An agent's `output_type` is sent as a JSON schema in `response_format`, which Fireworks AI + [supports](https://docs.fireworks.ai/structured-responses/structured-response-formatting). Fireworks AI turns off + reasoning output while it enforces a schema, and turns JSON mode on by itself for tool calls. +- Reasoning models return their thinking in a `reasoning_content` field, and for models that interleave thinking with + tool calls Fireworks AI + [expects each earlier assistant turn's `reasoning_content` back](https://docs.fireworks.ai/guides/reasoning) in the + next request. The Agents SDK replays it only for DeepSeek models by default; pass `should_replay_reasoning_content` to + `OpenAIChatCompletionsModel` to replay it for the model you use. +- [Prompt caching](https://docs.fireworks.ai/guides/prompt-caching) is on by default, so a retried attempt that re-sends + the same prompt pays the discounted cached rate for the prefix. On serverless the cache lives in one replica, so hits + need a session identifier, which you can send as an `x-session-affinity` header through + `ModelSettings(extra_headers=...)`; dedicated deployments share their cache across requests. +- Fireworks AI also serves the [Responses API](https://docs.fireworks.ai/guides/response-api) at the same base URL, with + stored conversation state and server-executed tools of its own: remote MCP servers, web search, and a Python tool. + Build an `OpenAIResponsesModel` instead of `OpenAIChatCompletionsModel` to use it. The sample does not, and those + tools were not exercised with the Agents SDK's hosted tool types. + +## Samples + +- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): + the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name. +- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, + and tracing with the plugin. \ No newline at end of file diff --git a/docs/develop/python/integrations/together-ai.mdx b/docs/develop/python/integrations/together-ai.mdx new file mode 100644 index 0000000000..58fb4e8a9e --- /dev/null +++ b/docs/develop/python/integrations/together-ai.mdx @@ -0,0 +1,308 @@ +--- +id: together-ai +title: Together AI integration +sidebar_label: Together AI +toc_max_heading_level: 2 +tags: + - Together AI + - Python SDK + - Temporal SDKs +description: + Run OpenAI Agents SDK agents on open-weight models served by Together AI, with every model call retried and recorded + as a Temporal Activity. +--- +[Together AI](https://www.together.ai/) serves open-weight models such as gpt-oss, DeepSeek, Qwen, and GLM on a +serverless per-token API and on dedicated endpoints through an API compatible with the OpenAI Chat Completions API. + +Temporal's [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) runs each model call as an +Activity. A call to Together AI gets retries with backoff, a timeout, and a durable record in Event History, and the +agent loop around it survives Worker restarts. Because Together AI speaks the OpenAI Chat Completions API, the +integration's model provider hook is all you need: a model provider that maps a model name such as +`together/openai/gpt-oss-120b` to Together AI's endpoint. One Worker can serve Together AI alongside other providers, +and each Workflow picks its provider through the model name it gives its agent. + +This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the +[OpenAI Agents SDK model provider samples](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers), +which run the same agent on Baseten, Fireworks AI, or Together AI from one Worker. + +## Prerequisites + +- This guide assumes you are already familiar with the + [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents) and with Together AI's + [OpenAI-compatible API](https://docs.together.ai/docs/inference/openai-compatibility). +- If you are new to Temporal, we recommend reading [Understanding Temporal](/evaluate/understanding-temporal) or taking + the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. +- Set up your local development environment by following + [Set up your local development environment](/develop/python/set-up-your-local-python) and leave the Temporal + development server running. +- Create a [Together AI API key](https://api.together.ai/settings/projects/~current/api-keys) and export it as + `TOGETHER_API_KEY` in the Worker's environment. The key stays in the Worker process; it is never part of Workflow + input or Event History. + +## Install + +```bash +uv add temporalio-openai-agents +``` + +The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the `openai` client. Together AI's own +[`together`](https://pypi.org/project/together/) package is not needed; the sample talks to Together AI through the +`openai` client. + +## Configure the model provider + +The sample describes each provider as a base URL, the environment variable that holds its API key, and an example model +with tool calling: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +TOGETHER = OpenAICompatibleProvider( + name="together", + base_url="https://api.together.ai/v1", + api_key_env="TOGETHER_API_KEY", + default_model="openai/gpt-oss-120b", +) +``` + + +Together AI model IDs follow the `/` convention, for example `openai/gpt-oss-120b`; OpenAI's +own model names return a 404. The [serverless models](https://docs.together.ai/docs/serverless/models) catalog lists +each model's ID and whether it supports function calling and structured outputs. The sample's example, +`openai/gpt-oss-120b`, is an open-weight reasoning model that supports both; a Workflow names it +`together/openai/gpt-oss-120b`. A [dedicated endpoint](https://docs.together.ai/docs/dedicated-endpoints/overview) is +addressed the same way: its endpoint string, `/`, goes in the `model` field, and requests +go to `https://api-inference.together.ai/v1` instead of the serverless base URL, so in the sample it is a second entry +in `PROVIDERS`. + +A `ModelProvider` routes a `/` name to the matching endpoint and returns the Chat Completions model +that the Agents SDK calls: + + +[openai_agents/model_providers/openai_compatible_providers.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/openai_compatible_providers.py) +```py +class OpenAICompatibleModelProvider(ModelProvider): + """Routes ``/`` names to the matching OpenAI-compatible endpoint. + + One Worker serves every provider whose API key is set, so Workflows on + different providers run side by side on the same Task Queue. Client + retries are off: the plugin runs each model call as a Temporal Activity, + and the Activity's Retry Policy owns the retries. + """ + + def __init__(self, providers: Iterable[OpenAICompatibleProvider]) -> None: + self._providers = {provider.name: provider for provider in providers} + self._clients: dict[str, AsyncOpenAI] = {} + for provider in self._providers.values(): + api_key = os.environ.get(provider.api_key_env) + if api_key: + self._clients[provider.name] = AsyncOpenAI( + base_url=provider.base_url, api_key=api_key, max_retries=0 + ) + + @property + def configured(self) -> list[str]: + """Names of the providers whose API key is set.""" + return sorted(self._clients) + + def get_model(self, model_name: str | None) -> Model: + provider_name, _, model = (model_name or "").partition("/") + provider = self._providers.get(provider_name) + if provider is None or not model: + # A mistake in the Workflow's input: fail the Activity without retrying. + raise ApplicationError( + f"Model name {model_name!r} must be / with a " + f"provider in {sorted(self._providers)}", + non_retryable=True, + ) + client = self._clients.get(provider_name) + if client is None: + raise ApplicationError( + f"Set {provider.api_key_env} and restart the Worker to use {provider_name}", + non_retryable=True, + ) + # Chat Completions is the API all of these providers serve. The Agents + # SDK defaults to the Responses API, which not every provider offers. + return OpenAIChatCompletionsModel(model=model, openai_client=client) +``` + + +Three details matter here. The provider builds a client only for the providers whose API key is set, and a model name +for a provider without one, or with an unknown prefix, fails the Activity without retrying. Client retries are off +(`max_retries=0`), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a +hidden retry inside the HTTP client. And the provider builds an `OpenAIChatCompletionsModel` rather than the Agents +SDK's default Responses API model, which Together AI's OpenAI compatibility guide does not cover. + +## Configure the Worker + +Pass the provider to the plugin. The Worker serves every provider whose key is set and prints which ones at startup. A +custom model provider must set a model Activity timeout; the sample uses 60 seconds. Serverless performance on Together +AI is best-effort, and a dedicated endpoint that was stopped or scaled down pays a cold start on its next request, so +size the timeout for the slowest call you expect rather than the median. The Heartbeat Timeout lets Temporal notice a +Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin heartbeats on its +own. + + +[openai_agents/model_providers/run_openai_compatible_worker.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/run_openai_compatible_worker.py) +```py +# One provider for every endpoint whose API key is set. The model name on +# each Workflow's agent selects the endpoint. +model_provider = OpenAICompatibleModelProvider(PROVIDERS.values()) +if not model_provider.configured: + env_vars = ", ".join(provider.api_key_env for provider in PROVIDERS.values()) + sys.exit(f"No provider API key is set; export at least one of {env_vars}.") +print(f"Serving providers: {', '.join(model_provider.configured)}", flush=True) + +client = await Client.connect( + "localhost:7233", + plugins=[ + OpenAIAgentsPlugin( + model_params=ModelActivityParameters( + # A custom model provider must set a timeout. Raise it for + # slow models or deployments that scale up from zero. + start_to_close_timeout=timedelta(seconds=60), + # The plugin heartbeats, so a Worker that dies mid-call is + # noticed in seconds rather than after the full timeout. + heartbeat_timeout=timedelta(seconds=10), + ), + model_provider=model_provider, + ), + ], +) + +worker = Worker( + client, + task_queue="openai-agents-model-providers-task-queue", + workflows=[OpenAICompatibleAgentWorkflow], +) +await worker.run() +``` + + +The Client that starts the Workflow also needs `OpenAIAgentsPlugin()`, for its Data Converter, but no model provider: +only the Worker talks to Together AI. + +## Write the Workflow + +The Workflow is ordinary Agents SDK code. The model name arrives in the Workflow input and goes on the `Agent`, so one +Worker runs Workflows on Together AI and on other providers at the same time. Any model Together AI serves works, +prefixed with `together/`. + + +[openai_agents/model_providers/workflows/openai_compatible_workflow.py](https://github.com/temporalio/samples-python/blob/main/openai_agents/model_providers/workflows/openai_compatible_workflow.py) +```py +@dataclass +class OpenAICompatibleAgentInput: + prompt: str + # "/", such as "together/openai/gpt-oss-120b". The Worker's + # model provider routes the agent's model calls to that provider. + model: str + + +@workflow.defn +class OpenAICompatibleAgentWorkflow: + @workflow.run + async def run(self, input: OpenAICompatibleAgentInput) -> str: + # Tools that run inside the Workflow must be async: the Agents SDK runs + # sync tools in a thread, which the Workflow sandbox does not allow. + @function_tool + async def get_weather(city: str) -> str: + workflow.logger.debug(f"Getting weather for {city}") + return f"The weather in {city} is sunny." + + agent = Agent( + name="Assistant", + instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.", + model=input.model, + tools=[get_weather], + ) + + result = await Runner.run(agent, input.prompt) + return result.final_output +``` + + +Name the model on the agent, as the sample does. An agent that names no model inherits the Agents SDK's default settings +for OpenAI's current default model, which include `verbosity` and a reasoning effort; `temporalio-openai-agents` 1.1.0 +forwards them to the provider unchanged, and Fireworks AI, for example, rejects `verbosity` with a 400. On that release, +give such an agent an explicit `model_settings=ModelSettings()`. + +Tools that run inside the Workflow must be `async`: the Agents SDK runs sync tools in a thread, which the Workflow +sandbox does not allow. Tools that do I/O belong in Activities; see +[Activity-backed tools](/develop/python/integrations/openai-agents#activity-backed-tools). + +## Run the sample + +Export your Together AI key and start the Worker: + +```bash +export TOGETHER_API_KEY="your_api_key" +uv run openai_agents/model_providers/run_openai_compatible_worker.py +``` + +Then start a Workflow on Together AI in a second terminal: + +```bash +uv run openai_agents/model_providers/run_openai_compatible_workflow.py --model together/openai/gpt-oss-120b +``` + +Export keys for other providers and pass several `--model` values to run one Workflow per model at the same time on +the same Worker. + +## Retries, rate limits, and timeouts + +The plugin classifies Together AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry +Policy, and a `Retry-After` header becomes the next retry delay. Connection errors and client-side timeouts retry as +well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying, as +does a model name whose provider has no key on the Worker. Each attempt, its failure, and the final result are in Event +History. + +Under high demand, Together AI's serverless API answers `429 Too Many Requests` or `503 Service Unavailable` and asks +clients to retry with exponential backoff, which is what the default Activity Retry Policy does; Together AI's +documentation does not describe a `Retry-After` header. Its [error codes](https://docs.together.ai/docs/error-codes) +page lists 500, 503, 504, 524, and 529 as retryable and the rest as request, credential, or billing problems to fix +first, including 402 when the account reaches its monthly spending limit and 403 when the prompt plus `max_tokens` +exceeds the model's context length. Serverless performance is best-effort; see +[rate limits](https://docs.together.ai/docs/serverless/rate-limits) for the current policy and +[provisioned throughput](https://docs.together.ai/docs/inference/provisioned-throughput) for reserved capacity with a +throughput and reliability commitment. + +## What works through Together AI + +Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the +Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter, +and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat +Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the +[OpenTelemetry tracing](/develop/python/integrations/openai-agents#opentelemetry) path works as usual. + +## Together AI notes + +- Dedicated endpoints are served at `https://api-inference.together.ai/v1`, not the serverless base URL. Deploy a model, + wait for the deployment to reach `DEPLOYMENT_STATE_READY` (first-time provisioning takes minutes), then use the + endpoint string as the model name. A running deployment keeps at least one replica; stopping it ends the cost and + makes the next request pay a cold start. See [autoscaling](https://docs.together.ai/docs/dedicated-endpoints/scaling) + for the replica bounds. +- Regional inference: `https://us-api.together.ai/v1` keeps serverless processing in the United States and + `https://ca-api.together.ai/v1` in Canada, each with its own model list. See + [geo-specific inference](https://docs.together.ai/docs/serverless/geo-specific-inference). +- `openai/gpt-oss-120b` is a reasoning model. Together AI accepts `reasoning_effort` (`low`, `medium`, or `high`; the + default is `medium`), which the Agents SDK sends when an agent sets + `ModelSettings(reasoning=Reasoning(effort="low"))`. + Together AI returns the model's chain of thought in a `reasoning` field, which the Agents SDK's Chat Completions model + does not read, so prior reasoning is not passed back between tool calls. See + [reasoning](https://docs.together.ai/docs/inference/chat/reasoning). +- Structured outputs: an agent's `output_type` becomes a `response_format` JSON schema, which + [Together AI supports](https://docs.together.ai/docs/inference/chat/structured-outputs) on models marked for it in the + catalog. Together AI recommends also describing the schema in the prompt, and a response that stops with + `finish_reason` of `length` leaves the JSON truncated, so give structured answers enough `max_tokens`. +- Model discovery: the `openai` client's `models.list()` works against Together AI, and the serverless catalog marks + function calling and structured outputs per model. Together AI retires serverless models over time (for example, + `openai/gpt-oss-20b` in September 2026), so check the [changelog](https://docs.together.ai/docs/changelog) before + pinning a model in production. + +## Samples + +- [openai_agents/model_providers](https://github.com/temporalio/samples-python/tree/main/openai_agents/model_providers): + the same agent on Baseten, Fireworks AI, or Together AI from one Worker, selected per Workflow by the model name. +- [OpenAI Agents SDK integration](/develop/python/integrations/openai-agents): tools, MCP servers, conversation history, + and tracing with the plugin. \ No newline at end of file diff --git a/sidebars.js b/sidebars.js index 50ff0e2984..adc29054dd 100644 --- a/sidebars.js +++ b/sidebars.js @@ -744,7 +744,9 @@ const developPythonCategory = { id: 'develop/python/integrations/index', }, items: [ + 'develop/python/integrations/baseten', 'develop/python/integrations/deepagents', + 'develop/python/integrations/fireworks-ai', 'develop/python/integrations/google-adk', 'develop/python/integrations/google-genai', 'develop/python/integrations/langgraph', @@ -753,6 +755,7 @@ const developPythonCategory = { 'develop/python/integrations/openai-agents', 'develop/python/integrations/openrouter', 'develop/python/integrations/strands-agents', + 'develop/python/integrations/together-ai', ], }, ], diff --git a/src/components/IntegrationsGrid/integrations-data.json b/src/components/IntegrationsGrid/integrations-data.json index 1678261d77..1f36a3a59e 100644 --- a/src/components/IntegrationsGrid/integrations-data.json +++ b/src/components/IntegrationsGrid/integrations-data.json @@ -395,5 +395,32 @@ ], "sdk": "TypeScript", "href": "https://github.com/temporalio/samples-typescript/tree/main/openrouter" + }, + { + "name": "Baseten", + "description": "Run OpenAI Agents SDK agents on open-weight models served by Baseten, with every model call as a durable Temporal Activity.", + "tags": [ + "Model provider" + ], + "sdk": "Python", + "href": "/develop/python/integrations/baseten" + }, + { + "name": "Fireworks AI", + "description": "Run OpenAI Agents SDK agents on open-weight models served by Fireworks AI, with every model call as a durable Temporal Activity.", + "tags": [ + "Model provider" + ], + "sdk": "Python", + "href": "/develop/python/integrations/fireworks-ai" + }, + { + "name": "Together AI", + "description": "Run OpenAI Agents SDK agents on open-weight models served by Together AI, with every model call as a durable Temporal Activity.", + "tags": [ + "Model provider" + ], + "sdk": "Python", + "href": "/develop/python/integrations/together-ai" } ] diff --git a/vale/styles/Temporal/Headings.yml b/vale/styles/Temporal/Headings.yml index 14bbb61775..9231894011 100644 --- a/vale/styles/Temporal/Headings.yml +++ b/vale/styles/Temporal/Headings.yml @@ -371,6 +371,9 @@ exceptions: - NOT_FOUND # AI vendor and model-platform proper nouns - AI + - Baseten + - Fireworks + - Together - Google - Gemini - Vertex