Add provider-agnostic deferred tool loading and tool discovery support

[superdav42]

Context

Function declarations are currently sent to the model in full on every request. This works well for a small set of functions, but becomes expensive when an application exposes dozens or hundreds of tools, such as a large WordPress Abilities API catalog.

OpenAI and Anthropic now support deferred tool loading. Applications still declare the available tools, but selected definitions are withheld from the model context until the model searches for and loads them.

This issue proposes a provider-agnostic representation for:

• marking function declarations as deferred;
• requesting tool discovery;
• representing tool discovery calls and results when they need to be carried across turns;
• declaring whether a model or provider supports native deferred loading.

Provider implementations would translate this common configuration into their native API formats.

Value

Lower token usage and cost

Tool parameter schemas count as input tokens. Sending a large tool catalog on every request can consume a material portion of the context window before the conversation begins.

Deferred loading lets the model see only the tools needed for the current task.

Better prompt caching

OpenAI and Anthropic append discovered tools later in the context instead of changing the cached prompt prefix. This preserves cache reuse as tools are discovered.

Larger tool catalogs

Applications could register a broad catalog of WordPress abilities without manually choosing a small subset before every prompt.

A portable API

Both OpenAI and Anthropic use the defer_loading concept, although their discovery protocols differ. Representing the intent in PHP AI Client would prevent applications from depending on provider-specific request options.

Providers without native support would work as they always have by including all tools in the system prompt without the benefits of deferred loading and support. Alternatively we could implement the behavior client side with a dedicated search_tools tool but this would probably be best in a follow up issue.

Provider behavior

OpenAI

OpenAI supports tool search in the Responses API on GPT-5.4 and later models.

To use it, a request:

1. adds {"type": "tool_search"} to tools;
2. marks deferred functions or MCP servers with defer_loading: true;
3. optionally groups functions into namespaces.

OpenAI supports two execution modes:

• hosted search, where OpenAI selects and loads tools within the same response;
• client-executed search, where the model returns a tool_search_call and the client supplies a tool_search_output.

Hosted responses can include tool_search_call and tool_search_output items before the eventual function call. These items should not be silently lost if they are needed for conversation replay.

OpenAI doc says the hosted search would be preferred if all the available tools are known and defined so we should focus on the hosted search.

Documentation:

• OpenAI tool search
• OpenAI function calling

Anthropic

Anthropic also supports defer_loading: true. A request includes a server-side tool search implementation, such as its regex or BM25 search tool, and marks other tools as deferred.

Anthropic returns tool_reference blocks for matching tools and expands those references into full definitions. Conversation history must preserve the related server tool-use and search-result blocks.

Anthropic also supports custom client-side search, including embedding-based discovery.

Documentation:

• Anthropic tool search tool
• Anthropic tool definition properties

Provider follow-ups

Provider repositories would then map the common configuration:

• WordPress/ai-provider-for-openai
  • serialize tool_search, namespaces, and defer_loading;
  • parse and preserve discovery output items;
  • limit support to compatible models.
• WordPress/ai-provider-for-anthropic
  • serialize the selected search tool and defer_loading;
  • preserve tool_reference and server tool-use blocks.

Backwards compatibility

• Existing FunctionDeclaration construction must remain valid.
• Deferred loading must default to false.
• Existing prompts must continue sending all function declarations eagerly.
• Providers that do not implement tool search must behave as they do today unless the caller explicitly requires deferred loading.
• DTO toArray() and fromArray() behavior must remain compatible with previously serialized configuration.

Open to feedback, but it seems a pretty simple feature that belongs in the client to provide a common interface that would be useable by any provider. When using a provider that does not have native support for deferred tools it will just work the way it is now.

[i-am-chitti]

Thanks for the detailed writeup. Token cost and caching are real problems once you have more than a few tools. Using the same idea as OpenAI's and Anthropic's defer_loading makes sense, so apps don't have to write different code for each provider.

One thing to flag first: this looks related to #256. That issue asks for a prepareFunctionDeclarationForRequest() hook, so providers can change a FunctionDeclaration before it is sent. Right now that happens in prepareToolsParam(). But that method only handles the older Chat Completions format. Deferred loading on the OpenAI side needs the newer Responses API, which isn't in this repo yet. So the two issues may not touch the same code. But both want to change how tools get sent to a provider. It would help to check they don't end up as two separate systems doing similar things.

A few questions I still have:

• The real provider code lives in ai-provider-for-openai and ai-provider-for-anthropic, not here. Should the changes in this repo ship first, or should all three repos be built together?
• The hardest part looks like saving tool_search_call / tool_reference data across turns. Can the current Message/MessagePart model hold that, or do we need a new part type?
• Is the client-side search_tools idea part of this issue, or a separate future issue? Worth saying clearly either way.

Not trying to block this. Just want to flag the overlap with #256 before any code gets written. Happy to help work out the DTO design once there's agreement on direction.

[superdav42]

Thanks, these are the right questions. I reviewed the current client and the OpenAI and Anthropic provider implementations.

On #256: I see it as adjacent rather than overlapping. #256 and PR #280 add a hook for changing a function declaration before the shared OpenAI-compatible Chat Completions serializer runs. The first-party OpenAI provider already uses the Responses API in ai-provider-for-openai, and Anthropic has its own serializer.

The real provider code lives in ai-provider-for-openai and ai-provider-for-anthropic, not here. Should the changes in this repo ship first, or should all three repos be built together?

The changes should be merged first to the php-ai-client but they do not need to be released before we add them to the providers. The provider plugins need to be able to work with older versions of the php-ai-client so we will need to do a version check or feature detection around the new code.

The hardest part looks like saving tool_search_call / tool_reference data across turns. Can the current Message/MessagePart model hold that, or do we need a new part type?

We need a new part type. The current Message / MessagePart model cannot losslessly represent the discovery state. It supports text, files, function calls, and function responses. The OpenAI provider skips unknown output items, while the Anthropic provider does not preserve these server-tool blocks. This matters because both provider documents require discovery state to remain in its original position when conversation history is replayed.

I do not think we should force the OpenAI and Anthropic protocols into provider-neutral tool_search_call DTOs. Their item shapes and replay rules differ. A provider-scoped, opaque, serializable conversation item may be safer: the originating provider can replay it unchanged, while another provider cannot accidentally interpret it. That mechanism could also support future provider-native response items without adding a new core message type for each API feature.

Is the client-side search_tools idea part of this issue, or a separate future issue? Worth saying clearly either way.

I would keep client-executed search_tools out of this issue. It was an idea we could implement to provide something like a polyfill for providers that do not support native tool search.

I did want to ask if we should silently falling back to eager loading when tool search is explicitly requested and not available or if we should return an error that no providers supports the tool search.

[zahidgh]

To keep this feature reviewable and backward-compatible, I suggest splitting the work into staged pieces:

1. Add a deferred-loading marker to FunctionDeclaration, defaulting to false; omit it from serialized output unless enabled so existing payloads remain unchanged.
2. Add a small provider/model capability contract for native tool discovery, without making unsupported providers advertise support.
3. Add protocol-specific discovery-call/result representations only after the OpenAI and Anthropic wire shapes are agreed, since their replay semantics differ.

The first PR could therefore establish the common intent and serialization contract while preserving eager loading for every existing caller. Providers could then opt in independently, and the conversation DTOs would not need speculative fields before maintainers settle how hosted and client-executed discovery should be represented. Would this decomposition fit the intended direction for #281?

[superdav42]

I don't actually think we need to split this up into 3 pieces.
I have made a PR #282 that implements a solution that not to complex to review. The main item that needs agreement on is the new ProviderData DTO As it is implemented now it would be useful not just for the deferred function loading but for issue WordPress/ai-provider-for-google#32 it could be used in Google PR #37
Parse server-tool parts as:

return new MessagePart(
    new ProviderData('google', $partData)
);