[Feature]: Azure LLM calls cannot be costed — need a cost policy for Azure OpenAI and Azure AI Foundry

[Irash-Perera]

Please select the area the issue is related to

Gateway Operator

Please select the aspect the issue is related to

Aspect/Monitoring (Metrics, observability, health checks, alerts)

Suggested Feature

The llm-cost policy has provider calculators for OpenAI, Anthropic, Gemini, Mistral and AWS Bedrock, but none for Azure. A request to an Azure OpenAI or Azure AI Foundry backend therefore resolves no calculator and records a cost of 0.

The effect is that any deployment fronting Azure models has no per-request cost. There is no spend attribution in analytics, and cost-based rate limiting cannot be used on those routes at all.

Azure also does not fit the assumptions the existing calculators make, so this needs its own policy rather than another branch of llm-cost. Azure prices by deployment rather than by model, and on several endpoints the response reports only the deployment name, which matches no pricing key. It keeps separate pricing catalogs for Azure OpenAI and Foundry models, varies rates by deployment type without reporting which one served the request, and returns token usage in several different shapes depending on the endpoint and model family.

What is needed is a cost policy for Azure OpenAI and Azure AI Foundry that prices requests from the token usage in the response and publishes the result the same way llm-cost does, so analytics and cost-based rate limiting work unchanged.

Related Issues

No response

Steps to Verify

• Design Document — A detailed design document has been created and reviewed, covering architecture, data flow, and edge cases.
• Design Mail — A design summary email has been sent to relevant stakeholders for awareness and feedback.
• Code Review — All code changes have been peer-reviewed and approved according to the project's review standards.
• Testing Complete — Adequate unit, integration, and/or end-to-end tests have been written and are passing.
• Documentation Review — User-facing and/or developer documentation has been updated to reflect the new feature and reviewed.
• Feature Complete — The feature is fully implemented, all checklist items above are done, and it is ready for release.

[Irash-Perera]

Update 12 Aug 2026

Started Working on Azure OpenAI and Azure AI Foundry cost policy

To understand what Azure cost support really needs, we built a working cost calculator for Azure OpenAI and Azure AI Foundry using the gateway's existing LLM cost policy pattern. Staying inside that pattern was deliberate, so we could see how far it stretches. It works, and it took roughly 2,200 lines including tests.

The Go code is bound to the JSON paths

In that pattern, every location a provider reports a number at lives in the Go code as a struct tag. Around nineteen JSON paths end up compiled into the policy, so it is built against one vendor's exact response shape and the only way to change a location is to change Go.

The calculator kept growing conditions

Azure is not one provider with one response shape.

• The model name sits in a different place on each API surface. The legacy surface puts the deployment name in the URL with an explicit api-version, the v1 surface puts it in the body with none:

POST /openai/deployments/{deployment-name}/chat/completions?api-version=2024-06-01
POST /openai/v1/chat/completions            body: { "model": "<deployment-name>", ... }

• Azure reports the deployment name, not always the model. A deployment is an operator-chosen alias. Chat completions usually resolves it to the underlying model, but the v1 Responses API echoes the deployment name back, which Microsoft's own docs show. Resolving it took a three candidate chain, response body then request body then path segment, each checked against an operator supplied mapping first.

• Azure never reports the pricing tier. The same model costs different amounts by deployment type and nothing in the response says which applies, so the operator has to supply a mapping. This part is unavoidable in any design.

• Two products, two pricing namespaces, ordered by path. Azure OpenAI and Foundry models use different key prefixes, so the search order is computed per request.

Each branch is reasonable alone. Stacked together they make the calculator hard to follow and hard to be confident in, and every one is Azure specific.

The cons

• Vendor bound. The paths, the resolution chain and the namespace ordering are all Azure specific and reusable by nothing. A sixth provider means a sixth calculator with its own branches.\
• Onboarding is a code change. A new surface or a renamed field means editing Go, rebuilding and releasing, even though the platform already keeps a per provider YAML template describing that provider.
• Nothing is shared. The policy opens the response body and finds the numbers itself. Any other policy needing the same numbers opens the same body again, so the knowledge is duplicated per policy and the copies drift
• Failures are silent. A wrong or moved path returns zero rather than an error, so a mispriced request is indistinguishable from a cheap one
• The target moves. Azure is migrating its own inference surface towards the OpenAI v1 route and retiring the older Azure AI Inference SDK, so a parser hardcoded to today's shapes ages badly.

[Irash-Perera]

Update 17 Aug 2026

Agreed direction: one policy extracts, the rest read the result

Following a discussion with Pubudu and Arshardh, we agreed to change the shape of the policy stream rather than write another vendor bound calculator. One policy reads the response and extracts the usage once, publishes it for the rest of the chain, and every later policy reads that result instead of opening the body again.

The design doc is available here: https://docs.google.com/document/d/13Vqx6ob5lnT9vEBUp2MZsbaGqt5EntejBZBysTQcs_Y/edit?usp=sharing

In short:

• Locations move out of Go and into the provider template. Where a provider reports each number becomes configuration in the LlmProviderTemplate instead of a struct tag compiled into a policy.

• One shared library does the reading. It resolves the route's template, reads the declared locations, and normalises provider quirks. Streaming and buffered responses go through the same path.

• The result is published in shared metadata. The first policy needing usage pays for the parse and writes the normalised result under a versioned key. Cost, rate limiting, quota alerting and analytics read that instead of reparsing.

• The cost policy only prices. It applies the rates per category and writes the total plus the breakdown.

• Rules stay in Go, addresses move to YAML. Arithmetic, billing quirks and tier thresholds remain code.

For this issue it means Azure support becomes a template plus a little arithmetic rather than a sixth calculator, while the deployment mapping and pricing tier that Azure never reports stay as operator configuration.

[Irash-Perera]

Update 19 Aug 2026

Developed the dedicated SDK library for LLM usage extraction and released it as sdk/ai/v0.2.2.
wso2/api-platform/pull/3239

The library reads token usage from a provider response using the field locations declared in that provider's LlmProviderTemplate, instead of hardcoding each provider's response shape in Go. The template says where a value lives, the library says how to read it, and policies just ask for the result.

What it brings:

• Provider onboarding becomes configuration. A new provider, or an existing one adding a field, is a template change rather than a Go change, a rebuild and a release.
• One parse per request. The result is memoized in SharedContext, so cost, rate limiting and analytics share a single extraction instead of each reopening the body and hunting for the same numbers.
• Streaming and buffered responses read alike. Server sent events are merged into one view, including payloads split across multiple data fields, so a policy writes one code path.
• Values can come from the request URL, not just the body. This is how Bedrock model IDs, ARNs and Gemini model names resolve.
• Provider vocabularies are translated in the template. Service tier names and similar provider specific spellings map through valueMap rather than a switch statement in code.
• Failure is a distinct state. Usage that could not be resolved is reported as such rather than returned as a silent zero, so a genuinely free request can be told apart from a failed extraction.

Covers token counts, cached reads, cache writes, reasoning and audio tokens, the resolved model name and the service tier.

It ships as a package inside sdk/ai, so it releases under the existing sdk/ai tags and needs no new module or workspace entry. Nothing consumes it yet.

[Irash-Perera]

Update 21 Aug 2026

Opened the PR for the core changes that happens in the API Platform in order to handle newly introduced fields.
wso2/api-platform/pull/3275

P.S. Merged

[Irash-Perera]

Update 23 Aug 2026

Opened the PR for the azure-llm-cost policy which is compatible with the old architecture of the policy stream
wso2/gateway-controllers#277

Started working on the new version of LLM Cost Policy with the upgraded policy stream architecture.