[Improvement]: Retry mechanism for oauth2 upstream-auth policy

[Thenujan-Nagaratnam]

Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/API (API backends, definitions, contracts, interfaces, OpenAPI)

Suggested Improvement

We need a retry mechanism for the oauth2 upstream-auth policy so that a failed authenticated call to the upstream (e.g. a 401 due to an expired/invalid token) can be retried.

This is applicable to the model failover policy as well. The same problem we have where we need to retry with the fallback model/endpoint when the request fails.

Related Issues

No response

[Thenujan-Nagaratnam]

Retry Mechanism: Internal Discussion Summary

This document summarizes our architectural discussion regarding the implementation of a retry mechanism to support OAuth2 Token Refresh and LLM Model Failover.

1. Primary Use Cases & Requirements

The retry mechanism must handle two distinct scenarios with varying levels of complexity:

• OAuth2 Policy (Token Refresh):
  • Requires catching authentication failures (e.g., 401s), fetching a new token, modifying the request headers, and re-invoking the endpoint.
• Model Failover (Cross-Provider Routing):
  • Requires falling back to alternative LLM providers (e.g., OpenAI → Anthropic) when the primary model fails.
  • Required Modifications during Retry:
    • Endpoint & Path: Routing to an entirely different provider URL (e.g., /chat/completions to /messages).
    • Body Transformation: Translating the payload to match the fallback provider’s expected schema.
    • Authentication: Swapping credentials to use the correct auth method for the fallback provider.
    • Headers: Adding or modifying headers as required by the new target (future-proofing).

2. Envoy Native Capabilities Analysis

We analyzed Envoy’s native RetryPolicy combined with upstream ext_proc (which sits between the router filter and the backend) and Aggregate Clusters.

• Upstream ext_proc: Can handle modifications to headers, body, and path. Limitation: Cannot modify the target endpoint (retries hit the same upstream).
• Aggregate Clusters: Allows defining an ordered list of fallback endpoints per retry attempt, solving the dynamic endpoint routing issue.
• Outstanding Gaps: While combining these features routes the request, native Envoy does not easily solve cross-provider authentication swapping and body transformations.

3. Approaches Considered

Approach	Status	Description & Assessment
1. Upstream ext_proc Retry	🛑 Blocked	Handle retries natively via Envoy's upstream hook. Drawbacks: The upstream ext_proc is still alpha and unfit for GA. Furthermore, standardizing transformation and authentication logic for multiple providers within this hook is complex and diverges from our existing policy logic.
2. Policy-Engine Re-route (Response Path)	🟡 Have to evaluate	On a failure (e.g., 401) in the response path, the policy engine re-invokes the route. Concept: If an API has an OAuth2 policy, create a parallel "bare" route (no policy chain). On 401, the engine refreshes the token and calls the bare route. Pros: Credential material is already accessible from the policy engine. Cons: Generalizing this to cross-provider failover (where the retry needs completely different auth/payloads, not just a refresh) remains an open challenge.
3. Lua Filters	❌ Ruled Out	Reasoning: The team is discussing about deprecating Lua filters in the gateway.

This is the branch I used for the Approach 01's POC: https://github.com/Thenujan-Nagaratnam/api-platform/tree/decoupled-retry-source