feat: Add support for OpenAI-compatible LLM endpoints (Groq, Ollama, Azure OpenAI, etc.) #1198

Activity

1. hemasekhar-p

   self-assigned this

   on May 18, 2026
2. hemasekhar-p

   added the

   Feature

   issue type on May 18, 2026
3. hemasekhar-p commented on May 18, 2026

   

   hemasekhar-p

   on May 18, 2026

   Contributor

   More actions

   Hi @Shaamam, Thank you for submitting this feature request. It is currently under review by our team, we will keep you updated if any additional information is required. We appreciate your input.
4. hemasekhar-p commented on May 18, 2026

   

   hemasekhar-p

   on May 18, 2026

   Contributor

   More actions

   @anFatum, Please take a look into it.
5. hemasekhar-p

   added

   needs review

   on May 18, 2026
6. Shaamam commented on May 18, 2026

   

   Shaamam

   on May 18, 2026

   Author

   More actions

   Hi @hemasekhar-p and @anFatum,

   Thank you for reviewing this! I've submitted PR #1202 with a complete implementation to facilitate the code review.

   The PR includes:

   • Core OpenAiCompatibleLlm implementation
   • 13 unit tests (all passing)
   • Integration tests with Ollama
   • Manual verification with Groq API (tested successfully)
   • Documentation updates

   All tests pass locally and the implementation has been verified with real Groq API. Happy to make any adjustments based on your feedback!
7. svetanis commented on May 19, 2026

   

   svetanis

   on May 19, 2026

   Last edited by svetanis

   Contributor

   More actions

   Hello Team! I saw this issue and wanted to flag that I've been working on a related solution.
   I submitted a PR (feat:contrib/model-prism: pluggable LLM backend SPI for ADK Java via ServiceLoader #1199) recently that covers similar ground — OpenAiCompatibleLlm as a base class, plus provider discovery via ServiceLoader SPI as a contrib module. It also includes streaming support and 11 demo classes.

   Happy to collaborate, align approaches, or break the PR into smaller pieces if that's easier to review. Just wanted to make sure the efforts are visible to each other.
8. FuturMix commented on May 19, 2026

   

   FuturMix

   on May 19, 2026

   More actions

   Great feature request and solid PR work by @Shaamam and @svetanis. The multi-provider pattern is essential for production agent systems — here are a few architectural considerations from running similar setups:

   Model routing by task type is the biggest cost win

   The issue author mentioned switching between Gemini (complex reasoning) and Groq (fast responses) based on task type — this is exactly the right pattern. In practice, a well-configured routing layer can reduce API costs by 40-70% without quality loss by sending:

   • Planning/reasoning → expensive model (Gemini, Claude Opus)
   • Code generation → mid-tier (Sonnet, GPT-5.5)
   • Summarization/formatting → cheap/fast model (Haiku, DeepSeek, Groq)

   Consider the provider abstraction carefully

   Both PRs take slightly different approaches. A few things worth considering for the final design:

   1. Credential isolation: Each provider may need different auth headers (Authorization: Bearer vs x-api-key vs custom). The builder should support a generic header map rather than assuming Authorization only.

   2. Model name normalization: Some providers accept gpt-5.5 while others need openai/gpt-5.5. Having an optional model alias map in the config avoids hardcoding provider-specific naming.

   3. Streaming support: The initial non-streaming implementation is fine for an MVP, but agent workflows with long outputs really benefit from streaming. The SPI approach in PR feat:contrib/model-prism: pluggable LLM backend SPI for ADK Java via ServiceLoader #1199 seems better positioned for this.

   4. Fallback chain: For production reliability, being able to define primary → fallback provider chains (e.g., Groq → Ollama if Groq is rate-limited) is very valuable.

   The Python ADK's LiteLLM integration handles some of this by delegating to LiteLLM's provider registry — the Java equivalent could follow a similar pattern where an OpenAI-compatible gateway handles provider abstraction externally, and ADK-Java only needs to talk to one endpoint.
9. svetanis commented on May 20, 2026

   

   svetanis

   on May 20, 2026

   Contributor

   More actions

   @FuturMix , Thanks for the thoughtful feedback! Just a couple of points regarding your feedback:

   Model name normalization. Fair point in a general framework, but the provider/model prefix convention sidesteps it here. The developer writes "groq/llama3" in the agent config, and GroqModelProvider just strips groq/ before forwarding "llama3" to Groq's endpoint — so the string the developer writes is exactly what the target API expects, no alias map needed. The prefix is purely a routing namespace: "groq/llama3" and "ollama/llama3" are distinct keys in LlmRegistry, so two providers can share the same bare model name without conflict. If we want to go further, prefix stripping could move into ModelProviderRegistry itself to remove even that boilerplate from individual providers.

   On the LiteLLM suggestion. LiteLLM's zero-code switching works because Python's dynamic type system can resolve a string like groq/llama3 to a live module at runtime — ServiceLoader is the Java-idiomatic way to achieve the same effect explicitly. With the ModelProvider SPI, a provider JAR drops its registration into META-INF/services, ModelProviderRegistry.registerAll() picks it up at startup, and switching providers becomes one property change plus one dependency swap with no app code touched - the same zero-code outcome LiteLLM gives Python developers, through Java's explicit infrastructure rather than dynamic imports.
10. Shaamam commented on May 20, 2026

    

    Shaamam

    on May 20, 2026

    Author

    More actions

    @FuturMix Thank you for the detailed feedback!

    Regarding streaming support - my implementation already supports it! The generateContent method passes the stream parameter directly to ChatCompletionsHttpClient.complete(), which handles Streaming per the OpenAI specification.

    From my implementation (line 96-97):

    public Flowable<LlmResponse> generateContent(LlmRequest llmRequest, boolean stream) {
      return client.complete(llmRequest, stream);  // ← Streaming supported
    }

    The documentation mentions "non-streaming only" because I focused on testing non-streaming cases, but the underlying infrastructure supports both modes.

    Design philosophy: My approach prioritizes simplicity - pattern-based registration is straightforward and reuses existing ADK
    infrastructure (ChatCompletionsHttpClient, LlmRegistry). Advanced features like fallback chains and cost-based routing could be added incrementally in follow-up PRs.

    @svetanis I see you also submitted PR #1199 with a ServiceLoader approach. Both have merit - would be happy to discuss with maintainers which direction fits ADK's architecture better!

    Looking forward to feedback from @hemasekhar-p and @anFatum on preferred approach.
11. kingpro1985 commented on May 20, 2026

    

    kingpro1985

    on May 20, 2026

    More actions

    I saw this while looking at custom model/provider setup issues: "feat: Add support for OpenAI-compatible LLM endpoints (Groq, Ollama, Azure OpenAI, etc.)".

    If it helps your testing, CloudGPU exposes a hosted OpenAI-compatible endpoint that can route to GPT, Claude, Gemini, DeepSeek, Kimi and GLM behind one API key.

    Base URL: https://gpuapis.com/api/v1
    Setup guide: https://gpuapis.com/use-cases/deepseek-gemini-claude-api/

    I can share a small test key/credit and a cURL request if you want to verify it in your current client before doing any integration work.
12. svetanis commented on May 21, 2026

    

    svetanis

    on May 21, 2026

    Contributor

    More actions

    @Shaamam Thanks for the clarification on streaming — good to know the underlying plumbing supports it.

    Looking more carefully at your OpenAiCompatibleLlm, I think both PRs actually agree on more than it might appear at first:

    Where we agree:
    Both proposals put an OpenAiCompatibleLlm-style class into ADK core to encapsulate the OpenAI Chat Completions protocol. That's a solid foundation.

    Where they diverge — the key difference:

    Your approach is a utility class: every ADK application that wants to use Groq, Ollama, or any other provider must write wiring code in their own codebase:

    OpenAiCompatibleLlm groq = OpenAiCompatibleLlm.builder()
        .baseUrl("https://api.groq.com/openai/v1/")
        .headers(ImmutableMap.of("Authorization", "Bearer " + apiKey))
        .modelName("groq-llama3-70b")
        .build();
    groq.registerWithPattern("groq-.*");

    This is clean and simple, but it means every application that uses Groq carries this wiring. If the Groq endpoint changes, every app must be updated independently.

    The model-prism SPI approach (a working prototype) takes this one step further. Three small additions go into google-adk.jar itself:

    • ModelProvider — the SPI interface that provider JARs implement
    • ModelProviderRegistry — calls ServiceLoader to discover all providers on the classpath; registerAll() is proposed to be called once inside ADK's Runner and AdkWebServer.start()
    • OpenAiCompatibleLlm — reusable base class for any POST /v1/chat/completions provider

    Provider JARs (e.g., adk-provider-groq) live outside ADK, depend on google-adk.jar, and self-register via META-INF/services. From the application developer's perspective, switching to Groq requires exactly two things:

    1. A Maven dependency:

    <dependency>
      <groupId>com.google.adk</groupId>
      <artifactId>adk-provider-groq</artifactId>
    </dependency>

    2. An application property:

    agent.model=groq/llama-3.1-8b-instant
    

    That's it. No Java code in the application at all. The provider JAR self-registers via META-INF/services, ADK's startup discovers it via ServiceLoader, and the model name string routes to the right backend automatically. Switching from Groq to Ollama is a dependency swap plus a config change — identical to the LiteLLM experience for Python developers.

    Summary:

    	Shaamam's approach	model-prism approach
    ADK core addition	OpenAiCompatibleLlm (1 class)	ModelProvider SPI + ModelProviderRegistry + OpenAiCompatibleLlm base class
    registerAll() call	N/A — no such mechanism	Inside ADK Runner / AdkWebServer.start() — zero app involvement
    Provider wiring	In every application's code	In a reusable, published provider JAR
    Adding a new provider	Write wiring in each app that uses it	Publish a provider JAR once; apps just add a dependency
    App-level code change	builder()...registerWithPattern() per provider	None — dependency + config property only
    Discovery	Manual, explicit	ServiceLoader (META-INF/services)

    Looking forward to hearing what direction fits ADK's roadmap.
13. akbarkanso commented on Aug 14, 2026

    

    akbarkanso

    on Aug 14, 2026

    More actions

    Hi all, we hit this from the other direction: contrib/spring-ai is not a working alternative for OpenAI-compatible endpoints on Spring AI 2.0.0, tool results never reach the model #1429. So there is currently no supported path for these gateways, which is the gap this would close.

    We ended up wrapping core's ChatCompletionsHttpClient in a small BaseLlm, the same shape proposed here, about 50-80 lines plus a retry policy. If this lands, ours becomes a deletion.

    One design question worth settling early: does an official connector own retry/backoff, or delegate it? Spring AI gave us spring.ai.retry.* for 429s, and shared gateways return 429 routinely, so a straight port of the request/response mapping silently loses rate-limit handling.
14. airouter-dev commented on Aug 14, 2026

    

    airouter-dev

    on Aug 14, 2026

    More actions

    A provider-neutral builder is a good fit for this request. A few interoperability cases are worth making explicit in the contract:

    • Normalize a base URL ending in /v1 exactly once, while also accepting an origin URL; avoid provider-specific path assumptions.
    • Treat model names as opaque IDs and keep model-pattern registration optional rather than requiring a vendor prefix.
    • Make authenticated GET /v1/models optional: some compatible endpoints do not expose discovery, so manual model selection should remain valid.
    • Keep Chat Completions support independent from Responses, embeddings, or live APIs so unsupported surfaces fail clearly.
    • Redact API keys and custom headers from logs and preserve upstream HTTP status/request IDs for diagnostics.

    Disclosure: I work on AI Router. Its English-root service can be used as an external HTTPS fixture for authenticated model discovery and OpenAI-compatible Chat Completions: https://ai-router.dev. This does not require a first-party ADK provider preset.