feat: serve custom endpoints their declared models intersected with the live catalog - #72
Open
sindriii wants to merge 11 commits into
Open
feat: serve custom endpoints their declared models intersected with the live catalog#72sindriii wants to merge 11 commits into
sindriii wants to merge 11 commits into
Conversation
…fetched catalog `models.default` has only ever been a fallback: with `fetch: true` a successful fetch replaces it wholesale, so every endpoint pointed at one gateway advertises that gateway's entire catalog. Deployments that put several endpoints over a single OpenAI-compatible gateway — one per provider dialect, which is the only way to send each provider the parameters it needs — therefore cannot give those endpoints different model lists. Curating `models.default` with `fetch: false` gets the lists apart, but then nothing reflects what the gateway actually serves: a model retired upstream lingers in the picker, and a per-user catalog collapses to one static list for everyone. Add `models.filter`. With it, `models.default` is an allowlist rather than a fallback and the endpoint serves `declared ∩ fetched`, in declared order. Declaring a model the gateway does not have is inert, so one shared list can cover a fleet of deployments that each hold a subset, and an endpoint's contents follow the gateway without a config edit. An empty declared list is now meaningful — an endpoint template that a deployment fills in — so `models.default` no longer requires one entry, and the two prefilters that decide whether an endpoint has any model source at all now test that list's length. They were testing it for truthiness, and `[]` is truthy, which would have admitted an endpoint that can never produce a model. Both prefilters were separate copies of that predicate; they now share one, next to the intersection it guards. A fetch that never answered is kept distinct from a fetch that answered with nothing. Transport failure falls back to the declared list, for every endpoint: an empty list is about to mean "no endpoint", and treating an unreachable gateway as an authoritative empty would take endpoints away mid-outage. This is the one behaviour change for endpoints that send an `authorization` header — a rejected fetch used to collapse to `[]` for them, because it was flattened into an empty answer before that check ran. An answer of `[]` still yields nothing, which is what the header check already did for a successful empty response, now reached for filtered endpoints without inspecting headers at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`getModelsConfig` is the accessor for a request's available models and is already reached from seven places — the models route, the startup-config route pruning model specs, model validation on submit, token config, agent initialization and both agent response controllers — each of which re-runs the whole resolution, including a live `/models` call per gateway. The shared `MODEL_QUERIES` cache cannot absorb that. `fetchModels` skips it whenever an endpoint forwards user-bound headers, and it has to: the response is scoped to the caller's identity, so one user's list must never be served to another. Memoize on the request object instead, which is the exact scope that makes the result reusable — same identity, same moment — and lets the entry be collected with the request. A failure evicts, so a later caller in the same request retries rather than inheriting a settled rejection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An endpoint's existence is decided by its declaration, never by whether it can serve anything, so an endpoint with an empty model list still renders. `useEndpoints` computes `hasModels` but drops only `agents`, and the agent builder offers the endpoint as a provider with no model to pick. That is already reachable: for an endpoint sending an `authorization` header, a fetch returning nothing yields an empty list rather than falling back to declared models, so a user whose token grants no models gets an endpoint that cannot answer. `models.filter` makes it ordinary — an endpoint whose declared list intersects the catalog to nothing is an endpoint this deployment does not have. Withhold those from the endpoints config, which is the one object the model selector, the agent builder and model-spec pruning all derive from, so a single decision covers all three instead of three that can drift. Custom endpoints only; built-in model lists are not per-request. Fails open — only an explicit empty list withholds an endpoint, while a models config that cannot be resolved, or that has no entry for an endpoint, leaves every declared endpoint in place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
`validateModel` treats any model absent from an endpoint's list as an `ILLEGAL_MODEL_REQUEST`, which carries a configurable violation score and contributes toward a ban. That reading only holds when the endpoint has models and the requested one is not among them. An endpoint whose list is empty is unavailable — the gateway serves none of what it declares, or none of it for this user's grants. Every model named against it is unserveable rather than illegitimate, and the requests that arrive are stored conversations and agents pointing at an endpoint that has since stopped serving them. Their owners would collect violations, and eventually a ban, for a change in configuration they had no part in. Reject those without logging a violation, and say the endpoint is unavailable rather than the request illegal. An unlisted model on an endpoint that does have models still logs, unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An endpoint's model list renders raw ids — `claude-opus-4-8`, `gpt-5.4-nano`. Only agents and assistants carry a name to show instead; every other endpoint shows the id it was configured with, which is the wrong string to put in front of a non-technical user. Adds `modelLabels`, an endpoint-level `Record<modelId, label>` alongside `modelDisplayLabel` and `tokenConfig`, and passes it through to the client. It is display-only. The id stays what is declared, fetched, intersected, selected, stored on the conversation and sent upstream, so a label is safe to change at any time and a model with no entry renders its id. Keeping the map at endpoint level rather than in `models.default` also keeps `modelsConfig` a `Record<string, string[]>`: `modelItemSchema` has no `label`, and both the declared and the fetched path resolve to bare id strings. Because a label for a model the endpoint does not serve is simply never rendered, one map can cover a fleet of deployments that each serve a subset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolves the name to show through one helper, `getModelName`, which returns the agent name, the assistant name or the endpoint's declared label for the model — and `undefined` when there is none, so each caller keeps its own fallback. An empty string counts as no name, which is what `agentNames` stores for an unnamed agent. Five sites read it: the endpoint's model list, a search result row, the selector's closed trigger, the announcement made on selecting a model, and the Agent Builder's model combobox. The last one matters most on deployments that set `modelSpecs.addedEndpoints: [agents]`, since that keeps provider endpoints out of the selector entirely and the Agent Builder becomes the only place their users meet a model id. Search covers the label and the id, across all three filters — which endpoints survive a query, the open endpoint's list, and the search results' own predicate. A label is additive there: both strings match. An agent or assistant name still replaces the id, unchanged. Selection is untouched: `handleSelectModel` and `ControlCombobox` both keep the id as the value they hand back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`withholdEmptyCustomEndpoints` ran on every `getEndpointsConfig` call, and that function has seven callers — most of which want the endpoint's *configuration*, not a view of what a user may be offered. Two of them make this actively wrong, not merely slow: - `buildEndpointOption` reads `defaultParamsEndpoint` off the merged config to pick a request builder. Withhold the endpoint and it reads `undefined`, and the conversation falls back to the OpenAI schema — the exact dialect degradation the per-provider split exists to prevent, reachable on the message path whenever a user's catalog comes back empty. - `validateModel` reads `userProvide` to let a user-keyed endpoint through. A withheld endpoint loses that and takes the wrong branch. And `checkCapability` shares the same function, so a capability check — three of which sit on the file-upload path — waited on a model catalog it never reads. Withholding is now opt-in, and `/api/endpoints` is the only caller that opts in: it is the one answering "what may this user be offered", and the selector, the Agent Builder and spec pruning all derive from its response. Cache the user-scoped catalog instead of skipping the cache. Skipping it was right about the hazard — keyed by baseURL+apiKey alone, one user's filtered list is served to the next request over that gateway — and wrong about the remedy: it made every request pay a live fetch, and a page load issues three. The key now carries the requesting user and the header templates in play, with a thirty-second TTL rather than the catalog's two minutes, so a page load costs one round trip and a revoked grant surfaces within half a minute. The gateway remains the enforcement point either way. With no identity to key on, nothing is cached. Cut this path's timeout from 5s to 2s. Every caller of the config loader degrades to the declared list, so the only thing a longer wait buys is a more accurate list — paid for with a blank picker in front of a waiting user. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Thirty seconds was picked to cover a page load and nothing more, on the assumption that a short window was the safe choice. It buys less safety than it looks: a stale entry cannot grant anything, because the gateway rejects a call for a model the user lacks regardless. What a long window actually costs is a list that lags — a revoked grant, or a model just rolled out gateway-side. Five minutes trades a little of that lag for far fewer round trips, which is the right way round while the cache has no shared store: without Redis it is an in-memory map per task, wiped on restart, so entries have to survive longer than one page load to be worth anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 5s ceiling was cut to 2s on the reasoning that a caller which degrades gracefully should not make a user wait — but the premise was wrong. `fetchModels` catches its own transport errors and returns an empty list, so a timeout is not a failure this code sees: it is an empty catalog, which intersects to nothing and withholds the endpoint. A shorter timeout therefore produces *more* empty pickers, which is the symptom it was meant to relieve. Removes the parameter as well as the value: nothing else wanted it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An empty model list on a user-provided endpoint reflects the user's stored key — missing, expired, or granted nothing — and the picker entry is the only way to set or fix that key. Withholding it would lock the user out, so it stays visible even when empty, mirroring validateModel's exemption. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
We front LiteLLM with several custom endpoints — one per provider dialect, because
customParams.defaultParamsEndpointis what selects the Anthropic/Google request builder, and Claude models are measurably worse on the OpenAI builder. All of those endpoints share onebaseURL,apiKeyand header set.Today they cannot have different model lists:
fetch: true, a successful fetch replacesmodels.defaultwholesale, so every endpoint advertises LiteLLM's entire catalog — including embeddings, rerankers and OCR models, each on the wrong dialect.fetch: falseand curated lists, the lists come apart but nothing tracks reality any more: a model retired in LiteLLM lingers in the picker, and the per-user catalog our OIDC endpoints already return collapses to one static list for everyone.There is no third option in config, which is what this PR adds.
The same split leaves a second gap. Once the model list comes from the gateway, every row in the picker is a raw model id —
claude-opus-4-8,gpt-5.4-nano. Agents and assistants get a name to show instead; no other endpoint does. Commits 5 and 6 give an endpoint somewhere to put a human label.What
Six commits, reviewable in order.
1.
models.filter— declared ∩ fetchedmodels.defaultbecomes an allowlist rather than a fallback: the endpoint serves the declared names the gateway actually returned, in declared order. Declaring a model a deployment does not have is inert, so one shared list covers a fleet where each deployment holds a subset, and rolling a model out becomes a gateway-side change with no config edit.Supporting changes:
models.defaultno longer requires one entry — an empty declared list is now meaningful (a template a deployment fills in).models.defaultfor truthiness, and[]is truthy. They now test length, and share one predicate instead of holding two copies of it.2. One models-config resolution per request
getModelsConfigis reached from seven places per page (models route, startup-config spec pruning,validateModel, token config, agent initialization, both agent response controllers) and each re-ran the full resolution including a live/modelscall per gateway.MODEL_QUERIEScannot absorb it —fetchModelsskips that cache whenever an endpoint forwards user-bound headers, and must, since the response is identity-scoped. Memoized on the request object instead; a failure evicts so a later caller can retry.3. Withhold endpoints with nothing to serve
An endpoint's existence is decided by declaration, never content, so an empty endpoint still renders —
useEndpointscomputeshasModelsbut drops onlyagents, and the agent builder offers it as a provider with no model to pick. Already reachable today: an endpoint sending anauthorizationheader whose fetch returns nothing gets[]rather than the declared fallback. Now withheld from the endpoints config, which is the single object the selector, the agent builder and spec pruning all derive from.Custom endpoints only. Fails open — only an explicit empty list withholds an endpoint.
4. No violation for a request to an endpoint with nothing to offer
validateModeltreats any model absent from an endpoint's list as anILLEGAL_MODEL_REQUEST, which carries a violation score and contributes toward a ban. That reading only holds when the endpoint has models and the requested one is not among them.An endpoint whose list is empty is unavailable, and the requests that arrive are stored conversations and agents naming an endpoint that has stopped serving them — commit 3 makes that an ordinary state. Their owners would collect violations for a configuration change they had no part in. Those are now rejected as "Endpoint unavailable" without logging. An unlisted model on an endpoint that does have models still logs, unchanged.
5.
modelLabels— display labels on the endpointAn endpoint-level
Record<modelId, label>, alongsidemodelDisplayLabeland followingtokenConfig, passed through to the client.Display-only. The id stays what is declared, fetched, intersected, selected, stored on the conversation and sent upstream, so a label is safe to change at any time and a model with no entry renders its id.
Not routed through
models.default:modelItemSchemahas nolabel, and both the declared and the fetched path resolve to bare id strings, somodelsConfigis aRecord<string, string[]>and stays that way. Keeping the map on the endpoint also means a label for a model that endpoint does not serve is simply never rendered — one map covers a fleet where each deployment serves a subset, which is the same property commit 1 gives the declared list.6. Render the label
One helper,
getModelName, returns the agent name, the assistant name or the declared label for a model, andundefinedwhen there is none so each caller keeps its own fallback. An empty string counts as no name — that is whatagentNamesstores for an unnamed agent.Five sites read it: the endpoint's model list, a search result row, the selector's closed trigger, the announcement made on selecting a model, and the Agent Builder's model combobox. The last matters most on deployments setting
modelSpecs.addedEndpoints: [agents], where provider endpoints are kept out of the selector entirely and the Agent Builder is the only place a user meets a model id.Search covers the label and the id, across all three filters — which endpoints survive a query, the open endpoint's list, and the search results' own predicate. A label is additive there: both strings match. An agent or assistant name still replaces the id, unchanged.
Selection is untouched:
handleSelectModelandControlComboboxboth keep the id as the value they hand back.Behaviour change to review deliberately
A rejected fetch now falls back to the declared list for every endpoint, including those sending an
authorizationheader. Previously a rejected promise was flattened to[]before the header check, so those endpoints went empty during an outage.This is intentional and paired with commit 3: once an empty list can remove an endpoint, treating an unreachable gateway as an authoritative empty would delete endpoints mid-blip and raise
INVALID_AGENT_PROVIDERfor stored agents naming them. The gateway, not this list, is the enforcement point — it rejects a call for a model the caller lacks regardless.An answer of
[]still yields nothing. For filtered endpoints that now falls out of the intersection without inspecting headers, sohasAuthorizationHeadershould become redundant on the fulfilled path once every endpoint carriesfilter: true— left in place until that is confirmed on a live deployment.Blast radius
useEndpointsandAgentPanelboth derive fromendpointsConfig/modelsConfig. Commits 5–6 touch the client, but only the string rendered for a model.filterModelSpecsByAvailabilityuntouched — it starts pruning against real availability again as a consequence of getting a better list. Note it fails open on a missing key and closed on[], which is why commit 1 keeps the key present.endpoints.agents.allowedProvidersunaffected —initialize.tsrejects a provider missing from the list, so extra entries stay harmless.filterbehave as before on the fulfilled path, OIDC empty-answer handling included.Tests
packages/api/src/endpoints/config/availability.spec.ts— 16 new: intersection and declared order, two endpoints sharing one coalesced fetch, empty answer, rejected fetch with and without anauthorizationheader,filterwith no fetch, the empty-declaration prefilter, and three pinning unfiltered behaviour.packages/api/src/endpoints/config/endpoints.spec.ts— 7 new: withholding, built-ins never withheld, fail-open on error / on a missing key / with no resolver supplied.api/server/services/Config/__tests__/getModelsConfig.spec.js— 4 new: merge, one resolution per request, no sharing between requests, retry after failure.api/server/middleware/__tests__/validateModel.spec.js— 2 new: no violation on an empty list, violation still logged for an unlisted model.packages/data-provider/specs/config-schemas.spec.ts— 4 new:modelLabelssurvives parsing rather than being stripped as an unknown key, accepts labels for undeclared models, stays optional, rejects a non-string label.packages/api/src/endpoints/custom/config.spec.ts— 2 new: passed through to the client, absent when undeclared.client/.../Endpoints/__tests__/utils.test.ts— 9 new:getModelNameprecedence and its empty-name and null handling, label-and-id search throughfilterItemsandfilterModels,getDisplayValuelabelled and unlabelled.client/.../components/__tests__/EndpointModelItem.test.tsx— 3 new: label rendered instead of the id, unlabelled model falls back, selection still keys on the id.client/.../components/__tests__/SearchResults.test.tsx— 3 new: found by label, found by id, selected by id.Green:
packages/api6576 tests (the 20 remaining failures are the Redis*_integrationsuites plus one timing-sensitiveflow/managerspec that passes in isolation — no local Redis),packages/data-provider1254, and the client'sEndpoints/SidePanel/Agents/hooks/Endpointsuites, 168.The
apisuite is flaky on this baseline — three consecutive runs gave 2, 4 and 3 failures out of 2976, all instrategies/openIdJwtStrategy, the OpenID cookie suites,requestPasswordResetand the CloudFront cookie suite. None sits in a path this branch touches. The two stable ones were reproduced at650e62089with this branch's changes reverted and both packages rebuilt. The subset covering everything changed here —server/services/Config,server/middleware,server/controllers,server/routes, 1349 tests — passes.Not yet exercised against a live gateway; that is the next step on
apro-sandbox.One thing this does not fix
RemoteAgents/Remote-style endpoints —fetch: true, nofilter, authenticating as a shared key — still replace their declared list with the gateway's whole catalog. That is whatfilteris for; adopting it there is a config change, not a code one.