model-gateway: derive pin-alias names from the deployed GGUFs - #177
Merged
Conversation
The pickable pin aliases were hardcoded model-generation names (qwen3.6-27b, qwen3.6-35b-a3b-cpu) that went stale on every model swap — after swapping the GPU model the gateway kept advertising an alias named for the previous generation. Same class of drift for the GPU entries' supports_vision flag and the served_by build name. Make the whole model_list deployment-derived, using the same .env-driven entrypoint templating the model_info values already use: - __GPU_MODEL_NAME__ / __CPU_MODEL_NAME__: derived from LLAMACPP_MODEL / LLAMACPP_CPU_MODEL basenames (lowercased, .gguf stripped, CPU suffixed -cpu). A model swap + gateway recreate renames the aliases automatically. - router_settings fallbacks target the templated CPU alias, so the failover contract can never name a model that isn't declared (test_the_failover_router_is_configured_to_route_there still verifies this placeholder-to-placeholder on the raw template). - __GPU_SUPPORTS_VISION__: true iff LLAMACPP_MMPROJ is non-empty — the vision flag now states whether an mmproj is actually loaded. - __GPU_IMAGE__: served_by reports the actual LLAMACPP_IMAGE instead of a hardcoded build name. - Model-family prose in descriptions replaced with weights_file references; README documents the derived names and that only local-chat/local-embed are stable ids. local-chat and local-embed are unchanged by contract. Validated live: /v1/models advertises qwen3.8-27b-q6_k + qwen3.6-35b-a3b-ud-q4_k_m-cpu after a swap, completions succeed through both local-chat and the derived pin alias. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The gateway's pickable pin aliases were hardcoded model-generation names (
qwen3.6-27b,qwen3.6-35b-a3b-cpu). After an active-model swap the gateway kept advertising an alias named for the previous generation (observed live:qwen3.6-27bserving Qwen3.8 weights). The GPU entries'supports_visionflag and theserved_bybuild name had the same hardcoded-drift problem.Fix — make the model list deployment-derived
Uses the same
.env-driven entrypoint templating themodel_infovalues already use, so alias names can no more drift thanweights_filecan:__GPU_MODEL_NAME__/__CPU_MODEL_NAME__derived fromLLAMACPP_MODEL/LLAMACPP_CPU_MODELbasenames (lowercased,.ggufstripped, CPU suffixed-cpu). A model swap + gateway recreate renames the aliases automatically.router_settings.fallbackstargets the templated CPU alias — the failover contract can never name an undeclared model;test_the_failover_router_is_configured_to_route_therestill verifies this placeholder-to-placeholder on the raw template.__GPU_SUPPORTS_VISION__=trueiffLLAMACPP_MMPROJis non-empty.__GPU_IMAGE__—served_byreports the actualLLAMACPP_IMAGEinstead of a hardcoded build name.weights_filereferences; README documents the derived names and that onlylocal-chat/local-embedare stable ids.local-chatandlocal-embedare unchanged by contract. Repo grep found no consumers of the old version-named aliases outside the gateway's own files.Validation
ruff check .clean;tests/substrate/test_gpu_arbitration.py23/23./v1/modelsnow advertisesqwen3.8-27b-q6_k+qwen3.6-35b-a3b-ud-q4_k_m-cpu; completions succeed through bothlocal-chatand the derived pin alias.🤖 Generated with Claude Code