Skip to content

model-gateway: derive pin-alias names from the deployed GGUFs - #177

Merged
AlienWalker1995 merged 1 commit into
mainfrom
feat/model-gateway-dynamic-aliases
Aug 14, 2026
Merged

model-gateway: derive pin-alias names from the deployed GGUFs#177
AlienWalker1995 merged 1 commit into
mainfrom
feat/model-gateway-dynamic-aliases

Conversation

@AlienWalker1995

Copy link
Copy Markdown
Owner

Problem

The gateway's pickable pin aliases were hardcoded model-generation names (qwen3.6-27b, qwen3.6-35b-a3b-cpu). After an active-model swap the gateway kept advertising an alias named for the previous generation (observed live: qwen3.6-27b serving Qwen3.8 weights). The GPU entries' supports_vision flag and the served_by build name had the same hardcoded-drift problem.

Fix — make the model list deployment-derived

Uses the same .env-driven entrypoint templating the model_info values already use, so alias names can no more drift than weights_file can:

  • __GPU_MODEL_NAME__ / __CPU_MODEL_NAME__ derived from LLAMACPP_MODEL / LLAMACPP_CPU_MODEL basenames (lowercased, .gguf stripped, CPU suffixed -cpu). A model swap + gateway recreate renames the aliases automatically.
  • router_settings.fallbacks targets the templated CPU alias — the failover contract can never name an undeclared model; test_the_failover_router_is_configured_to_route_there still verifies this placeholder-to-placeholder on the raw template.
  • __GPU_SUPPORTS_VISION__ = true iff LLAMACPP_MMPROJ is non-empty.
  • __GPU_IMAGE__served_by reports the actual LLAMACPP_IMAGE instead of a hardcoded build name.
  • Model-family prose replaced with weights_file references; README documents the derived names and that only local-chat/local-embed are stable ids.

local-chat and local-embed are unchanged by contract. Repo grep found no consumers of the old version-named aliases outside the gateway's own files.

Validation

  • ruff check . clean; tests/substrate/test_gpu_arbitration.py 23/23.
  • Template render simulated with production env values → YAML parses, fallback target ∈ declared names, vision flags are real booleans.
  • Deployed live: /v1/models now advertises qwen3.8-27b-q6_k + qwen3.6-35b-a3b-ud-q4_k_m-cpu; completions succeed through both local-chat and the derived pin alias.

🤖 Generated with Claude Code

The pickable pin aliases were hardcoded model-generation names
(qwen3.6-27b, qwen3.6-35b-a3b-cpu) that went stale on every model swap —
after swapping the GPU model the gateway kept advertising an alias named
for the previous generation. Same class of drift for the GPU entries'
supports_vision flag and the served_by build name.

Make the whole model_list deployment-derived, using the same .env-driven
entrypoint templating the model_info values already use:

- __GPU_MODEL_NAME__ / __CPU_MODEL_NAME__: derived from LLAMACPP_MODEL /
  LLAMACPP_CPU_MODEL basenames (lowercased, .gguf stripped, CPU suffixed
  -cpu). A model swap + gateway recreate renames the aliases automatically.
- router_settings fallbacks target the templated CPU alias, so the
  failover contract can never name a model that isn't declared
  (test_the_failover_router_is_configured_to_route_there still verifies
  this placeholder-to-placeholder on the raw template).
- __GPU_SUPPORTS_VISION__: true iff LLAMACPP_MMPROJ is non-empty — the
  vision flag now states whether an mmproj is actually loaded.
- __GPU_IMAGE__: served_by reports the actual LLAMACPP_IMAGE instead of a
  hardcoded build name.
- Model-family prose in descriptions replaced with weights_file
  references; README documents the derived names and that only
  local-chat/local-embed are stable ids.

local-chat and local-embed are unchanged by contract.

Validated live: /v1/models advertises qwen3.8-27b-q6_k +
qwen3.6-35b-a3b-ud-q4_k_m-cpu after a swap, completions succeed through
both local-chat and the derived pin alias.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AlienWalker1995
AlienWalker1995 merged commit 99fba14 into main Aug 14, 2026
3 checks passed
@AlienWalker1995
AlienWalker1995 deleted the feat/model-gateway-dynamic-aliases branch August 14, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant