From dacf1435c16dc89df0856c5b330feeb1b8356e25 Mon Sep 17 00:00:00 2001 From: Hermes Bot Date: Fri, 14 Aug 2026 12:24:48 -0400 Subject: [PATCH] model-gateway: derive pin-alias names from the deployed GGUFs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pickable pin aliases were hardcoded model-generation names (qwen3.6-27b, qwen3.6-35b-a3b-cpu) that went stale on every model swap — after swapping the GPU model the gateway kept advertising an alias named for the previous generation. Same class of drift for the GPU entries' supports_vision flag and the served_by build name. Make the whole model_list deployment-derived, using the same .env-driven entrypoint templating the model_info values already use: - __GPU_MODEL_NAME__ / __CPU_MODEL_NAME__: derived from LLAMACPP_MODEL / LLAMACPP_CPU_MODEL basenames (lowercased, .gguf stripped, CPU suffixed -cpu). A model swap + gateway recreate renames the aliases automatically. - router_settings fallbacks target the templated CPU alias, so the failover contract can never name a model that isn't declared (test_the_failover_router_is_configured_to_route_there still verifies this placeholder-to-placeholder on the raw template). - __GPU_SUPPORTS_VISION__: true iff LLAMACPP_MMPROJ is non-empty — the vision flag now states whether an mmproj is actually loaded. - __GPU_IMAGE__: served_by reports the actual LLAMACPP_IMAGE instead of a hardcoded build name. - Model-family prose in descriptions replaced with weights_file references; README documents the derived names and that only local-chat/local-embed are stable ids. local-chat and local-embed are unchanged by contract. Validated live: /v1/models advertises qwen3.8-27b-q6_k + qwen3.6-35b-a3b-ud-q4_k_m-cpu after a swap, completions succeed through both local-chat and the derived pin alias. Co-Authored-By: Claude Fable 5 --- ordo/gpu.py | 6 +-- services/model-gateway/README.md | 21 +++++++--- services/model-gateway/entrypoint.sh | 16 +++++++- services/model-gateway/litellm_config.yaml | 46 ++++++++++++---------- 4 files changed, 58 insertions(+), 31 deletions(-) diff --git a/ordo/gpu.py b/ordo/gpu.py index 0885c88..691e64f 100644 --- a/ordo/gpu.py +++ b/ordo/gpu.py @@ -68,9 +68,9 @@ CAPABILITY when it does. Not all preemptible services are alike: - A burst render has no degraded path — reclaiming it means the work stops. ``strategy: stop``. - - The resident LLM does: it MIGRATES rather than dies. ``llamacpp-cpu`` runs the same Qwen3.6 - A3B on CPU at the same 131072 window, and the model-gateway (LiteLLM) already carries - ``fallbacks: [{local-chat: [qwen3.6-35b-a3b-cpu]}]`` with a 30s cooldown that routes back to + - The resident LLM does: it MIGRATES rather than dies. ``llamacpp-cpu`` runs its own CPU + chat model at the same context window, and the model-gateway (LiteLLM) already carries + ``fallbacks: [{local-chat: []}]`` with a 30s cooldown that routes back to the GPU model as soon as it is healthy again. So llama.cpp yielding the 5090 is a GPU→CPU migration: availability is preserved, only throughput degrades. That is declared as ``strategy: failover`` with the degraded target and the component that reroutes named as diff --git a/services/model-gateway/README.md b/services/model-gateway/README.md index 833b3cd..20fe54f 100644 --- a/services/model-gateway/README.md +++ b/services/model-gateway/README.md @@ -9,11 +9,16 @@ entrypoint that templates the config placeholders at startup. | id | mode | backend | notes | |----|------|---------|-------| -| `local-chat` | chat | `llamacpp` (GPU) | Canonical auto-routing alias; fails over to `qwen3.6-35b-a3b-cpu` on GPU eviction. | -| `qwen3.6-27b` | chat | `llamacpp` (GPU) | Explicit pin of the GPU deployment — no failover. | -| `qwen3.6-35b-a3b-cpu` | chat | `llamacpp-cpu` | Always-on CPU MoE fallback (opt-in `cpu-fallback` profile). | +| `local-chat` | chat | `llamacpp` (GPU) | Canonical auto-routing alias; fails over to the CPU pin on GPU eviction. | +| *\* | chat | `llamacpp` (GPU) | Explicit pin of the GPU deployment — no failover. Name derived from `LLAMACPP_MODEL` (basename, lowercased; e.g. `qwen3.8-27b-q6_k`). | +| *\*`-cpu` | chat | `llamacpp-cpu` | Always-on CPU fallback (opt-in `cpu-fallback` profile). Name derived from `LLAMACPP_CPU_MODEL`. | | `local-embed` | embedding | `llamacpp-embed` | nomic-embed-text-v1.5, 768-dim, ctx 8192. | +The two pin-alias names are **derived at startup from the deployed GGUF filenames** — a +model swap renames them automatically, so nothing version-named is hardcoded in the config. +Only `local-chat` and `local-embed` are stable ids; anything that must survive a model swap +should use those. + Every entry carries a fully populated `model_info` block (mode, context window, output cap, capability flags, description, backing GGUF) — that block IS the gateway's model documentation for clients. Deployment-variable values are placeholder-templated by the @@ -29,10 +34,14 @@ metadata tracks the running deployment instead of drifting: | `__GPU_WEIGHTS__` | `LLAMACPP_MODEL` (model.gguf) | | `__CPU_WEIGHTS__` | `LLAMACPP_CPU_MODEL` (Qwen3.6-35B-A3B-UD-Q4_K_M.gguf) | | `__EMBED_WEIGHTS__` | `LLAMACPP_EMBED_MODEL` (nomic-embed-text-v1.5.Q4_K_M.gguf) | +| `__GPU_IMAGE__` | `LLAMACPP_IMAGE` (llama.cpp) | +| `__GPU_MODEL_NAME__` | derived: `LLAMACPP_MODEL` basename, lowercased, `.gguf` stripped | +| `__CPU_MODEL_NAME__` | derived: `LLAMACPP_CPU_MODEL` basename, lowercased, + `-cpu` | +| `__GPU_SUPPORTS_VISION__` | derived: `true` iff `LLAMACPP_MMPROJ` is non-empty | -The `supports_*` capability flags (tools, vision, reasoning) describe the deployed Qwen3.6 -family and are static in the config — revisit them if the active model swaps to a different -model family. +`supports_vision` is derived from whether the GPU server actually loads an mmproj. The +remaining `supports_*` flags (tools, reasoning) describe the llama-server invocation +(`--jinja`, `--reasoning-format`) rather than a model family, and are static in the config. The Ordo stack references it as a **project buildable image** (`ordo/model-gateway:latest`) — pinned by its build context, not pulled from a registry — so `ordo preflight` reports a missing one as diff --git a/services/model-gateway/entrypoint.sh b/services/model-gateway/entrypoint.sh index 02fd3da..e5a14cd 100644 --- a/services/model-gateway/entrypoint.sh +++ b/services/model-gateway/entrypoint.sh @@ -16,6 +16,16 @@ CPU_CTX_SIZE="${LLAMACPP_CPU_CTX:-131072}" GPU_WEIGHTS="${LLAMACPP_MODEL:-model.gguf}" CPU_WEIGHTS="${LLAMACPP_CPU_MODEL:-Qwen3.6-35B-A3B-UD-Q4_K_M.gguf}" EMBED_WEIGHTS="${LLAMACPP_EMBED_MODEL:-nomic-embed-text-v1.5.Q4_K_M.gguf}" +GPU_IMAGE="${LLAMACPP_IMAGE:-llama.cpp}" + +# The pickable pin-alias NAMES derive from the deployed weights (basename, lowercased, +# .gguf stripped) — a model swap renames them automatically, so the template never +# hardcodes a model generation. `local-chat`/`local-embed` stay stable by contract. +GPU_MODEL_NAME="$(basename "${GPU_WEIGHTS}" .gguf | tr '[:upper:]' '[:lower:]')" +CPU_MODEL_NAME="$(basename "${CPU_WEIGHTS}" .gguf | tr '[:upper:]' '[:lower:]')-cpu" + +# Vision support is a fact about the deployment (is an mmproj loaded?), not the template. +if [ -n "${LLAMACPP_MMPROJ:-}" ]; then GPU_SUPPORTS_VISION=true; else GPU_SUPPORTS_VISION=false; fi sed -e "s|__MASTER_KEY__|${MASTER_KEY}|g" \ -e "s|__CTX_SIZE__|${CTX_SIZE}|g" \ @@ -23,7 +33,11 @@ sed -e "s|__MASTER_KEY__|${MASTER_KEY}|g" \ -e "s|__CPU_CTX_SIZE__|${CPU_CTX_SIZE}|g" \ -e "s|__GPU_WEIGHTS__|${GPU_WEIGHTS}|g" \ -e "s|__CPU_WEIGHTS__|${CPU_WEIGHTS}|g" \ - -e "s|__EMBED_WEIGHTS__|${EMBED_WEIGHTS}|g" /app/config.template.yaml > /tmp/config.yaml + -e "s|__EMBED_WEIGHTS__|${EMBED_WEIGHTS}|g" \ + -e "s|__GPU_IMAGE__|${GPU_IMAGE}|g" \ + -e "s|__GPU_MODEL_NAME__|${GPU_MODEL_NAME}|g" \ + -e "s|__CPU_MODEL_NAME__|${CPU_MODEL_NAME}|g" \ + -e "s|__GPU_SUPPORTS_VISION__|${GPU_SUPPORTS_VISION}|g" /app/config.template.yaml > /tmp/config.yaml # LiteLLM's proxy callback importer (get_instance_fn in # litellm/proxy/types_utils/utils.py) resolves "module.attr" relative to the diff --git a/services/model-gateway/litellm_config.yaml b/services/model-gateway/litellm_config.yaml index a175a2c..35763cb 100644 --- a/services/model-gateway/litellm_config.yaml +++ b/services/model-gateway/litellm_config.yaml @@ -5,11 +5,12 @@ model_list: # # model_info blocks are the gateway's model documentation: everything /model/info and # /v1/models consumers see. All deployment-variable values (__CTX_SIZE__, __N_PREDICT__, - # __CPU_CTX_SIZE__, __*_WEIGHTS__) are entrypoint-substituted from the SAME .env the - # backend llama-server containers read, so this metadata cannot drift from what's running. - # The supports_* flags describe the deployed Qwen3.6 family (vision mmproj loaded on the - # GPU server, --jinja tool templates, --reasoning-format deepseek) — revisit them if the - # active model swaps to a different family. + # __CPU_CTX_SIZE__, __*_WEIGHTS__, __GPU_IMAGE__, __*_MODEL_NAME__, + # __GPU_SUPPORTS_VISION__) are entrypoint-substituted from the SAME .env the backend + # llama-server containers read, so this metadata — including the pickable pin-alias + # NAMES and the vision flag — cannot drift from what's running. The remaining static + # supports_* flags describe the llama-server invocation itself (--jinja tool templates, + # --reasoning-format), not any particular model family. - model_name: "local-chat" litellm_params: model: "openai/local-chat" @@ -28,21 +29,23 @@ model_list: supports_function_calling: true supports_tool_choice: true supports_response_schema: true - supports_vision: true + supports_vision: __GPU_SUPPORTS_VISION__ supports_reasoning: true description: >- Stable auto-routing alias — the id every Ordo consumer should use. Backed by the active GPU chat model on the RTX 5090 (see weights_file for the exact GGUF); fails - over to qwen3.6-35b-a3b-cpu when the scheduler leases the GPU to a media render, and - recovers automatically once the GPU model is healthy again. + over to the CPU deployment (__CPU_MODEL_NAME__) when the scheduler leases the GPU + to a media render, and recovers automatically once the GPU model is healthy again. weights_file: __GPU_WEIGHTS__ - served_by: llamacpp (patched qwen36-swa llama.cpp build, RTX 5090) + served_by: llamacpp (__GPU_IMAGE__, RTX 5090) # Explicit, pickable REAL models so callers/UI see WHICH model they're using, not just the - # opaque `local-chat`. qwen3.6-27b = the GPU chat model; qwen3.6-35b-a3b-cpu = the always-on - # CPU fallback (also local-chat's failover target). The CPU one is only reachable when the + # opaque `local-chat`. Their NAMES are derived from the deployed GGUFs by the entrypoint + # (never hardcode a model generation here — it goes stale on the next swap): + # __GPU_MODEL_NAME__ = the GPU chat model; __CPU_MODEL_NAME__ = the always-on CPU + # fallback (also local-chat's failover target). The CPU one is only reachable when the # opt-in `llamacpp-cpu` service is running; otherwise it just sits unavailable (harmless). - - model_name: "qwen3.6-27b" + - model_name: "__GPU_MODEL_NAME__" litellm_params: model: "openai/local-chat" api_base: "http://llamacpp:8080/v1" @@ -59,16 +62,17 @@ model_list: supports_function_calling: true supports_tool_choice: true supports_response_schema: true - supports_vision: true + supports_vision: __GPU_SUPPORTS_VISION__ supports_reasoning: true description: >- Explicit pin of the GPU chat deployment (same backend as local-chat but with NO CPU - failover routing — errors when the GPU model is evicted for a render). Qwen3.6-27B - finetune, Q6_K, hybrid attention/SSM, vision via the shared Qwen3.6-27B mmproj. + failover routing — errors when the GPU model is evicted for a render). See + weights_file for the exact GGUF; supports_vision reflects whether an mmproj is + loaded on the GPU server. weights_file: __GPU_WEIGHTS__ - served_by: llamacpp (patched qwen36-swa llama.cpp build, RTX 5090) + served_by: llamacpp (__GPU_IMAGE__, RTX 5090) - - model_name: "qwen3.6-35b-a3b-cpu" + - model_name: "__CPU_MODEL_NAME__" litellm_params: model: "openai/local-chat" api_base: "http://llamacpp-cpu:8080/v1" @@ -88,9 +92,9 @@ model_list: supports_vision: false supports_reasoning: true description: >- - Always-on CPU fallback and local-chat's failover target: Qwen3.6-35B-A3B MoE - (~3B active params) running CPU-only on stock llama.cpp. Context matches the GPU - window so failover never truncates a conversation. Only reachable while the opt-in + Always-on CPU fallback and local-chat's failover target (see weights_file for the + exact GGUF), running CPU-only on stock llama.cpp. Context matches the GPU window + so failover never truncates a conversation. Only reachable while the opt-in cpu-fallback profile (llamacpp-cpu) is running. Slow but survives GPU eviction. weights_file: __CPU_WEIGHTS__ served_by: llamacpp-cpu (stock llama.cpp, CPU-only) @@ -126,7 +130,7 @@ general_settings: # the always-on CPU A3B so chat/agents keep working instead of erroring — then recovers to the # GPU model as soon as it's healthy again (short cooldown). No effect until llamacpp-cpu runs. router_settings: - fallbacks: [{"local-chat": ["qwen3.6-35b-a3b-cpu"]}] + fallbacks: [{"local-chat": ["__CPU_MODEL_NAME__"]}] cooldown_time: 30 num_retries: 1