throughput: authoritative model-state tracking (real-GGUF attribution, timestamped store, gateway-recreate drift fix) - #182
Merged
Conversation
…hroughput store Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… clean reset of alias-conflated v1 data Also stops /api/throughput/benchmark double-recording under the requested alias (the gateway callback already records every completion). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_throughput_record_accepts_alias_and_backend previously only checked the POST returned ok and the model appeared in /stats — both true even without alias/backend support (pydantic silently drops unknown fields). Now asserts the recorded _service_usage event carries the real alias and backend values, so a future refactor can't silently drop attribution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…el from ops /model-config Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…urrent ops-controller calls Fixes a check-then-act race in _throughput_active_model: two concurrent /stats requests landing after TTL expiry could both pass the staleness check and both call ops-controller. Guards the fetch path with _active_model_fetch_lock and re-checks staleness after acquiring so the second waiter reuses the first's result, restoring "one upstream call per TTL window". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nge, not only ctx A model swap without a ctx change left the gateway advertising the OLD model (stale GGUF-derived pin-aliases, weights_file, vision flag) - violating its "cannot drift from what's running" contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… served, not the alias The local-chat bucket conflated every model ever active behind it and mixed CPU-failover tok/s into GPU percentiles. Resolution uses the served deployment model_info.weights_file the router attaches to kwargs (verified against the pinned LiteLLM v1.82.3 source), with the legacy naming as fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…heuristics deleted Honest states replace inference: "no traffic yet" for an idle active model, "control plane unreachable" when ops is down, and a recent-models history list instead of mislabeling stale models Active. Benchmark targets the active model's pin-alias. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s derivation Final-review fix wave for the throughput/model-state tab: - Distinguish dashboard-API failure from control-plane unreachability from reachable-but-unconfigured, instead of collapsing all three into a single null active_model that blamed the control plane for the wrong thing. - Derive the gateway pin-alias server-side once (_gateway_pin_alias) and serve it as active_model_alias so the UI stops re-implementing it. - Reduce the ops /model-config timeout to 3s (the negative cache absorbs failures) and persist the v1->v2 store reset so idle dashboards don't re-reset and re-WARN on every restart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The throughput tab guessed "the active model" from sample recency with a lifetime-count fallback, over a store that keyed samples by routing ALIAS (one local-chat bucket conflating every model ever active behind it, CPU-failover tok/s included), kept no timestamps, and never evicted. Verified live before this change: sample keys were local-chat / test-model (a literal pytest payload that leaked into prod state) / a long-retired model.
Fix (root causes, per layer)
Validation
Deployed to the live stack and validated E2E: a completion via local-chat landed keyed by the real GGUF (Qwen3.8-27B-Uncensored-Q6_K.gguf) with a fresh timestamp while the gateway response still said local-chat; the v1 store clean-reset was observed in logs; stats active_model matches ops /model-config; the served bundle renders it; benchmark via the pin-alias attributes to the GGUF. Full suite: 644 passed, only the 5 known pre-existing failures (operator catalog WIP + local container state). Whole-branch review + scoped re-review clean.
🤖 Generated with Claude Code