feat(chart): PodMonitor for router-proxy metrics - #1473
Conversation
The metrics listener defilantech#1457 adds (container port `metrics`, default :9090) has nothing scraping it: inference-podmonitor.yaml selects inference.llmkube.dev/service on a port named http, and router-proxy pods carry neither. Mirrors the inference PodMonitor's shape against routerProxySelectorLabels' inference.llmkube.dev/model-router label. Fixes defilantech#1472 Refs defilantech#1427, defilantech#1457 Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Sibling inference-podmonitor.yaml explains why Exists is the whole selector (an empty-value matchLabels entry would be ANDed in and match nothing); this file only pointed at the sibling for the reasoning instead of stating it. Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
Defilan
left a comment
There was a problem hiding this comment.
Approving. I checked the two claims this rests on rather than taking the
description's word:
routerProxySelectorLabelsdoes setinference.llmkube.dev/model-router: mr.Name, so the selector matches every router-proxy pod and nothing else.- #1457 does add a container port named
metricsbacked by
routerProxyMetricsPort int32 = 9090, with--metrics-bind-address :9090.
Selector and port name line up, the relabel to job: llmkube-router matches
what the inference and controller monitors already do, and defaulting
enabled: false is consistent with inferencePodMonitor and serviceMonitor.
One sequencing note: this is strictly downstream of #1457. Merging it first is
harmless since it ships disabled, but until #1457 lands the PodMonitor would
select pods with no metrics port, so anyone enabling it early gets empty
targets rather than an error. I will merge #1457 first.
Good catch on #1427 too. That dashboard has been advertising "renders empty
today" for a while and this is the half that was actually missing.
The endpoint served the Prometheus default registry while every llmkube collector registers into controller-runtime's, so it exposed none of the series it exists for. That shipped in defilantech#1457 and was fixed in defilantech#1393; nothing stops it regressing again. The failure is silent, which is the reason it needs a test rather than care. The scrape returns HTTP 200 with Go runtime and process metrics, the PodMonitor from defilantech#1473 reports a healthy target, and the dashboards are simply empty. Touching a real counter before scraping is load-bearing: a labelled collector emits no family until it has a child series, so an idle scrape is identical under either registry and the obvious version of this test passes on the bug. Verified by reintroducing promhttp.Handler() and confirming the test fails, rather than assuming a new test must catch something. newMetricsHandler() is extracted from main() only so there is something to assert against. Co-authored-by: Sylvain Niles <540991+sylvainsf@users.noreply.github.com> Signed-off-by: Christopher Maher <chris@mahercode.io>
…mkube (0.9.14 ➔ 0.9.16) (#294) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/home-operations/charts-mirror/llmkube](https://github.com/defilantech/LLMKube) | patch | `0.9.14` → `0.9.16` | --- ### Release Notes <details> <summary>defilantech/LLMKube (ghcr.io/home-operations/charts-mirror/llmkube)</summary> ### [`v0.9.16`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0916-2026-08-10) [Compare Source](defilantech/LLMKube@v0.9.15...v0.9.16) ##### Features - **chart:** PodMonitor for router-proxy metrics ([#​1473](defilantech/LLMKube#1473)) ([fd7c640](defilantech/LLMKube@fd7c640)) - **foreman:** advisory two-site parity signal in the coder gate ([#​1456](defilantech/LLMKube#1456)) ([944f90d](defilantech/LLMKube@944f90d)) - **foreman:** image input, so a vision-capable coder can see what it rendered ([#​1471](defilantech/LLMKube#1471)) ([6c122be](defilantech/LLMKube@6c122be)) - **foreman:** read-only fetch\_pull\_request tool ([#​1458](defilantech/LLMKube#1458)) ([7f4910b](defilantech/LLMKube@7f4910b)) - **modelpool:** cross-pod sticky model swapping on a shared GPU slot ([#​1393](defilantech/LLMKube#1393)) ([ba7777d](defilantech/LLMKube@ba7777d)) - **router-proxy:** expose llmkube\_router\_\* metrics on a dedicated listener ([#​1457](defilantech/LLMKube#1457)) ([ca3730d](defilantech/LLMKube@ca3730d)) ##### Bug Fixes - **controller:** make the custom CA additive to the system trust store ([#​1469](defilantech/LLMKube#1469)) ([d490059](defilantech/LLMKube@d490059)) - **controller:** stage files and mmproj from an s3:// object store ([#​1475](defilantech/LLMKube#1475)) ([769b60d](defilantech/LLMKube@769b60d)) - **dashboard:** AMD dashboard vendor hygiene — remove runtime-agnostic llama.cpp panels, standardize on DS\_PROMETHEUS ([#​1446](defilantech/LLMKube#1446)) ([c6c0670](defilantech/LLMKube@c6c0670)) - **dashboard:** retarget amd-gpu-observability off amdgpu\_\*-only queries ([#​1474](defilantech/LLMKube#1474)) ([a02e8e8](defilantech/LLMKube@a02e8e8)) ### [`v0.9.15`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0915-2026-08-08) [Compare Source](defilantech/LLMKube@v0.9.14...v0.9.15) ##### Features - **chart:** expose --ca-cert-configmap as controllerManager.caCertConfigMap ([#​1440](defilantech/LLMKube#1440)) ([606c7f9](defilantech/LLMKube@606c7f9)) - **controller:** diagnose CUDA driver/runtime mismatch from crashed pods ([#​1425](defilantech/LLMKube#1425)) ([06670f9](defilantech/LLMKube@06670f9)) - **controller:** diagnose model-transfer failures from init containers ([#​1463](defilantech/LLMKube#1463)) ([a092de3](defilantech/LLMKube@a092de3)) - **foreman:** advisory gate for command-string changes tested only by shape ([#​1421](defilantech/LLMKube#1421)) ([b580e18](defilantech/LLMKube@b580e18)) - **inferenceservice:** add spec.modelCache.persistence for an ephemeral model cache ([#​1452](defilantech/LLMKube#1452)) ([c4ae7ee](defilantech/LLMKube@c4ae7ee)) ##### Bug Fixes - **api:** reject unservable speculativeDecoding type draft at admission ([#​1455](defilantech/LLMKube#1455)) ([5d3ab73](defilantech/LLMKube@5d3ab73)) - **chart:** default the router-proxy image tag to the chart appVersion ([#​1431](defilantech/LLMKube#1431)) ([3e07b81](defilantech/LLMKube@3e07b81)) - **controller:** clean up orphaned .tmp files from interrupted model transfers ([#​1459](defilantech/LLMKube#1459)) ([2f453d6](defilantech/LLMKube@2f453d6)) - **controller:** emit draft-simple for speculativeDecoding type draft ([#​1417](defilantech/LLMKube#1417)) ([ccccd02](defilantech/LLMKube@ccccd02)) - **controller:** publish model downloads atomically so an interrupted transfer is never cached ([#​1432](defilantech/LLMKube#1432)) ([206b6c3](defilantech/LLMKube@206b6c3)) - **controller:** route s3:// model sources to the runtime-resolved path ([#​1450](defilantech/LLMKube#1450)) ([f054960](defilantech/LLMKube@f054960)) - **controller:** set TerminationMessagePolicy on generated init containers ([#​1460](defilantech/LLMKube#1460)) ([bc732b3](defilantech/LLMKube@bc732b3)) - **dashboard:** select the latency success series by le!="" so llmkube-slo error-budget panels render for any threshold ([#​1444](defilantech/LLMKube#1444)) ([d381112](defilantech/LLMKube@d381112)) - **foreman:** add chart validation to the coder gate, and install helm so it can run ([#​1442](defilantech/LLMKube#1442)) ([b26c6ae](defilantech/LLMKube@b26c6ae)) - **foreman:** detect assertion-value churn in the test-dilution gate ([#​1416](defilantech/LLMKube#1416)) ([b53df14](defilantech/LLMKube@b53df14)) - **foreman:** preserve the coder Job name's uniqueness suffix on truncation ([#​1412](defilantech/LLMKube#1412)) ([929f0a8](defilantech/LLMKube@929f0a8)) - ground the PR body summary against the branch diff ([#​1448](defilantech/LLMKube#1448)) ([6f24527](defilantech/LLMKube@6f24527)) - **metrics:** correct the AMD doc's metric contract and guard prose docs ([#​1420](defilantech/LLMKube#1420)) ([36631a9](defilantech/LLMKube@36631a9)) - **runtime:** disable the llama.cpp prompt cache for embedding and rerank ([#​1413](defilantech/LLMKube#1413)) ([407de59](defilantech/LLMKube@407de59)) ##### Documentation - **b200:** record GB10 partial-proxy reachability and why NVLink/MIG/sm\_100 stay gated ([#​1422](defilantech/LLMKube#1422)) ([e6ab2e9](defilantech/LLMKube@e6ab2e9)) - **chart:** dashboard datasources examples must use the datasource UID, not its display name ([#​1433](defilantech/LLMKube#1433)) ([a57d499](defilantech/LLMKube@a57d499)) </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC43LjQiLCJ1cGRhdGVkSW5WZXIiOiI0NC43LjQiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbInJlbm92YXRlL2NvbnRhaW5lciIsInR5cGUvcGF0Y2giXX0=--> Reviewed-on: https://git.greyrock.io/greyrock-labs/home-ops/pulls/294
What
A
PodMonitorfor router-proxy pods, scraping the metrics listener #1457 adds.Why
Fixes #1472
#1457 adds a
--metrics-bind-addresslistener (container portmetrics,default
:9090) but nothing scrapes it:inference-podmonitor.yamlselectsinference.llmkube.dev/serviceon a port namedhttp, and router-proxy podscarry neither that label nor that port name. Without this,
model-router-dashboard.jsonstill can't populate even once #1457 merges — its own PR body says so and
deliberately left #1427 open for that reason.
How
router-podmonitor.yamlmirrorsinference-podmonitor.yaml's shape:inference.llmkube.dev/model-router: Exists, the stable labelrouterProxySelectorLabels(internal/controller/router_common.go:68)already puts on every router-proxy pod.
port: metrics,path: /metrics.job: llmkube-routerrelabeling, so a futureRouterProxyDownrule has astable label to match — same pattern the inference/controller monitors use.
prometheus.routerPodMonitor.enabled(defaultfalse, matchinginferencePodMonitorandserviceMonitor's existing default).This PR is chart-only and depends on #1457 for the port to exist; opening as
draft until #1457 merges (port name/number final).
Assisted-by: OpenCode (helped draft this PR; I reviewed the final change and stand behind it.)
Checklist
make testpasses locallymake lintpasses locallygit commit -s) per DCO