Skip to content

feat(chart): PodMonitor for router-proxy metrics - #1473

Merged
Defilan merged 2 commits into
defilantech:mainfrom
Tanguille:feat/router-podmonitor
Aug 9, 2026
Merged

feat(chart): PodMonitor for router-proxy metrics#1473
Defilan merged 2 commits into
defilantech:mainfrom
Tanguille:feat/router-podmonitor

Conversation

@Tanguille

@Tanguille Tanguille commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

What

A PodMonitor for router-proxy pods, scraping the metrics listener #1457 adds.

Why

Fixes #1472

#1457 adds a --metrics-bind-address listener (container port metrics,
default :9090) but nothing scrapes it: inference-podmonitor.yaml selects
inference.llmkube.dev/service on a port named http, and router-proxy pods
carry neither that label nor that port name. Without this, model-router-dashboard.json
still can't populate even once #1457 merges — its own PR body says so and
deliberately left #1427 open for that reason.

How

router-podmonitor.yaml mirrors inference-podmonitor.yaml's shape:

  • Selector: inference.llmkube.dev/model-router: Exists, the stable label
    routerProxySelectorLabels (internal/controller/router_common.go:68)
    already puts on every router-proxy pod.
  • port: metrics, path: /metrics.
  • job: llmkube-router relabeling, so a future RouterProxyDown rule has a
    stable label to match — same pattern the inference/controller monitors use.
  • Gated on prometheus.routerPodMonitor.enabled (default false, matching
    inferencePodMonitor and serviceMonitor's existing default).

This PR is chart-only and depends on #1457 for the port to exist; opening as
draft until #1457 merges (port name/number final).

Assisted-by: OpenCode (helped draft this PR; I reviewed the final change and stand behind it.)

Checklist

  • Tests added/updated
  • make test passes locally
  • make lint passes locally
  • Commit messages follow conventional commits
  • All commits are signed off (git commit -s) per DCO
  • AI assistance (if any) is disclosed above, per CONTRIBUTING.md
  • Documentation updated (if user-facing change)

The metrics listener defilantech#1457 adds (container port `metrics`, default
:9090) has nothing scraping it: inference-podmonitor.yaml selects
inference.llmkube.dev/service on a port named http, and router-proxy
pods carry neither. Mirrors the inference PodMonitor's shape against
routerProxySelectorLabels' inference.llmkube.dev/model-router label.

Fixes defilantech#1472
Refs defilantech#1427, defilantech#1457

Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>
@codecov

codecov Bot commented Aug 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Tanguille
Tanguille marked this pull request as ready for review August 9, 2026 15:05
@Tanguille
Tanguille requested a review from Defilan as a code owner August 9, 2026 15:05
Sibling inference-podmonitor.yaml explains why Exists is the whole
selector (an empty-value matchLabels entry would be ANDed in and match
nothing); this file only pointed at the sibling for the reasoning
instead of stating it.

Signed-off-by: Tanguille <91473554+Tanguille@users.noreply.github.com>

@Defilan Defilan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I checked the two claims this rests on rather than taking the
description's word:

  • routerProxySelectorLabels does set inference.llmkube.dev/model-router: mr.Name, so the selector matches every router-proxy pod and nothing else.
  • #1457 does add a container port named metrics backed by
    routerProxyMetricsPort int32 = 9090, with --metrics-bind-address :9090.

Selector and port name line up, the relabel to job: llmkube-router matches
what the inference and controller monitors already do, and defaulting
enabled: false is consistent with inferencePodMonitor and serviceMonitor.

One sequencing note: this is strictly downstream of #1457. Merging it first is
harmless since it ships disabled, but until #1457 lands the PodMonitor would
select pods with no metrics port, so anyone enabling it early gets empty
targets rather than an error. I will merge #1457 first.

Good catch on #1427 too. That dashboard has been advertising "renders empty
today" for a while and this is the half that was actually missing.

@Defilan
Defilan merged commit fd7c640 into defilantech:main Aug 9, 2026
24 checks passed
@github-actions github-actions Bot mentioned this pull request Aug 9, 2026
Defilan added a commit to Defilan/LLMKube that referenced this pull request Aug 10, 2026
The endpoint served the Prometheus default registry while every llmkube
collector registers into controller-runtime's, so it exposed none of the series
it exists for. That shipped in defilantech#1457 and was fixed in defilantech#1393; nothing stops it
regressing again.

The failure is silent, which is the reason it needs a test rather than care.
The scrape returns HTTP 200 with Go runtime and process metrics, the PodMonitor
from defilantech#1473 reports a healthy target, and the dashboards are simply empty.

Touching a real counter before scraping is load-bearing: a labelled collector
emits no family until it has a child series, so an idle scrape is identical
under either registry and the obvious version of this test passes on the bug.
Verified by reintroducing promhttp.Handler() and confirming the test fails,
rather than assuming a new test must catch something.

newMetricsHandler() is extracted from main() only so there is something to
assert against.

Co-authored-by: Sylvain Niles <540991+sylvainsf@users.noreply.github.com>
Signed-off-by: Christopher Maher <chris@mahercode.io>
doonga pushed a commit to greyrock-labs/home-ops that referenced this pull request Aug 10, 2026
…mkube (0.9.14 ➔ 0.9.16) (#294)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [ghcr.io/home-operations/charts-mirror/llmkube](https://github.com/defilantech/LLMKube) | patch | `0.9.14` → `0.9.16` |

---

### Release Notes

<details>
<summary>defilantech/LLMKube (ghcr.io/home-operations/charts-mirror/llmkube)</summary>

### [`v0.9.16`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0916-2026-08-10)

[Compare Source](defilantech/LLMKube@v0.9.15...v0.9.16)

##### Features

- **chart:** PodMonitor for router-proxy metrics ([#&#8203;1473](defilantech/LLMKube#1473)) ([fd7c640](defilantech/LLMKube@fd7c640))
- **foreman:** advisory two-site parity signal in the coder gate ([#&#8203;1456](defilantech/LLMKube#1456)) ([944f90d](defilantech/LLMKube@944f90d))
- **foreman:** image input, so a vision-capable coder can see what it rendered ([#&#8203;1471](defilantech/LLMKube#1471)) ([6c122be](defilantech/LLMKube@6c122be))
- **foreman:** read-only fetch\_pull\_request tool ([#&#8203;1458](defilantech/LLMKube#1458)) ([7f4910b](defilantech/LLMKube@7f4910b))
- **modelpool:** cross-pod sticky model swapping on a shared GPU slot ([#&#8203;1393](defilantech/LLMKube#1393)) ([ba7777d](defilantech/LLMKube@ba7777d))
- **router-proxy:** expose llmkube\_router\_\* metrics on a dedicated listener ([#&#8203;1457](defilantech/LLMKube#1457)) ([ca3730d](defilantech/LLMKube@ca3730d))

##### Bug Fixes

- **controller:** make the custom CA additive to the system trust store ([#&#8203;1469](defilantech/LLMKube#1469)) ([d490059](defilantech/LLMKube@d490059))
- **controller:** stage files and mmproj from an s3:// object store ([#&#8203;1475](defilantech/LLMKube#1475)) ([769b60d](defilantech/LLMKube@769b60d))
- **dashboard:** AMD dashboard vendor hygiene — remove runtime-agnostic llama.cpp panels, standardize on DS\_PROMETHEUS ([#&#8203;1446](defilantech/LLMKube#1446)) ([c6c0670](defilantech/LLMKube@c6c0670))
- **dashboard:** retarget amd-gpu-observability off amdgpu\_\*-only queries ([#&#8203;1474](defilantech/LLMKube#1474)) ([a02e8e8](defilantech/LLMKube@a02e8e8))

### [`v0.9.15`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0915-2026-08-08)

[Compare Source](defilantech/LLMKube@v0.9.14...v0.9.15)

##### Features

- **chart:** expose --ca-cert-configmap as controllerManager.caCertConfigMap ([#&#8203;1440](defilantech/LLMKube#1440)) ([606c7f9](defilantech/LLMKube@606c7f9))
- **controller:** diagnose CUDA driver/runtime mismatch from crashed pods ([#&#8203;1425](defilantech/LLMKube#1425)) ([06670f9](defilantech/LLMKube@06670f9))
- **controller:** diagnose model-transfer failures from init containers ([#&#8203;1463](defilantech/LLMKube#1463)) ([a092de3](defilantech/LLMKube@a092de3))
- **foreman:** advisory gate for command-string changes tested only by shape ([#&#8203;1421](defilantech/LLMKube#1421)) ([b580e18](defilantech/LLMKube@b580e18))
- **inferenceservice:** add spec.modelCache.persistence for an ephemeral model cache ([#&#8203;1452](defilantech/LLMKube#1452)) ([c4ae7ee](defilantech/LLMKube@c4ae7ee))

##### Bug Fixes

- **api:** reject unservable speculativeDecoding type draft at admission ([#&#8203;1455](defilantech/LLMKube#1455)) ([5d3ab73](defilantech/LLMKube@5d3ab73))
- **chart:** default the router-proxy image tag to the chart appVersion ([#&#8203;1431](defilantech/LLMKube#1431)) ([3e07b81](defilantech/LLMKube@3e07b81))
- **controller:** clean up orphaned .tmp files from interrupted model transfers ([#&#8203;1459](defilantech/LLMKube#1459)) ([2f453d6](defilantech/LLMKube@2f453d6))
- **controller:** emit draft-simple for speculativeDecoding type draft ([#&#8203;1417](defilantech/LLMKube#1417)) ([ccccd02](defilantech/LLMKube@ccccd02))
- **controller:** publish model downloads atomically so an interrupted transfer is never cached ([#&#8203;1432](defilantech/LLMKube#1432)) ([206b6c3](defilantech/LLMKube@206b6c3))
- **controller:** route s3:// model sources to the runtime-resolved path ([#&#8203;1450](defilantech/LLMKube#1450)) ([f054960](defilantech/LLMKube@f054960))
- **controller:** set TerminationMessagePolicy on generated init containers ([#&#8203;1460](defilantech/LLMKube#1460)) ([bc732b3](defilantech/LLMKube@bc732b3))
- **dashboard:** select the latency success series by le!="" so llmkube-slo error-budget panels render for any threshold ([#&#8203;1444](defilantech/LLMKube#1444)) ([d381112](defilantech/LLMKube@d381112))
- **foreman:** add chart validation to the coder gate, and install helm so it can run ([#&#8203;1442](defilantech/LLMKube#1442)) ([b26c6ae](defilantech/LLMKube@b26c6ae))
- **foreman:** detect assertion-value churn in the test-dilution gate ([#&#8203;1416](defilantech/LLMKube#1416)) ([b53df14](defilantech/LLMKube@b53df14))
- **foreman:** preserve the coder Job name's uniqueness suffix on truncation ([#&#8203;1412](defilantech/LLMKube#1412)) ([929f0a8](defilantech/LLMKube@929f0a8))
- ground the PR body summary against the branch diff ([#&#8203;1448](defilantech/LLMKube#1448)) ([6f24527](defilantech/LLMKube@6f24527))
- **metrics:** correct the AMD doc's metric contract and guard prose docs ([#&#8203;1420](defilantech/LLMKube#1420)) ([36631a9](defilantech/LLMKube@36631a9))
- **runtime:** disable the llama.cpp prompt cache for embedding and rerank ([#&#8203;1413](defilantech/LLMKube#1413)) ([407de59](defilantech/LLMKube@407de59))

##### Documentation

- **b200:** record GB10 partial-proxy reachability and why NVLink/MIG/sm\_100 stay gated ([#&#8203;1422](defilantech/LLMKube#1422)) ([e6ab2e9](defilantech/LLMKube@e6ab2e9))
- **chart:** dashboard datasources examples must use the datasource UID, not its display name ([#&#8203;1433](defilantech/LLMKube#1433)) ([a57d499](defilantech/LLMKube@a57d499))

</details>

---

### Configuration

📅 **Schedule**: (in timezone America/New_York)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC43LjQiLCJ1cGRhdGVkSW5WZXIiOiI0NC43LjQiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbInJlbm92YXRlL2NvbnRhaW5lciIsInR5cGUvcGF0Y2giXX0=-->

Reviewed-on: https://git.greyrock.io/greyrock-labs/home-ops/pulls/294
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] PodMonitor for router-proxy metrics (closes the gap #1457 leaves open on #1427)

2 participants