Complete the metric surface described in the telemetry design

[dennis-upbound]

design/telemetry.md describes a surface of 50 metrics. #470 composes the collector and the renames that produce 23 of them: the engine, GPU and gateway-latency core, everything reachable by renaming one metric to one name and converting its unit.

The other 27 each need something that doesn't exist yet. Grouped by what actually blocks them, because the groups are independent and not equally sized.

Per-object state, 11 metrics

Needs resource-state-metrics on the control plane reading Modelplane's own objects, plus kube-state-metrics for container restarts and gang status. None of these come from scraping a workload; they're the state of a ModelReplica, a ModelDeployment or an InferenceCluster.

• modelplane_replica_gpus, modelplane_replica_gpu{gpu_uuid}, modelplane_replica_allocated_time_seconds
• modelplane_replicas_desired, modelplane_replicas_ready, modelplane_replica_ready_duration_seconds
• modelplane_replica_cache_hit, modelplane_cluster_gpus_allocatable, modelplane_cluster_connected
• modelplane_gang_incomplete (LWS/Grove status), modelplane_engine_restarts_total (kube-state-metrics)

This is the largest group, and modelplane_gpu_seconds_total — the metric a cost question actually needs — is derived from modelplane_replica_allocated_time_seconds, so it's here too.

Folding several series under one label, 8 metrics

Needs MetricMapping to grow the two fields left out of #470: a labels form that folds two metrics into one distinguished by a fixed value, or renames a label and remaps its value vocabulary; and a part form that takes a histogram's count as a counter.

• modelplane_requests_total{status}, modelplane_tokens_total{direction} — fleet gateway
• modelplane_responses_total{reason}, modelplane_tool_call_parses_total{outcome} — engine
• modelplane_route_requests_total{decision}, modelplane_route_pd_pairings_total{status} — picker
• modelplane_gpu_ecc_errors_total{type}, modelplane_gpu_interconnect_errors_total{link} — DCGM, and also gated on fields that are off by default

The API shape is sketched in #470 (comment). It was left out of #470 because six of these eight depend on source metric names at the gateway, the picker and DCGM that nobody has read off a running deployment yet, and the seventh and eighth need DCGM reconfigured. modelplane_responses_total{reason} is the one implementable today, from vllm:request_success_total{finished_reason}.

First step: capture the gateway's and the picker's actual metric names from a live cluster. The API follows from those, not the other way round.

Merging replicas twice, 2 metrics

A saturation gauge needs a mean and a maximum, because capacity planning reads the mean and an alert fires on the hot replica. The collector merges replicas once today.

• modelplane_kv_cache_utilization_ratio_max, modelplane_requests_waiting_max

A pipeline change in compose-serving-stack, not an API change.

Instrumentation that doesn't emit yet, 6 metrics

• modelplane_replica_staging_seconds, modelplane_replica_warmup_seconds — ModelExpress
• modelplane_requests_throttled_total — Envoy rate limits
• modelplane_dra_allocation_errors_total — DRA driver
• modelplane_gpu_thermal_throttle_seconds_total — GPU exporter
• modelplane_request_kv_transfer_seconds — engine, disaggregated only

Order

Per-object state first: it's eleven metrics, it's the only group blocking cost questions, and it needs no live measurement to design. The label folding wants a cluster to read names off first. The double merge is small and can go any time.

[dennis-upbound]

One thing to fold in from the #470 review, raised by @lsviben: the per-object-state group above assumes resource-state-metrics runs on the control plane, but #470 composes a collector onto each inference cluster and none onto the control plane. So there is nothing there to scrape it, and that needs deciding before those 11 metrics can be built — either a collector on the control plane, or RSM exporting directly, or those metrics come from somewhere else entirely.

Also worth recording against the engine group: SGLang publishes no per-request queue time and no preemption counters at v0.4.6, v0.4.9 or v0.5.0, so modelplane_request_queue_seconds and modelplane_requests_preempted_total are vLLM-only until it does. Checked against a running engine.

[dennis-upbound]

Research pass on this, prompted by the model delivery design (design/modelcache-sources.md, on dennis/oci-source-poc) wanting four cache metrics that land in the per-object-state group. Three things have changed since this was written, and the approach is now a decision rather than an open question.

The folding group is unblocked

Needs MetricMapping to grow the two fields left out of #470

Both fields shipped in v0.5. labels and part are in apis/metricmappings/definition.yaml, and both are exercised on a live cluster: a mapping folding SGLang's two token counters gave modelplane_tokens_total{direction="input"} 398 and {direction="output"} 607, and part: Count over its latency histogram gave modelplane_engine_requests_total 12.

The first step this issue names is also done. The gateway and the picker have been read off a running GKE cluster:

• Gateway: gen_ai_server_request_duration_seconds, gen_ai_server_time_to_first_token_seconds, gen_ai_server_time_per_output_token_seconds. These are on the ext-proc sidecar's port, not the Envoy admin port — 55 gen_ai_* lines on :1064, zero on :19001.
• Picker: llm_d_epp_scheduler_e2e_duration_seconds, llm_d_epp_scheduler_attempts_total. The inference_extension_* names are deprecated twins of the same series.

So for this group what remains is DCGM's two (still gated on fields that are off by default) and writing the mappings.

The per-object-state group has a blocker this issue does not name

Every metric here reads an XR's status, and those XRs live on the control plane. Modelplane composes no collector onto the control plane. compose_collector is called only from the ServingStack path, and composes onto each inference cluster.

So the group needs two things, not one: something to turn XR status into metrics, and somewhere for it to be scraped and exported from. design/telemetry.md:166-177 already specifies both, and neither exists.

There is no control-plane-scoped XR to compose it from. ServingStack is namespaced and per-cluster, and every function composes through a provider that targets a workload cluster. So a control-plane collector is either a new XR and function, or something the install guide tells a user to run.

One piece is already reusable: TelemetryDestination is cluster-scoped on the control plane, its own function composes nothing, and its schema is plain exporter configuration. A control-plane collector could read the same list that compose_collector reads. The only friction is collector.py:540, which stamps a per-cluster cluster attribute that a control-plane collector would set differently or omit.

Two ways to get XR status into metrics

kube-state-metrics CustomResourceStateMetrics (CRS). The decisive property: metricNamePrefix replaces the whole default prefix, so metricNamePrefix: modelplane with name: replicas_ready emits modelplane_replicas_ready directly. That passes filter/modelplane with no mapping at all. It reads nested paths, emits one series per array element or map key, selects conditions by [type=Ready], coerces RFC3339 timestamps to epoch seconds and quantities like 512Gi, and has a StateSet type that fans an enum out to one 0/1 series per value — which fits ModelCache.status.clusters[].phase exactly.

- groupVersionKind: {group: modelplane.ai, version: v1alpha1, kind: ModelDeployment}
  metricNamePrefix: modelplane
  labelsFromPath: {deployment: [metadata, name], namespace: [metadata, namespace]}
  metrics:
    - name: replicas_ready
      each: {type: Gauge, gauge: {path: [status, replicas, ready], nilIsZero: true}}

Limits: Gauge, StateSet and Info only, no expressions, and no count-items-matching-a-predicate. You emit per-item 0/1 gauges and sum in PromQL. It needs explicit RBAC per custom resource, and a second ksm on the control plane, since the one in the serving stack runs on workload clusters.

resource-state-metrics (RSM), which is what design/telemetry.md already assumes. crossplane-contrib/resource-state-metrics packages it as a Crossplane Configuration (v0.7.0, pre-GA, 13 stars). Namespaced ResourceMetricsMonitor resources with CEL or Starlark resolvers, so predicate counts, timestamp arithmetic and gang completeness are expressible where CRS cannot reach them.

Its catch is the mirror image of CRS's advantage: the prefix kube_customresource_ is hardcoded in internal/family.go:44 with no override. So RSM output needs a rename before filter/modelplane. That is cheap here, because transform/modelplane runs before filter/modelplane and a MetricMapping per metric is exactly the mechanism #470 built.

Crossplane core gives nothing usable. Its metrics are function-run and controller counters; crossplane_managed_resource_{exists,ready,synced} covers managed resources, not composites.

Suggested shape

1. Decide the control-plane collector first. It blocks all 11 metrics and is the larger question, since it is the first thing Modelplane would compose or install on the control plane itself.
2. Prefer CRS for the metrics it can express, because it emits modelplane_* natively and is upstream-stable. Keep RSM for the ones needing an expression, and pay one MetricMapping rename for those.
3. modelplane_engine_restarts_total is plain kube-state-metrics (kube_pod_container_status_restarts_total) on the workload cluster, not CRS, and it is the metric #498 wants for a cause that outlives the pod.

Cannot be settled without a cluster: whether LWS and Grove expose enough status for gang_incomplete without CEL, and whether the DRA driver publishes a parent GPU UUID for modelplane_replica_gpu.

Cache metrics to add to this issue

That design, in section 6.2, adds four, all in this group and blocked by it:

• modelplane_cache_cluster_ready — a gauge for each cache and cluster, with the phase as a label
• modelplane_cache_ready_seconds — created until readable
• modelplane_cache_staged_bytes and modelplane_cache_size_bytes — progress

#494 spent 40 minutes in one phase with no series to show it, and #495 had to read staging progress from a file listing. The StateSet type covers the first of these directly.