feat(modelpool): cross-pod sticky model swapping on a shared GPU slot

[sylvainsf]

What

Adds ModelPool, a CRD plus controller that lets several single-model InferenceServices share one exclusive GPU slot. At most one member is Ready at a time; the controller drains and unloads the incumbent (freeing VRAM) before the next member loads. The ModelRouter proxy fronts the members, activates a stopped member on demand, and enforces a sticky, anti-thrash swap policy across the slot.

Four commits:

1. feat(modelpool): add ModelPool CRD and exclusive-slot reconciler
2. feat(router): activate ModelPool members with sticky idle-gated swapping
3. feat(modelpool): decouple swap hold budget and self-heal residency
4. feat(router): pin the router-proxy with spec.proxy.nodeSelector

Why

Fixes #1111.
Fixes #1123.
Refs #516.

ModelPool is tracked as its own feature issue (#1111), distinct from the in-process runtime: llamacpp-router shape in #516. ModelPool is the cross-pod shape: N independent single-model services sharing one exclusive GPU slot, so LLMKube owns the queue and swap policy in Go and the anti-thrash behavior needs no llama.cpp change. Members can also diverge in runtime (different images, context sizes, vLLM vs llama.cpp). The two shapes compose: a ModelRouter can front both transparently.

On a single-GPU node only one model can be resident, and there was no safe primitive to switch models on demand. ModelPool provides scale-from-zero on request, a VRAM-gated exclusive slot, and a sticky policy that will not thrash under mixed demand. The residency self-heal in this PR also closes the stale-resident activator bug (#1123): a member whose pod is gone no longer keeps serving stale 502s until a proxy restart.

Priority-reclaim swap mode and metal-backed members are deliberately deferred and tracked in #1394.

How

Design notes (full write-up in docs/proposals/516-modelpool-cross-pod-swapping.md):

• Exclusive slot via Kubernetes device-plugin gating: a busy incumbent is never scaled down, and the displaced member stays Pending until the incumbent releases the device. v1 is Kubernetes-GPU-only; metal-backed members are refused with MetalSupported=False.
• Drain-before-unload reuses the runtime idle-probe abstraction (IdleDetector.IdleProbe, /slots for llama.cpp): an unreachable idle check fails closed (the incumbent stays resident). There is no controller-side force-unload; the only bound on a waiting request is the router hold budget, surfaced as 503 plus Retry-After (never 429).
• The activator runs swaps under a base context, not the caller context, so a client disconnecting mid-swap never aborts an in-progress load. Cross-model requests are held open as real connection state; same-model demand coalesces onto the warm member to avoid GPU ping-pong.
• spec.swapBudget (default 300s) bounds the swap hold independently of the response-header (generation) timeout, so a large cold load does not force operators to inflate the generation cap for every request. The dispatch clock starts only after the member is resident.
• Residency self-heal: the activator re-verifies its cached resident at a bounded interval and on a connection-level dispatch failure, so a member whose pod is gone does not keep serving stale 502s until a proxy restart.
• Least-privilege activation RBAC: the operator provisions a per-router ServiceAccount, Role, and RoleBinding (get/list/watch/update/patch on InferenceServices) owner-referenced to the ModelRouter; the proxy runs with --enable-activation only when the router actually fronts pools.
• spec.proxy.nodeSelector pins the router-proxy pod, which fixes a node-local-registry setup where the proxy scheduled onto a node that lacked its image.

Metrics

New ModelPool observability, exported by the router-proxy at :8081/metrics (registered via the shared AllCollectors list so nothing is silently dropped):

• llmkube_modelpool_resident{namespace,pool,member}: which member owns the slot (1 resident, 0 not).
• llmkube_modelpool_swaps_total{router,pool,from,to}: completed slot swaps by direction.
• llmkube_modelpool_swap_duration_seconds{router,pool}: incumbent unload plus target load.
• llmkube_modelpool_hold_duration_seconds{router,pool,member}: how long a request was held open waiting for activation.
• llmkube_modelpool_coalesced_total{router,pool,member}: requests served on the warm member without a swap.
• llmkube_modelpool_held_requests{router,pool,member}: requests currently held open per member.

Testing

Cluster validation (author, on the ModelPool functionality)

Environment: k3s v1.36.2+k3s1, two nodes (helpy control-plane; corsair agent, Ubuntu 24.04.4, kernel 6.17-oem). Single AMD Radeon 8050S iGPU on corsair (gfx1151 / Strix Halo), served via the operator's Vulkan image ghcr.io/defilantech/llmkube-llama-vulkan scheduled against devices.es/dri-render (generic-device-plugin, count 1), not amd.com/gpu or NVIDIA. Node label lab.sylvain.dev/inference=heavy on corsair. Models pre-staged on a local NVMe hostPath PVC (pvc://), no download.

Live-tested commit: ad4d8e0 (the pre-rebase "decouple swap hold budget and self-heal residency" tip). Built controller and router-proxy with podman, imported into containerd via k3s ctr images import, rolled the controller-manager onto the new image, and applied the branch's ModelPool CRD (the live CRD predated swapBudget).

ModelPool heavy-slot (namespace lab, gpu: 1, swapPolicy: sticky, default: coder, nodeSelector: {lab.sylvain.dev/inference: heavy}, swapBudget: 300s) with two Vulkan members on the single GPU:

• coder = Qwen3.6-27B-Q6_K (about 21 GB)
• gemma-longctx = gemma-4-31B-it-Q4_K_M (about 18 GB)

Fronted by a ModelRouter (defaultRouteStrategy: BackendNameMatch); proxy and per-backend responseHeaderTimeout initially 60s, later dropped to 15s for the decoupling proof.

Results:

• swapBudget compiles into the proxy: the generated router-proxy ConfigMap showed "swapBudget": 300000000000 (300s) per backend pool alongside the backend "timeout", confirming ModelPool.spec.swapBudget reaches BackendPool.SwapBudget, decoupled from the dispatch timeout.
• Cold-pool convergence and on-demand activation: coder warmed to Ready, gemma-longctx held Stopped, pool Ready with residentMember=coder.
• Cross-model swap and drain-before-unload: requesting gemma-longctx drained coder to Stopped, then loaded gemma-longctx to Ready. First run (60s timeout): HTTP 200 in 37s, server swap_duration about 34s.
• swapBudget decoupling (the definitive test): dropping the dispatch timeout to 15s (below the about 34s swap) and triggering coder -> gemma-longctx returned HTTP 200 in 35.2s with swap_duration about 32s. A 32s hold succeeding under a 15s generation cap is only possible because swapBudget (300s) governs the hold; under the old coupling that request would 503 at 15s.
• Never-two-resident invariant: member phases never showed both Ready during or after swaps; the residency gauge showed exactly one resident.
• Metrics scraped from the proxy pod IP :8081/metrics: llmkube_modelpool_swaps_total{from="coder",to="gemma-longctx"} 1, swap_duration_seconds sum about 32-34s (count 1), resident{member="coder"} 0 and resident{member="gemma-longctx"} 1; the full family (swaps_total, swap_duration, resident, coalesced_total, hold_duration, held_requests) was present.
• Activation RBAC: the operator provisioned the proxy's ServiceAccount/Role/RoleBinding and the proxy Deployment came up with --enable-activation (after the RBAC note below).
• Sticky: the pool held the resident member until a different member was requested; repeats did not swap.

Issues hit and worked around during the live run:

• Stale operator ClusterRole: the deployed manager-role predated the activation rules, so the proxy Deployment was never created (operator logged serviceaccounts is forbidden ... cannot list). Applying the branch's updated config/rbac/role.yaml and restarting the operator fixed it. A fresh install from this branch's chart provisions it correctly.
• Proxy placement: ad4d8e0 had no proxy placement field, so the proxy scheduled onto helpy, which lacked the node-local image (ImagePullBackOff). Cordoning helpy rescheduled it to corsair. This is exactly the gap spec.proxy.nodeSelector (commit 4) closes.

Not live-exercised (verified at the code/test level only, stated to avoid overclaiming):

• spec.proxy.nodeSelector pinning: implemented in commit 4, verified via build/vet/tests and generated CRD/deepcopy match, but not run on the cluster (the live run used the helpy-cordon workaround instead).
• The [BUG] ModelRouter activator serves a stale resident member and never self-heals after an out-of-band residency change (instant 502s until proxy restart) #1123 stale-resident self-heal: present and covered by unit/controller tests, but no deliberate out-of-band residency change was staged live; the live swaps completed without stale-502 symptoms.

Local validation (on the rebased branch)

This branch was rebased onto current upstream/main (0 behind), resolving the conflicts from upstream's idle-check refactor (the ModelPool drain now routes through IdleDetector.IdleProbe instead of a private /slots helper) and the deployment-update path (merged upstream's suspension-aware HPA handling with the retry-on-conflict update that absorbs concurrent ModelPool replica scaling). Manifests were regenerated with the pinned controller-gen v0.19.0 (this also dropped stale minResidency / priority drift from the ModelPool CRD) and the CRDs and ClusterRole were synced into the Helm chart.

On the rebased tip:

• go build ./... (linux): clean
• go vet ./... (linux): clean
• golangci-lint run v2.12.2 on the changed packages (internal/router/..., internal/metrics/..., internal/controller/..., api/v1alpha1/...): 0 issues
• go test ./internal/router/...: pass (about 122s), covering the activator sticky/coalesce/cancel-on-timeout paths, the swapBudget decoupling test, and the [BUG] ModelRouter activator serves a stale resident member and never self-heals after an out-of-band residency change (instant 502s until proxy restart) #1123 self-heal test
• The envtest controller suite (go test ./internal/controller/...) was not re-run on the rebase host (Windows, no envtest binaries); it passed in the author's lab on the pre-rebase code. Reviewers on Linux CI will exercise it.

Checklist

• Tests added/updated
• make lint passes (golangci-lint v2.12.2, 0 issues on the changed packages)
• Build and vet pass; router unit tests pass (envtest controller suite runs in CI, see Testing)
• Commit messages follow conventional commits
• All commits are signed off per DCO (author identity matches sign-off)
• Documentation updated (design proposal under docs/proposals/)

[codecov]

Codecov Report

❌ Patch coverage is 67.57050% with 299 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
api/v1alpha1/zz_generated.deepcopy.go	0.00%	94 Missing and 1 partial ⚠️
internal/controller/modelpool_controller.go	78.23%	54 Missing and 20 partials ⚠️
cmd/router-proxy/main.go	0.00%	48 Missing ⚠️
internal/controller/router_activation_rbac.go	75.90%	10 Missing and 10 partials ⚠️
internal/router/activator.go	90.25%	8 Missing and 7 partials ⚠️
internal/controller/modelrouter_controller.go	54.16%	9 Missing and 2 partials ⚠️
internal/controller/router_config_builder.go	71.79%	8 Missing and 3 partials ⚠️
cmd/main.go	0.00%	7 Missing ⚠️
internal/router/proxy.go	81.81%	3 Missing and 3 partials ⚠️
internal/controller/router_deployment_builder.go	82.60%	3 Missing and 1 partial ⚠️
... and 3 more

📢 Thoughts on this report? Let us know!

[Defilan]

Thanks for this, and apologies for the slow first pass. The shape is good: the RBAC is a namespaced Role rather than a ClusterRole, scoped to inferenceservices, bound to a per-router ServiceAccount and owner-referenced so it garbage-collects with the ModelRouter. CRD, chart CRD, and RBAC are all in sync, and the test coverage is substantial.

One thing I want to resolve before merging: activation does not appear to be coordinated across proxy replicas.

Activator.mu is a single in-process mutex, and spec.proxy.replicas is user-settable with nothing constraining it when a pool is attached (replicas := int32(1) in newRouterDeployment is only a default). The member writes in activator_k8s.go use client.Patch(ctx, patched, client.MergeFrom(cur)) without WithOptimisticLock, so two proxies racing to activate different members do not conflict; the later write silently wins.

The controller-side exclusive-slot invariant still holds, so this is not corruption and two members never go resident at once. The cost is thrash: at two or more replicas, proxies receiving requests for different members each drive a swap, and every flip is a full drain plus a multi-gigabyte reload. That is the GPU ping-pong the coalescing logic exists to prevent, reappearing at replicas >= 2, with nothing in status or events to indicate it is happening.

The Activator doc comment says all swaps for a pool are serialized so at most one member is ever made resident at a time, which a reader would reasonably take as a cluster-wide guarantee rather than a per-process one.

Two resolutions I would be happy with:

1. Validation. Reject or pin spec.proxy.replicas to 1 while a ModelPool references the router, and state the constraint in the CRD field docs. Smallest change, and honest about what the current design guarantees.
2. Real single-writer. client.MergeFromWithOptions(cur, client.MergeFromWithOptimisticLock{}) so competing activations conflict instead of silently last-write-winning, plus a lease so one replica owns swap decisions for a pool.

Which did you have in mind? If it is 1, that is fine for this PR and 2 can be a follow-up issue.

[sylvainsf]

Thanks for the thorough pass. I went with option 1 (pin) for this PR and filed #1477 to track option 2 (the real cross-replica single-writer).

What changed:

• The operator now pins spec.proxy.replicas to 1 whenever a router has pooled backends, in newRouterDeployment where hasPools is already known. I put it there rather than in the ModelRouter validating webhook because that webhook is pure-spec-only by design (pool resolution needs a cluster lookup, which its doc deliberately keeps reconciler-side).
• It logs the override at reconcile time when a user asked for more than 1, and the constraint is documented on the CRD field (RouterProxySpec.Replicas), regenerated into the base CRD and the chart.
• Test: TestNewRouterDeploymentPinsReplicasWhenPooled (pooled pins to 1 even with replicas: 3; unpooled honors it).

You are right that the Activator doc comment overreached by implying a cluster-wide guarantee. With the pin it is now an accurate per-process guarantee for the only topology the operator allows.

#1477 covers the real fix: MergeFromWithOptimisticLock member writes so competing activations conflict instead of last-write-winning, plus a Lease so one replica owns swap decisions for a pool, after which the replicas=1 pin can be lifted.

Also rebased onto latest main to clear the merge conflict. Worth flagging one resolution: the router-proxy metrics server collided with the new --metrics-bind-address work that landed on main. I resolved to main's flag name and port (9090), but kept the handler serving ctrlmetrics.Registry rather than the default registry, since that is where the router and ModelPool collectors actually register (internal/metrics init) and the default-registry handler would expose none of them.

[Defilan]

Option 1 is the right call for this PR, and #1477 is the right place for the
rest. Approving.

I verified each of the four claims rather than taking them:

• The pin is real and in the right place: if hasPools { replicas = 1 } in
  newRouterDeployment, with the override logged at reconcile time when a user
  asked for more. Putting it there rather than in the validating webhook is
  correct; pool resolution needs a cluster lookup and that webhook is
  deliberately pure-spec.
• TestNewRouterDeploymentPinsReplicasWhenPooled asserts both directions,
  pooled pinning to 1 with replicas: 3 set and unpooled still honouring 3.
  The negative half is what makes it a real test.
• The CRD field doc carries the constraint and it is regenerated into both the
  base CRD and the chart.
• The drain-path conflict was resolved by adapting memberIdle to main's newer
  IdleDetector interface rather than taking either side wholesale. That is the
  resolution I was most concerned about and it preserves the fail-closed
  behaviour.

On the metrics registry: you found a live bug on main, and you are right.

I checked this empirically because it is exactly the kind of thing that silently
serves an empty endpoint. Built both binaries and scraped them:

main: 38 metric families
PR:  149 metric families, including certwatcher_* and controller_runtime_*

Those are different registries. internal/metrics registers every collector
into ctrlmetrics.Registry in its init(), while main's handler is
promhttp.Handler(), which serves the Prometheus default gatherer. So on main
the llmkube_router_* collectors exist in a registry nothing exposes, and
/metrics returns Go runtime and process metrics only. #1427 is effectively
still open, and the PodMonitor in #1473 is currently scraping an endpoint with
none of the series it was added for.

Your promhttp.HandlerFor(ctrlmetrics.Registry, ...) is the correct fix.

One request, and it is not a change to this PR: that fix is small, independent,
and repairs something broken on main right now, while this PR is large and will
take longer to land. Would you mind pulling it out into its own PR against main?
Happy to do it myself and credit you if you would rather not carry another
branch. This one can keep the change too; the duplicate resolves trivially
whichever lands second.

One non-blocking observation for later, arguably pre-existing rather than yours.
GenericBackend.IdleProbe returns errIdleUnsupported when the
AnnotationIdleEndpoint annotation is absent, and memberIdle surfaces that as
an error, which the caller treats like any probe failure: defer with
ReasonIdleCheckFailed. For a generic-backend member without that annotation
that is a permanent wedge reported as a transient condition, since the probe can
never start succeeding. The six runtimes that implement IdleDetector are all
fine, so this only bites the annotation-less generic case. Worth distinguishing
"cannot ever be probed" from "probe failed this time" at some point, perhaps
alongside #1477.