From 278c6fdeeaec3d4e54da64929921297d7b5ec3a5 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 14 Oct 2026 01:42:03 +0300 Subject: [PATCH 01/32] docs(design-proposals): add GPUDevice proposal A tenant can only ask for a GPU by a device-plugin resource name, cannot hold one while a VM is stopped, and learns that an order does not fit only when virt-launcher stays Pending. Propose a hoster-defined GPU catalog (GPUClass), tenant-held GPU reservations that VMs and Kairos worker pools attach the same way (GPUDevice), a per-node inventory of physical GPUs and VFs (GPUNode), and usage records that an external billing system reads. Cover the host layer under them as a replaceable phase: passthrough and vGPU nodes in one cluster, declarative vGPU profiles, and NVIDIA/kubevirt-gpu-device-plugin#192 carried as a downstream patch until upstream settles. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 373 +++++++++++++++++++++++++++ 1 file changed, 373 insertions(+) create mode 100644 design-proposals/gpudevice/README.md diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md new file mode 100644 index 00000000..7eb8f1fe --- /dev/null +++ b/design-proposals/gpudevice/README.md @@ -0,0 +1,373 @@ +# GPUDevice: GPUs as a tenant resource + +- **Title:** `GPUDevice: a GPU catalog, per-tenant GPU reservations for VMs and worker pools, and the host layer under them` +- **Author(s):** `@lexfrei` +- **Date:** `2026-10-14` +- **Status:** Review + +## Overview + +A tenant who wants a GPU today writes a device-plugin resource name such as `nvidia.com/AD102GL_L40S` into `gpus` of a VM or a worker pool and hopes one is free. The `gpu` option source shows how many of each name the nodes advertise and on which nodes, but nothing says which model, mode or profile a name stands for, nothing holds a GPU while the VM is off, and an order that does not fit is accepted and then sits in `Pending`. A hoster selling GPUs needs the opposite: a catalog of GPU models and modes, GPUs a tenant takes within a quota and keeps until they release them, the same handling for VMs and managed Kubernetes worker pools, and a usage record of who held what and when, which the hoster's external billing system reads. + +This proposal adds three kinds. `GPUClass` is the hoster's catalog: one entry per offer, such as a whole H100 in passthrough or an L40S-24C vGPU profile. `GPUDevice` is a tenant's GPU, or a set of GPUs on one node: ordered against a class, bound to a node with free capacity, and attached to a VM by name or to a Kairos worker pool by a label selector. `GPUNode` is the inventory: every physical GPU and VF on a node, its configuration, and which tenant uses it. The proposal also covers the host layer under them, as its own phase: passthrough and vGPU nodes in one cluster, declarative vGPU profiles, and the device plugin that advertises GPUs to KubeVirt. + +NVIDIA's vGPU stack changes quickly. Within the last few releases the driver branches, the way vGPUs are exposed (mediated devices or SR-IOV virtual functions), the KubeVirt device plugins and the licensing rules all changed. This proposal describes that stack as it was in October 2026. Expect it to be updated as NVIDIA and KubeVirt move, with a decision record under it for each change of course, and check the versions named here against your own before relying on them. + +## Scope and related proposals + +- [`kairos-vgpu-workers`](../kairos-vgpu-workers/README.md) gives managed Kubernetes worker pools on Kairos a worker image per GPU mode, with the vGPU guest driver or the data center driver, and delivers the vGPU license token to the workers. [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers VMs with passthrough and vGPU: the hoster adds GPU guest images to the golden image collection through `extraImages`, marked with a `gpu` field. Both are proposed alongside this one and use `GPUDevice` from the start, through `device` entries and `deviceSelector`; the links resolve once they merge. +- [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and section 2 rebinds their `Lost` devices. The link resolves once it merges. +- [IP addresses as a first-class resource](https://github.com/cozystack/community/pull/35) solves the same problem for addresses with a class, a tenant claim and a backend contract. +- [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared reservations the quota authority. A `GPUDevice` is a reservation in that sense, and section 4 hands GPU budgets over to it only if its gate takes over subtree caps for GPU keys. +- Host-side discovery of vGPU VFs grouped by profile is [NVIDIA/kubevirt-gpu-device-plugin#192](https://github.com/NVIDIA/kubevirt-gpu-device-plugin/pull/192), still open. Section 6 ships it downstream in the host layer, behind a boundary the rest of the design does not cross. + +## Decisions + +## Context + +### What exists + +The `cozystack.gpu-operator` package has three variants, selected by `bundles.iaas.gpuOperatorVariant` ([`docs/gpu-vgpu.md`](https://github.com/cozystack/cozystack/blob/main/docs/gpu-vgpu.md)). `default` binds whole GPUs to `vfio-pci` for VMs, `vgpu` runs the NVIDIA vGPU host driver, and `container` serves GPUs to pods. For the two VM variants the platform writes the `HostDevices` feature gate and, when it has entries, a `permittedHostDevices` table into the KubeVirt CR (`packages/core/platform/templates/bundles/iaas.yaml`). Passthrough GPUs are advertised by NVIDIA's sandbox device plugin. + +How NVIDIA's vGPU host driver exposes vGPUs depends on the GPU, the host OS and the driver branch. Older setups use mediated devices (mdev). SR-IOV GPUs on current releases, such as Ubuntu 24.04 with the vGPU 17 or newer branch that Cozystack's guide uses, get VFs, and each VF gets a profile when `current_vgpu_type` is written in its sysfs. KubeVirt v1.9, which Cozystack bundles, advertises such VFs through virt-handler's own device plugin ([kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890)), matched by PCI ID; on L40S every VF reports the PF's ID. On GPU Operator nodes nothing in Cozystack writes VF profiles; the `cozystack-nvidia-vgpu-restore` boot unit from cozystack/ansible-cozystack#70 covers only hosts with a host-installed driver. + +Both consumers take GPUs the same way: `vm-instance` and `kubernetes-nodes` read `gpus` as a list of `{name: }`, one entry per GPU. The deprecated `gn1.*` instance types carry GPU entries of their own in KubeVirt's shape, `{name: gpu1, deviceName: nvidia.com/A400}`, and neither default `permittedHostDevices` table lists that name. Tenant quotas become a namespace `ResourceQuota`, which refuses a GPU only when virt-launcher's pod is created, and the usage records (`Workload`) show a GPU only while a VM runs. + +### The problem + +- A tenant sees resource names, not offers. The dashboard's `gpu` option source lists the names KubeVirt permits; they say nothing about mode, profile or price. +- A GPU cannot be held. When a VM stops, its GPU goes back to the pool and the next start can fail. +- An order that does not fit is accepted, and the tenant learns it from pod events. +- Nobody can tell which tenant uses a physical GPU without reading pods on the node. +- In the `vgpu` variant a node labelled for passthrough can serve whole GPUs of another model, but a model whose whole GPU and VFs share a PCI ID, such as L40S, is offered either whole or as vGPU, not both, because two device plugins would claim that ID. The limit is per GPU model, not per cluster. +- VF profiles on GPU Operator nodes are written by hand after every reboot. + +## Goals + +- A hoster publishes a catalog of GPU classes, each with a model, a mode (passthrough or vGPU) and, for vGPU, a profile. Tenants read the catalog with free capacity and their remaining budget before ordering. +- A tenant orders a number of GPUs of a class within their quota and holds them, attached or not, until they release them. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden` (section 7). That is the end state, not the default: `forbidden` needs every Talos GPU pool replaced by a Kairos pool, which waits for cozystack/cozystack#4911, and every ComputePlane GPU group gone, which waits for ComputePlane on Kairos workers. Until then a cluster with either keeps raw names, and the hold is not guaranteed there. An order that cannot be satisfied is refused with a message naming the class. +- VMs and Kairos worker pools attach held GPUs the same way. A restarted VM gets the same held device back, and a replaced worker gets one of its pool's devices back unless the tenant's own VM or another pool uses it. +- Every physical GPU and VF can be traced to its node, its configuration and the tenant using it. +- Held GPUs appear in usage records (`Workload`) by tenant, class and time, which an external billing system reads. Cozystack charges nothing and stores no prices. +- One cluster can have passthrough and vGPU nodes of the same GPU model, with vGPU profiles applied declaratively and kept across reboots. +- Existing `gpus` entries keep working, with a documented path to `GPUDevice` for VMs; existing Talos GPU pools stay on `name` entries. +- The catalog can later describe MIG, other vendors and DRA-backed devices without changing the tenant API. + +### Non-goals + +- Prices and charging. They live in an external billing system, keyed by class name. +- The vGPU guest driver, the license server and license tokens. The hoster provides them (section 5). +- The catalog UI. This proposal provides the data a portal reads. +- GPUs for pods inside tenant clusters (gpu-operator in a tenant cluster, HAMi). +- Time-slicing one device between tenants, and live migration of GPU VMs. + +## Design + +### 1. Object model + +```yaml +# Hoster. One per offer. Cluster-scoped, readable by every authenticated user. +apiVersion: gpu.cozystack.io/v1alpha1 +kind: GPUClass +metadata: + name: l40s-24c +spec: + displayName: "NVIDIA L40S, 24 GB vGPU (compute)" + vendor: nvidia + model: L40S + mode: vgpu # passthrough | vgpu + profile: L40S-24C # vgpu only + memory: 24Gi + maxCount: 2 # vGPUs of this class per VM + onCordon: Move # Move | Stay; can change + nodeSelector: # nodes carrying this profile + gpu.cozystack.io/10de-26b9: L40S-24C + source: + devicePlugin: + resourceName: nvidia.com/NVIDIA_L40S-24C +status: + free: 12 + maxFreePerNode: 4 # most free units on one node +--- +# Tenant. One per GPU or per set of GPUs used together on one node. +apiVersion: gpu.cozystack.io/v1alpha1 +kind: GPUDevice +metadata: + name: train-0 + namespace: tenant-acme +spec: + className: l40s-24c + count: 1 # GPUs of the class, always on one node +status: + phase: Attached # Pending | Bound | Attached | Lost + nodeName: gpu-node-3 + attachedTo: + vmi: vm-instance-trainer + boundAt: "2026-10-14T09:00:00Z" +--- +# Inventory. One per GPU node. Cluster-scoped, admin only. +apiVersion: gpu.cozystack.io/v1alpha1 +kind: GPUNode +metadata: + name: gpu-node-3 +status: + devices: + - pciAddress: "0000:41:00.0" + model: L40S + function: pf # pf | vf | whole + vfs: 16 + - pciAddress: "0000:41:00.4" + function: vf + profile: L40S-24C + healthy: true + usedBy: {namespace: tenant-acme, pod: virt-launcher-vm-instance-trainer-x7k2p} +``` + +A class maps an offer to a backend through a `source` union and to the nodes that serve it through `nodeSelector`. The first release has one source, `devicePlugin`; `dra` is reserved for section 8. Two classes may share a resource name when their node selectors do not overlap, which lets a host layer that cannot tell two offers apart by name tell them apart by node. A node that matches two classes with the same resource name gets a condition on both classes and on its `GPUNode`, which names the two classes, and is not used for binding. Class names are DNS labels, since they become part of quota keys. + +A class referenced by held devices cannot disappear or change its meaning. A finalizer keeps a deleted class until its last `Bound`, `Attached` or `Lost` device is gone, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `vendor`, `model`, `mode`, `profile`, `memory`, `maxCount` and the kind of `source` cannot change while the finalizer is present. `resourceName` and `nodeSelector` can change, for changes in the host layer (section 6) and for moving the offer to other nodes. + +A `GPUDevice` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate devices. How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 6), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPU devices of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. Admission refuses a device above `maxCount`. The spec is immutable, so a change is a new order. + +The inventory is written by a node agent from sysfs and from kubelet's pod-resources API, and goes away with its `Node`. Ownership is exact for running GPUs: a `usedBy` pod leads to its VM, and the VM's devices name it in `status.attachedTo`. A held but not running reservation is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. PCI addresses stay in `GPUNode`, which tenants cannot read. + +What `Bound` guarantees depends on the class's source: + +| Source | A `Bound` device holds | The physical device is picked by | +| --- | --- | --- | +| `devicePlugin` | `count` free units of the class on one node | kubelet, when the pod starts | +| `dra` (section 8) | one `ResourceClaim` within budget, and its devices once the claim is allocated at the first start | the scheduler, at allocation | + +A tenant sees a class, a count, a phase and, once there is one, a node, never which card it holds, so its API stays the same when an offer moves to a class of the other source. + +### 2. Binding and capacity + +A controller in `cozystack-controller` binds each new `GPUDevice` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`. The taint rule leaves out the `node.kubernetes.io/*` taints Kubernetes sets itself for a cordon or a node that is not `Ready`, which the first two rules already cover. A device already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI creation (section 3): it may take an unattached device off a node that is not `Ready`, because no VMI uses that device and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound devices, minus the units pods hold beyond what their devices cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. + +Admission refuses an order that cannot be met at that moment, by capacity or by budget (section 4), with a message naming the class, the count and the limit it hit: a budget of the tenant or an ancestor, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. + +The phases decide what a device holds and costs: + +| Phase | Holds GPUs | Counts against budget | In usage records | +| --- | --- | --- | --- | +| `Pending` | no | no | no | +| `Bound` | yes, on a node | yes | yes | +| `Attached` | yes, used by a VMI | yes | yes | +| `Lost` | no | yes | no | + +A `dra` device before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 8). + +A device leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 deletes a final VMI before it creates the VM's next one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)), so a new VMI never finds its device attached to the old one. + +A device keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached device that no longer fits there moves to another node of its class, or to `Lost` when none has room. The devices one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A device or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a device's node is deleted, the controller also deletes the device's unscheduled VMI, so the device becomes unattached and moves. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves devices by the class's `onCordon` policy. With `Move`, the default, an unattached device on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its device becomes unattached, moves, and the VM's next VMI starts on the new node. Kairos workers use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a device by the pool rules of section 3. With `Stay` the device keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any device of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a device of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI creation, when one is needed, refuses such a set (section 3). + +A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity when the agent reports a device missing or unhealthy for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no device. Then: + +- the node and the affected classes get a `LostCapacity` condition; +- unattached devices that no longer fit move to another node of their class, or to `Lost` when none has room; +- on a `NotReady` node, attached devices go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the device goes back to `Attached` there; a device that a gather at VMI creation is moving off the node is the exception: it finishes the move, or loses its attachment and follows these rules when the gather fails; +- a VM started on such a node after the condition is set is refused (section 3), and the controller deletes the unscheduled VMI of a device that no longer fits, so its device becomes unattached and can move. What the VM does next follows its `runStrategy` for a deleted VMI: `Always` and `Once` create a new VMI and start once the device has moved or been rebound from `Lost`, while `Manual` and `RerunOnFailure` stay stopped until the tenant starts the VM (Failure cases); unlike a cordon, the node cannot serve the device, so waiting is no option. A VM restarted earlier, on a node that is not `Ready`, is gathered at creation instead (section 3). + +Devices in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a device whose VMI is still live waits until that VMI is final or removed (above). The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` device already counts. + +### 3. Attaching + +VMs and Kairos worker pools reference devices in `gpus`: + +```yaml +# vm-instance +gpus: +- device: train-0 +--- +# kubernetes-nodes, Kairos pool: each worker takes one matching device +gpus: +- deviceSelector: + matchLabels: + pool: inference +``` + +The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named devices. For a pool worker it takes the device the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` device on a `Ready` node that matches the selector and that no VM names in a `device` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same device, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the device, so two VMIs can never hold one device, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A device still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A device that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved device; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Devices a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the devices a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template devices, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template devices included. This runs after the other refusal checks below and after the template devices pass their budget and entitlement checks, so every device in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a device of it, and each device keeps its hold. A device of a `Stay` class on a cordoned node does not move. When all of the VMI's devices are on that node and the VMI has no template devices, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. Otherwise the set cannot be gathered, and the VMI is refused. With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each device and its node. A move that does not finish within the admission timeout refuses the VMI the same way; devices already moved keep their new node, and KubeVirt's retry continues from there. Template devices are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the device's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. + +The webhook refuses to start a VMI, and names each device and its node, when a referenced device: + +- is `Pending`, `Lost`, being deleted, or attached to another VMI; +- is bound to a node that is gone, no longer matches the class, has `LostCapacity` the device does not fit under, or has a pending layout change (section 6); +- cannot be brought, with the VMI's other devices and template devices, onto one node with room for all of them (above), for example because a device of a `Stay` class is on a cordoned node while other devices need gathering; +- is a vGPU device next to vGPU devices of another class, or brings the VMI's GPUs of its class above `maxCount`. + +A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are the resource names of `GPUClass` objects and the names in the hoster's list `gpu.resourceNames`, which defaults to the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. The list also takes every resource name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin, as the agent reports it, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU device for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its devices unattached, so they move by the rules of section 2. Both webhooks see only VMIs and pods carrying the chart's reference label and fail closed, so an outage blocks the start of VMs with device references and nothing else. + +A device cannot disappear under a running VM: a deleted `GPUDevice` stays, with its binding and usage record, until no VMI uses it, and is not attached again. + +A `deviceTemplate` entry names a class and a count instead of a held device. At VMI creation the webhook creates a `GPUDevice` from it, named after the VMI and the entry's index and marked as a template device. A repeated call or a recreated VMI finds the same device and reuses it when it carries that mark, has no owner yet and matches the template; a device of that name still being deleted, or still owned by the VM's previous VMI, refuses the VMI, and KubeVirt retries. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked device that already has the name refuses the VMI, and the message names it. The webhook has the binder bind the device under the same capacity and budget checks as any order, within the admission timeout, and attaches it. The node for template devices is picked with room for all of the VMI's devices, held and template: the node of the gathered set when the VMI has held devices, and otherwise one node with room for all its template devices. The VMI is refused if no node has room for all of them. A bind that does not fit or does not finish in time refuses the VMI, and KubeVirt retries. On a dry run the webhook creates, attaches and moves nothing, so it declares `sideEffects: NoneOnDryRun`. The VMI has no UID yet at that point, so the controller sets the VMI as the device's owner once the VMI exists, and deletes a template device whose VMI never appears after the same grace period as a stale attachment. The controller deletes a template device as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the device lives and is billed only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any device, but only while the VMI exists: an order that does not fit is refused when the VM or worker starts, not ahead of time. A template counts as a device reference under `forbidden`. It lets an autoscaled pool pay for running workers only. + +A `vm-instance` entry has exactly one of `name`, `device` or `deviceTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `deviceSelector` or `deviceTemplate`. A list holds only `name` entries or only references, and a pool has at most one `deviceSelector` or one `deviceTemplate`, never both, and a selector's devices must share a class and a `count`; the webhook checks this at each attach, since device labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to Kairos is separate work. + +Kairos pools take GPUs only through `deviceSelector` or `deviceTemplate`, from classes of both modes. The chart refuses `name` entries on them. The pool's mode, `kairos.gpuMode`, must match the class, and the chart sets it as the `gpu.cozystack.io/mode` label on the worker VMI template ([Choosing the worker image by mode](../kairos-vgpu-workers/README.md#choosing-the-worker-image-by-mode)). At VMI creation the webhook refuses a device, selected or from a template, whose class `mode` differs from that label, and a pool reference on a VMI without the label. The `kubernetes-nodes` chart, from Rollout step 2, fails to render a `deviceSelector` or `deviceTemplate` on a pool with `os: talos`, the field cozystack/cozystack#4911 adds, so the pool's HelmRelease fails before any machine is created. A pool on `deviceSelector` never runs more GPU workers than it has matching devices. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer devices sets it. A pool without spare devices rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a device, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools, existing and new, keep working as today on `name` entries, and `GPUDevice` does not extend them. + +### 4. Quotas + +A tenant's GPU budget is declared in `resourceQuotas`, per class as `gpu.cozystack.io/: ` and in total as `gpu.cozystack.io/total: `; the class name `total` is reserved. These keys are not rendered into the namespace `ResourceQuota`, since `GPUDevice` is not a pod, and a tenant with only GPU keys gets no `ResourceQuota` or `LimitRange`. That is a change to the tenant chart, which today renders both for any `resourceQuotas` key. + +GPU budgets follow their own rule, not the pool and carve-out rules for other resources: every GPU key caps the whole subtree of the tenant that declares it. Today's pools cannot carry GPU keys, since a tenant that declares any key becomes its own pool root and leaves its parent's limits for every key it does not declare, so a child with only `cpu` would escape its parent's GPU budget. The tenant declaration gate skips `gpu.cozystack.io/*` keys, because under this rule a child's key may overlap its siblings'. + +- An order is refused if, for the tenant or any ancestor that declares the class key or `total`, the devices counting against the budget (table in section 2) in that ancestor's subtree would exceed the key. +- A child's key only adds a limit; it never lifts an ancestor's limit or a missing entitlement. +- A raw GPU key (section 7) on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has the class key or `total`. While the raw key exists, it also caps those classes' devices together with the raw-name VMs it covers, so an existing GPU limit is never bypassed. +- A tenant with no GPU key of either kind anywhere above it is limited only by capacity, for vGPU classes once it is entitled. +- A vGPU class is ordered only by a tenant entitled to vGPU. The Tenant field `gpu.vgpu` grants it. The parent tenant's administrator sets it, or the hoster on a top-level tenant. A tenant is entitled only when the flag is set on it and its parent is entitled. The root tenant is rendered by the platform, not by the tenant chart, so its value comes from platform values. The flag is off by default everywhere, so the hoster turns it on for the root first. No tenant can grant itself vGPU, since its Tenant object lives in its parent's namespace. The tenant chart computes the effective value from the parent's, like its other namespace values, and stamps it on the namespace as the label `namespace.cozystack.io/vgpu`; a change reaches grandchildren as each level re-renders. The chart always writes the label, so only the value `"true"` means entitled; any other value, or no label, means not entitled. Entitlement is separate from budget because a vGPU license is counted per unit, which is a budget, while getting the vGPU guest driver and the license token at all is yes or no. Class keys and `total` then limit vGPU like any other GPU. Clearing the flag refuses new vGPU orders, including the device a `deviceTemplate` would create at the next VMI start, so such a VM or worker no longer starts. Devices already held keep counting and can still move or be rebound, so a pool on `deviceSelector` keeps its devices, and the tenant's Kairos pools lose their vGPU entitlement as `kairos-vgpu-workers` describes. Raw vGPU names in VMs stay as open as today until a raw key limits them or `forbidden` is set (section 7). +- GPU keys play no part in the existing pool membership and carve-outs. + +A parent with `total: 4` caps everything below it at four GPUs, each vGPU counting as one. A parent cannot reserve part of its budget for one child. #48 would add that, but as written it covers neither GPUs nor entitlement, so this section stays GPUDevice's own rule, and budgets move to #48 only if its gate takes over subtree caps for GPU keys. + +### 5. Usage accounting, catalog and licensing + +Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads. Each `GPUDevice` gets one with `kind: gpu-device` and resources `gpu.cozystack.io/: ` while it is `Bound` or `Attached`, a `dra` device only once its claim is allocated (section 8), next to the records Cozystack already keeps for CPU, memory and storage. The GPU controller writes the record itself, under a `WorkloadMonitor` it owns in the tenant's namespace, whose selector matches no object, so tenant labels cannot drop a device from the usage records. The selector is a reserved label key that admission refuses on every object in a tenant namespace, on create and update, whoever makes the request, so no object can match it, not even a virt-launcher pod or a DataVolume that inherits the label from a VM template. Each record carries the `workloads.cozystack.io/monitor` label naming that monitor: the `Workload` controller deletes a record only when that label is missing or names a monitor that is gone, and deletes nothing by selector. Like other `Workload` records it has no time fields: a held interval is the record's lifetime, which a billing system observes by watching and reconciles by sampling. A virt-launcher pod does not report GPUs its devices cover a second time. + +The catalog is the list of `GPUClass` objects with `displayName`, model, mode, profile, memory, `maxCount`, `status.free` and `status.maxFreePerNode`. Every authenticated user reads it, because a hoster publishes its offers and a tenant names a class when ordering, including from `kubectl` or GitOps without a portal. So the status holds only `free` and `maxFreePerNode`, and its conditions, such as `LostCapacity`, name no nodes and carry no counts. The class's capacity, bound units and `Lost` count show the hoster's inventory and load, and they are not in the status. The controller exports them as metrics that platform monitoring scrapes. Per node, a hoster reads device occupancy and health from `GPUNode`, and lists bound devices with a field selector on the `GPUDevice` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write. A tenant can still watch `free` and `maxFreePerNode` over time and see how busy a class is. The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest order, leaving out a field that is absent because it sets no limit. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. A `gpudevice` source lists a tenant's own devices. The existing `gpu` source keeps serving raw names while they are allowed. + +The remaining budget is computed over each ancestor's whole subtree, so a tenant under a shared parent `total` can see how many GPUs its siblings hold, as a number, without names. This is accepted: the alternative is a budget the tenant can only discover by a refused order, which is the failure this proposal removes. + +vGPU needs a license, and passthrough needs one only for graphics. Compute (C-series) profiles are licensed through NVIDIA AI Enterprise, graphics (Q-series) profiles through NVIDIA vGPU software. The hoster provides the license server, the token and the guest drivers. In worker pools `kairos-vgpu-workers` delivers the token and the guest license settings. In VMs the hoster does, through its image or at boot, as `vgpu-vms` describes. The class's `mode` tells a portal and a tenant which case applies. + +### 6. Host layer + +This proposal requires vGPU hosts to run Ubuntu or Kairos with `bundles.system.variant: isp-full-generic`: the vGPU host driver and the patched plugin run there. Cozystack has no Kairos hosts today, so installing the bundle on Kairos and a vGPU Manager image for it are new work of this phase. `docs/gpu-vgpu.md` does not recommend Talos for vGPU, so this phase does not cover Talos hosts; they keep passthrough through the `default` variant as today. NVIDIA AI Enterprise lists vGPU for Compute on Linux with KVM as supported only through specific hypervisor vendors and, on Ubuntu and RHEL with KVM, for single-node hosts only ([support matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/support/support-matrix-8/8.2.html)), so support for such a host is a matter between the hoster and NVIDIA. + +The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ships as its own phase (Rollout, step 4), and it is replaceable. The API above it depends on four things only: each class's `resourceName` is advertised on the nodes its `nodeSelector` picks, one name never covers two different offers on one node, `GPUNode` reports devices and their health, and node labels mark each node's mode and profiles. `GPUClass`, `GPUDevice`, `GPUNode`, quotas, both webhooks, usage records and migration use no plugin-specific name, discovery format or configuration; the agent maps whatever device IDs a plugin reports to PCI addresses through sysfs. A different plugin, or a different version of the one chosen here, changes only this phase and the `resourceName` of affected classes, which can change while devices are held. Tenants keep their devices and `device` entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to `GPUDevice` first. + +Until the phase ships, a cluster keeps the modes its variant gives it today, one per model for models whose whole GPU and VFs share a PCI ID, and vGPU classes keep the experimental status `docs/gpu-vgpu.md` gives the `vgpu` variant: the hoster writes profiles and node labels by hand. On clusters without a layout, which includes the `default` variant and Talos hosts, node labels stay hand-written. + +The device plugin is the decision reviewers should look at first. Cozystack ships NVIDIA's kubevirt-gpu-device-plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, as the sandbox device plugin of the `vgpu` variant, until NVIDIA/kubevirt-gpu-device-plugin#192 merges upstream or upstream takes another approach. NVIDIA/kubevirt-gpu-device-plugin#192 adds discovery of vGPU VFs on the vendor-specific VFIO framework and advertises them with one resource name per profile, next to the passthrough names the plugin already advertises for GPUs bound to `vfio-pci`. One plugin then serves both modes of every model under distinct names, so the per-model limit of the problem list goes away. The branch is validated on hardware. Its upstream review has had no maintainer response since 2026-07-10, which is why it ships as a downstream patch rather than waiting. On a card whose VFs all carry a profile, the plugin can read profile names only through NVML, so the platform runs it as its own DaemonSet with the host driver's `libnvidia-ml.so.1` and the `/dev/nvidia*` nodes mounted, privileged; the GPU Operator's DaemonSet has no field for those mounts. The patch leaves when NVIDIA/kubevirt-gpu-device-plugin#192 merges upstream, in this or another form, or when the NVIDIA DRA driver serves vGPU VFs to KubeVirt (kubernetes-sigs/dra-driver-nvidia-gpu#1402); a merged NVIDIA/kubevirt-gpu-device-plugin#192 changes only this phase, while a move to DRA adds new `dra` classes (Rollout, step 6). KubeVirt's built-in plugin was the other candidate (Alternatives). + +The phase adds a node layout to the `vgpu` variant, in platform values. For each node selector it names a mode and, for vGPU, the profile of each GPU model. Nodes outside every selector are passthrough, through the `defaultWorkload: vm-passthrough` the layout renders. With a layout: + +- The platform runs the patched plugin and renders the KubeVirt `permittedHostDevices` entries for the names it advertises, as entries with an external provider. Other vendors' entries, such as an FPGA, stay as written. +- A node agent writes the VF profiles at boot and when the layout changes. The controller sets each node's workload label and one label per GPU model, `gpu.cozystack.io/-` (for L40S `gpu.cozystack.io/10de-26b9`), to the profile or to `passthrough`; class node selectors match these labels. +- The plugin does not pick up VFs that gain a profile after it started, so the controller restarts the plugin on a node when devices there become ready later, for example after a reboot. Running VMs are not affected. + +The layout and mdev GPUs coexist: GPUs on the mdev path keep today's configuration through `vgpu-device-manager`, and the agent writes profiles only on nodes that use VFs. New deployments of SR-IOV GPUs on current host releases use VFs. + +A layout change on a node, of mode or of profile, waits until the node has no GPU VMs. The node gets a `LayoutPending` condition: new binds and starts there are refused, unattached devices move away, and running VMs keep their GPUs until the hoster drains the node. A GPU VM created before Cozystack switched to `LiveMigrateIfPossible` needs one restart before the drain can evict it. A node whose hand-written profiles already match the layout is not a change. A `vgpu` cluster adopts the layout with vGPU selectors that cover its existing vGPU nodes, and labels those nodes `vm-vgpu` before setting it, so the layout's passthrough default does not change them in between. Its vGPU classes, raw `name` entries and raw quota keys move from the hand-written VF names to the plugin's per-profile names in one maintenance window, or the raw-name consumers move to `GPUDevice` first. KubeVirt's built-in plugin and the patched plugin report different device IDs for the same VF, and kubelet would otherwise hand out VFs in use, so affected nodes are cordoned and free of GPU VMs until the patched plugin has registered there. + +### 7. Migration of existing `gpus` + +Raw `name` entries keep working, and nothing converts them automatically, because a conversion would bind GPUs a tenant never ordered and put them in the usage records. The binder counts their usage, so it never books a unit a raw-name VM holds. The reverse is not protected: the scheduler does not see a device that is bound but not attached, so while raw names are allowed, a raw-name VM can take that unit. A reservation is a hard guarantee only with `gpu.rawResourceNames: forbidden`. + +A tenant moves a VM by stopping it, ordering devices of the matching class, replacing each `name` entry with a `device` entry and starting it. On a full pool the device can only be ordered after the stop; with spare units, ordering first puts both in the usage records until the restart. A Talos GPU pool has no in-place path. The tenant creates a Kairos pool with `kairos.gpuMode` and a `deviceSelector`, which needs a hoster-built worker image for the class's mode, and then deletes the Talos pool. On a full cluster the Talos pool scales down first, so its GPUs are free to order. ComputePlane GPU groups have no such path until ComputePlane runs Kairos workers, so a cluster with them keeps raw names allowed. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes `device` entries. A Talos pool on one takes the Kairos path above. + +Quota keys move too. A raw GPU key is a quota key on a GPU resource name, such as `nvidia.com/AD102GL_L40S`. A device-backed VM's pod still requests that name, so while a tenant migrates, its raw key covers both kinds of VM; after its last raw-name VM has moved, the key is replaced by `gpu.cozystack.io/*` keys. A tenant whose raw-name VMs use vGPU can order their devices only after it is entitled to vGPU (section 4), and within any class keys set above it. + +When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. Preconditions: no Talos GPU pools in the cluster, ComputePlane GPU groups included (a hoster without a passthrough worker image keeps passthrough Talos pools and raw names), no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: + +- an admission policy refuses GPU entries (section 3) on every VMI created without a device reference, on every path; +- the same policy refuses every pod created in a tenant namespace that requests a GPU resource name, unless it is a virt-launcher pod created by KubeVirt's virt-controller service account with an owner reference to a VirtualMachineInstance. Tenants cannot forge such a pod, since no tenant role can create pods or VMIs, so app charts that copy free-form `resources` into pods cannot request GPUs either; +- tenant validation refuses raw GPU keys; +- the `gpu` option source returns nothing, and the `instancetype` source hides instance types that carry GPU entries. + +### 8. Extensions + +The catalog is designed so later backends add `source` members and class fields, not tenant fields. + +- DRA. Kubernetes DRA has been GA since 1.34, and KubeVirt v1.9 has `GPUsWithDRA` and `HostDevicesWithDRA` in beta. A `dra` source names a `DeviceClass` and selectors, and under it a `GPUDevice` maps one to one onto a `ResourceClaim` created at order time. The scheduler allocates the claim, the GPU controller keeps it allocated through its own `reservedFor` entry, which the resourceclaim controller leaves in place, and the webhook renders the claim reference into the VMI; the binder does not allocate. Three gaps remain. The NVIDIA DRA driver has no vGPU yet (kubernetes-sigs/dra-driver-nvidia-gpu#1402, milestone v0.6.0), and its passthrough is alpha. A claim is allocated only when a pod first needs it, so nothing is held before the first boot: until then a `dra` device is `Bound` without a node, counts against the budget, holds no GPU and stays out of the usage records, and the webhook attaches it like any `Bound` device without pinning the pod to a node. DRA quota is per namespace, without subtree caps. KubeVirt VEP 344 (`persistWhenStopped`) holds a claim for a stopped VM from the VM side; a `dra` class and VEP 344 must not both manage one claim. When a DRA GPU driver is installed, the raw-name policy has to cover claim references too. +- MIG-backed vGPU profiles get their own resource names from the patched plugin, so they are ordinary classes; with DRA, partitionable devices (beta in Kubernetes 1.36) describe MIG natively. +- A non-NVIDIA GPU is a class with another `vendor`; Q, B and A series vGPU profiles are classes like C series. +- NVLink sets that must be attached together are left to the DRA source. +- On-demand slicing. A hoster may let tenants take any size from a shared pool, from a fraction of a GPU, such as a MIG-backed vGPU on a 1/7 slice of an A100, up to several whole GPUs of one node, taken from GPUs the layout keeps out of MIG mode. The tenant contract stays a class, one per size, and a `count`, and a device still lives on one node. The host layer and the binder change: the node agent creates the profile, and on MIG GPUs the MIG instance under it, when a device binds instead of following a fixed layout, so the binder counts capacity from free GPU resources rather than from advertised names. Enabling MIG mode needs an idle GPU and, on Ampere, a GPU reset, so the layout enables it ahead of time. On Hopper and newer the mode does not survive a reboot, so the layout sets it again at boot. A plugin restart per bind, as section 6 does for late profiles, would leave the node's allocatable at zero until the plugin registers again on every bind, so the plugin needs to rediscover profiles without one, separate work on top of NVIDIA/kubevirt-gpu-device-plugin#192. + +## User-facing changes + +- Tenants get `GPUDevice`, a GPU catalog with free capacity and remaining budget per class, `device` entries in `vm-instance`, `deviceSelector` entries for Kairos pools in `kubernetes-nodes`, and `deviceTemplate` entries in both. Raw resource names keep working until a hoster forbids them. +- A held device is billed while held, whether a VM runs on it or not. An autoscaled Kairos pool on `deviceSelector` holds, and pays for, every device its selector matches, which is at least `maxReplicas`; under `forbidden` nothing runs on GPUs without a device. A tenant who wants to pay only while running uses `deviceTemplate`, and gets the refusal at start instead of at order time. +- Hosters get `GPUClass`, `GPUNode`, GPU budgets in `resourceQuotas` and the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout, `onCordon` on classes, the `gpu.resourceNames` list and `gpu.rawResourceNames`; an external billing system gets a usage record (`Workload`) per held device. + +## Upgrade and rollback compatibility + +The change is additive. Existing `gpus` entries render as before, and the existing variants work as today until a layout is set. + +Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, and the tenants' GPU limits, with four exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only limits were `gpu.cozystack.io/*` keys lose them (rules below), and tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through `GPUDevice`, so they lose their GPUs and run on the plain worker image. A child tenant that declares its own quotas no longer shares its parent's GPU limit, because a raw key is carved out of the parent's pool rather than capping the subtree. + +The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `device` and `deviceTemplate` entries of VMs become raw `name` entries and those VMs restart. `deviceSelector` and `deviceTemplate` entries and `kairos.gpuMode` are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUDevice` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the GPU controller's `WorkloadMonitor` objects, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. + +Three rules hold throughout: + +- Names move back by the window rule of section 6 in reverse: vGPU classes return to the hand-written VF names, and raw `name` entries and raw keys on the per-profile names move with them in the same window. +- Raw keys go through the existing tenant quota pools, not the subtree rule of section 4: a declared key makes a tenant its own pool root for every resource and a child's key is carved out of its parent's pool. So raw keys go only to tenants that already declare quotas other than `gpu.cozystack.io/*`, and raw names elsewhere are unlimited, as before this proposal. +- A raw key also renders the tenant chart's default `LimitRange`, whose memory limit can sit below a VM's memory request, because KubeVirt sets virt-launcher limits from the namespace quota only when it limits `cpu` and `memory`, so a tenant given a raw key also declares `cpu` and `memory`. + +## Security + +- Tenant access. Tenants with the `admin` role or above manage `GPUDevice` in their namespaces; lower roles read it. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. Tenants cannot read `GPUNode`. +- What a tenant controls. A tenant never supplies a resource name or a node through a device reference: both come from `GPUClass` and from the device's status, which tenants cannot write. Tenants cannot add or remove finalizers on devices, so a held device cannot leave the usage records. Admission refuses the reserved label of the GPU usage monitor on any object in a tenant namespace, on create and update (section 5). A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes devices only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a device reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. +- Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. +- Node agent. It reads sysfs and kubelet's pod-resources socket, writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can write only its own node's `GPUNode`. +- Webhook. For `deviceTemplate` entries it creates `GPUDevice` objects in tenant namespaces, only of the class and count the template names. +- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights: patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its devices. +- Visibility. A tenant sees the node name of its own devices and, through the remaining budget, its siblings' GPU count under a shared parent budget (section 5). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. + +## Failure and edge cases + +Each case gives what is seen and what to do. + +- A tenant's order is refused at create → the message names the class, the count and the limit, or, for a vGPU class, that the tenant is not entitled to vGPU; order a smaller count, another class, or ask for more budget. +- A device stays `Pending` after an order → its condition names the shortage; it binds when capacity or budget frees up, or the tenant deletes it. +- A VM with devices does not start → the webhook message names each device and its node and the reason. When the reason is that the VM's devices, template devices included, cannot be brought onto one node, either no node can take all of them or a device of a `Stay` class is on a cordoned node while others need gathering. Then uncordon that node, free capacity on one node, drop a device the VM does not need, or, within one class, replace them with one device of the combined `count`. A refusal for the admission timeout usually clears on KubeVirt's retry. +- A node is drained or cordoned → under `Move` its unattached devices move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the devices stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. +- A cordoned node is deleted while a drained VM's VMI still waits for it → the controller deletes that unscheduled VMI, the device moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. +- A node dies but stays registered → after the grace period its attached devices go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost` (section 2); once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the devices are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached devices onto a `Ready` node with room (section 3) and starts there at once. That holds for a VM's held devices. A pool worker takes only a spare device of its pool, template devices follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node (all in section 3). After a `Node` deletion the VM starts on a retry instead, once the controller has moved the devices. Only when no node fits is the VMI refused, and then it is not created at all. KubeVirt v1.9.0 retries the creation with backoff for `Always`, which deletes the failed VMI and starts a new one, and for `RerunOnFailure`, whose failed VMI leaves a start request that stays until a VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)); such a VM starts once a node has room for its set. The retry goes through the VM controller's rate-limited queue, whose per-item backoff grows to 1000 seconds, so the start can come up to about 17 minutes after room appears. `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its devices unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. +- A node loses capacity while a VM's VMI waits for it unscheduled → the controller deletes that VMI so its devices can move; an `Always` or `Once` VM starts again once its devices have moved or been rebound, a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. +- A GPU fails under a running VM → the device stays `Attached` and in the usage records while the VM runs; when the VM restarts it is refused there, and the device moves or goes to `Lost`. +- A layout change does not apply → the node shows `LayoutPending`; drain it. A VM with a raw `name` entry on the node holds the change until it stops. +- A Talos GPU pool, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a Kairos pool with `deviceSelector`, or allow raw names again while the cluster has Talos GPU pools. +- A pool with `os: talos` and a `deviceSelector` or `deviceTemplate` fails to render → Talos pools take GPUs only through `name` entries; use a Kairos pool for devices. +- A Kairos worker VMI is refused because its device's class `mode` differs from the pool's `gpu.cozystack.io/mode` label → set `kairos.gpuMode` to the class's mode, or select devices of a class in the pool's mode. +- A tenant's VM is `Pending` while its device is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. +- A pool worker is refused → no matching device is free, because of a surge machine during a rollout, devices still `Pending` or `Lost`, matching devices only on nodes that are not `Ready`, or another pool or a VM using the matching devices; wait, add devices, give each pool its own labels, or lower `maxReplicas`. + +## Testing + +- Unit tests for the binder and both webhooks: capacity with raw-name usage, budgets over the ancestor walk, the phase table, every move and refusal in sections 2 and 3, concurrent starts, and the pod pin after a LiveUpdate sync. +- Webhook tests for cordons, templates and NICs: under `Move` a device moves at VMI creation and the pod gets a required node affinity, an attached device with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity the unscheduled VMI is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, the devices of one VM move, go to `Lost` and are rebound as a set, devices of two classes on different nodes, and a device whose node has no room for the VMI's template device, are moved onto one node at VMI creation, refused when no node fits, a `Stay` device on a cordoned node pins the VMI when it holds all of the VM's devices and refuses it when other devices need gathering, devices on a node that is not `Ready` are gathered, two VMIs whose sets share a device do not both gather it, a VMI with only template devices gets one node with room for all of them, and under `Stay` a single device waits; a `deviceTemplate` device is created, refused when it does not fit, deleted with its VMI, deleted when its VMI is refused later in admission, reused on a repeated call, and on a dry run neither created nor attached nor moved; a VM with a GPU device and `nvidia.com/hostdev` passes. +- Chart tests for `kubernetes-nodes`: a `deviceSelector` or `deviceTemplate` with `os: talos` fails to render, and a Kairos pool gets the `gpu.cozystack.io/mode` label on its worker VMI template. +- Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held devices and refuses the next template device, and `remainingBudget` is 0 without the entitlement. +- Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no device reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. +- Controller tests for the class metrics: capacity, bound units and the `Lost` count match the devices, and none of them appears in `GPUClass` status. +- Binder tests: a node with a `NoSchedule` taint that virt-launcher pods do not tolerate gets no binds, moves or gathers and its units leave `free`, while the Fence Agents Remediation taint alone does not make a node cordoned. +- Admission tests for the reserved label of the GPU usage monitor, refused on any object in a tenant namespace, on create and update, including a virt-launcher pod and a DataVolume that get it from a VM template. +- Controller tests for the usage record: present exactly while `Bound` or `Attached`, independent of tenant labels, not reported twice, and kept through reconciles of its `WorkloadMonitor` and of the `Workload` controller. +- Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, `WorkloadMonitor`, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. +- E2E with passthrough GPUs on the GPU runner: order, attach, stop, confirm another tenant cannot take the GPU, start again, and check the usage record. +- Manual on SR-IOV vGPU hosts: both modes of one model in one cluster, profiles across reboots, a layout change through a drain, and adoption from `default` and from `vgpu`, on Ubuntu and Kairos hosts. + +## Rollout + +1. `GPUClass`, `GPUDevice`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, both webhooks, the policies, the option sources, `device` and `deviceTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants. Before tenants order vGPU devices, the `gpu.vgpu` flag is set from the root tenant down. +2. `deviceSelector` and `deviceTemplate` in `kubernetes-nodes`, after cozystack/cozystack#4911. +3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, `kairos.gpuMode` and the mode label, and `vgpu-vms` adds GPU guest images for VMs with `device` entries. A cluster with Talos GPU pools stays on raw names and has no hard hold guarantee. +4. The host layer: the node layout, profile writes, node labels, the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, and Kairos vGPU hosts (the bundle on Kairos and a vGPU Manager image for it). +5. Budgets through #48, only if its gate takes over subtree caps for GPU keys. +6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by ordering new devices. + +## Open questions + +- Release on tenant suspension. A hoster may want a suspended tenant's GPUs released after a GPU-specific grace period; Cozystack has no platform-wide notion of suspension yet. +- Names. Aligning with #35 would make `GPUDevice` into `GPUClaim`; the meaning is the same. +- Packing policy. Fullest-node packing is the default; hosters may need spreading across failure domains as a class option. + +## Alternatives considered + +DRA as the whole API, with `DeviceClass` as the catalog and `ResourceClaim` as the tenant object, or DRA first for passthrough with vGPU later. Rejected for now: a non-Pod `reservedFor` entry does keep a claim allocated for a stopped VM, but the NVIDIA DRA driver does not serve vGPU yet and its passthrough is alpha, and vGPU is the case hosters sell first. DRA is the planned backend under `GPUClass` (section 8), mapped one to one onto `GPUDevice`, so tenants order devices of a `dra` class through the same API. + +A cluster-scoped object per reservation unit, like `IPAddress` in #35. Rejected for the device-plugin backend, where kubelet picks the physical device and such an object would claim a precision the backend does not have. `GPUNode` gives the per-device view where it is real. + +Tenant quota on the resource name only. Stock `requests.` quotas need no new code, but they cannot hold a GPU for a stopped VM and still refuse orders only at pod creation. + +Placeholder pods that hold the device until the VM starts. Rejected because preemption picks victims by priority, not by owner, so one tenant's VM could evict another tenant's placeholder. + +Class-level reservations for worker pools, without per-worker devices. Rejected because a pool would get a promise of capacity, not GPUs. + +A cluster-wide reservation per tenant and class, a count without a node. It needs no binding, moves or pod pin, and for single-GPU offers it would be enough. Rejected because the count holds only in sum. With free units spread one per node, a held set of two fits nowhere. A vGPU profile exists only on the nodes the layout gives it. A lost node silently shrinks every tenant's promise instead of showing which devices are `Lost`. + +NVIDIA vgpu-device-manager as the profile writer. At v0.5.1 it writes mdev profiles only and has no path for `current_vgpu_type` on SR-IOV VFs, and the pull requests that added VF support were closed unmerged, NVIDIA/vgpu-device-manager#139 as not planned, and NVIDIA/vgpu-device-manager#195 after NVIDIA named gaps on the KubeVirt side and had no resources for that test round. The node agent writes VF profiles instead; nodes on the mdev path keep vgpu-device-manager (section 6). + +KubeVirt's built-in device plugin for VFs (kubevirt/kubevirt#16890), with the sandbox plugin for passthrough. It ships in KubeVirt v1.9 and needs no patch, but it gives every vGPU profile of a model one resource name, so classes tell profiles apart only by node. On GPUs whose whole GPU and VFs share a PCI ID it clashes with the sandbox plugin, so both modes of such a model need the built-in plugin for passthrough too, both modes share one name, and raw resource names would have to be forbidden on any cluster with vGPU nodes. Switching between it and the sandbox plugin is a maintenance window each time. Rejected in favour of the downstream patch, whose cost is carrying one patch until upstream settles and running the plugin privileged with NVML access. From 23241ddbd50a7fd7d48d256fe0df96a7135f2fca Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 21 Oct 2026 04:28:03 +0300 Subject: [PATCH 02/32] docs(design-proposals): record the GPUDevice implementation decisions Building the first step of GPUDevice settled several questions the proposal left open or answered differently: how admission handles a class KubeVirt does not permit, what runs on a cluster without GPUs for VMs, how the usage-monitor label is reserved, what GPUNode health means, which taints count, when a Stay pin gives way, who can order template devices, and when the controller gets its host-layer rights. Record each one next to the proposal, so the reasoning survives the proposal text being brought in line. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 9 ++++ ...ttedhostdevices-is-refused-at-admission.md | 35 ++++++++++++++++ ...y-where-gpu-operator-serves-gpus-to-vms.md | 39 +++++++++++++++++ ...l-reservation-is-always-on-and-ratchets.md | 35 ++++++++++++++++ ...punode-health-is-kubelet-allocatability.md | 42 +++++++++++++++++++ ...e-skips-exactly-three-kubernetes-taints.md | 31 ++++++++++++++ ...-to-a-node-that-cannot-serve-the-device.md | 32 ++++++++++++++ ...-write-a-vm-orders-its-template-devices.md | 33 +++++++++++++++ ...s-host-layer-rights-with-the-host-layer.md | 31 ++++++++++++++ 9 files changed, 287 insertions(+) create mode 100644 design-proposals/gpudevice/decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md create mode 100644 design-proposals/gpudevice/decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md create mode 100644 design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md create mode 100644 design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md create mode 100644 design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md create mode 100644 design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md create mode 100644 design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md create mode 100644 design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 7eb8f1fe..8b0ac848 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -23,6 +23,15 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Decisions +- [0008. The GPU controller gets its node-label and plugin-restart rights with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) +- [0007. Whoever can write a VM orders its template devices, while held devices stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md) +- [0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the device](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md) +- [0005. The taint rule skips exactly the three taints that mirror a cordon or readiness](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md) +- [0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts](./decisions/0004-gpunode-health-is-kubelet-allocatability.md) +- [0003. The usage-monitor label reservation is always on, and it ratchets](./decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md) +- [0002. GPUDevice works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run](./decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md) +- [0001. A device whose class resource name KubeVirt does not permit is refused at VMI admission](./decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md) + ## Context ### What exists diff --git a/design-proposals/gpudevice/decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md b/design-proposals/gpudevice/decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md new file mode 100644 index 00000000..52cce3b3 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md @@ -0,0 +1,35 @@ +# 0001. A device whose class resource name KubeVirt does not permit is refused at VMI admission + +- **Number:** `0001` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** not yet + +## Context + +On the `default` and `vgpu` variants the platform writes a `permittedHostDevices` table into the KubeVirt CR whenever it has entries. The proposal assumed that KubeVirt refuses a GPU outside that table when the VMI is admitted. It does not. In KubeVirt v1.9.0 no admitter refuses a GPU or host-device name the table leaves out; the GraceIOVirtualization admitter reads the table only to apply its rules to names listed there, and the refusal is `validatePermittedHostDevices` in [`renderresources.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/renderresources.go), which runs when virt-controller renders the virt-launcher pod. A render error creates no pod, and the VMI stays `Pending` with `Synchronized=False`, reason `FailedCreate`, retried with backoff. It never reaches a final phase. + +That defeats the attachment rules of section 2. A device leaves `Attached` only when its VMI is final or removed, so a device of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry. + +## Decision + +The VMI webhook reads the effective table from the KubeVirt CR, the resource names of its `pciHostDevices` and `mediatedDevices` entries, and refuses a VMI whose device set includes a class whose `resourceName` is not in it, with a message naming the class and the name. An unset table means KubeVirt checks nothing, and then neither does the webhook. The controller also sets a `ResourceNameNotPermitted` condition on such a class, so the hoster sees the gap before a tenant hits it. + +## Why not the alternatives + +- **Rely on KubeVirt's own refusal**, as the proposal did. Rejected on the mechanism above: the refusal comes after admission, as a `Pending` VMI that holds its devices, not as an error the tenant sees at start. +- **Treat a `Pending` VMI with `Synchronized=False` and `FailedCreate` as final, and collect it.** It would release the devices, but only after the tenant has started a VM that cannot run, and the reason would surface as a VMI condition rather than in the start request. `FailedCreate` is also the reason KubeVirt gives for other failures to render or create the pod, so the rule would have to parse messages to tell this one apart. +- **Refuse such a class when it is created.** A class may legitimately come before its table entry, and the table can change after the class exists, so a create-time check catches only one of the two paths. + +## Consequences + +- A tenant learns at start, from the webhook message, that the offer is not usable yet, and the device stays unattached and moves by the usual rules. +- The webhook and the controller condition read the same lists of the table, so the condition predicts the refusal. KubeVirt's own check also covers `usb` entries, which no GPU class uses. +- Every VMI admission with device references reads the KubeVirt CR, through the controller's cache. + +## Revisit if + +KubeVirt starts refusing a device outside `permittedHostDevices` at VMI admission, or the table is dropped for DRA-backed devices. diff --git a/design-proposals/gpudevice/decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md b/design-proposals/gpudevice/decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md new file mode 100644 index 00000000..2a1955ef --- /dev/null +++ b/design-proposals/gpudevice/decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md @@ -0,0 +1,39 @@ +# 0002. GPUDevice works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run + +- **Number:** `0002` +- **Date:** `2026-10-21` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#5008`](https://github.com/cozystack/cozystack/pull/5008) (the gate and the policies under it); the rest: not yet + +## Context + +The proposal rolls step 1 out "on the existing variants" and says nothing about a cluster where GPUs do not reach VMs: gpu-operator under the `container` variant, gpu-operator not enabled, or no iaas bundle. On such a cluster KubeVirt runs without the `HostDevices` feature gate and without a `permittedHostDevices` table, and nothing there resolves a device reference. + +The `vm-instance` chart renders only the reference, on the VMI template, and leaves it to the VMI webhook to fill in the GPUs (section 3). Without the webhook a `device` or `deviceTemplate` entry would boot the VM with no GPU and no error. The CRDs, on the other hand, ship with the controller on every cluster, so a `GPUClass` or a `GPUDevice` can exist on a cluster that has no placement running, and both carry finalizers. + +## Decision + +The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the binder, the VMI and virt-launcher webhooks, the controller's watches on KubeVirt objects, the GPU admission policies other than the label reservation of [0003](./0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md), the metrics scrape, and the tenant admin's write access to `GPUDevice`. Elsewhere: + +- `vm-instance` refuses to render `device` and `deviceTemplate` entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today. +- The platform refuses `gpu.rawResourceNames: forbidden`, because the policy that enforces it does not render there. +- The controller still adds and removes the finalizers of `GPUClass` and `GPUDevice` and keeps the usage records of devices, so deleting a class, a device or a tenant namespace never hangs. + +## Why not the alternatives + +- **Run everything on every cluster.** The binder and the webhooks read KubeVirt objects, which do not exist without KubeVirt, and under the `container` variant the GPUs gpu-operator advertises are meant for pods, not VFIO devices a VM can take. A device bound there would be held and recorded for a GPU no VM can use. +- **Gate the finalizer paths with the rest.** A hoster who disables GPUs, or moves to the `container` variant, would then leave every existing class and device stuck on its finalizer, and with it the tenant namespaces that hold devices. +- **Let `vm-instance` render device references anyway and rely on the webhook.** On these clusters no webhook exists, so the failure is a VM that runs without the GPU it asked for, which the tenant may notice only from inside the guest. + +## Consequences + +- On a cluster without GPUs for VMs, a tenant reads the catalog and its devices but cannot order, and a VM with device references fails at render with a message that says GPUs for VMs are not served. +- Turning GPUs for VMs on writes one new cluster value, which every tenant release reads, so each upgrades once, and only on such clusters. +- The controller keeps a resync loop while a deleted device waits for its VMI, because with the feature off nothing watches VMIs. + +## Revisit if + +GPUs reach VMs through another path than gpu-operator's VM variants, for example a DRA driver installed on its own (section 8). diff --git a/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md b/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md new file mode 100644 index 00000000..713591f9 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md @@ -0,0 +1,35 @@ +# 0003. The usage-monitor label reservation is always on, and it ratchets + +- **Number:** `0003` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#5008`](https://github.com/cozystack/cozystack/pull/5008); refusing a changed value: not yet + +## Context + +Section 5 keeps the GPU usage records under a `WorkloadMonitor` whose selector is a reserved label key, `gpu.cozystack.io/usage-monitor`, and has admission refuse that key on every object in a tenant namespace, on create and update, so no object can match the monitor and no tenant can shape the GPU usage records. + +Two cases were left open. First, the GPU admission policies render only where GPUs serve VMs ([0002](./0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). Had the reservation followed them, a tenant could label its objects on a cluster without GPUs, and they would match the monitor the day the hoster turns GPUs on. Second, an object can already carry the key when the policy first renders, from before the upgrade that brought it. A policy that refuses every write of an object carrying the key would refuse every update of that object that keeps the label, including the one in which a controller removes its finalizer, so its deletion would hang until someone drops the label by hand. + +## Decision + +The reservation renders on every cluster where the `ValidatingAdmissionPolicy` API exists, whether or not GPUs are on, and it ratchets: it refuses a create that carries the key and an update that adds the key or changes its value, and it allows an update that keeps the value or drops the key. The rule lives in `packages/system/cozystack-basics/templates/gpu-usage-monitor-label-policy.yaml`, with its cases in `internal/gpu/policytest/usagemonitor_test.go`. + +## Why not the alternatives + +- **Render it with the other GPU policies.** That leaves the window above open, so the guarantee of section 5 would hold only for labels written after GPUs were enabled. +- **Refuse the key on every create and update, as section 5 has it.** It blocks every update of an object that predates the policy and carries the key, finalizer removal included, so controllers that know nothing about the label stall on it until a person removes the label. +- **Strip the key at admission instead of refusing it.** A mutating webhook on every object in every tenant namespace makes the controller's availability a dependency of every tenant write, for a label no legitimate writer sets. + +## Consequences + +- No new object in a tenant namespace can match the GPU usage monitor, on any cluster. +- An object that carried the key before the policy existed keeps it until someone removes it, and until then, if its value is `true`, the monitor's selector matches it. The policy cannot see such objects; the hoster finds them by listing objects with the label. +- The policy matches every create and update of every namespaced object in every tenant namespace, whether or not the cluster uses GPUs, so its expression reads only metadata. + +## Revisit if + +Cozystack gets a way to reserve label keys for its own controllers across tenant namespaces, so each reserved key no longer needs its own policy. diff --git a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md new file mode 100644 index 00000000..37fbb11e --- /dev/null +++ b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md @@ -0,0 +1,42 @@ +# 0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts + +- **Number:** `0004` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) (the fields); the node agent: not yet + +## Context + +Section 1 has the node agent write `GPUNode` from sysfs and kubelet's pod-resources API, with a `healthy` flag per PCI function, and section 2 decides lost capacity from devices the agent reports missing or unhealthy. Section 3 adds every resource name a GPU device plugin advertises to the GPU resource names, "as the agent reports it". Building the agent showed three gaps in that picture. + +The pod-resources API carries no device health. It lists the allocatable devices per resource name and the devices each pod holds, and kubelet lists as allocatable only the devices their plugin reports healthy. + +One PCI function is not always one unit. An mdev parent serves several plugin devices, so counting one healthy unit per function would make a node with more than one held mdev device look as if it had lost capacity. + +Not every GPU device ID maps to a PCI function. The GPU Operator's device plugin of the `container` variant reports NVML UUIDs, `GPU-…` or `MIG-…`, which sysfs cannot resolve, so the agent has no function to put such a GPU under, and section 3 still needs its resource name, such as `nvidia.com/gpu`. + +## Decision + +- A function is healthy when every plugin device on it is allocatable, and unhealthy when a pod that has not finished holds one of its devices that is not. A function no plugin device maps to has no health. +- Each function reports `allocatable`, the number of its plugin devices kubelet lists as allocatable: 1 for a VF or a GPU passed through whole, N for an mdev parent. The controller sums it over healthy functions, and a node has lost capacity for a resource name when it holds more units than that sum, or when a function is unhealthy. +- `GPUNode.status.resourceNames` lists every resource name on the node whose devices are GPUs, including names whose device IDs are NVML UUIDs the agent cannot place under a function. The names join the cluster's GPU resource names of section 3. +- A function serving several pods, such as an mdev parent, names one of them in `usedBy`, the smallest by namespace and name, so the report is stable. + +## Why not the alternatives + +- **Read health from each device plugin's socket.** The agent would talk to every plugin the host layer runs, while kubelet already folds their health into what it lists as allocatable, and kubelet's view is the one the scheduler acts on. A judgement call, not a hard limit. +- **Count one unit per function.** Wrong for mdev parents, as above, and it would fire lost capacity on nodes that have lost nothing. +- **Leave GPUs the agent cannot place out of the report.** Section 3 then misses `nvidia.com/gpu`, so a VMI or a pod that names it would not count as carrying a GPU entry. + +## Consequences + +- An unhealthy device that no pod holds disappears from the allocatable list and looks the same as a missing one. Both lower `allocatable`, and both become lost capacity after the grace period once the node holds more units than remain. +- `usedBy` traces only one pod per function. Section 1's "ownership is exact for running GPUs" holds per function for VFs and whole GPUs, and for an mdev parent only through the devices' own `status.attachedTo`. +- The agent does not fill `profile` yet; writing and reading VF profiles is host-layer work (Rollout, step 4). + +## Revisit if + +kubelet's pod-resources API starts reporting device health, or a device plugin that Cozystack ships reports IDs the agent can neither resolve through sysfs nor recognise as NVML UUIDs. diff --git a/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md new file mode 100644 index 00000000..c3aab6a7 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md @@ -0,0 +1,31 @@ +# 0005. The taint rule skips exactly the three taints that mirror a cordon or readiness + +- **Number:** `0005` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) + +## Context + +Section 2 keeps binds, moves, rebinds and gathers off nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and leaves out "the `node.kubernetes.io/*` taints Kubernetes sets itself for a cordon or a node that is not `Ready`", because the cordon and readiness rules already cover those nodes. Read as a prefix, that sentence would also skip the other well-known taints Kubernetes defines under `node.kubernetes.io/`: `memory-pressure`, `disk-pressure`, `pid-pressure`, `network-unavailable` and `out-of-service`. None of them mirrors a cordon or readiness, and a node carrying one of them as `NoSchedule` or `NoExecute` would still be bound to, though the scheduler keeps a virt-launcher pod that does not tolerate it off the node. + +## Decision + +The rule skips exactly `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`. Every other `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate counts, under `node.kubernetes.io/` or not. The rule and its cases live in `internal/gpu/model/node.go` and `node_test.go`. + +## Why not the alternatives + +- **Skip the whole `node.kubernetes.io/` prefix.** It binds devices onto nodes that the scheduler then refuses, and keeps their units in `free`, which is the failure the taint rule exists to prevent. +- **Count the three as well.** Kubernetes sets `node.kubernetes.io/unschedulable` on every cordoned node, so a cordoned node would also read as tainted. A taint keeps a bound device where it is, so under `Move` a cordon would never move a device, and `onCordon` would stop meaning anything. + +## Consequences + +- A node under memory, disk or PID pressure, one whose network is unavailable, and one tainted `out-of-service` get no binds, moves or gathers while the taint holds and virt-launcher pods do not tolerate it, and their units leave `free` and `maxFreePerNode`. Outside a gather at VMI creation, a device already bound there stays, as on any tainted node. +- A pressure taint comes and goes with the node's condition, so `free` on a busy class can drop and return without any device moving. + +## Revisit if + +Kubernetes adds a taint of its own that mirrors a cordon or readiness, or Cozystack's virt-launcher pods start tolerating taints by default. diff --git a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md new file mode 100644 index 00000000..235ad1d6 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md @@ -0,0 +1,32 @@ +# 0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the device + +- **Number:** `0006` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** not yet + +## Context + +Section 3 pins a VMI to a cordoned node when a device of a `Stay` class is there and all of the VMI's devices are on that node, with no template devices: there is no gather, "even if the node is also not `Ready`", and the VMI waits. Section 2 has a `Stay` device keep its node through a cordon, while lost capacity, a node that is gone and a node that no longer serves the class are hard reasons for a device to leave. The proposal did not say which of the two wins when a pinned node also has a hard reason, so a node under maintenance that then died could hold a VM's devices, and keep them in the usage records, for as long as the dead node stayed registered. + +## Decision + +The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint virt-launcher pods do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (sections 2 and 3): a new VMI is refused when the node is gone, no longer serves the class, or has lost capacity the device does not fit under, and otherwise its set is gathered onto one node with room for all of it; a VMI already waiting there unscheduled is deleted when its devices must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The binder at admission and the controller's background pass read the pin through one function, so admission never pins a VMI that the next pass deletes. + +## Why not the alternatives + +- **The pin always wins.** A `Stay` class exists for a hoster who wants VMs back on the same hardware after maintenance. A dead node is not coming back to the same state on any known schedule, and the VM's devices would stay held and recorded against hardware that serves nothing. +- **Break the pin on `NotReady` alone.** It undoes the point of `Stay` for the most common maintenance step, a reboot, and section 2 already uses the lost-capacity grace period so that a short outage moves nothing. +- **Break the pin on any taint.** Apart from a gather at VMI creation, a taint never moves a bound device; it only keeps new binds off the node (section 2). Breaking the pin on one would make a `Stay` device leave in a case where a device of a `Move` class on an uncordoned node stays. + +## Consequences + +- A `Stay` VM waits through a reboot of its cordoned node and comes back on it, and after a node failure its devices move like any other VM's once the grace period has passed. +- A pinned VM waits on a dead node for the lost-capacity grace period, five minutes by default, before its devices can move. + +## Revisit if + +Hosters ask for a `Stay` policy that also outlasts a node failure, for example to wait for a repaired node rather than move off it. diff --git a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md new file mode 100644 index 00000000..db240bc7 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md @@ -0,0 +1,33 @@ +# 0007. Whoever can write a VM orders its template devices, while held devices stay with tenant admins + +- **Number:** `0007` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** not yet + +## Context + +The Security section gives `GPUDevice` writes to tenants with the `admin` role or above, and has the webhook create `GPUDevice` objects for `deviceTemplate` entries, only of the class and count the template names. It does not say who that webhook acts for. The VMI is created by KubeVirt's virt-controller, not by the person who wrote the VM, so by the time the webhook sees it the requester is KubeVirt and the identity behind the VM is gone. + +In Cozystack more identities can write a `vm-instance` than can write a `GPUDevice`. The tenant role that tenant ServiceAccounts get grants every verb on `apps.cozystack.io`, which includes `vm-instance`, and the `use` role can start and restart a VM, which creates a new VMI. + +## Decision + +A `deviceTemplate` entry is part of the VM, so whoever can write the VM can order its template devices, and whoever can start it makes them exist and enter the usage records for as long as the VMI runs. The webhook applies the tenant's budgets, the vGPU entitlement and capacity to every template device, as to any order, and does not check the identity behind the VM. Ordering a held `GPUDevice` directly stays with tenant admins and above. + +## Why not the alternatives + +- **Check the identity behind the VM at VMI admission.** The webhook sees KubeVirt's service account. The VM's author is recorded nowhere it could read reliably, and the identity that starts a VM can differ from the one that wrote it. +- **Refuse `deviceTemplate` entries in `vm-instance` for identities that cannot write `GPUDevice`.** The aggregated API that serves `vm-instance` would need a per-field authorization step for one field of one kind, and the rule would still not cover a start by a `use` role holder, who changes no field at all. The cost is a judgement call, not a hard limit. + +## Consequences + +- A tenant admin who hands out a tenant ServiceAccount also hands out the ability to order GPUs through template devices, up to the tenant's budget, and the `use` role lets its holder start VMs that already carry template devices. The budget, the class keys and the vGPU entitlement are the controls that limit both. +- A held device, which is held and recorded whether or not anything runs on it, still needs a tenant admin. + +## Revisit if + +KubeVirt starts recording the VM's author on the VMI in a form admission can trust, or Cozystack gains per-field authorization on application kinds. diff --git a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md new file mode 100644 index 00000000..cd15d5f8 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md @@ -0,0 +1,31 @@ +# 0008. The GPU controller gets its node-label and plugin-restart rights with the host layer, not before + +- **Number:** `0008` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** not yet + +## Context + +The Security section lists three rights the GPU controller gains beyond its own kinds and the usage records: patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its devices. Only the third is used by Rollout step 1. Node labels and plugin restarts belong to the node layout of section 6, which ships as Rollout step 4, and until then the hoster writes node labels by hand. + +## Decision + +Of the three, Rollout step 1 grants the controller only the VMI deletion. The node-label patch right, the policy that restricts it, and the right to restart the device plugin pod ship with the host layer, together with the code that uses them. + +## Why not the alternatives + +- **Grant all three in step 1, as the Security section lists them.** Every cluster would carry a write right on every `Node` that no code uses, and the policy that narrows it would have no writer to test against. +- **Grant the label right without its policy until the host layer ships.** That is a broader right than the proposal ever allowed. + +## Consequences + +- Between step 1 and step 4 the controller cannot change any node, so the GPU node labels that classes match are the ones the hoster writes. +- The rights and their policy are reviewed in step 4, next to the code that uses them. + +## Revisit if + +The host layer is dropped or replaced by one that does not need the controller to write nodes. From a08936f361625fb657a6b3e9b68d49cda1cc5aed Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 21 Oct 2026 04:28:44 +0300 Subject: [PATCH 03/32] docs(design-proposals): align the GPUDevice proposal with its decisions Several sentences of the proposal now contradict what the decision records settled, or leave out the rule the records add: the taint rule read as a whole prefix, the usage-monitor label as refused on every update, and the Security section as granting the controller its node rights and tenant admins their writes on every cluster. Change those sentences and point each one at its record, so a reader of the proposal does not get the old rule. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 29 +++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 8b0ac848..c2e5955c 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -129,7 +129,9 @@ status: function: vf profile: L40S-24C healthy: true + allocatable: 1 # N for an mdev parent usedBy: {namespace: tenant-acme, pod: virt-launcher-vm-instance-trainer-x7k2p} + resourceNames: [nvidia.com/NVIDIA_L40S-24C] ``` A class maps an offer to a backend through a `source` union and to the nodes that serve it through `nodeSelector`. The first release has one source, `devicePlugin`; `dra` is reserved for section 8. Two classes may share a resource name when their node selectors do not overlap, which lets a host layer that cannot tell two offers apart by name tell them apart by node. A node that matches two classes with the same resource name gets a condition on both classes and on its `GPUNode`, which names the two classes, and is not used for binding. Class names are DNS labels, since they become part of quota keys. @@ -138,7 +140,7 @@ A class referenced by held devices cannot disappear or change its meaning. A fin A `GPUDevice` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate devices. How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 6), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPU devices of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. Admission refuses a device above `maxCount`. The spec is immutable, so a change is a new order. -The inventory is written by a node agent from sysfs and from kubelet's pod-resources API, and goes away with its `Node`. Ownership is exact for running GPUs: a `usedBy` pod leads to its VM, and the VM's devices name it in `status.attachedTo`. A held but not running reservation is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. PCI addresses stay in `GPUNode`, which tenants cannot read. +The inventory is written by a node agent from sysfs and from kubelet's pod-resources API, and goes away with its `Node`. That API carries no device health, so a function is healthy when kubelet lists its devices as allocatable, each function reports how many of them are, one for a VF or a whole GPU and several for an mdev parent, and `resourceNames` lists the node's GPU resource names, including those whose devices the agent cannot place under a PCI function ([0004](./decisions/0004-gpunode-health-is-kubelet-allocatability.md)). Ownership is exact for running GPUs: a `usedBy` pod leads to its VM, and the VM's devices name it in `status.attachedTo`; an mdev parent that serves several pods names one of them. A held but not running reservation is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. PCI addresses stay in `GPUNode`, which tenants cannot read. What `Bound` guarantees depends on the class's source: @@ -151,7 +153,7 @@ A tenant sees a class, a count, a phase and, once there is one, a node, never wh ### 2. Binding and capacity -A controller in `cozystack-controller` binds each new `GPUDevice` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`. The taint rule leaves out the `node.kubernetes.io/*` taints Kubernetes sets itself for a cordon or a node that is not `Ready`, which the first two rules already cover. A device already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI creation (section 3): it may take an unattached device off a node that is not `Ready`, because no VMI uses that device and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound devices, minus the units pods hold beyond what their devices cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. +A controller in `cozystack-controller` binds each new `GPUDevice` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`. The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included ([0005](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md)). A device already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI creation (section 3): it may take an unattached device off a node that is not `Ready`, because no VMI uses that device and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound devices, minus the units pods hold beyond what their devices cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. Admission refuses an order that cannot be met at that moment, by capacity or by budget (section 4), with a message naming the class, the count and the limit it hit: a budget of the tenant or an ancestor, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. @@ -195,14 +197,15 @@ gpus: pool: inference ``` -The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named devices. For a pool worker it takes the device the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` device on a `Ready` node that matches the selector and that no VM names in a `device` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same device, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the device, so two VMIs can never hold one device, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A device still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A device that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved device; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Devices a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the devices a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template devices, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template devices included. This runs after the other refusal checks below and after the template devices pass their budget and entitlement checks, so every device in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a device of it, and each device keeps its hold. A device of a `Stay` class on a cordoned node does not move. When all of the VMI's devices are on that node and the VMI has no template devices, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. Otherwise the set cannot be gathered, and the VMI is refused. With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each device and its node. A move that does not finish within the admission timeout refuses the VMI the same way; devices already moved keep their new node, and KubeVirt's retry continues from there. Template devices are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the device's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. +The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named devices. For a pool worker it takes the device the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` device on a `Ready` node that matches the selector and that no VM names in a `device` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same device, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the device, so two VMIs can never hold one device, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A device still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A device that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved device; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Devices a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the devices a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template devices, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template devices included. This runs after the other refusal checks below and after the template devices pass their budget and entitlement checks, so every device in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a device of it, and each device keeps its hold. A device of a `Stay` class on a cordoned node does not move. When all of the VMI's devices are on that node and the VMI has no template devices, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. When some of its devices are elsewhere, or it has template devices, the set cannot be gathered, and the VMI is refused. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a device of the set has lost capacity there, the node is gone, or it no longer serves a device's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md)). With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each device and its node. A move that does not finish within the admission timeout refuses the VMI the same way; devices already moved keep their new node, and KubeVirt's retry continues from there. Template devices are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the device's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. The webhook refuses to start a VMI, and names each device and its node, when a referenced device: - is `Pending`, `Lost`, being deleted, or attached to another VMI; - is bound to a node that is gone, no longer matches the class, has `LostCapacity` the device does not fit under, or has a pending layout change (section 6); - cannot be brought, with the VMI's other devices and template devices, onto one node with room for all of them (above), for example because a device of a `Stay` class is on a cordoned node while other devices need gathering; -- is a vGPU device next to vGPU devices of another class, or brings the VMI's GPUs of its class above `maxCount`. +- is a vGPU device next to vGPU devices of another class, or brings the VMI's GPUs of its class above `maxCount`; +- is of a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` ([0001](./decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md)). A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are the resource names of `GPUClass` objects and the names in the hoster's list `gpu.resourceNames`, which defaults to the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. The list also takes every resource name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin, as the agent reports it, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU device for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its devices unattached, so they move by the rules of section 2. Both webhooks see only VMIs and pods carrying the chart's reference label and fail closed, so an outage blocks the start of VMs with device references and nothing else. @@ -210,7 +213,7 @@ A device cannot disappear under a running VM: a deleted `GPUDevice` stays, with A `deviceTemplate` entry names a class and a count instead of a held device. At VMI creation the webhook creates a `GPUDevice` from it, named after the VMI and the entry's index and marked as a template device. A repeated call or a recreated VMI finds the same device and reuses it when it carries that mark, has no owner yet and matches the template; a device of that name still being deleted, or still owned by the VM's previous VMI, refuses the VMI, and KubeVirt retries. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked device that already has the name refuses the VMI, and the message names it. The webhook has the binder bind the device under the same capacity and budget checks as any order, within the admission timeout, and attaches it. The node for template devices is picked with room for all of the VMI's devices, held and template: the node of the gathered set when the VMI has held devices, and otherwise one node with room for all its template devices. The VMI is refused if no node has room for all of them. A bind that does not fit or does not finish in time refuses the VMI, and KubeVirt retries. On a dry run the webhook creates, attaches and moves nothing, so it declares `sideEffects: NoneOnDryRun`. The VMI has no UID yet at that point, so the controller sets the VMI as the device's owner once the VMI exists, and deletes a template device whose VMI never appears after the same grace period as a stale attachment. The controller deletes a template device as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the device lives and is billed only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any device, but only while the VMI exists: an order that does not fit is refused when the VM or worker starts, not ahead of time. A template counts as a device reference under `forbidden`. It lets an autoscaled pool pay for running workers only. -A `vm-instance` entry has exactly one of `name`, `device` or `deviceTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `deviceSelector` or `deviceTemplate`. A list holds only `name` entries or only references, and a pool has at most one `deviceSelector` or one `deviceTemplate`, never both, and a selector's devices must share a class and a `count`; the webhook checks this at each attach, since device labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to Kairos is separate work. +A `vm-instance` entry has exactly one of `name`, `device` or `deviceTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `deviceSelector` or `deviceTemplate`. A list holds only `name` entries or only references, and a pool has at most one `deviceSelector` or one `deviceTemplate`, never both, and a selector's devices must share a class and a `count`; the webhook checks this at each attach, since device labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `device` and `deviceTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to Kairos is separate work. Kairos pools take GPUs only through `deviceSelector` or `deviceTemplate`, from classes of both modes. The chart refuses `name` entries on them. The pool's mode, `kairos.gpuMode`, must match the class, and the chart sets it as the `gpu.cozystack.io/mode` label on the worker VMI template ([Choosing the worker image by mode](../kairos-vgpu-workers/README.md#choosing-the-worker-image-by-mode)). At VMI creation the webhook refuses a device, selected or from a template, whose class `mode` differs from that label, and a pool reference on a VMI without the label. The `kubernetes-nodes` chart, from Rollout step 2, fails to render a `deviceSelector` or `deviceTemplate` on a pool with `os: talos`, the field cozystack/cozystack#4911 adds, so the pool's HelmRelease fails before any machine is created. A pool on `deviceSelector` never runs more GPU workers than it has matching devices. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer devices sets it. A pool without spare devices rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a device, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools, existing and new, keep working as today on `name` entries, and `GPUDevice` does not extend them. @@ -231,7 +234,7 @@ A parent with `total: 4` caps everything below it at four GPUs, each vGPU counti ### 5. Usage accounting, catalog and licensing -Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads. Each `GPUDevice` gets one with `kind: gpu-device` and resources `gpu.cozystack.io/: ` while it is `Bound` or `Attached`, a `dra` device only once its claim is allocated (section 8), next to the records Cozystack already keeps for CPU, memory and storage. The GPU controller writes the record itself, under a `WorkloadMonitor` it owns in the tenant's namespace, whose selector matches no object, so tenant labels cannot drop a device from the usage records. The selector is a reserved label key that admission refuses on every object in a tenant namespace, on create and update, whoever makes the request, so no object can match it, not even a virt-launcher pod or a DataVolume that inherits the label from a VM template. Each record carries the `workloads.cozystack.io/monitor` label naming that monitor: the `Workload` controller deletes a record only when that label is missing or names a monitor that is gone, and deletes nothing by selector. Like other `Workload` records it has no time fields: a held interval is the record's lifetime, which a billing system observes by watching and reconciles by sampling. A virt-launcher pod does not report GPUs its devices cover a second time. +Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads. Each `GPUDevice` gets one with `kind: gpu-device` and resources `gpu.cozystack.io/: ` while it is `Bound` or `Attached`, a `dra` device only once its claim is allocated (section 8), next to the records Cozystack already keeps for CPU, memory and storage. The GPU controller writes the record itself, under a `WorkloadMonitor` it owns in the tenant's namespace, whose selector matches no object, so tenant labels cannot drop a device from the usage records. The selector is a reserved label key that admission refuses on every object in a tenant namespace, on create and on an update that adds it or changes its value, whoever makes the request and whether or not GPUs are on, so no new object can match it, not even a virt-launcher pod or a DataVolume that inherits the label from a VM template; an update that keeps or drops a label set before the policy existed passes ([0003](./decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md)). Each record carries the `workloads.cozystack.io/monitor` label naming that monitor: the `Workload` controller deletes a record only when that label is missing or names a monitor that is gone, and deletes nothing by selector. Like other `Workload` records it has no time fields: a held interval is the record's lifetime, which a billing system observes by watching and reconciles by sampling. A virt-launcher pod does not report GPUs its devices cover a second time. The catalog is the list of `GPUClass` objects with `displayName`, model, mode, profile, memory, `maxCount`, `status.free` and `status.maxFreePerNode`. Every authenticated user reads it, because a hoster publishes its offers and a tenant names a class when ordering, including from `kubectl` or GitOps without a portal. So the status holds only `free` and `maxFreePerNode`, and its conditions, such as `LostCapacity`, name no nodes and carry no counts. The class's capacity, bound units and `Lost` count show the hoster's inventory and load, and they are not in the status. The controller exports them as metrics that platform monitoring scrapes. Per node, a hoster reads device occupancy and health from `GPUNode`, and lists bound devices with a field selector on the `GPUDevice` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write. A tenant can still watch `free` and `maxFreePerNode` over time and see how busy a class is. The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest order, leaving out a field that is absent because it sets no limit. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. A `gpudevice` source lists a tenant's own devices. The existing `gpu` source keeps serving raw names while they are allowed. @@ -267,7 +270,7 @@ A tenant moves a VM by stopping it, ordering devices of the matching class, repl Quota keys move too. A raw GPU key is a quota key on a GPU resource name, such as `nvidia.com/AD102GL_L40S`. A device-backed VM's pod still requests that name, so while a tenant migrates, its raw key covers both kinds of VM; after its last raw-name VM has moved, the key is replaced by `gpu.cozystack.io/*` keys. A tenant whose raw-name VMs use vGPU can order their devices only after it is entitled to vGPU (section 4), and within any class keys set above it. -When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. Preconditions: no Talos GPU pools in the cluster, ComputePlane GPU groups included (a hoster without a passthrough worker image keeps passthrough Talos pools and raw names), no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: +When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. Preconditions: no Talos GPU pools in the cluster, ComputePlane GPU groups included (a hoster without a passthrough worker image keeps passthrough Talos pools and raw names), no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: - an admission policy refuses GPU entries (section 3) on every VMI created without a device reference, on every path; - the same policy refuses every pod created in a tenant namespace that requests a GPU resource name, unless it is a virt-launcher pod created by KubeVirt's virt-controller service account with an owner reference to a VirtualMachineInstance. Tenants cannot forge such a pod, since no tenant role can create pods or VMIs, so app charts that copy free-form `resources` into pods cannot request GPUs either; @@ -306,12 +309,12 @@ Three rules hold throughout: ## Security -- Tenant access. Tenants with the `admin` role or above manage `GPUDevice` in their namespaces; lower roles read it. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. Tenants cannot read `GPUNode`. -- What a tenant controls. A tenant never supplies a resource name or a node through a device reference: both come from `GPUClass` and from the device's status, which tenants cannot write. Tenants cannot add or remove finalizers on devices, so a held device cannot leave the usage records. Admission refuses the reserved label of the GPU usage monitor on any object in a tenant namespace, on create and update (section 5). A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes devices only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a device reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. +- Tenant access. Tenants with the `admin` role or above manage `GPUDevice` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. Tenants cannot read `GPUNode`. +- What a tenant controls. A tenant never supplies a resource name or a node through a device reference: both come from `GPUClass` and from the device's status, which tenants cannot write. Tenants cannot add or remove finalizers on devices, so a held device cannot leave the usage records. Admission refuses setting the reserved label of the GPU usage monitor on any object in a tenant namespace (section 5). A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes devices only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a device reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. - Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. - Node agent. It reads sysfs and kubelet's pod-resources socket, writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can write only its own node's `GPUNode`. -- Webhook. For `deviceTemplate` entries it creates `GPUDevice` objects in tenant namespaces, only of the class and count the template names. -- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights: patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its devices. +- Webhook. For `deviceTemplate` entries it creates `GPUDevice` objects in tenant namespaces, only of the class and count the template names. It does not check who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template devices, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md)). +- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its devices. It also writes one ConfigMap, the GPU settings in `cozy-system`. - Visibility. A tenant sees the node name of its own devices and, through the remaining budget, its siblings' GPU count under a shared parent budget (section 5). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. ## Failure and edge cases @@ -342,7 +345,7 @@ Each case gives what is seen and what to do. - Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no device reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. - Controller tests for the class metrics: capacity, bound units and the `Lost` count match the devices, and none of them appears in `GPUClass` status. - Binder tests: a node with a `NoSchedule` taint that virt-launcher pods do not tolerate gets no binds, moves or gathers and its units leave `free`, while the Fence Agents Remediation taint alone does not make a node cordoned. -- Admission tests for the reserved label of the GPU usage monitor, refused on any object in a tenant namespace, on create and update, including a virt-launcher pod and a DataVolume that get it from a VM template. +- Admission tests for the reserved label of the GPU usage monitor, refused on any object in a tenant namespace on create and on an update that adds it or changes its value, including a virt-launcher pod and a DataVolume that get it from a VM template, and allowed on an update that keeps or drops it. - Controller tests for the usage record: present exactly while `Bound` or `Attached`, independent of tenant labels, not reported twice, and kept through reconciles of its `WorkloadMonitor` and of the `Workload` controller. - Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, `WorkloadMonitor`, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. - E2E with passthrough GPUs on the GPU runner: order, attach, stop, confirm another tenant cannot take the GPU, start again, and check the usage record. @@ -350,7 +353,7 @@ Each case gives what is seen and what to do. ## Rollout -1. `GPUClass`, `GPUDevice`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, both webhooks, the policies, the option sources, `device` and `deviceTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants. Before tenants order vGPU devices, the `gpu.vgpu` flag is set from the root tenant down. +1. `GPUClass`, `GPUDevice`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, both webhooks, the policies, the option sources, `device` and `deviceTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRDs ship and finalizers and usage records still run. Before tenants order vGPU devices, the `gpu.vgpu` flag is set from the root tenant down. 2. `deviceSelector` and `deviceTemplate` in `kubernetes-nodes`, after cozystack/cozystack#4911. 3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, `kairos.gpuMode` and the mode label, and `vgpu-vms` adds GPU guest images for VMs with `device` entries. A cluster with Talos GPU pools stays on raw names and has no hard hold guarantee. 4. The host layer: the node layout, profile writes, node labels, the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, and Kairos vGPU hosts (the bundle on Kairos and a vGPU Manager image for it). From 057cc3fb8b97feefdf542247585d8aa4dd0ef1a7 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 21 Oct 2026 04:28:58 +0300 Subject: [PATCH 04/32] docs(design-proposals): cover a Stay-pinned VM on a node that dies The failure cases did not say what happens to a VM pinned by a Stay device to a cordoned node when that node then fails, and section 3 alone reads as if such a VM waits on the dead node indefinitely. Describe the implemented behaviour: the pin holds through NotReady and gives way once the node has lost capacity, after which the set moves or goes to Lost as any other set does. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index c2e5955c..fb60e0ed 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -328,6 +328,7 @@ Each case gives what is seen and what to do. - A cordoned node is deleted while a drained VM's VMI still waits for it → the controller deletes that unscheduled VMI, the device moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. - A node dies but stays registered → after the grace period its attached devices go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost` (section 2); once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the devices are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached devices onto a `Ready` node with room (section 3) and starts there at once. That holds for a VM's held devices. A pool worker takes only a spare device of its pool, template devices follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node (all in section 3). After a `Node` deletion the VM starts on a retry instead, once the controller has moved the devices. Only when no node fits is the VMI refused, and then it is not created at all. KubeVirt v1.9.0 retries the creation with backoff for `Always`, which deletes the failed VMI and starts a new one, and for `RerunOnFailure`, whose failed VMI leaves a start request that stays until a VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)); such a VM starts once a node has room for its set. The retry goes through the VM controller's rate-limited queue, whose per-item backoff grows to 1000 seconds, so the start can come up to about 17 minutes after room appears. `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its devices unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. - A node loses capacity while a VM's VMI waits for it unscheduled → the controller deletes that VMI so its devices can move; an `Always` or `Once` VM starts again once its devices have moved or been rebound, a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. +- A cordoned node dies while a VM's VMI waits on it, pinned there by a device of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its devices stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md)): the controller deletes the unscheduled VMI, and the VM's devices move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has (section 2). An `Always` or `Once` VM then starts on the new node once its devices have moved or been rebound; a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. - A GPU fails under a running VM → the device stays `Attached` and in the usage records while the VM runs; when the VM restarts it is refused there, and the device moves or goes to `Lost`. - A layout change does not apply → the node shows `LayoutPending`; drain it. A VM with a raw `name` entry on the node holds the change until it stops. - A Talos GPU pool, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a Kairos pool with `deviceSelector`, or allow raw names again while the cluster has Talos GPU pools. From 2c36d72ec28fa35c3fea97d26e62963440cd93b5 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 21 Oct 2026 14:29:40 +0300 Subject: [PATCH 05/32] docs(design-proposals): state the GPU budget rule without the pool details Section 4 and the rollback rules explained why GPU keys stay out of the tenant quota pools by walking through how pool membership treats undeclared keys. The rule itself is what the design needs: GPU keys stay out of the pool and carve-out model, so a budget covers every descendant, and raw keys keep following the pools. The Stay-pin sentence in section 3 now says "no template devices still to place", which is the condition the implementation checks. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index fb60e0ed..6c170b5b 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -197,7 +197,7 @@ gpus: pool: inference ``` -The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named devices. For a pool worker it takes the device the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` device on a `Ready` node that matches the selector and that no VM names in a `device` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same device, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the device, so two VMIs can never hold one device, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A device still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A device that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved device; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Devices a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the devices a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template devices, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template devices included. This runs after the other refusal checks below and after the template devices pass their budget and entitlement checks, so every device in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a device of it, and each device keeps its hold. A device of a `Stay` class on a cordoned node does not move. When all of the VMI's devices are on that node and the VMI has no template devices, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. When some of its devices are elsewhere, or it has template devices, the set cannot be gathered, and the VMI is refused. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a device of the set has lost capacity there, the node is gone, or it no longer serves a device's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md)). With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each device and its node. A move that does not finish within the admission timeout refuses the VMI the same way; devices already moved keep their new node, and KubeVirt's retry continues from there. Template devices are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the device's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. +The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named devices. For a pool worker it takes the device the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` device on a `Ready` node that matches the selector and that no VM names in a `device` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same device, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the device, so two VMIs can never hold one device, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A device still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A device that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved device; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Devices a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the devices a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template devices, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template devices included. This runs after the other refusal checks below and after the template devices pass their budget and entitlement checks, so every device in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a device of it, and each device keeps its hold. A device of a `Stay` class on a cordoned node does not move. When all of the VMI's devices are on that node and the VMI has no template devices still to place, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. When some of its devices are elsewhere, or it has template devices, the set cannot be gathered, and the VMI is refused. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a device of the set has lost capacity there, the node is gone, or it no longer serves a device's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md)). With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each device and its node. A move that does not finish within the admission timeout refuses the VMI the same way; devices already moved keep their new node, and KubeVirt's retry continues from there. Template devices are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the device's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. The webhook refuses to start a VMI, and names each device and its node, when a referenced device: @@ -221,7 +221,7 @@ Kairos pools take GPUs only through `deviceSelector` or `deviceTemplate`, from c A tenant's GPU budget is declared in `resourceQuotas`, per class as `gpu.cozystack.io/: ` and in total as `gpu.cozystack.io/total: `; the class name `total` is reserved. These keys are not rendered into the namespace `ResourceQuota`, since `GPUDevice` is not a pod, and a tenant with only GPU keys gets no `ResourceQuota` or `LimitRange`. That is a change to the tenant chart, which today renders both for any `resourceQuotas` key. -GPU budgets follow their own rule, not the pool and carve-out rules for other resources: every GPU key caps the whole subtree of the tenant that declares it. Today's pools cannot carry GPU keys, since a tenant that declares any key becomes its own pool root and leaves its parent's limits for every key it does not declare, so a child with only `cpu` would escape its parent's GPU budget. The tenant declaration gate skips `gpu.cozystack.io/*` keys, because under this rule a child's key may overlap its siblings'. +GPU budgets follow their own rule, not the pool and carve-out rules for other resources: every GPU key caps the whole subtree of the tenant that declares it. GPU keys stay out of the pool and carve-out model, so a GPU budget covers every descendant of the tenant that declares it. The tenant declaration gate skips `gpu.cozystack.io/*` keys, because under this rule a child's key may overlap its siblings'. - An order is refused if, for the tenant or any ancestor that declares the class key or `total`, the devices counting against the budget (table in section 2) in that ancestor's subtree would exceed the key. - A child's key only adds a limit; it never lifts an ancestor's limit or a missing entitlement. @@ -304,7 +304,7 @@ The phases run in this order. The hoster removes the node layout and chooses the Three rules hold throughout: - Names move back by the window rule of section 6 in reverse: vGPU classes return to the hand-written VF names, and raw `name` entries and raw keys on the per-profile names move with them in the same window. -- Raw keys go through the existing tenant quota pools, not the subtree rule of section 4: a declared key makes a tenant its own pool root for every resource and a child's key is carved out of its parent's pool. So raw keys go only to tenants that already declare quotas other than `gpu.cozystack.io/*`, and raw names elsewhere are unlimited, as before this proposal. +- Raw keys go through the existing tenant quota pools, not the subtree rule of section 4, so adding one changes how the tenant's other quotas are pooled. Raw keys therefore go only to tenants that already declare quotas other than `gpu.cozystack.io/*`, and raw names elsewhere are unlimited, as before this proposal. - A raw key also renders the tenant chart's default `LimitRange`, whose memory limit can sit below a VM's memory request, because KubeVirt sets virt-launcher limits from the namespace quota only when it limits `cpu` and `memory`, so a tenant given a raw key also declares `cpu` and `memory`. ## Security From 5fad72255c787e3e6443c84c49c789cbfb1efae1 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:15:06 +0300 Subject: [PATCH 06/32] docs(design-proposals): rename GPUDevice to GPUClaim A GPUDevice with count 2 is a claim on units of a node, not a device, and kubelet picks the physical device only when the pod starts. The kind, its gpus entries, the reference label and annotation, the template mark, the finalizer, the option source and the usage record kind follow the new name, so renaming a v1alpha1 kind later is not needed. "Device" stays where it means hardware. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 193 +++++++++--------- ...tedhostdevices-is-refused-at-admission.md} | 16 +- ...-where-gpu-operator-serves-gpus-to-vms.md} | 22 +- ...l-reservation-is-always-on-and-ratchets.md | 2 +- ...punode-health-is-kubelet-allocatability.md | 2 +- ...e-skips-exactly-three-kubernetes-taints.md | 8 +- ...y-to-a-node-that-cannot-serve-the-claim.md | 32 +++ ...-to-a-node-that-cannot-serve-the-device.md | 32 --- ...n-write-a-vm-orders-its-template-claims.md | 33 +++ ...-write-a-vm-orders-its-template-devices.md | 33 --- ...s-host-layer-rights-with-the-host-layer.md | 2 +- 11 files changed, 187 insertions(+), 188 deletions(-) rename design-proposals/gpudevice/decisions/{0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md => 0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md} (54%) rename design-proposals/gpudevice/decisions/{0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md => 0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md} (58%) create mode 100644 design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md delete mode 100644 design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md create mode 100644 design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md delete mode 100644 design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 6c170b5b..eb9dfd98 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -1,6 +1,6 @@ -# GPUDevice: GPUs as a tenant resource +# GPUClaim: GPUs as a tenant resource -- **Title:** `GPUDevice: a GPU catalog, per-tenant GPU reservations for VMs and worker pools, and the host layer under them` +- **Title:** `GPUClaim: a GPU catalog, per-tenant GPU reservations for VMs and worker pools, and the host layer under them` - **Author(s):** `@lexfrei` - **Date:** `2026-10-14` - **Status:** Review @@ -9,28 +9,28 @@ A tenant who wants a GPU today writes a device-plugin resource name such as `nvidia.com/AD102GL_L40S` into `gpus` of a VM or a worker pool and hopes one is free. The `gpu` option source shows how many of each name the nodes advertise and on which nodes, but nothing says which model, mode or profile a name stands for, nothing holds a GPU while the VM is off, and an order that does not fit is accepted and then sits in `Pending`. A hoster selling GPUs needs the opposite: a catalog of GPU models and modes, GPUs a tenant takes within a quota and keeps until they release them, the same handling for VMs and managed Kubernetes worker pools, and a usage record of who held what and when, which the hoster's external billing system reads. -This proposal adds three kinds. `GPUClass` is the hoster's catalog: one entry per offer, such as a whole H100 in passthrough or an L40S-24C vGPU profile. `GPUDevice` is a tenant's GPU, or a set of GPUs on one node: ordered against a class, bound to a node with free capacity, and attached to a VM by name or to a Kairos worker pool by a label selector. `GPUNode` is the inventory: every physical GPU and VF on a node, its configuration, and which tenant uses it. The proposal also covers the host layer under them, as its own phase: passthrough and vGPU nodes in one cluster, declarative vGPU profiles, and the device plugin that advertises GPUs to KubeVirt. +This proposal adds three kinds. `GPUClass` is the hoster's catalog: one entry per offer, such as a whole H100 in passthrough or an L40S-24C vGPU profile. `GPUClaim` is a tenant's GPU, or a set of GPUs on one node: ordered against a class, bound to a node with free capacity, and attached to a VM by name or to a Kairos worker pool by a label selector. `GPUNode` is the inventory: every physical GPU and VF on a node, its configuration, and which tenant uses it. The proposal also covers the host layer under them, as its own phase: passthrough and vGPU nodes in one cluster, declarative vGPU profiles, and the device plugin that advertises GPUs to KubeVirt. NVIDIA's vGPU stack changes quickly. Within the last few releases the driver branches, the way vGPUs are exposed (mediated devices or SR-IOV virtual functions), the KubeVirt device plugins and the licensing rules all changed. This proposal describes that stack as it was in October 2026. Expect it to be updated as NVIDIA and KubeVirt move, with a decision record under it for each change of course, and check the versions named here against your own before relying on them. ## Scope and related proposals -- [`kairos-vgpu-workers`](../kairos-vgpu-workers/README.md) gives managed Kubernetes worker pools on Kairos a worker image per GPU mode, with the vGPU guest driver or the data center driver, and delivers the vGPU license token to the workers. [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers VMs with passthrough and vGPU: the hoster adds GPU guest images to the golden image collection through `extraImages`, marked with a `gpu` field. Both are proposed alongside this one and use `GPUDevice` from the start, through `device` entries and `deviceSelector`; the links resolve once they merge. -- [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and section 2 rebinds their `Lost` devices. The link resolves once it merges. +- [`kairos-vgpu-workers`](../kairos-vgpu-workers/README.md) gives managed Kubernetes worker pools on Kairos a worker image per GPU mode, with the vGPU guest driver or the data center driver, and delivers the vGPU license token to the workers. [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers VMs with passthrough and vGPU: the hoster adds GPU guest images to the golden image collection through `extraImages`, marked with a `gpu` field. Both are proposed alongside this one and use `GPUClaim` from the start, through `claim` entries and `claimSelector`; the links resolve once they merge. +- [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and section 2 rebinds their `Lost` claims. The link resolves once it merges. - [IP addresses as a first-class resource](https://github.com/cozystack/community/pull/35) solves the same problem for addresses with a class, a tenant claim and a backend contract. -- [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared reservations the quota authority. A `GPUDevice` is a reservation in that sense, and section 4 hands GPU budgets over to it only if its gate takes over subtree caps for GPU keys. +- [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared reservations the quota authority. A `GPUClaim` is a reservation in that sense, and section 4 hands GPU budgets over to it only if its gate takes over subtree caps for GPU keys. - Host-side discovery of vGPU VFs grouped by profile is [NVIDIA/kubevirt-gpu-device-plugin#192](https://github.com/NVIDIA/kubevirt-gpu-device-plugin/pull/192), still open. Section 6 ships it downstream in the host layer, behind a boundary the rest of the design does not cross. ## Decisions - [0008. The GPU controller gets its node-label and plugin-restart rights with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) -- [0007. Whoever can write a VM orders its template devices, while held devices stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md) -- [0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the device](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md) +- [0007. Whoever can write a VM orders its template claims, while held claims stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md) +- [0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md) - [0005. The taint rule skips exactly the three taints that mirror a cordon or readiness](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md) - [0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts](./decisions/0004-gpunode-health-is-kubelet-allocatability.md) - [0003. The usage-monitor label reservation is always on, and it ratchets](./decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md) -- [0002. GPUDevice works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run](./decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md) -- [0001. A device whose class resource name KubeVirt does not permit is refused at VMI admission](./decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md) +- [0002. GPUClaim works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md) +- [0001. A claim whose class resource name KubeVirt does not permit is refused at VMI admission](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md) ## Context @@ -48,18 +48,18 @@ Both consumers take GPUs the same way: `vm-instance` and `kubernetes-nodes` read - A GPU cannot be held. When a VM stops, its GPU goes back to the pool and the next start can fail. - An order that does not fit is accepted, and the tenant learns it from pod events. - Nobody can tell which tenant uses a physical GPU without reading pods on the node. -- In the `vgpu` variant a node labelled for passthrough can serve whole GPUs of another model, but a model whose whole GPU and VFs share a PCI ID, such as L40S, is offered either whole or as vGPU, not both, because two device plugins would claim that ID. The limit is per GPU model, not per cluster. +- In the `vgpu` variant a node labelled for passthrough can serve whole GPUs of another model, but a model whose whole GPU and VFs share a PCI ID, such as L40S, is offered either whole or as vGPU, not both, because two device plugins would both take that ID. The limit is per GPU model, not per cluster. - VF profiles on GPU Operator nodes are written by hand after every reboot. ## Goals - A hoster publishes a catalog of GPU classes, each with a model, a mode (passthrough or vGPU) and, for vGPU, a profile. Tenants read the catalog with free capacity and their remaining budget before ordering. - A tenant orders a number of GPUs of a class within their quota and holds them, attached or not, until they release them. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden` (section 7). That is the end state, not the default: `forbidden` needs every Talos GPU pool replaced by a Kairos pool, which waits for cozystack/cozystack#4911, and every ComputePlane GPU group gone, which waits for ComputePlane on Kairos workers. Until then a cluster with either keeps raw names, and the hold is not guaranteed there. An order that cannot be satisfied is refused with a message naming the class. -- VMs and Kairos worker pools attach held GPUs the same way. A restarted VM gets the same held device back, and a replaced worker gets one of its pool's devices back unless the tenant's own VM or another pool uses it. +- VMs and Kairos worker pools attach held GPUs the same way. A restarted VM gets the same held claim back, and a replaced worker gets one of its pool's claims back unless the tenant's own VM or another pool uses it. - Every physical GPU and VF can be traced to its node, its configuration and the tenant using it. - Held GPUs appear in usage records (`Workload`) by tenant, class and time, which an external billing system reads. Cozystack charges nothing and stores no prices. - One cluster can have passthrough and vGPU nodes of the same GPU model, with vGPU profiles applied declaratively and kept across reboots. -- Existing `gpus` entries keep working, with a documented path to `GPUDevice` for VMs; existing Talos GPU pools stay on `name` entries. +- Existing `gpus` entries keep working, with a documented path to `GPUClaim` for VMs; existing Talos GPU pools stay on `name` entries. - The catalog can later describe MIG, other vendors and DRA-backed devices without changing the tenant API. ### Non-goals @@ -68,7 +68,7 @@ Both consumers take GPUs the same way: `vm-instance` and `kubernetes-nodes` read - The vGPU guest driver, the license server and license tokens. The hoster provides them (section 5). - The catalog UI. This proposal provides the data a portal reads. - GPUs for pods inside tenant clusters (gpu-operator in a tenant cluster, HAMi). -- Time-slicing one device between tenants, and live migration of GPU VMs. +- Time-slicing one claim between tenants, and live migration of GPU VMs. ## Design @@ -100,7 +100,7 @@ status: --- # Tenant. One per GPU or per set of GPUs used together on one node. apiVersion: gpu.cozystack.io/v1alpha1 -kind: GPUDevice +kind: GPUClaim metadata: name: train-0 namespace: tenant-acme @@ -136,15 +136,15 @@ status: A class maps an offer to a backend through a `source` union and to the nodes that serve it through `nodeSelector`. The first release has one source, `devicePlugin`; `dra` is reserved for section 8. Two classes may share a resource name when their node selectors do not overlap, which lets a host layer that cannot tell two offers apart by name tell them apart by node. A node that matches two classes with the same resource name gets a condition on both classes and on its `GPUNode`, which names the two classes, and is not used for binding. Class names are DNS labels, since they become part of quota keys. -A class referenced by held devices cannot disappear or change its meaning. A finalizer keeps a deleted class until its last `Bound`, `Attached` or `Lost` device is gone, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `vendor`, `model`, `mode`, `profile`, `memory`, `maxCount` and the kind of `source` cannot change while the finalizer is present. `resourceName` and `nodeSelector` can change, for changes in the host layer (section 6) and for moving the offer to other nodes. +A class referenced by held claims cannot disappear or change its meaning. A finalizer keeps a deleted class until its last `Bound`, `Attached` or `Lost` claim is gone, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `vendor`, `model`, `mode`, `profile`, `memory`, `maxCount` and the kind of `source` cannot change while the finalizer is present. `resourceName` and `nodeSelector` can change, for changes in the host layer (section 6) and for moving the offer to other nodes. -A `GPUDevice` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate devices. How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 6), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPU devices of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. Admission refuses a device above `maxCount`. The spec is immutable, so a change is a new order. +A `GPUClaim` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate claims. How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 6), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPU claims of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. Admission refuses a claim above `maxCount`. The spec is immutable, so a change is a new order. -The inventory is written by a node agent from sysfs and from kubelet's pod-resources API, and goes away with its `Node`. That API carries no device health, so a function is healthy when kubelet lists its devices as allocatable, each function reports how many of them are, one for a VF or a whole GPU and several for an mdev parent, and `resourceNames` lists the node's GPU resource names, including those whose devices the agent cannot place under a PCI function ([0004](./decisions/0004-gpunode-health-is-kubelet-allocatability.md)). Ownership is exact for running GPUs: a `usedBy` pod leads to its VM, and the VM's devices name it in `status.attachedTo`; an mdev parent that serves several pods names one of them. A held but not running reservation is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. PCI addresses stay in `GPUNode`, which tenants cannot read. +The inventory is written by a node agent from sysfs and from kubelet's pod-resources API, and goes away with its `Node`. That API carries no device health, so a function is healthy when kubelet lists its devices as allocatable, each function reports how many of them are, one for a VF or a whole GPU and several for an mdev parent, and `resourceNames` lists the node's GPU resource names, including those whose devices the agent cannot place under a PCI function ([0004](./decisions/0004-gpunode-health-is-kubelet-allocatability.md)). Ownership is exact for running GPUs: a `usedBy` pod leads to its VM, and the VM's claims name it in `status.attachedTo`; an mdev parent that serves several pods names one of them. A held but not running reservation is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. PCI addresses stay in `GPUNode`, which tenants cannot read. What `Bound` guarantees depends on the class's source: -| Source | A `Bound` device holds | The physical device is picked by | +| Source | A `Bound` claim holds | The physical device is picked by | | --- | --- | --- | | `devicePlugin` | `count` free units of the class on one node | kubelet, when the pod starts | | `dra` (section 8) | one `ResourceClaim` within budget, and its devices once the claim is allocated at the first start | the scheduler, at allocation | @@ -153,11 +153,11 @@ A tenant sees a class, a count, a phase and, once there is one, a node, never wh ### 2. Binding and capacity -A controller in `cozystack-controller` binds each new `GPUDevice` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`. The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included ([0005](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md)). A device already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI creation (section 3): it may take an unattached device off a node that is not `Ready`, because no VMI uses that device and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound devices, minus the units pods hold beyond what their devices cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. +A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`. The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included ([0005](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md)). A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI creation (section 3): it may take an unattached claim off a node that is not `Ready`, because no VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. Admission refuses an order that cannot be met at that moment, by capacity or by budget (section 4), with a message naming the class, the count and the limit it hit: a budget of the tenant or an ancestor, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. -The phases decide what a device holds and costs: +The phases decide what a claim holds and costs: | Phase | Holds GPUs | Counts against budget | In usage records | | --- | --- | --- | --- | @@ -166,77 +166,77 @@ The phases decide what a device holds and costs: | `Attached` | yes, used by a VMI | yes | yes | | `Lost` | no | yes | no | -A `dra` device before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 8). +A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 8). -A device leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 deletes a final VMI before it creates the VM's next one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)), so a new VMI never finds its device attached to the old one. +A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 deletes a final VMI before it creates the VM's next one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)), so a new VMI never finds its claim attached to the old one. -A device keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached device that no longer fits there moves to another node of its class, or to `Lost` when none has room. The devices one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A device or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a device's node is deleted, the controller also deletes the device's unscheduled VMI, so the device becomes unattached and moves. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves devices by the class's `onCordon` policy. With `Move`, the default, an unattached device on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its device becomes unattached, moves, and the VM's next VMI starts on the new node. Kairos workers use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a device by the pool rules of section 3. With `Stay` the device keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any device of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a device of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI creation, when one is needed, refuses such a set (section 3). +A claim keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted, the controller also deletes the claim's unscheduled VMI, so the claim becomes unattached and moves. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Kairos workers use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules of section 3. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI creation, when one is needed, refuses such a set (section 3). -A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity when the agent reports a device missing or unhealthy for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no device. Then: +A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity when the agent reports a device missing or unhealthy for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no claim. Then: - the node and the affected classes get a `LostCapacity` condition; -- unattached devices that no longer fit move to another node of their class, or to `Lost` when none has room; -- on a `NotReady` node, attached devices go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the device goes back to `Attached` there; a device that a gather at VMI creation is moving off the node is the exception: it finishes the move, or loses its attachment and follows these rules when the gather fails; -- a VM started on such a node after the condition is set is refused (section 3), and the controller deletes the unscheduled VMI of a device that no longer fits, so its device becomes unattached and can move. What the VM does next follows its `runStrategy` for a deleted VMI: `Always` and `Once` create a new VMI and start once the device has moved or been rebound from `Lost`, while `Manual` and `RerunOnFailure` stay stopped until the tenant starts the VM (Failure cases); unlike a cordon, the node cannot serve the device, so waiting is no option. A VM restarted earlier, on a node that is not `Ready`, is gathered at creation instead (section 3). +- unattached claims that no longer fit move to another node of their class, or to `Lost` when none has room; +- on a `NotReady` node, attached claims go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI creation is moving off the node is the exception: it finishes the move, or loses its attachment and follows these rules when the gather fails; +- a VM started on such a node after the condition is set is refused (section 3), and the controller deletes the unscheduled VMI of a claim that no longer fits, so its claim becomes unattached and can move. What the VM does next follows its `runStrategy` for a deleted VMI: `Always` and `Once` create a new VMI and start once the claim has moved or been rebound from `Lost`, while `Manual` and `RerunOnFailure` stay stopped until the tenant starts the VM (Failure cases); unlike a cordon, the node cannot serve the claim, so waiting is no option. A VM restarted earlier, on a node that is not `Ready`, is gathered at creation instead (section 3). -Devices in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a device whose VMI is still live waits until that VMI is final or removed (above). The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` device already counts. +Claims in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a claim whose VMI is still live waits until that VMI is final or removed (above). The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` claim already counts. ### 3. Attaching -VMs and Kairos worker pools reference devices in `gpus`: +VMs and Kairos worker pools reference claims in `gpus`: ```yaml # vm-instance gpus: -- device: train-0 +- claim: train-0 --- -# kubernetes-nodes, Kairos pool: each worker takes one matching device +# kubernetes-nodes, Kairos pool: each worker takes one matching claim gpus: -- deviceSelector: +- claimSelector: matchLabels: pool: inference ``` -The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named devices. For a pool worker it takes the device the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` device on a `Ready` node that matches the selector and that no VM names in a `device` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same device, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the device, so two VMIs can never hold one device, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A device still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A device that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved device; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Devices a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the devices a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template devices, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template devices included. This runs after the other refusal checks below and after the template devices pass their budget and entitlement checks, so every device in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a device of it, and each device keeps its hold. A device of a `Stay` class on a cordoned node does not move. When all of the VMI's devices are on that node and the VMI has no template devices still to place, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. When some of its devices are elsewhere, or it has template devices, the set cannot be gathered, and the VMI is refused. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a device of the set has lost capacity there, the node is gone, or it no longer serves a device's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md)). With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each device and its node. A move that does not finish within the admission timeout refuses the VMI the same way; devices already moved keep their new node, and KubeVirt's retry continues from there. Template devices are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the device's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. +The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named claims. For a pool worker it takes the claim the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the claim, so two VMIs can never hold one claim, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A claim still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template claims, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template claims included. This runs after the other refusal checks below and after the template claims pass their budget and entitlement checks, so every claim in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a claim of it, and each claim keeps its hold. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims still to place, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered, and the VMI is refused. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each claim and its node. A move that does not finish within the admission timeout refuses the VMI the same way; claims already moved keep their new node, and KubeVirt's retry continues from there. Template claims are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the claim's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. -The webhook refuses to start a VMI, and names each device and its node, when a referenced device: +The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim: - is `Pending`, `Lost`, being deleted, or attached to another VMI; -- is bound to a node that is gone, no longer matches the class, has `LostCapacity` the device does not fit under, or has a pending layout change (section 6); -- cannot be brought, with the VMI's other devices and template devices, onto one node with room for all of them (above), for example because a device of a `Stay` class is on a cordoned node while other devices need gathering; -- is a vGPU device next to vGPU devices of another class, or brings the VMI's GPUs of its class above `maxCount`; -- is of a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` ([0001](./decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md)). +- is bound to a node that is gone, no longer matches the class, has `LostCapacity` the claim does not fit under, or has a pending layout change (section 6); +- cannot be brought, with the VMI's other claims and template claims, onto one node with room for all of them (above), for example because a claim of a `Stay` class is on a cordoned node while other claims need gathering; +- is a vGPU claim next to vGPU claims of another class, or brings the VMI's GPUs of its class above `maxCount`; +- is of a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). -A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are the resource names of `GPUClass` objects and the names in the hoster's list `gpu.resourceNames`, which defaults to the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. The list also takes every resource name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin, as the agent reports it, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU device for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its devices unattached, so they move by the rules of section 2. Both webhooks see only VMIs and pods carrying the chart's reference label and fail closed, so an outage blocks the start of VMs with device references and nothing else. +A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are the resource names of `GPUClass` objects and the names in the hoster's list `gpu.resourceNames`, which defaults to the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. The list also takes every resource name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin, as the agent reports it, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU claim for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its claims unattached, so they move by the rules of section 2. Both webhooks see only VMIs and pods carrying the chart's reference label, `gpu.cozystack.io/claim-reference`, next to the `gpu.cozystack.io/claims` annotation that lists the references, and fail closed, so an outage blocks the start of VMs with claim references and nothing else. -A device cannot disappear under a running VM: a deleted `GPUDevice` stays, with its binding and usage record, until no VMI uses it, and is not attached again. +A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and usage record, under the `gpu.cozystack.io/claim-protection` finalizer until no VMI uses it, and is not attached again. -A `deviceTemplate` entry names a class and a count instead of a held device. At VMI creation the webhook creates a `GPUDevice` from it, named after the VMI and the entry's index and marked as a template device. A repeated call or a recreated VMI finds the same device and reuses it when it carries that mark, has no owner yet and matches the template; a device of that name still being deleted, or still owned by the VM's previous VMI, refuses the VMI, and KubeVirt retries. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked device that already has the name refuses the VMI, and the message names it. The webhook has the binder bind the device under the same capacity and budget checks as any order, within the admission timeout, and attaches it. The node for template devices is picked with room for all of the VMI's devices, held and template: the node of the gathered set when the VMI has held devices, and otherwise one node with room for all its template devices. The VMI is refused if no node has room for all of them. A bind that does not fit or does not finish in time refuses the VMI, and KubeVirt retries. On a dry run the webhook creates, attaches and moves nothing, so it declares `sideEffects: NoneOnDryRun`. The VMI has no UID yet at that point, so the controller sets the VMI as the device's owner once the VMI exists, and deletes a template device whose VMI never appears after the same grace period as a stale attachment. The controller deletes a template device as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the device lives and is billed only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any device, but only while the VMI exists: an order that does not fit is refused when the VM or worker starts, not ahead of time. A template counts as a device reference under `forbidden`. It lets an autoscaled pool pay for running workers only. +A `claimTemplate` entry names a class and a count instead of a held claim. At VMI creation the webhook creates a `GPUClaim` from it, named after the VMI and the entry's index and marked as a template claim by the `gpu.cozystack.io/template-claim` label. A repeated call or a recreated VMI finds the same claim and reuses it when it carries that mark, has no owner yet and matches the template; a claim of that name still being deleted, or still owned by the VM's previous VMI, refuses the VMI, and KubeVirt retries. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name refuses the VMI, and the message names it. The webhook has the binder bind the claim under the same capacity and budget checks as any order, within the admission timeout, and attaches it. The node for template claims is picked with room for all of the VMI's claims, held and template: the node of the gathered set when the VMI has held claims, and otherwise one node with room for all its template claims. The VMI is refused if no node has room for all of them. A bind that does not fit or does not finish in time refuses the VMI, and KubeVirt retries. On a dry run the webhook creates, attaches and moves nothing, so it declares `sideEffects: NoneOnDryRun`. The VMI has no UID yet at that point, so the controller sets the VMI as the claim's owner once the VMI exists, and deletes a template claim whose VMI never appears after the same grace period as a stale attachment. The controller deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives and is billed only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any claim, but only while the VMI exists: an order that does not fit is refused when the VM or worker starts, not ahead of time. A template counts as a claim reference under `forbidden`. It lets an autoscaled pool pay for running workers only. -A `vm-instance` entry has exactly one of `name`, `device` or `deviceTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `deviceSelector` or `deviceTemplate`. A list holds only `name` entries or only references, and a pool has at most one `deviceSelector` or one `deviceTemplate`, never both, and a selector's devices must share a class and a `count`; the webhook checks this at each attach, since device labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `device` and `deviceTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to Kairos is separate work. +A `vm-instance` entry has exactly one of `name`, `claim` or `claimTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `claimSelector` or `claimTemplate`. A list holds only `name` entries or only references, and a pool has at most one `claimSelector` or one `claimTemplate`, never both, and a selector's claims must share a class and a `count`; the webhook checks this at each attach, since claim labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `claim` and `claimTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to Kairos is separate work. -Kairos pools take GPUs only through `deviceSelector` or `deviceTemplate`, from classes of both modes. The chart refuses `name` entries on them. The pool's mode, `kairos.gpuMode`, must match the class, and the chart sets it as the `gpu.cozystack.io/mode` label on the worker VMI template ([Choosing the worker image by mode](../kairos-vgpu-workers/README.md#choosing-the-worker-image-by-mode)). At VMI creation the webhook refuses a device, selected or from a template, whose class `mode` differs from that label, and a pool reference on a VMI without the label. The `kubernetes-nodes` chart, from Rollout step 2, fails to render a `deviceSelector` or `deviceTemplate` on a pool with `os: talos`, the field cozystack/cozystack#4911 adds, so the pool's HelmRelease fails before any machine is created. A pool on `deviceSelector` never runs more GPU workers than it has matching devices. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer devices sets it. A pool without spare devices rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a device, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools, existing and new, keep working as today on `name` entries, and `GPUDevice` does not extend them. +Kairos pools take GPUs only through `claimSelector` or `claimTemplate`, from classes of both modes. The chart refuses `name` entries on them. The pool's mode, `kairos.gpuMode`, must match the class, and the chart sets it as the `gpu.cozystack.io/mode` label on the worker VMI template ([Choosing the worker image by mode](../kairos-vgpu-workers/README.md#choosing-the-worker-image-by-mode)). At VMI creation the webhook refuses a claim, selected or from a template, whose class `mode` differs from that label, and a pool reference on a VMI without the label. The `kubernetes-nodes` chart, from Rollout step 2, fails to render a `claimSelector` or `claimTemplate` on a pool with `os: talos`, the field cozystack/cozystack#4911 adds, so the pool's HelmRelease fails before any machine is created. A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools, existing and new, keep working as today on `name` entries, and `GPUClaim` does not extend them. ### 4. Quotas -A tenant's GPU budget is declared in `resourceQuotas`, per class as `gpu.cozystack.io/: ` and in total as `gpu.cozystack.io/total: `; the class name `total` is reserved. These keys are not rendered into the namespace `ResourceQuota`, since `GPUDevice` is not a pod, and a tenant with only GPU keys gets no `ResourceQuota` or `LimitRange`. That is a change to the tenant chart, which today renders both for any `resourceQuotas` key. +A tenant's GPU budget is declared in `resourceQuotas`, per class as `gpu.cozystack.io/: ` and in total as `gpu.cozystack.io/total: `; the class name `total` is reserved. These keys are not rendered into the namespace `ResourceQuota`, since `GPUClaim` is not a pod, and a tenant with only GPU keys gets no `ResourceQuota` or `LimitRange`. That is a change to the tenant chart, which today renders both for any `resourceQuotas` key. GPU budgets follow their own rule, not the pool and carve-out rules for other resources: every GPU key caps the whole subtree of the tenant that declares it. GPU keys stay out of the pool and carve-out model, so a GPU budget covers every descendant of the tenant that declares it. The tenant declaration gate skips `gpu.cozystack.io/*` keys, because under this rule a child's key may overlap its siblings'. -- An order is refused if, for the tenant or any ancestor that declares the class key or `total`, the devices counting against the budget (table in section 2) in that ancestor's subtree would exceed the key. +- An order is refused if, for the tenant or any ancestor that declares the class key or `total`, the claims counting against the budget (table in section 2) in that ancestor's subtree would exceed the key. - A child's key only adds a limit; it never lifts an ancestor's limit or a missing entitlement. -- A raw GPU key (section 7) on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has the class key or `total`. While the raw key exists, it also caps those classes' devices together with the raw-name VMs it covers, so an existing GPU limit is never bypassed. +- A raw GPU key (section 7) on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has the class key or `total`. While the raw key exists, it also caps those classes' claims together with the raw-name VMs it covers, so an existing GPU limit is never bypassed. - A tenant with no GPU key of either kind anywhere above it is limited only by capacity, for vGPU classes once it is entitled. -- A vGPU class is ordered only by a tenant entitled to vGPU. The Tenant field `gpu.vgpu` grants it. The parent tenant's administrator sets it, or the hoster on a top-level tenant. A tenant is entitled only when the flag is set on it and its parent is entitled. The root tenant is rendered by the platform, not by the tenant chart, so its value comes from platform values. The flag is off by default everywhere, so the hoster turns it on for the root first. No tenant can grant itself vGPU, since its Tenant object lives in its parent's namespace. The tenant chart computes the effective value from the parent's, like its other namespace values, and stamps it on the namespace as the label `namespace.cozystack.io/vgpu`; a change reaches grandchildren as each level re-renders. The chart always writes the label, so only the value `"true"` means entitled; any other value, or no label, means not entitled. Entitlement is separate from budget because a vGPU license is counted per unit, which is a budget, while getting the vGPU guest driver and the license token at all is yes or no. Class keys and `total` then limit vGPU like any other GPU. Clearing the flag refuses new vGPU orders, including the device a `deviceTemplate` would create at the next VMI start, so such a VM or worker no longer starts. Devices already held keep counting and can still move or be rebound, so a pool on `deviceSelector` keeps its devices, and the tenant's Kairos pools lose their vGPU entitlement as `kairos-vgpu-workers` describes. Raw vGPU names in VMs stay as open as today until a raw key limits them or `forbidden` is set (section 7). +- A vGPU class is ordered only by a tenant entitled to vGPU. The Tenant field `gpu.vgpu` grants it. The parent tenant's administrator sets it, or the hoster on a top-level tenant. A tenant is entitled only when the flag is set on it and its parent is entitled. The root tenant is rendered by the platform, not by the tenant chart, so its value comes from platform values. The flag is off by default everywhere, so the hoster turns it on for the root first. No tenant can grant itself vGPU, since its Tenant object lives in its parent's namespace. The tenant chart computes the effective value from the parent's, like its other namespace values, and stamps it on the namespace as the label `namespace.cozystack.io/vgpu`; a change reaches grandchildren as each level re-renders. The chart always writes the label, so only the value `"true"` means entitled; any other value, or no label, means not entitled. Entitlement is separate from budget because a vGPU license is counted per unit, which is a budget, while getting the vGPU guest driver and the license token at all is yes or no. Class keys and `total` then limit vGPU like any other GPU. Clearing the flag refuses new vGPU orders, including the claim a `claimTemplate` would create at the next VMI start, so such a VM or worker no longer starts. Claims already held keep counting and can still move or be rebound, so a pool on `claimSelector` keeps its claims, and the tenant's Kairos pools lose their vGPU entitlement as `kairos-vgpu-workers` describes. Raw vGPU names in VMs stay as open as today until a raw key limits them or `forbidden` is set (section 7). - GPU keys play no part in the existing pool membership and carve-outs. -A parent with `total: 4` caps everything below it at four GPUs, each vGPU counting as one. A parent cannot reserve part of its budget for one child. #48 would add that, but as written it covers neither GPUs nor entitlement, so this section stays GPUDevice's own rule, and budgets move to #48 only if its gate takes over subtree caps for GPU keys. +A parent with `total: 4` caps everything below it at four GPUs, each vGPU counting as one. A parent cannot reserve part of its budget for one child. #48 would add that, but as written it covers neither GPUs nor entitlement, so this section stays GPUClaim's own rule, and budgets move to #48 only if its gate takes over subtree caps for GPU keys. ### 5. Usage accounting, catalog and licensing -Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads. Each `GPUDevice` gets one with `kind: gpu-device` and resources `gpu.cozystack.io/: ` while it is `Bound` or `Attached`, a `dra` device only once its claim is allocated (section 8), next to the records Cozystack already keeps for CPU, memory and storage. The GPU controller writes the record itself, under a `WorkloadMonitor` it owns in the tenant's namespace, whose selector matches no object, so tenant labels cannot drop a device from the usage records. The selector is a reserved label key that admission refuses on every object in a tenant namespace, on create and on an update that adds it or changes its value, whoever makes the request and whether or not GPUs are on, so no new object can match it, not even a virt-launcher pod or a DataVolume that inherits the label from a VM template; an update that keeps or drops a label set before the policy existed passes ([0003](./decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md)). Each record carries the `workloads.cozystack.io/monitor` label naming that monitor: the `Workload` controller deletes a record only when that label is missing or names a monitor that is gone, and deletes nothing by selector. Like other `Workload` records it has no time fields: a held interval is the record's lifetime, which a billing system observes by watching and reconciles by sampling. A virt-launcher pod does not report GPUs its devices cover a second time. +Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads. Each `GPUClaim` gets one with `kind: gpu-claim` and resources `gpu.cozystack.io/: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 8), next to the records Cozystack already keeps for CPU, memory and storage. The GPU controller writes the record itself, under a `WorkloadMonitor` it owns in the tenant's namespace, whose selector matches no object, so tenant labels cannot drop a claim from the usage records. The selector is a reserved label key that admission refuses on every object in a tenant namespace, on create and on an update that adds it or changes its value, whoever makes the request and whether or not GPUs are on, so no new object can match it, not even a virt-launcher pod or a DataVolume that inherits the label from a VM template; an update that keeps or drops a label set before the policy existed passes ([0003](./decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md)). Each record carries the `workloads.cozystack.io/monitor` label naming that monitor: the `Workload` controller deletes a record only when that label is missing or names a monitor that is gone, and deletes nothing by selector. Like other `Workload` records it has no time fields: a held interval is the record's lifetime, which a billing system observes by watching and reconciles by sampling. A virt-launcher pod does not report GPUs its claims cover a second time. -The catalog is the list of `GPUClass` objects with `displayName`, model, mode, profile, memory, `maxCount`, `status.free` and `status.maxFreePerNode`. Every authenticated user reads it, because a hoster publishes its offers and a tenant names a class when ordering, including from `kubectl` or GitOps without a portal. So the status holds only `free` and `maxFreePerNode`, and its conditions, such as `LostCapacity`, name no nodes and carry no counts. The class's capacity, bound units and `Lost` count show the hoster's inventory and load, and they are not in the status. The controller exports them as metrics that platform monitoring scrapes. Per node, a hoster reads device occupancy and health from `GPUNode`, and lists bound devices with a field selector on the `GPUDevice` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write. A tenant can still watch `free` and `maxFreePerNode` over time and see how busy a class is. The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest order, leaving out a field that is absent because it sets no limit. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. A `gpudevice` source lists a tenant's own devices. The existing `gpu` source keeps serving raw names while they are allowed. +The catalog is the list of `GPUClass` objects with `displayName`, model, mode, profile, memory, `maxCount`, `status.free` and `status.maxFreePerNode`. Every authenticated user reads it, because a hoster publishes its offers and a tenant names a class when ordering, including from `kubectl` or GitOps without a portal. So the status holds only `free` and `maxFreePerNode`, and its conditions, such as `LostCapacity`, name no nodes and carry no counts. The class's capacity, bound units and `Lost` count show the hoster's inventory and load, and they are not in the status. The controller exports them as metrics that platform monitoring scrapes. Per node, a hoster reads device occupancy and health from `GPUNode`, and lists bound claims with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write. A tenant can still watch `free` and `maxFreePerNode` over time and see how busy a class is. The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest order, leaving out a field that is absent because it sets no limit. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. A `gpuclaim` source lists a tenant's own claims. The existing `gpu` source keeps serving raw names while they are allowed. The remaining budget is computed over each ancestor's whole subtree, so a tenant under a shared parent `total` can see how many GPUs its siblings hold, as a number, without names. This is accepted: the alternative is a budget the tenant can only discover by a refused order, which is the failure this proposal removes. @@ -246,7 +246,7 @@ vGPU needs a license, and passthrough needs one only for graphics. Compute (C-se This proposal requires vGPU hosts to run Ubuntu or Kairos with `bundles.system.variant: isp-full-generic`: the vGPU host driver and the patched plugin run there. Cozystack has no Kairos hosts today, so installing the bundle on Kairos and a vGPU Manager image for it are new work of this phase. `docs/gpu-vgpu.md` does not recommend Talos for vGPU, so this phase does not cover Talos hosts; they keep passthrough through the `default` variant as today. NVIDIA AI Enterprise lists vGPU for Compute on Linux with KVM as supported only through specific hypervisor vendors and, on Ubuntu and RHEL with KVM, for single-node hosts only ([support matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/support/support-matrix-8/8.2.html)), so support for such a host is a matter between the hoster and NVIDIA. -The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ships as its own phase (Rollout, step 4), and it is replaceable. The API above it depends on four things only: each class's `resourceName` is advertised on the nodes its `nodeSelector` picks, one name never covers two different offers on one node, `GPUNode` reports devices and their health, and node labels mark each node's mode and profiles. `GPUClass`, `GPUDevice`, `GPUNode`, quotas, both webhooks, usage records and migration use no plugin-specific name, discovery format or configuration; the agent maps whatever device IDs a plugin reports to PCI addresses through sysfs. A different plugin, or a different version of the one chosen here, changes only this phase and the `resourceName` of affected classes, which can change while devices are held. Tenants keep their devices and `device` entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to `GPUDevice` first. +The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ships as its own phase (Rollout, step 4), and it is replaceable. The API above it depends on four things only: each class's `resourceName` is advertised on the nodes its `nodeSelector` picks, one name never covers two different offers on one node, `GPUNode` reports devices and their health, and node labels mark each node's mode and profiles. `GPUClass`, `GPUClaim`, `GPUNode`, quotas, both webhooks, usage records and migration use no plugin-specific name, discovery format or configuration; the agent maps whatever device IDs a plugin reports to PCI addresses through sysfs. A different plugin, or a different version of the one chosen here, changes only this phase and the `resourceName` of affected classes, which can change while claims are held. Tenants keep their claims and `claim` entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to `GPUClaim` first. Until the phase ships, a cluster keeps the modes its variant gives it today, one per model for models whose whole GPU and VFs share a PCI ID, and vGPU classes keep the experimental status `docs/gpu-vgpu.md` gives the `vgpu` variant: the hoster writes profiles and node labels by hand. On clusters without a layout, which includes the `default` variant and Talos hosts, node labels stay hand-written. @@ -260,19 +260,19 @@ The phase adds a node layout to the `vgpu` variant, in platform values. For each The layout and mdev GPUs coexist: GPUs on the mdev path keep today's configuration through `vgpu-device-manager`, and the agent writes profiles only on nodes that use VFs. New deployments of SR-IOV GPUs on current host releases use VFs. -A layout change on a node, of mode or of profile, waits until the node has no GPU VMs. The node gets a `LayoutPending` condition: new binds and starts there are refused, unattached devices move away, and running VMs keep their GPUs until the hoster drains the node. A GPU VM created before Cozystack switched to `LiveMigrateIfPossible` needs one restart before the drain can evict it. A node whose hand-written profiles already match the layout is not a change. A `vgpu` cluster adopts the layout with vGPU selectors that cover its existing vGPU nodes, and labels those nodes `vm-vgpu` before setting it, so the layout's passthrough default does not change them in between. Its vGPU classes, raw `name` entries and raw quota keys move from the hand-written VF names to the plugin's per-profile names in one maintenance window, or the raw-name consumers move to `GPUDevice` first. KubeVirt's built-in plugin and the patched plugin report different device IDs for the same VF, and kubelet would otherwise hand out VFs in use, so affected nodes are cordoned and free of GPU VMs until the patched plugin has registered there. +A layout change on a node, of mode or of profile, waits until the node has no GPU VMs. The node gets a `LayoutPending` condition: new binds and starts there are refused, unattached claims move away, and running VMs keep their GPUs until the hoster drains the node. A GPU VM created before Cozystack switched to `LiveMigrateIfPossible` needs one restart before the drain can evict it. A node whose hand-written profiles already match the layout is not a change. A `vgpu` cluster adopts the layout with vGPU selectors that cover its existing vGPU nodes, and labels those nodes `vm-vgpu` before setting it, so the layout's passthrough default does not change them in between. Its vGPU classes, raw `name` entries and raw quota keys move from the hand-written VF names to the plugin's per-profile names in one maintenance window, or the raw-name consumers move to `GPUClaim` first. KubeVirt's built-in plugin and the patched plugin report different device IDs for the same VF, and kubelet would otherwise hand out VFs in use, so affected nodes are cordoned and free of GPU VMs until the patched plugin has registered there. ### 7. Migration of existing `gpus` -Raw `name` entries keep working, and nothing converts them automatically, because a conversion would bind GPUs a tenant never ordered and put them in the usage records. The binder counts their usage, so it never books a unit a raw-name VM holds. The reverse is not protected: the scheduler does not see a device that is bound but not attached, so while raw names are allowed, a raw-name VM can take that unit. A reservation is a hard guarantee only with `gpu.rawResourceNames: forbidden`. +Raw `name` entries keep working, and nothing converts them automatically, because a conversion would bind GPUs a tenant never ordered and put them in the usage records. The binder counts their usage, so it never books a unit a raw-name VM holds. The reverse is not protected: the scheduler does not see a claim that is bound but not attached, so while raw names are allowed, a raw-name VM can take that unit. A reservation is a hard guarantee only with `gpu.rawResourceNames: forbidden`. -A tenant moves a VM by stopping it, ordering devices of the matching class, replacing each `name` entry with a `device` entry and starting it. On a full pool the device can only be ordered after the stop; with spare units, ordering first puts both in the usage records until the restart. A Talos GPU pool has no in-place path. The tenant creates a Kairos pool with `kairos.gpuMode` and a `deviceSelector`, which needs a hoster-built worker image for the class's mode, and then deletes the Talos pool. On a full cluster the Talos pool scales down first, so its GPUs are free to order. ComputePlane GPU groups have no such path until ComputePlane runs Kairos workers, so a cluster with them keeps raw names allowed. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes `device` entries. A Talos pool on one takes the Kairos path above. +A tenant moves a VM by stopping it, ordering claims of the matching class, replacing each `name` entry with a `claim` entry and starting it. On a full pool the claim can only be ordered after the stop; with spare units, ordering first puts both in the usage records until the restart. A Talos GPU pool has no in-place path. The tenant creates a Kairos pool with `kairos.gpuMode` and a `claimSelector`, which needs a hoster-built worker image for the class's mode, and then deletes the Talos pool. On a full cluster the Talos pool scales down first, so its GPUs are free to order. ComputePlane GPU groups have no such path until ComputePlane runs Kairos workers, so a cluster with them keeps raw names allowed. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes `claim` entries. A Talos pool on one takes the Kairos path above. -Quota keys move too. A raw GPU key is a quota key on a GPU resource name, such as `nvidia.com/AD102GL_L40S`. A device-backed VM's pod still requests that name, so while a tenant migrates, its raw key covers both kinds of VM; after its last raw-name VM has moved, the key is replaced by `gpu.cozystack.io/*` keys. A tenant whose raw-name VMs use vGPU can order their devices only after it is entitled to vGPU (section 4), and within any class keys set above it. +Quota keys move too. A raw GPU key is a quota key on a GPU resource name, such as `nvidia.com/AD102GL_L40S`. A claim-backed VM's pod still requests that name, so while a tenant migrates, its raw key covers both kinds of VM; after its last raw-name VM has moved, the key is replaced by `gpu.cozystack.io/*` keys. A tenant whose raw-name VMs use vGPU can order their claims only after it is entitled to vGPU (section 4), and within any class keys set above it. When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. Preconditions: no Talos GPU pools in the cluster, ComputePlane GPU groups included (a hoster without a passthrough worker image keeps passthrough Talos pools and raw names), no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: -- an admission policy refuses GPU entries (section 3) on every VMI created without a device reference, on every path; +- an admission policy refuses GPU entries (section 3) on every VMI created without a claim reference, on every path; - the same policy refuses every pod created in a tenant namespace that requests a GPU resource name, unless it is a virt-launcher pod created by KubeVirt's virt-controller service account with an owner reference to a VirtualMachineInstance. Tenants cannot forge such a pod, since no tenant role can create pods or VMIs, so app charts that copy free-form `resources` into pods cannot request GPUs either; - tenant validation refuses raw GPU keys; - the `gpu` option source returns nothing, and the `instancetype` source hides instance types that carry GPU entries. @@ -281,25 +281,25 @@ When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden` The catalog is designed so later backends add `source` members and class fields, not tenant fields. -- DRA. Kubernetes DRA has been GA since 1.34, and KubeVirt v1.9 has `GPUsWithDRA` and `HostDevicesWithDRA` in beta. A `dra` source names a `DeviceClass` and selectors, and under it a `GPUDevice` maps one to one onto a `ResourceClaim` created at order time. The scheduler allocates the claim, the GPU controller keeps it allocated through its own `reservedFor` entry, which the resourceclaim controller leaves in place, and the webhook renders the claim reference into the VMI; the binder does not allocate. Three gaps remain. The NVIDIA DRA driver has no vGPU yet (kubernetes-sigs/dra-driver-nvidia-gpu#1402, milestone v0.6.0), and its passthrough is alpha. A claim is allocated only when a pod first needs it, so nothing is held before the first boot: until then a `dra` device is `Bound` without a node, counts against the budget, holds no GPU and stays out of the usage records, and the webhook attaches it like any `Bound` device without pinning the pod to a node. DRA quota is per namespace, without subtree caps. KubeVirt VEP 344 (`persistWhenStopped`) holds a claim for a stopped VM from the VM side; a `dra` class and VEP 344 must not both manage one claim. When a DRA GPU driver is installed, the raw-name policy has to cover claim references too. +- DRA. Kubernetes DRA has been GA since 1.34, and KubeVirt v1.9 has `GPUsWithDRA` and `HostDevicesWithDRA` in beta. A `dra` source names a `DeviceClass` and selectors, and under it a `GPUClaim` maps one to one onto a `ResourceClaim` created at order time. The scheduler allocates the `ResourceClaim`, the GPU controller keeps it allocated through its own `reservedFor` entry, which the resourceclaim controller leaves in place, and the webhook renders the `ResourceClaim` reference into the VMI; the binder does not allocate. Three gaps remain. The NVIDIA DRA driver has no vGPU yet (kubernetes-sigs/dra-driver-nvidia-gpu#1402, milestone v0.6.0), and its passthrough is alpha. A `ResourceClaim` is allocated only when a pod first needs it, so nothing is held before the first boot: until then a `dra` claim is `Bound` without a node, counts against the budget, holds no GPU and stays out of the usage records, and the webhook attaches it like any `Bound` claim without pinning the pod to a node. DRA quota is per namespace, without subtree caps. KubeVirt VEP 344 (`persistWhenStopped`) holds a claim for a stopped VM from the VM side; a `dra` class and VEP 344 must not both manage one claim. When a DRA GPU driver is installed, the raw-name policy has to cover `ResourceClaim` references too. - MIG-backed vGPU profiles get their own resource names from the patched plugin, so they are ordinary classes; with DRA, partitionable devices (beta in Kubernetes 1.36) describe MIG natively. - A non-NVIDIA GPU is a class with another `vendor`; Q, B and A series vGPU profiles are classes like C series. - NVLink sets that must be attached together are left to the DRA source. -- On-demand slicing. A hoster may let tenants take any size from a shared pool, from a fraction of a GPU, such as a MIG-backed vGPU on a 1/7 slice of an A100, up to several whole GPUs of one node, taken from GPUs the layout keeps out of MIG mode. The tenant contract stays a class, one per size, and a `count`, and a device still lives on one node. The host layer and the binder change: the node agent creates the profile, and on MIG GPUs the MIG instance under it, when a device binds instead of following a fixed layout, so the binder counts capacity from free GPU resources rather than from advertised names. Enabling MIG mode needs an idle GPU and, on Ampere, a GPU reset, so the layout enables it ahead of time. On Hopper and newer the mode does not survive a reboot, so the layout sets it again at boot. A plugin restart per bind, as section 6 does for late profiles, would leave the node's allocatable at zero until the plugin registers again on every bind, so the plugin needs to rediscover profiles without one, separate work on top of NVIDIA/kubevirt-gpu-device-plugin#192. +- On-demand slicing. A hoster may let tenants take any size from a shared pool, from a fraction of a GPU, such as a MIG-backed vGPU on a 1/7 slice of an A100, up to several whole GPUs of one node, taken from GPUs the layout keeps out of MIG mode. The tenant contract stays a class, one per size, and a `count`, and a claim still lives on one node. The host layer and the binder change: the node agent creates the profile, and on MIG GPUs the MIG instance under it, when a claim binds instead of following a fixed layout, so the binder counts capacity from free GPU resources rather than from advertised names. Enabling MIG mode needs an idle GPU and, on Ampere, a GPU reset, so the layout enables it ahead of time. On Hopper and newer the mode does not survive a reboot, so the layout sets it again at boot. A plugin restart per bind, as section 6 does for late profiles, would leave the node's allocatable at zero until the plugin registers again on every bind, so the plugin needs to rediscover profiles without one, separate work on top of NVIDIA/kubevirt-gpu-device-plugin#192. ## User-facing changes -- Tenants get `GPUDevice`, a GPU catalog with free capacity and remaining budget per class, `device` entries in `vm-instance`, `deviceSelector` entries for Kairos pools in `kubernetes-nodes`, and `deviceTemplate` entries in both. Raw resource names keep working until a hoster forbids them. -- A held device is billed while held, whether a VM runs on it or not. An autoscaled Kairos pool on `deviceSelector` holds, and pays for, every device its selector matches, which is at least `maxReplicas`; under `forbidden` nothing runs on GPUs without a device. A tenant who wants to pay only while running uses `deviceTemplate`, and gets the refusal at start instead of at order time. -- Hosters get `GPUClass`, `GPUNode`, GPU budgets in `resourceQuotas` and the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout, `onCordon` on classes, the `gpu.resourceNames` list and `gpu.rawResourceNames`; an external billing system gets a usage record (`Workload`) per held device. +- Tenants get `GPUClaim`, a GPU catalog with free capacity and remaining budget per class, `claim` entries in `vm-instance`, `claimSelector` entries for Kairos pools in `kubernetes-nodes`, and `claimTemplate` entries in both. Raw resource names keep working until a hoster forbids them. +- A held claim is billed while held, whether a VM runs on it or not. An autoscaled Kairos pool on `claimSelector` holds, and pays for, every claim its selector matches, which is at least `maxReplicas`; under `forbidden` nothing runs on GPUs without a claim. A tenant who wants to pay only while running uses `claimTemplate`, and gets the refusal at start instead of at order time. +- Hosters get `GPUClass`, `GPUNode`, GPU budgets in `resourceQuotas` and the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout, `onCordon` on classes, the `gpu.resourceNames` list and `gpu.rawResourceNames`; an external billing system gets a usage record (`Workload`) per held claim. ## Upgrade and rollback compatibility The change is additive. Existing `gpus` entries render as before, and the existing variants work as today until a layout is set. -Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, and the tenants' GPU limits, with four exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only limits were `gpu.cozystack.io/*` keys lose them (rules below), and tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through `GPUDevice`, so they lose their GPUs and run on the plain worker image. A child tenant that declares its own quotas no longer shares its parent's GPU limit, because a raw key is carved out of the parent's pool rather than capping the subtree. +Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, and the tenants' GPU limits, with four exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only limits were `gpu.cozystack.io/*` keys lose them (rules below), and tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through `GPUClaim`, so they lose their GPUs and run on the plain worker image. A child tenant that declares its own quotas no longer shares its parent's GPU limit, because a raw key is carved out of the parent's pool rather than capping the subtree. -The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `device` and `deviceTemplate` entries of VMs become raw `name` entries and those VMs restart. `deviceSelector` and `deviceTemplate` entries and `kairos.gpuMode` are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUDevice` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the GPU controller's `WorkloadMonitor` objects, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. +The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `claim` and `claimTemplate` entries of VMs become raw `name` entries and those VMs restart. `claimSelector` and `claimTemplate` entries and `kairos.gpuMode` are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUClaim` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the GPU controller's `WorkloadMonitor` objects, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. Three rules hold throughout: @@ -309,42 +309,42 @@ Three rules hold throughout: ## Security -- Tenant access. Tenants with the `admin` role or above manage `GPUDevice` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. Tenants cannot read `GPUNode`. -- What a tenant controls. A tenant never supplies a resource name or a node through a device reference: both come from `GPUClass` and from the device's status, which tenants cannot write. Tenants cannot add or remove finalizers on devices, so a held device cannot leave the usage records. Admission refuses setting the reserved label of the GPU usage monitor on any object in a tenant namespace (section 5). A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes devices only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a device reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. +- Tenant access. Tenants with the `admin` role or above manage `GPUClaim` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. Tenants cannot read `GPUNode`. +- What a tenant controls. A tenant never supplies a resource name or a node through a claim reference: both come from `GPUClass` and from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. Admission refuses setting the reserved label of the GPU usage monitor on any object in a tenant namespace (section 5). A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes claims only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. - Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. - Node agent. It reads sysfs and kubelet's pod-resources socket, writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can write only its own node's `GPUNode`. -- Webhook. For `deviceTemplate` entries it creates `GPUDevice` objects in tenant namespaces, only of the class and count the template names. It does not check who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template devices, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md)). -- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its devices. It also writes one ConfigMap, the GPU settings in `cozy-system`. -- Visibility. A tenant sees the node name of its own devices and, through the remaining budget, its siblings' GPU count under a shared parent budget (section 5). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. +- Webhook. For `claimTemplate` entries it creates `GPUClaim` objects in tenant namespaces, only of the class and count the template names. It does not check who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template claims, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). +- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. It also writes one ConfigMap, the GPU settings in `cozy-system`. +- Visibility. A tenant sees the node name of its own claims and, through the remaining budget, its siblings' GPU count under a shared parent budget (section 5). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. ## Failure and edge cases Each case gives what is seen and what to do. - A tenant's order is refused at create → the message names the class, the count and the limit, or, for a vGPU class, that the tenant is not entitled to vGPU; order a smaller count, another class, or ask for more budget. -- A device stays `Pending` after an order → its condition names the shortage; it binds when capacity or budget frees up, or the tenant deletes it. -- A VM with devices does not start → the webhook message names each device and its node and the reason. When the reason is that the VM's devices, template devices included, cannot be brought onto one node, either no node can take all of them or a device of a `Stay` class is on a cordoned node while others need gathering. Then uncordon that node, free capacity on one node, drop a device the VM does not need, or, within one class, replace them with one device of the combined `count`. A refusal for the admission timeout usually clears on KubeVirt's retry. -- A node is drained or cordoned → under `Move` its unattached devices move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the devices stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. -- A cordoned node is deleted while a drained VM's VMI still waits for it → the controller deletes that unscheduled VMI, the device moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. -- A node dies but stays registered → after the grace period its attached devices go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost` (section 2); once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the devices are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached devices onto a `Ready` node with room (section 3) and starts there at once. That holds for a VM's held devices. A pool worker takes only a spare device of its pool, template devices follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node (all in section 3). After a `Node` deletion the VM starts on a retry instead, once the controller has moved the devices. Only when no node fits is the VMI refused, and then it is not created at all. KubeVirt v1.9.0 retries the creation with backoff for `Always`, which deletes the failed VMI and starts a new one, and for `RerunOnFailure`, whose failed VMI leaves a start request that stays until a VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)); such a VM starts once a node has room for its set. The retry goes through the VM controller's rate-limited queue, whose per-item backoff grows to 1000 seconds, so the start can come up to about 17 minutes after room appears. `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its devices unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. -- A node loses capacity while a VM's VMI waits for it unscheduled → the controller deletes that VMI so its devices can move; an `Always` or `Once` VM starts again once its devices have moved or been rebound, a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. -- A cordoned node dies while a VM's VMI waits on it, pinned there by a device of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its devices stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md)): the controller deletes the unscheduled VMI, and the VM's devices move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has (section 2). An `Always` or `Once` VM then starts on the new node once its devices have moved or been rebound; a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. -- A GPU fails under a running VM → the device stays `Attached` and in the usage records while the VM runs; when the VM restarts it is refused there, and the device moves or goes to `Lost`. +- A claim stays `Pending` after an order → its condition names the shortage; it binds when capacity or budget frees up, or the tenant deletes it. +- A VM with claims does not start → the webhook message names each claim and its node and the reason. When the reason is that the VM's claims, template claims included, cannot be brought onto one node, either no node can take all of them or a claim of a `Stay` class is on a cordoned node while others need gathering. Then uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. A refusal for the admission timeout usually clears on KubeVirt's retry. +- A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. +- A cordoned node is deleted while a drained VM's VMI still waits for it → the controller deletes that unscheduled VMI, the claim moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. +- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost` (section 2); once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached claims onto a `Ready` node with room (section 3) and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node (all in section 3). After a `Node` deletion the VM starts on a retry instead, once the controller has moved the claims. Only when no node fits is the VMI refused, and then it is not created at all. KubeVirt v1.9.0 retries the creation with backoff for `Always`, which deletes the failed VMI and starts a new one, and for `RerunOnFailure`, whose failed VMI leaves a start request that stays until a VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)); such a VM starts once a node has room for its set. The retry goes through the VM controller's rate-limited queue, whose per-item backoff grows to 1000 seconds, so the start can come up to about 17 minutes after room appears. `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. +- A node loses capacity while a VM's VMI waits for it unscheduled → the controller deletes that VMI so its claims can move; an `Always` or `Once` VM starts again once its claims have moved or been rebound, a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. +- A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has (section 2). An `Always` or `Once` VM then starts on the new node once its claims have moved or been rebound; a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. +- A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts it is refused there, and the claim moves or goes to `Lost`. - A layout change does not apply → the node shows `LayoutPending`; drain it. A VM with a raw `name` entry on the node holds the change until it stops. -- A Talos GPU pool, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a Kairos pool with `deviceSelector`, or allow raw names again while the cluster has Talos GPU pools. -- A pool with `os: talos` and a `deviceSelector` or `deviceTemplate` fails to render → Talos pools take GPUs only through `name` entries; use a Kairos pool for devices. -- A Kairos worker VMI is refused because its device's class `mode` differs from the pool's `gpu.cozystack.io/mode` label → set `kairos.gpuMode` to the class's mode, or select devices of a class in the pool's mode. -- A tenant's VM is `Pending` while its device is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. -- A pool worker is refused → no matching device is free, because of a surge machine during a rollout, devices still `Pending` or `Lost`, matching devices only on nodes that are not `Ready`, or another pool or a VM using the matching devices; wait, add devices, give each pool its own labels, or lower `maxReplicas`. +- A Talos GPU pool, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a Kairos pool with `claimSelector`, or allow raw names again while the cluster has Talos GPU pools. +- A pool with `os: talos` and a `claimSelector` or `claimTemplate` fails to render → Talos pools take GPUs only through `name` entries; use a Kairos pool for claims. +- A Kairos worker VMI is refused because its claim's class `mode` differs from the pool's `gpu.cozystack.io/mode` label → set `kairos.gpuMode` to the class's mode, or select claims of a class in the pool's mode. +- A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. +- A pool worker is refused → no matching claim is free, because of a surge machine during a rollout, claims still `Pending` or `Lost`, matching claims only on nodes that are not `Ready`, or another pool or a VM using the matching claims; wait, add claims, give each pool its own labels, or lower `maxReplicas`. ## Testing - Unit tests for the binder and both webhooks: capacity with raw-name usage, budgets over the ancestor walk, the phase table, every move and refusal in sections 2 and 3, concurrent starts, and the pod pin after a LiveUpdate sync. -- Webhook tests for cordons, templates and NICs: under `Move` a device moves at VMI creation and the pod gets a required node affinity, an attached device with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity the unscheduled VMI is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, the devices of one VM move, go to `Lost` and are rebound as a set, devices of two classes on different nodes, and a device whose node has no room for the VMI's template device, are moved onto one node at VMI creation, refused when no node fits, a `Stay` device on a cordoned node pins the VMI when it holds all of the VM's devices and refuses it when other devices need gathering, devices on a node that is not `Ready` are gathered, two VMIs whose sets share a device do not both gather it, a VMI with only template devices gets one node with room for all of them, and under `Stay` a single device waits; a `deviceTemplate` device is created, refused when it does not fit, deleted with its VMI, deleted when its VMI is refused later in admission, reused on a repeated call, and on a dry run neither created nor attached nor moved; a VM with a GPU device and `nvidia.com/hostdev` passes. -- Chart tests for `kubernetes-nodes`: a `deviceSelector` or `deviceTemplate` with `os: talos` fails to render, and a Kairos pool gets the `gpu.cozystack.io/mode` label on its worker VMI template. -- Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held devices and refuses the next template device, and `remainingBudget` is 0 without the entitlement. -- Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no device reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. -- Controller tests for the class metrics: capacity, bound units and the `Lost` count match the devices, and none of them appears in `GPUClass` status. +- Webhook tests for cordons, templates and NICs: under `Move` a claim moves at VMI creation and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity the unscheduled VMI is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI creation, refused when no node fits, a `Stay` claim on a cordoned node pins the VMI when it holds all of the VM's claims and refuses it when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only template claims gets one node with room for all of them, and under `Stay` a single claim waits; a `claimTemplate` claim is created, refused when it does not fit, deleted with its VMI, deleted when its VMI is refused later in admission, reused on a repeated call, and on a dry run neither created nor attached nor moved; a VM with a GPU claim and `nvidia.com/hostdev` passes. +- Chart tests for `kubernetes-nodes`: a `claimSelector` or `claimTemplate` with `os: talos` fails to render, and a Kairos pool gets the `gpu.cozystack.io/mode` label on its worker VMI template. +- Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held claims and refuses the next template claim, and `remainingBudget` is 0 without the entitlement. +- Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. +- Controller tests for the class metrics: capacity, bound units and the `Lost` count match the claims, and none of them appears in `GPUClass` status. - Binder tests: a node with a `NoSchedule` taint that virt-launcher pods do not tolerate gets no binds, moves or gathers and its units leave `free`, while the Fence Agents Remediation taint alone does not make a node cordoned. - Admission tests for the reserved label of the GPU usage monitor, refused on any object in a tenant namespace on create and on an update that adds it or changes its value, including a virt-launcher pod and a DataVolume that get it from a VM template, and allowed on an update that keeps or drops it. - Controller tests for the usage record: present exactly while `Bound` or `Attached`, independent of tenant labels, not reported twice, and kept through reconciles of its `WorkloadMonitor` and of the `Workload` controller. @@ -354,32 +354,31 @@ Each case gives what is seen and what to do. ## Rollout -1. `GPUClass`, `GPUDevice`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, both webhooks, the policies, the option sources, `device` and `deviceTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRDs ship and finalizers and usage records still run. Before tenants order vGPU devices, the `gpu.vgpu` flag is set from the root tenant down. -2. `deviceSelector` and `deviceTemplate` in `kubernetes-nodes`, after cozystack/cozystack#4911. -3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, `kairos.gpuMode` and the mode label, and `vgpu-vms` adds GPU guest images for VMs with `device` entries. A cluster with Talos GPU pools stays on raw names and has no hard hold guarantee. +1. `GPUClass`, `GPUClaim`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, both webhooks, the policies, the option sources, `claim` and `claimTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRDs ship and finalizers and usage records still run. Before tenants order vGPU claims, the `gpu.vgpu` flag is set from the root tenant down. +2. `claimSelector` and `claimTemplate` in `kubernetes-nodes`, after cozystack/cozystack#4911. +3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, `kairos.gpuMode` and the mode label, and `vgpu-vms` adds GPU guest images for VMs with `claim` entries. A cluster with Talos GPU pools stays on raw names and has no hard hold guarantee. 4. The host layer: the node layout, profile writes, node labels, the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, and Kairos vGPU hosts (the bundle on Kairos and a vGPU Manager image for it). 5. Budgets through #48, only if its gate takes over subtree caps for GPU keys. -6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by ordering new devices. +6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by ordering new claims. ## Open questions - Release on tenant suspension. A hoster may want a suspended tenant's GPUs released after a GPU-specific grace period; Cozystack has no platform-wide notion of suspension yet. -- Names. Aligning with #35 would make `GPUDevice` into `GPUClaim`; the meaning is the same. - Packing policy. Fullest-node packing is the default; hosters may need spreading across failure domains as a class option. ## Alternatives considered -DRA as the whole API, with `DeviceClass` as the catalog and `ResourceClaim` as the tenant object, or DRA first for passthrough with vGPU later. Rejected for now: a non-Pod `reservedFor` entry does keep a claim allocated for a stopped VM, but the NVIDIA DRA driver does not serve vGPU yet and its passthrough is alpha, and vGPU is the case hosters sell first. DRA is the planned backend under `GPUClass` (section 8), mapped one to one onto `GPUDevice`, so tenants order devices of a `dra` class through the same API. +DRA as the whole API, with `DeviceClass` as the catalog and `ResourceClaim` as the tenant object, or DRA first for passthrough with vGPU later. Rejected for now: a non-Pod `reservedFor` entry does keep a `ResourceClaim` allocated for a stopped VM, but the NVIDIA DRA driver does not serve vGPU yet and its passthrough is alpha, and vGPU is the case hosters sell first. DRA is the planned backend under `GPUClass` (section 8), mapped one to one onto `GPUClaim`, so tenants order claims of a `dra` class through the same API. -A cluster-scoped object per reservation unit, like `IPAddress` in #35. Rejected for the device-plugin backend, where kubelet picks the physical device and such an object would claim a precision the backend does not have. `GPUNode` gives the per-device view where it is real. +A cluster-scoped object per reservation unit, like `IPAddress` in #35. Rejected for the device-plugin backend, where kubelet picks the physical device and such an object would promise a precision the backend does not have. `GPUNode` gives the per-claim view where it is real. Tenant quota on the resource name only. Stock `requests.` quotas need no new code, but they cannot hold a GPU for a stopped VM and still refuse orders only at pod creation. -Placeholder pods that hold the device until the VM starts. Rejected because preemption picks victims by priority, not by owner, so one tenant's VM could evict another tenant's placeholder. +Placeholder pods that hold the claim until the VM starts. Rejected because preemption picks victims by priority, not by owner, so one tenant's VM could evict another tenant's placeholder. -Class-level reservations for worker pools, without per-worker devices. Rejected because a pool would get a promise of capacity, not GPUs. +Class-level reservations for worker pools, without per-worker claims. Rejected because a pool would get a promise of capacity, not GPUs. -A cluster-wide reservation per tenant and class, a count without a node. It needs no binding, moves or pod pin, and for single-GPU offers it would be enough. Rejected because the count holds only in sum. With free units spread one per node, a held set of two fits nowhere. A vGPU profile exists only on the nodes the layout gives it. A lost node silently shrinks every tenant's promise instead of showing which devices are `Lost`. +A cluster-wide reservation per tenant and class, a count without a node. It needs no binding, moves or pod pin, and for single-GPU offers it would be enough. Rejected because the count holds only in sum. With free units spread one per node, a held set of two fits nowhere. A vGPU profile exists only on the nodes the layout gives it. A lost node silently shrinks every tenant's promise instead of showing which claims are `Lost`. NVIDIA vgpu-device-manager as the profile writer. At v0.5.1 it writes mdev profiles only and has no path for `current_vgpu_type` on SR-IOV VFs, and the pull requests that added VF support were closed unmerged, NVIDIA/vgpu-device-manager#139 as not planned, and NVIDIA/vgpu-device-manager#195 after NVIDIA named gaps on the KubeVirt side and had no resources for that test round. The node agent writes VF profiles instead; nodes on the mdev path keep vgpu-device-manager (section 6). diff --git a/design-proposals/gpudevice/decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md similarity index 54% rename from design-proposals/gpudevice/decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md rename to design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md index 52cce3b3..b8e24b06 100644 --- a/design-proposals/gpudevice/decisions/0001-a-device-outside-permittedhostdevices-is-refused-at-admission.md +++ b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md @@ -1,4 +1,4 @@ -# 0001. A device whose class resource name KubeVirt does not permit is refused at VMI admission +# 0001. A claim whose class resource name KubeVirt does not permit is refused at VMI admission - **Number:** `0001` - **Date:** `2026-10-20` @@ -12,24 +12,24 @@ On the `default` and `vgpu` variants the platform writes a `permittedHostDevices` table into the KubeVirt CR whenever it has entries. The proposal assumed that KubeVirt refuses a GPU outside that table when the VMI is admitted. It does not. In KubeVirt v1.9.0 no admitter refuses a GPU or host-device name the table leaves out; the GraceIOVirtualization admitter reads the table only to apply its rules to names listed there, and the refusal is `validatePermittedHostDevices` in [`renderresources.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/renderresources.go), which runs when virt-controller renders the virt-launcher pod. A render error creates no pod, and the VMI stays `Pending` with `Synchronized=False`, reason `FailedCreate`, retried with backoff. It never reaches a final phase. -That defeats the attachment rules of section 2. A device leaves `Attached` only when its VMI is final or removed, so a device of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry. +That defeats the attachment rules of section 2. A claim leaves `Attached` only when its VMI is final or removed, so a claim of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry. ## Decision -The VMI webhook reads the effective table from the KubeVirt CR, the resource names of its `pciHostDevices` and `mediatedDevices` entries, and refuses a VMI whose device set includes a class whose `resourceName` is not in it, with a message naming the class and the name. An unset table means KubeVirt checks nothing, and then neither does the webhook. The controller also sets a `ResourceNameNotPermitted` condition on such a class, so the hoster sees the gap before a tenant hits it. +The VMI webhook reads the effective table from the KubeVirt CR, the resource names of its `pciHostDevices` and `mediatedDevices` entries, and refuses a VMI whose claim set includes a class whose `resourceName` is not in it, with a message naming the class and the name. An unset table means KubeVirt checks nothing, and then neither does the webhook. The controller also sets a `ResourceNameNotPermitted` condition on such a class, so the hoster sees the gap before a tenant hits it. ## Why not the alternatives -- **Rely on KubeVirt's own refusal**, as the proposal did. Rejected on the mechanism above: the refusal comes after admission, as a `Pending` VMI that holds its devices, not as an error the tenant sees at start. -- **Treat a `Pending` VMI with `Synchronized=False` and `FailedCreate` as final, and collect it.** It would release the devices, but only after the tenant has started a VM that cannot run, and the reason would surface as a VMI condition rather than in the start request. `FailedCreate` is also the reason KubeVirt gives for other failures to render or create the pod, so the rule would have to parse messages to tell this one apart. +- **Rely on KubeVirt's own refusal**, as the proposal did. Rejected on the mechanism above: the refusal comes after admission, as a `Pending` VMI that holds its claims, not as an error the tenant sees at start. +- **Treat a `Pending` VMI with `Synchronized=False` and `FailedCreate` as final, and collect it.** It would release the claims, but only after the tenant has started a VM that cannot run, and the reason would surface as a VMI condition rather than in the start request. `FailedCreate` is also the reason KubeVirt gives for other failures to render or create the pod, so the rule would have to parse messages to tell this one apart. - **Refuse such a class when it is created.** A class may legitimately come before its table entry, and the table can change after the class exists, so a create-time check catches only one of the two paths. ## Consequences -- A tenant learns at start, from the webhook message, that the offer is not usable yet, and the device stays unattached and moves by the usual rules. +- A tenant learns at start, from the webhook message, that the offer is not usable yet, and the claim stays unattached and moves by the usual rules. - The webhook and the controller condition read the same lists of the table, so the condition predicts the refusal. KubeVirt's own check also covers `usb` entries, which no GPU class uses. -- Every VMI admission with device references reads the KubeVirt CR, through the controller's cache. +- Every VMI admission with claim references reads the KubeVirt CR, through the controller's cache. ## Revisit if -KubeVirt starts refusing a device outside `permittedHostDevices` at VMI admission, or the table is dropped for DRA-backed devices. +KubeVirt starts refusing a claim outside `permittedHostDevices` at VMI admission, or the table is dropped for DRA-backed devices. diff --git a/design-proposals/gpudevice/decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md similarity index 58% rename from design-proposals/gpudevice/decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md rename to design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md index 2a1955ef..d2ba606f 100644 --- a/design-proposals/gpudevice/decisions/0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md +++ b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md @@ -1,4 +1,4 @@ -# 0002. GPUDevice works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run +# 0002. GPUClaim works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run - **Number:** `0002` - **Date:** `2026-10-21` @@ -10,29 +10,29 @@ ## Context -The proposal rolls step 1 out "on the existing variants" and says nothing about a cluster where GPUs do not reach VMs: gpu-operator under the `container` variant, gpu-operator not enabled, or no iaas bundle. On such a cluster KubeVirt runs without the `HostDevices` feature gate and without a `permittedHostDevices` table, and nothing there resolves a device reference. +The proposal rolls step 1 out "on the existing variants" and says nothing about a cluster where GPUs do not reach VMs: gpu-operator under the `container` variant, gpu-operator not enabled, or no iaas bundle. On such a cluster KubeVirt runs without the `HostDevices` feature gate and without a `permittedHostDevices` table, and nothing there resolves a claim reference. -The `vm-instance` chart renders only the reference, on the VMI template, and leaves it to the VMI webhook to fill in the GPUs (section 3). Without the webhook a `device` or `deviceTemplate` entry would boot the VM with no GPU and no error. The CRDs, on the other hand, ship with the controller on every cluster, so a `GPUClass` or a `GPUDevice` can exist on a cluster that has no placement running, and both carry finalizers. +The `vm-instance` chart renders only the reference, on the VMI template, and leaves it to the VMI webhook to fill in the GPUs (section 3). Without the webhook a `claim` or `claimTemplate` entry would boot the VM with no GPU and no error. The CRDs, on the other hand, ship with the controller on every cluster, so a `GPUClass` or a `GPUClaim` can exist on a cluster that has no placement running, and both carry finalizers. ## Decision -The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the binder, the VMI and virt-launcher webhooks, the controller's watches on KubeVirt objects, the GPU admission policies other than the label reservation of [0003](./0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md), the metrics scrape, and the tenant admin's write access to `GPUDevice`. Elsewhere: +The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the binder, the VMI and virt-launcher webhooks, the controller's watches on KubeVirt objects, the GPU admission policies other than the label reservation of [0003](./0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md), the metrics scrape, and the tenant admin's write access to `GPUClaim`. Elsewhere: -- `vm-instance` refuses to render `device` and `deviceTemplate` entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today. +- `vm-instance` refuses to render `claim` and `claimTemplate` entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today. - The platform refuses `gpu.rawResourceNames: forbidden`, because the policy that enforces it does not render there. -- The controller still adds and removes the finalizers of `GPUClass` and `GPUDevice` and keeps the usage records of devices, so deleting a class, a device or a tenant namespace never hangs. +- The controller still adds and removes the finalizers of `GPUClass` and `GPUClaim` and keeps the usage records of claims, so deleting a class, a claim or a tenant namespace never hangs. ## Why not the alternatives -- **Run everything on every cluster.** The binder and the webhooks read KubeVirt objects, which do not exist without KubeVirt, and under the `container` variant the GPUs gpu-operator advertises are meant for pods, not VFIO devices a VM can take. A device bound there would be held and recorded for a GPU no VM can use. -- **Gate the finalizer paths with the rest.** A hoster who disables GPUs, or moves to the `container` variant, would then leave every existing class and device stuck on its finalizer, and with it the tenant namespaces that hold devices. -- **Let `vm-instance` render device references anyway and rely on the webhook.** On these clusters no webhook exists, so the failure is a VM that runs without the GPU it asked for, which the tenant may notice only from inside the guest. +- **Run everything on every cluster.** The binder and the webhooks read KubeVirt objects, which do not exist without KubeVirt, and under the `container` variant the GPUs gpu-operator advertises are meant for pods, not VFIO devices a VM can take. A claim bound there would be held and recorded for a GPU no VM can use. +- **Gate the finalizer paths with the rest.** A hoster who disables GPUs, or moves to the `container` variant, would then leave every existing class and claim stuck on its finalizer, and with it the tenant namespaces that hold claims. +- **Let `vm-instance` render claim references anyway and rely on the webhook.** On these clusters no webhook exists, so the failure is a VM that runs without the GPU it asked for, which the tenant may notice only from inside the guest. ## Consequences -- On a cluster without GPUs for VMs, a tenant reads the catalog and its devices but cannot order, and a VM with device references fails at render with a message that says GPUs for VMs are not served. +- On a cluster without GPUs for VMs, a tenant reads the catalog and its claims but cannot order, and a VM with claim references fails at render with a message that says GPUs for VMs are not served. - Turning GPUs for VMs on writes one new cluster value, which every tenant release reads, so each upgrades once, and only on such clusters. -- The controller keeps a resync loop while a deleted device waits for its VMI, because with the feature off nothing watches VMIs. +- The controller keeps a resync loop while a deleted claim waits for its VMI, because with the feature off nothing watches VMIs. ## Revisit if diff --git a/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md b/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md index 713591f9..9ab2cb76 100644 --- a/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md +++ b/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md @@ -12,7 +12,7 @@ Section 5 keeps the GPU usage records under a `WorkloadMonitor` whose selector is a reserved label key, `gpu.cozystack.io/usage-monitor`, and has admission refuse that key on every object in a tenant namespace, on create and update, so no object can match the monitor and no tenant can shape the GPU usage records. -Two cases were left open. First, the GPU admission policies render only where GPUs serve VMs ([0002](./0002-gpudevice-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). Had the reservation followed them, a tenant could label its objects on a cluster without GPUs, and they would match the monitor the day the hoster turns GPUs on. Second, an object can already carry the key when the policy first renders, from before the upgrade that brought it. A policy that refuses every write of an object carrying the key would refuse every update of that object that keeps the label, including the one in which a controller removes its finalizer, so its deletion would hang until someone drops the label by hand. +Two cases were left open. First, the GPU admission policies render only where GPUs serve VMs ([0002](./0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). Had the reservation followed them, a tenant could label its objects on a cluster without GPUs, and they would match the monitor the day the hoster turns GPUs on. Second, an object can already carry the key when the policy first renders, from before the upgrade that brought it. A policy that refuses every write of an object carrying the key would refuse every update of that object that keeps the label, including the one in which a controller removes its finalizer, so its deletion would hang until someone drops the label by hand. ## Decision diff --git a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md index 37fbb11e..cd43c7be 100644 --- a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md +++ b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md @@ -34,7 +34,7 @@ Not every GPU device ID maps to a PCI function. The GPU Operator's device plugin ## Consequences - An unhealthy device that no pod holds disappears from the allocatable list and looks the same as a missing one. Both lower `allocatable`, and both become lost capacity after the grace period once the node holds more units than remain. -- `usedBy` traces only one pod per function. Section 1's "ownership is exact for running GPUs" holds per function for VFs and whole GPUs, and for an mdev parent only through the devices' own `status.attachedTo`. +- `usedBy` traces only one pod per function. Section 1's "ownership is exact for running GPUs" holds per function for VFs and whole GPUs, and for an mdev parent only through the claims' own `status.attachedTo`. - The agent does not fill `profile` yet; writing and reading VF profiles is host-layer work (Rollout, step 4). ## Revisit if diff --git a/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md index c3aab6a7..f8c28592 100644 --- a/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md +++ b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md @@ -18,13 +18,13 @@ The rule skips exactly `node.kubernetes.io/unschedulable`, `node.kubernetes.io/n ## Why not the alternatives -- **Skip the whole `node.kubernetes.io/` prefix.** It binds devices onto nodes that the scheduler then refuses, and keeps their units in `free`, which is the failure the taint rule exists to prevent. -- **Count the three as well.** Kubernetes sets `node.kubernetes.io/unschedulable` on every cordoned node, so a cordoned node would also read as tainted. A taint keeps a bound device where it is, so under `Move` a cordon would never move a device, and `onCordon` would stop meaning anything. +- **Skip the whole `node.kubernetes.io/` prefix.** It binds claims onto nodes that the scheduler then refuses, and keeps their units in `free`, which is the failure the taint rule exists to prevent. +- **Count the three as well.** Kubernetes sets `node.kubernetes.io/unschedulable` on every cordoned node, so a cordoned node would also read as tainted. A taint keeps a bound claim where it is, so under `Move` a cordon would never move a claim, and `onCordon` would stop meaning anything. ## Consequences -- A node under memory, disk or PID pressure, one whose network is unavailable, and one tainted `out-of-service` get no binds, moves or gathers while the taint holds and virt-launcher pods do not tolerate it, and their units leave `free` and `maxFreePerNode`. Outside a gather at VMI creation, a device already bound there stays, as on any tainted node. -- A pressure taint comes and goes with the node's condition, so `free` on a busy class can drop and return without any device moving. +- A node under memory, disk or PID pressure, one whose network is unavailable, and one tainted `out-of-service` get no binds, moves or gathers while the taint holds and virt-launcher pods do not tolerate it, and their units leave `free` and `maxFreePerNode`. Outside a gather at VMI creation, a claim already bound there stays, as on any tainted node. +- A pressure taint comes and goes with the node's condition, so `free` on a busy class can drop and return without any claim moving. ## Revisit if diff --git a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md new file mode 100644 index 00000000..303e0e19 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md @@ -0,0 +1,32 @@ +# 0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim + +- **Number:** `0006` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** not yet + +## Context + +Section 3 pins a VMI to a cordoned node when a claim of a `Stay` class is there and all of the VMI's claims are on that node, with no template claims: there is no gather, "even if the node is also not `Ready`", and the VMI waits. Section 2 has a `Stay` claim keep its node through a cordon, while lost capacity, a node that is gone and a node that no longer serves the class are hard reasons for a claim to leave. The proposal did not say which of the two wins when a pinned node also has a hard reason, so a node under maintenance that then died could hold a VM's claims, and keep them in the usage records, for as long as the dead node stayed registered. + +## Decision + +The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint virt-launcher pods do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (sections 2 and 3): a new VMI is refused when the node is gone, no longer serves the class, or has lost capacity the claim does not fit under, and otherwise its set is gathered onto one node with room for all of it; a VMI already waiting there unscheduled is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The binder at admission and the controller's background pass read the pin through one function, so admission never pins a VMI that the next pass deletes. + +## Why not the alternatives + +- **The pin always wins.** A `Stay` class exists for a hoster who wants VMs back on the same hardware after maintenance. A dead node is not coming back to the same state on any known schedule, and the VM's claims would stay held and recorded against hardware that serves nothing. +- **Break the pin on `NotReady` alone.** It undoes the point of `Stay` for the most common maintenance step, a reboot, and section 2 already uses the lost-capacity grace period so that a short outage moves nothing. +- **Break the pin on any taint.** Apart from a gather at VMI creation, a taint never moves a bound claim; it only keeps new binds off the node (section 2). Breaking the pin on one would make a `Stay` claim leave in a case where a claim of a `Move` class on an uncordoned node stays. + +## Consequences + +- A `Stay` VM waits through a reboot of its cordoned node and comes back on it, and after a node failure its claims move like any other VM's once the grace period has passed. +- A pinned VM waits on a dead node for the lost-capacity grace period, five minutes by default, before its claims can move. + +## Revisit if + +Hosters ask for a `Stay` policy that also outlasts a node failure, for example to wait for a repaired node rather than move off it. diff --git a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md deleted file mode 100644 index 235ad1d6..00000000 --- a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-device.md +++ /dev/null @@ -1,32 +0,0 @@ -# 0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the device - -- **Number:** `0006` -- **Date:** `2026-10-20` -- **Status:** Accepted -- **Deciders:** `@lexfrei` -- **Proposal:** [`../README.md`](../README.md) -- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** not yet - -## Context - -Section 3 pins a VMI to a cordoned node when a device of a `Stay` class is there and all of the VMI's devices are on that node, with no template devices: there is no gather, "even if the node is also not `Ready`", and the VMI waits. Section 2 has a `Stay` device keep its node through a cordon, while lost capacity, a node that is gone and a node that no longer serves the class are hard reasons for a device to leave. The proposal did not say which of the two wins when a pinned node also has a hard reason, so a node under maintenance that then died could hold a VM's devices, and keep them in the usage records, for as long as the dead node stayed registered. - -## Decision - -The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint virt-launcher pods do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (sections 2 and 3): a new VMI is refused when the node is gone, no longer serves the class, or has lost capacity the device does not fit under, and otherwise its set is gathered onto one node with room for all of it; a VMI already waiting there unscheduled is deleted when its devices must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The binder at admission and the controller's background pass read the pin through one function, so admission never pins a VMI that the next pass deletes. - -## Why not the alternatives - -- **The pin always wins.** A `Stay` class exists for a hoster who wants VMs back on the same hardware after maintenance. A dead node is not coming back to the same state on any known schedule, and the VM's devices would stay held and recorded against hardware that serves nothing. -- **Break the pin on `NotReady` alone.** It undoes the point of `Stay` for the most common maintenance step, a reboot, and section 2 already uses the lost-capacity grace period so that a short outage moves nothing. -- **Break the pin on any taint.** Apart from a gather at VMI creation, a taint never moves a bound device; it only keeps new binds off the node (section 2). Breaking the pin on one would make a `Stay` device leave in a case where a device of a `Move` class on an uncordoned node stays. - -## Consequences - -- A `Stay` VM waits through a reboot of its cordoned node and comes back on it, and after a node failure its devices move like any other VM's once the grace period has passed. -- A pinned VM waits on a dead node for the lost-capacity grace period, five minutes by default, before its devices can move. - -## Revisit if - -Hosters ask for a `Stay` policy that also outlasts a node failure, for example to wait for a repaired node rather than move off it. diff --git a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md new file mode 100644 index 00000000..39f58d25 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md @@ -0,0 +1,33 @@ +# 0007. Whoever can write a VM orders its template claims, while held claims stay with tenant admins + +- **Number:** `0007` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** not yet + +## Context + +The Security section gives `GPUClaim` writes to tenants with the `admin` role or above, and has the webhook create `GPUClaim` objects for `claimTemplate` entries, only of the class and count the template names. It does not say who that webhook acts for. The VMI is created by KubeVirt's virt-controller, not by the person who wrote the VM, so by the time the webhook sees it the requester is KubeVirt and the identity behind the VM is gone. + +In Cozystack more identities can write a `vm-instance` than can write a `GPUClaim`. The tenant role that tenant ServiceAccounts get grants every verb on `apps.cozystack.io`, which includes `vm-instance`, and the `use` role can start and restart a VM, which creates a new VMI. + +## Decision + +A `claimTemplate` entry is part of the VM, so whoever can write the VM can order its template claims, and whoever can start it makes them exist and enter the usage records for as long as the VMI runs. The webhook applies the tenant's budgets, the vGPU entitlement and capacity to every template claim, as to any order, and does not check the identity behind the VM. Ordering a held `GPUClaim` directly stays with tenant admins and above. + +## Why not the alternatives + +- **Check the identity behind the VM at VMI admission.** The webhook sees KubeVirt's service account. The VM's author is recorded nowhere it could read reliably, and the identity that starts a VM can differ from the one that wrote it. +- **Refuse `claimTemplate` entries in `vm-instance` for identities that cannot write `GPUClaim`.** The aggregated API that serves `vm-instance` would need a per-field authorization step for one field of one kind, and the rule would still not cover a start by a `use` role holder, who changes no field at all. The cost is a judgement call, not a hard limit. + +## Consequences + +- A tenant admin who hands out a tenant ServiceAccount also hands out the ability to order GPUs through template claims, up to the tenant's budget, and the `use` role lets its holder start VMs that already carry template claims. The budget, the class keys and the vGPU entitlement are the controls that limit both. +- A held claim, which is held and recorded whether or not anything runs on it, still needs a tenant admin. + +## Revisit if + +KubeVirt starts recording the VM's author on the VMI in a form admission can trust, or Cozystack gains per-field authorization on application kinds. diff --git a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md deleted file mode 100644 index db240bc7..00000000 --- a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-devices.md +++ /dev/null @@ -1,33 +0,0 @@ -# 0007. Whoever can write a VM orders its template devices, while held devices stay with tenant admins - -- **Number:** `0007` -- **Date:** `2026-10-20` -- **Status:** Accepted -- **Deciders:** `@lexfrei` -- **Proposal:** [`../README.md`](../README.md) -- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** not yet - -## Context - -The Security section gives `GPUDevice` writes to tenants with the `admin` role or above, and has the webhook create `GPUDevice` objects for `deviceTemplate` entries, only of the class and count the template names. It does not say who that webhook acts for. The VMI is created by KubeVirt's virt-controller, not by the person who wrote the VM, so by the time the webhook sees it the requester is KubeVirt and the identity behind the VM is gone. - -In Cozystack more identities can write a `vm-instance` than can write a `GPUDevice`. The tenant role that tenant ServiceAccounts get grants every verb on `apps.cozystack.io`, which includes `vm-instance`, and the `use` role can start and restart a VM, which creates a new VMI. - -## Decision - -A `deviceTemplate` entry is part of the VM, so whoever can write the VM can order its template devices, and whoever can start it makes them exist and enter the usage records for as long as the VMI runs. The webhook applies the tenant's budgets, the vGPU entitlement and capacity to every template device, as to any order, and does not check the identity behind the VM. Ordering a held `GPUDevice` directly stays with tenant admins and above. - -## Why not the alternatives - -- **Check the identity behind the VM at VMI admission.** The webhook sees KubeVirt's service account. The VM's author is recorded nowhere it could read reliably, and the identity that starts a VM can differ from the one that wrote it. -- **Refuse `deviceTemplate` entries in `vm-instance` for identities that cannot write `GPUDevice`.** The aggregated API that serves `vm-instance` would need a per-field authorization step for one field of one kind, and the rule would still not cover a start by a `use` role holder, who changes no field at all. The cost is a judgement call, not a hard limit. - -## Consequences - -- A tenant admin who hands out a tenant ServiceAccount also hands out the ability to order GPUs through template devices, up to the tenant's budget, and the `use` role lets its holder start VMs that already carry template devices. The budget, the class keys and the vGPU entitlement are the controls that limit both. -- A held device, which is held and recorded whether or not anything runs on it, still needs a tenant admin. - -## Revisit if - -KubeVirt starts recording the VM's author on the VMI in a form admission can trust, or Cozystack gains per-field authorization on application kinds. diff --git a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md index cd15d5f8..5a77174a 100644 --- a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md +++ b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md @@ -10,7 +10,7 @@ ## Context -The Security section lists three rights the GPU controller gains beyond its own kinds and the usage records: patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its devices. Only the third is used by Rollout step 1. Node labels and plugin restarts belong to the node layout of section 6, which ships as Rollout step 4, and until then the hoster writes node labels by hand. +The Security section lists three rights the GPU controller gains beyond its own kinds and the usage records: patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Only the third is used by Rollout step 1. Node labels and plugin restarts belong to the node layout of section 6, which ships as Rollout step 4, and until then the hoster writes node labels by hand. ## Decision From a55622298ec4de6e8e78c6a7688ff0a5592bc051 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:16:17 +0300 Subject: [PATCH 07/32] docs(design-proposals): make GPUClass mode immutable and name the class in claimSelector A worker pool stated its GPU mode in its own field, mirrored it into a label on the worker VMI, and the webhook refused a claim whose class disagreed. With the mode fixed for a class's lifetime and the selector naming the class, the pool takes the mode from the class, and the label, its admission check and the undefined case of a VMI without the label go away. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 19 +++++----- ...is-immutable-and-a-pool-names-its-class.md | 35 +++++++++++++++++++ 2 files changed, 45 insertions(+), 9 deletions(-) create mode 100644 design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index eb9dfd98..91d770fd 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -23,6 +23,7 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Decisions +- [0009. `GPUClass.mode` never changes, and a pool's `claimSelector` names its class](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md) - [0008. The GPU controller gets its node-label and plugin-restart rights with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) - [0007. Whoever can write a VM orders its template claims, while held claims stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md) - [0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md) @@ -84,7 +85,7 @@ spec: displayName: "NVIDIA L40S, 24 GB vGPU (compute)" vendor: nvidia model: L40S - mode: vgpu # passthrough | vgpu + mode: vgpu # passthrough | vgpu; never changes profile: L40S-24C # vgpu only memory: 24Gi maxCount: 2 # vGPUs of this class per VM @@ -136,7 +137,7 @@ status: A class maps an offer to a backend through a `source` union and to the nodes that serve it through `nodeSelector`. The first release has one source, `devicePlugin`; `dra` is reserved for section 8. Two classes may share a resource name when their node selectors do not overlap, which lets a host layer that cannot tell two offers apart by name tell them apart by node. A node that matches two classes with the same resource name gets a condition on both classes and on its `GPUNode`, which names the two classes, and is not used for binding. Class names are DNS labels, since they become part of quota keys. -A class referenced by held claims cannot disappear or change its meaning. A finalizer keeps a deleted class until its last `Bound`, `Attached` or `Lost` claim is gone, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `vendor`, `model`, `mode`, `profile`, `memory`, `maxCount` and the kind of `source` cannot change while the finalizer is present. `resourceName` and `nodeSelector` can change, for changes in the host layer (section 6) and for moving the offer to other nodes. +A class referenced by held claims cannot disappear or change its meaning. A finalizer keeps a deleted class until its last `Bound`, `Attached` or `Lost` claim is gone, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `vendor`, `model`, `profile`, `memory`, `maxCount` and the kind of `source` cannot change while the finalizer is present. `mode` never changes, held claims or not, so a class name always stands for one mode, and whatever names a class, such as a worker pool's selector, knows the mode from the name ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). `resourceName` and `nodeSelector` can change, for changes in the host layer (section 6) and for moving the offer to other nodes. A `GPUClaim` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate claims. How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 6), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPU claims of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. Admission refuses a claim above `maxCount`. The spec is immutable, so a change is a new order. @@ -193,6 +194,7 @@ gpus: # kubernetes-nodes, Kairos pool: each worker takes one matching claim gpus: - claimSelector: + className: l40s-24c matchLabels: pool: inference ``` @@ -213,9 +215,9 @@ A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with it A `claimTemplate` entry names a class and a count instead of a held claim. At VMI creation the webhook creates a `GPUClaim` from it, named after the VMI and the entry's index and marked as a template claim by the `gpu.cozystack.io/template-claim` label. A repeated call or a recreated VMI finds the same claim and reuses it when it carries that mark, has no owner yet and matches the template; a claim of that name still being deleted, or still owned by the VM's previous VMI, refuses the VMI, and KubeVirt retries. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name refuses the VMI, and the message names it. The webhook has the binder bind the claim under the same capacity and budget checks as any order, within the admission timeout, and attaches it. The node for template claims is picked with room for all of the VMI's claims, held and template: the node of the gathered set when the VMI has held claims, and otherwise one node with room for all its template claims. The VMI is refused if no node has room for all of them. A bind that does not fit or does not finish in time refuses the VMI, and KubeVirt retries. On a dry run the webhook creates, attaches and moves nothing, so it declares `sideEffects: NoneOnDryRun`. The VMI has no UID yet at that point, so the controller sets the VMI as the claim's owner once the VMI exists, and deletes a template claim whose VMI never appears after the same grace period as a stale attachment. The controller deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives and is billed only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any claim, but only while the VMI exists: an order that does not fit is refused when the VM or worker starts, not ahead of time. A template counts as a claim reference under `forbidden`. It lets an autoscaled pool pay for running workers only. -A `vm-instance` entry has exactly one of `name`, `claim` or `claimTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `claimSelector` or `claimTemplate`. A list holds only `name` entries or only references, and a pool has at most one `claimSelector` or one `claimTemplate`, never both, and a selector's claims must share a class and a `count`; the webhook checks this at each attach, since claim labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `claim` and `claimTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to Kairos is separate work. +A `vm-instance` entry has exactly one of `name`, `claim` or `claimTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `claimSelector` or `claimTemplate`. A list holds only `name` entries or only references, and a pool has at most one `claimSelector` or one `claimTemplate`, never both, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `claim` and `claimTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to Kairos is separate work. -Kairos pools take GPUs only through `claimSelector` or `claimTemplate`, from classes of both modes. The chart refuses `name` entries on them. The pool's mode, `kairos.gpuMode`, must match the class, and the chart sets it as the `gpu.cozystack.io/mode` label on the worker VMI template ([Choosing the worker image by mode](../kairos-vgpu-workers/README.md#choosing-the-worker-image-by-mode)). At VMI creation the webhook refuses a claim, selected or from a template, whose class `mode` differs from that label, and a pool reference on a VMI without the label. The `kubernetes-nodes` chart, from Rollout step 2, fails to render a `claimSelector` or `claimTemplate` on a pool with `os: talos`, the field cozystack/cozystack#4911 adds, so the pool's HelmRelease fails before any machine is created. A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools, existing and new, keep working as today on `name` entries, and `GPUClaim` does not extend them. +Kairos pools take GPUs only through `claimSelector` or `claimTemplate`, from classes of both modes. The chart refuses `name` entries on them. A pool states no mode of its own: its `claimSelector` or `claimTemplate` names the class, and the class's `mode` never changes, so the chart and the webhook take the mode from the class. The `kubernetes-nodes` chart, from Rollout step 2, fails to render a `claimSelector` or `claimTemplate` on a pool with `os: talos`, the field cozystack/cozystack#4911 adds, so the pool's HelmRelease fails before any machine is created. A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools, existing and new, keep working as today on `name` entries, and `GPUClaim` does not extend them. ### 4. Quotas @@ -266,7 +268,7 @@ A layout change on a node, of mode or of profile, waits until the node has no GP Raw `name` entries keep working, and nothing converts them automatically, because a conversion would bind GPUs a tenant never ordered and put them in the usage records. The binder counts their usage, so it never books a unit a raw-name VM holds. The reverse is not protected: the scheduler does not see a claim that is bound but not attached, so while raw names are allowed, a raw-name VM can take that unit. A reservation is a hard guarantee only with `gpu.rawResourceNames: forbidden`. -A tenant moves a VM by stopping it, ordering claims of the matching class, replacing each `name` entry with a `claim` entry and starting it. On a full pool the claim can only be ordered after the stop; with spare units, ordering first puts both in the usage records until the restart. A Talos GPU pool has no in-place path. The tenant creates a Kairos pool with `kairos.gpuMode` and a `claimSelector`, which needs a hoster-built worker image for the class's mode, and then deletes the Talos pool. On a full cluster the Talos pool scales down first, so its GPUs are free to order. ComputePlane GPU groups have no such path until ComputePlane runs Kairos workers, so a cluster with them keeps raw names allowed. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes `claim` entries. A Talos pool on one takes the Kairos path above. +A tenant moves a VM by stopping it, ordering claims of the matching class, replacing each `name` entry with a `claim` entry and starting it. On a full pool the claim can only be ordered after the stop; with spare units, ordering first puts both in the usage records until the restart. A Talos GPU pool has no in-place path. The tenant creates a Kairos pool with a `claimSelector`, which needs a hoster-built worker image for the class's mode, and then deletes the Talos pool. On a full cluster the Talos pool scales down first, so its GPUs are free to order. ComputePlane GPU groups have no such path until ComputePlane runs Kairos workers, so a cluster with them keeps raw names allowed. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes `claim` entries. A Talos pool on one takes the Kairos path above. Quota keys move too. A raw GPU key is a quota key on a GPU resource name, such as `nvidia.com/AD102GL_L40S`. A claim-backed VM's pod still requests that name, so while a tenant migrates, its raw key covers both kinds of VM; after its last raw-name VM has moved, the key is replaced by `gpu.cozystack.io/*` keys. A tenant whose raw-name VMs use vGPU can order their claims only after it is entitled to vGPU (section 4), and within any class keys set above it. @@ -299,7 +301,7 @@ The change is additive. Existing `gpus` entries render as before, and the existi Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, and the tenants' GPU limits, with four exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only limits were `gpu.cozystack.io/*` keys lose them (rules below), and tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through `GPUClaim`, so they lose their GPUs and run on the plain worker image. A child tenant that declares its own quotas no longer shares its parent's GPU limit, because a raw key is carved out of the parent's pool rather than capping the subtree. -The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `claim` and `claimTemplate` entries of VMs become raw `name` entries and those VMs restart. `claimSelector` and `claimTemplate` entries and `kairos.gpuMode` are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUClaim` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the GPU controller's `WorkloadMonitor` objects, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. +The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `claim` and `claimTemplate` entries of VMs become raw `name` entries and those VMs restart. `claimSelector` and `claimTemplate` entries are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUClaim` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the GPU controller's `WorkloadMonitor` objects, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. Three rules hold throughout: @@ -333,7 +335,6 @@ Each case gives what is seen and what to do. - A layout change does not apply → the node shows `LayoutPending`; drain it. A VM with a raw `name` entry on the node holds the change until it stops. - A Talos GPU pool, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a Kairos pool with `claimSelector`, or allow raw names again while the cluster has Talos GPU pools. - A pool with `os: talos` and a `claimSelector` or `claimTemplate` fails to render → Talos pools take GPUs only through `name` entries; use a Kairos pool for claims. -- A Kairos worker VMI is refused because its claim's class `mode` differs from the pool's `gpu.cozystack.io/mode` label → set `kairos.gpuMode` to the class's mode, or select claims of a class in the pool's mode. - A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. - A pool worker is refused → no matching claim is free, because of a surge machine during a rollout, claims still `Pending` or `Lost`, matching claims only on nodes that are not `Ready`, or another pool or a VM using the matching claims; wait, add claims, give each pool its own labels, or lower `maxReplicas`. @@ -341,7 +342,7 @@ Each case gives what is seen and what to do. - Unit tests for the binder and both webhooks: capacity with raw-name usage, budgets over the ancestor walk, the phase table, every move and refusal in sections 2 and 3, concurrent starts, and the pod pin after a LiveUpdate sync. - Webhook tests for cordons, templates and NICs: under `Move` a claim moves at VMI creation and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity the unscheduled VMI is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI creation, refused when no node fits, a `Stay` claim on a cordoned node pins the VMI when it holds all of the VM's claims and refuses it when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only template claims gets one node with room for all of them, and under `Stay` a single claim waits; a `claimTemplate` claim is created, refused when it does not fit, deleted with its VMI, deleted when its VMI is refused later in admission, reused on a repeated call, and on a dry run neither created nor attached nor moved; a VM with a GPU claim and `nvidia.com/hostdev` passes. -- Chart tests for `kubernetes-nodes`: a `claimSelector` or `claimTemplate` with `os: talos` fails to render, and a Kairos pool gets the `gpu.cozystack.io/mode` label on its worker VMI template. +- Chart tests for `kubernetes-nodes`: a `claimSelector` or `claimTemplate` with `os: talos` fails to render. - Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held claims and refuses the next template claim, and `remainingBudget` is 0 without the entitlement. - Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. - Controller tests for the class metrics: capacity, bound units and the `Lost` count match the claims, and none of them appears in `GPUClass` status. @@ -356,7 +357,7 @@ Each case gives what is seen and what to do. 1. `GPUClass`, `GPUClaim`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, both webhooks, the policies, the option sources, `claim` and `claimTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRDs ship and finalizers and usage records still run. Before tenants order vGPU claims, the `gpu.vgpu` flag is set from the root tenant down. 2. `claimSelector` and `claimTemplate` in `kubernetes-nodes`, after cozystack/cozystack#4911. -3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, `kairos.gpuMode` and the mode label, and `vgpu-vms` adds GPU guest images for VMs with `claim` entries. A cluster with Talos GPU pools stays on raw names and has no hard hold guarantee. +3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, and `vgpu-vms` adds GPU guest images for VMs with `claim` entries. A cluster with Talos GPU pools stays on raw names and has no hard hold guarantee. 4. The host layer: the node layout, profile writes, node labels, the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, and Kairos vGPU hosts (the bundle on Kairos and a vGPU Manager image for it). 5. Budgets through #48, only if its gate takes over subtree caps for GPU keys. 6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by ordering new claims. diff --git a/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md b/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md new file mode 100644 index 00000000..7f664f51 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md @@ -0,0 +1,35 @@ +# 0009. `GPUClass.mode` never changes, and a pool's `claimSelector` names its class + +- **Number:** `0009` +- **Date:** `2026-10-23` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) +- **Implemented in:** not yet + +## Context + +Section 1 froze a class's `mode` only while its finalizer was present, that is while claims of the class were held. A worker pool selected claims by labels alone, and a selector's claims had to share a class anyway. The mode the pool's workers needed, passthrough or vGPU, came from a separate pool field, `kairos.gpuMode`, which the chart set as a `gpu.cozystack.io/mode` label on the worker VMI template, and the webhook refused a claim whose class mode differed from that label. The implementation accepted a pool reference on a VMI without the label, so the text and the code disagreed. + +@kvaps asked in review of #99 and [#101](https://github.com/cozystack/community/pull/101#pullrequestreview-5616165668) for `mode` to be immutable always and for the selector to name the class, so that a pool does not state its mode at all. + +## Decision + +`GPUClass.mode` cannot change after the class is created, whether or not claims of it exist. A `claimSelector` carries `className` next to its label selector and matches only claims of that class, as a `claimTemplate` already names its class. A pool states no mode: the chart and admission read it from the named class. The mode label and the mode check at VMI admission are dropped. + +## Why not the alternatives + +- **Keep the pool's mode field and the label check.** Two fields say the same thing, one of them can be wrong, and the webhook has to refuse the mismatch at every worker start instead of the chart getting it right once. A VMI without the label was a third state nobody had defined. +- **Freeze `mode` only while claims exist, as section 1 had it.** A class without claims could switch mode, and a pool that names it would then render workers for the other mode, so the chart could not rely on the name. The hoster loses nothing by the stricter rule: a different mode is a new class with a new name, which a tenant has to order against anyway because budgets and prices are per class. + +## Consequences + +- A class name stands for one mode for the class's lifetime, so a pool's worker template follows from the class it names. +- A hoster who created a class in the wrong mode deletes it and creates another, once no claim of it is held. +- A selector cannot span classes, which section 3 already required in another form. +- A pool rendered against a class keeps that class's worker image until it renders again, so a hoster who deletes a class does not create another class of the same name in the other mode. Nothing in the platform enforces this; the hoster documentation states it. + +## Revisit if + +A class needs to serve both modes, for example if one resource name ever covers whole GPUs and vGPUs on the same node. From 0c9e002e9c4df936249fe81c926920d6c2574622 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:17:24 +0300 Subject: [PATCH 08/32] docs(design-proposals): take Kairos out of the GPUClaim proposal The proposal tied three things to Kairos: Kairos as a vGPU host OS, claimSelector and claimTemplate only for Kairos pools, and no Talos GPU pools as a precondition of forbidden. Kairos as a host OS of the management cluster is a decision of its own, and Kairos worker pools wait for their own proposal. Talos workers take passthrough GPUs today, so Talos pools can take claims of passthrough classes, step 2 no longer waits for the Kairos worker pools, and forbidden only needs no pools on raw names. The host layer is stated without a host OS. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 44 ++++++++++++++-------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 91d770fd..9c762c02 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -9,7 +9,7 @@ A tenant who wants a GPU today writes a device-plugin resource name such as `nvidia.com/AD102GL_L40S` into `gpus` of a VM or a worker pool and hopes one is free. The `gpu` option source shows how many of each name the nodes advertise and on which nodes, but nothing says which model, mode or profile a name stands for, nothing holds a GPU while the VM is off, and an order that does not fit is accepted and then sits in `Pending`. A hoster selling GPUs needs the opposite: a catalog of GPU models and modes, GPUs a tenant takes within a quota and keeps until they release them, the same handling for VMs and managed Kubernetes worker pools, and a usage record of who held what and when, which the hoster's external billing system reads. -This proposal adds three kinds. `GPUClass` is the hoster's catalog: one entry per offer, such as a whole H100 in passthrough or an L40S-24C vGPU profile. `GPUClaim` is a tenant's GPU, or a set of GPUs on one node: ordered against a class, bound to a node with free capacity, and attached to a VM by name or to a Kairos worker pool by a label selector. `GPUNode` is the inventory: every physical GPU and VF on a node, its configuration, and which tenant uses it. The proposal also covers the host layer under them, as its own phase: passthrough and vGPU nodes in one cluster, declarative vGPU profiles, and the device plugin that advertises GPUs to KubeVirt. +This proposal adds three kinds. `GPUClass` is the hoster's catalog: one entry per offer, such as a whole H100 in passthrough or an L40S-24C vGPU profile. `GPUClaim` is a tenant's GPU, or a set of GPUs on one node: ordered against a class, bound to a node with free capacity, and attached to a VM by name or to a worker pool by a label selector. `GPUNode` is the inventory: every physical GPU and VF on a node, its configuration, and which tenant uses it. The proposal also covers the host layer under them, as its own phase: passthrough and vGPU nodes in one cluster, declarative vGPU profiles, and the device plugin that advertises GPUs to KubeVirt. NVIDIA's vGPU stack changes quickly. Within the last few releases the driver branches, the way vGPUs are exposed (mediated devices or SR-IOV virtual functions), the KubeVirt device plugins and the licensing rules all changed. This proposal describes that stack as it was in October 2026. Expect it to be updated as NVIDIA and KubeVirt move, with a decision record under it for each change of course, and check the versions named here against your own before relying on them. @@ -55,12 +55,12 @@ Both consumers take GPUs the same way: `vm-instance` and `kubernetes-nodes` read ## Goals - A hoster publishes a catalog of GPU classes, each with a model, a mode (passthrough or vGPU) and, for vGPU, a profile. Tenants read the catalog with free capacity and their remaining budget before ordering. -- A tenant orders a number of GPUs of a class within their quota and holds them, attached or not, until they release them. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden` (section 7). That is the end state, not the default: `forbidden` needs every Talos GPU pool replaced by a Kairos pool, which waits for cozystack/cozystack#4911, and every ComputePlane GPU group gone, which waits for ComputePlane on Kairos workers. Until then a cluster with either keeps raw names, and the hold is not guaranteed there. An order that cannot be satisfied is refused with a message naming the class. -- VMs and Kairos worker pools attach held GPUs the same way. A restarted VM gets the same held claim back, and a replaced worker gets one of its pool's claims back unless the tenant's own VM or another pool uses it. +- A tenant orders a number of GPUs of a class within their quota and holds them, attached or not, until they release them. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden` (section 7). That is the end state, not the default: `forbidden` needs every GPU worker pool on claims instead of raw names, ComputePlane GPU groups included, which take only raw names today. Until then such a cluster keeps raw names, and the hold is not guaranteed there. An order that cannot be satisfied is refused with a message naming the class. +- VMs and worker pools attach held GPUs the same way. A restarted VM gets the same held claim back, and a replaced worker gets one of its pool's claims back unless the tenant's own VM or another pool uses it. - Every physical GPU and VF can be traced to its node, its configuration and the tenant using it. - Held GPUs appear in usage records (`Workload`) by tenant, class and time, which an external billing system reads. Cozystack charges nothing and stores no prices. - One cluster can have passthrough and vGPU nodes of the same GPU model, with vGPU profiles applied declaratively and kept across reboots. -- Existing `gpus` entries keep working, with a documented path to `GPUClaim` for VMs; existing Talos GPU pools stay on `name` entries. +- Existing `gpus` entries keep working, with a documented path to `GPUClaim` for VMs and for worker pools on passthrough GPUs. - The catalog can later describe MIG, other vendors and DRA-backed devices without changing the tenant API. ### Non-goals @@ -171,7 +171,7 @@ A `dra` claim before its first start is the exception: it is `Bound` without a n A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 deletes a final VMI before it creates the VM's next one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)), so a new VMI never finds its claim attached to the old one. -A claim keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted, the controller also deletes the claim's unscheduled VMI, so the claim becomes unattached and moves. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Kairos workers use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules of section 3. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI creation, when one is needed, refuses such a set (section 3). +A claim keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted, the controller also deletes the claim's unscheduled VMI, so the claim becomes unattached and moves. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules of section 3. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI creation, when one is needed, refuses such a set (section 3). A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity when the agent reports a device missing or unhealthy for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no claim. Then: @@ -184,14 +184,14 @@ Claims in `Lost` are rebound before new orders when capacity returns, a VM's set ### 3. Attaching -VMs and Kairos worker pools reference claims in `gpus`: +VMs and worker pools reference claims in `gpus`: ```yaml # vm-instance gpus: - claim: train-0 --- -# kubernetes-nodes, Kairos pool: each worker takes one matching claim +# kubernetes-nodes: each worker of the pool takes one matching claim gpus: - claimSelector: className: l40s-24c @@ -215,9 +215,9 @@ A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with it A `claimTemplate` entry names a class and a count instead of a held claim. At VMI creation the webhook creates a `GPUClaim` from it, named after the VMI and the entry's index and marked as a template claim by the `gpu.cozystack.io/template-claim` label. A repeated call or a recreated VMI finds the same claim and reuses it when it carries that mark, has no owner yet and matches the template; a claim of that name still being deleted, or still owned by the VM's previous VMI, refuses the VMI, and KubeVirt retries. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name refuses the VMI, and the message names it. The webhook has the binder bind the claim under the same capacity and budget checks as any order, within the admission timeout, and attaches it. The node for template claims is picked with room for all of the VMI's claims, held and template: the node of the gathered set when the VMI has held claims, and otherwise one node with room for all its template claims. The VMI is refused if no node has room for all of them. A bind that does not fit or does not finish in time refuses the VMI, and KubeVirt retries. On a dry run the webhook creates, attaches and moves nothing, so it declares `sideEffects: NoneOnDryRun`. The VMI has no UID yet at that point, so the controller sets the VMI as the claim's owner once the VMI exists, and deletes a template claim whose VMI never appears after the same grace period as a stale attachment. The controller deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives and is billed only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any claim, but only while the VMI exists: an order that does not fit is refused when the VM or worker starts, not ahead of time. A template counts as a claim reference under `forbidden`. It lets an autoscaled pool pay for running workers only. -A `vm-instance` entry has exactly one of `name`, `claim` or `claimTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `claimSelector` or `claimTemplate`. A list holds only `name` entries or only references, and a pool has at most one `claimSelector` or one `claimTemplate`, never both, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `claim` and `claimTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to Kairos is separate work. +A `vm-instance` entry has exactly one of `name`, `claim` or `claimTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `claimSelector` or `claimTemplate`. A list holds only `name` entries or only references, and a pool has at most one `claimSelector` or one `claimTemplate`, never both, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `claim` and `claimTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to claims is separate work. -Kairos pools take GPUs only through `claimSelector` or `claimTemplate`, from classes of both modes. The chart refuses `name` entries on them. A pool states no mode of its own: its `claimSelector` or `claimTemplate` names the class, and the class's `mode` never changes, so the chart and the webhook take the mode from the class. The `kubernetes-nodes` chart, from Rollout step 2, fails to render a `claimSelector` or `claimTemplate` on a pool with `os: talos`, the field cozystack/cozystack#4911 adds, so the pool's HelmRelease fails before any machine is created. A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools, existing and new, keep working as today on `name` entries, and `GPUClaim` does not extend them. +A pool states no mode of its own: its `claimSelector` or `claimTemplate` names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class; a hoster does not reuse a deleted class's name for the other mode ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes claims of passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a `claimSelector` or `claimTemplate` that names a vGPU class on a Talos pool, before any machine is created. Worker pools on another OS, such as the Kairos pools of `kairos-vgpu-workers`, take vGPU classes as that proposal describes. A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools on `name` entries keep working as today. ### 4. Quotas @@ -246,7 +246,7 @@ vGPU needs a license, and passthrough needs one only for graphics. Compute (C-se ### 6. Host layer -This proposal requires vGPU hosts to run Ubuntu or Kairos with `bundles.system.variant: isp-full-generic`: the vGPU host driver and the patched plugin run there. Cozystack has no Kairos hosts today, so installing the bundle on Kairos and a vGPU Manager image for it are new work of this phase. `docs/gpu-vgpu.md` does not recommend Talos for vGPU, so this phase does not cover Talos hosts; they keep passthrough through the `default` variant as today. NVIDIA AI Enterprise lists vGPU for Compute on Linux with KVM as supported only through specific hypervisor vendors and, on Ubuntu and RHEL with KVM, for single-node hosts only ([support matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/support/support-matrix-8/8.2.html)), so support for such a host is a matter between the hoster and NVIDIA. +The host layer does not depend on the host OS: it needs the vGPU host driver, `vfio-pci` and the device plugin, wherever they run. Which OSes Cozystack supports as hosts of the management cluster is decided outside this proposal. Today vGPU hosts run Ubuntu with `bundles.system.variant: isp-full-generic`, where the vGPU host driver and the patched plugin run. `docs/gpu-vgpu.md` does not recommend Talos for vGPU, so this phase does not cover Talos hosts; they keep passthrough through the `default` variant as today. NVIDIA AI Enterprise lists vGPU for Compute on Linux with KVM as supported only through specific hypervisor vendors and, on Ubuntu and RHEL with KVM, for single-node hosts only ([support matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/support/support-matrix-8/8.2.html)), so support for such a host is a matter between the hoster and NVIDIA. The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ships as its own phase (Rollout, step 4), and it is replaceable. The API above it depends on four things only: each class's `resourceName` is advertised on the nodes its `nodeSelector` picks, one name never covers two different offers on one node, `GPUNode` reports devices and their health, and node labels mark each node's mode and profiles. `GPUClass`, `GPUClaim`, `GPUNode`, quotas, both webhooks, usage records and migration use no plugin-specific name, discovery format or configuration; the agent maps whatever device IDs a plugin reports to PCI addresses through sysfs. A different plugin, or a different version of the one chosen here, changes only this phase and the `resourceName` of affected classes, which can change while claims are held. Tenants keep their claims and `claim` entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to `GPUClaim` first. @@ -268,11 +268,11 @@ A layout change on a node, of mode or of profile, waits until the node has no GP Raw `name` entries keep working, and nothing converts them automatically, because a conversion would bind GPUs a tenant never ordered and put them in the usage records. The binder counts their usage, so it never books a unit a raw-name VM holds. The reverse is not protected: the scheduler does not see a claim that is bound but not attached, so while raw names are allowed, a raw-name VM can take that unit. A reservation is a hard guarantee only with `gpu.rawResourceNames: forbidden`. -A tenant moves a VM by stopping it, ordering claims of the matching class, replacing each `name` entry with a `claim` entry and starting it. On a full pool the claim can only be ordered after the stop; with spare units, ordering first puts both in the usage records until the restart. A Talos GPU pool has no in-place path. The tenant creates a Kairos pool with a `claimSelector`, which needs a hoster-built worker image for the class's mode, and then deletes the Talos pool. On a full cluster the Talos pool scales down first, so its GPUs are free to order. ComputePlane GPU groups have no such path until ComputePlane runs Kairos workers, so a cluster with them keeps raw names allowed. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes `claim` entries. A Talos pool on one takes the Kairos path above. +A tenant moves a VM by stopping it, ordering claims of the matching class, replacing each `name` entry with a `claim` entry and starting it. On a full pool the claim can only be ordered after the stop; with spare units, ordering first puts both in the usage records until the restart. A Talos GPU pool on a passthrough name moves by ordering claims of the matching passthrough class and replacing its `name` entries with a `claimSelector` or a `claimTemplate`, which rolls its workers; on a full cluster the pool scales down first, so its GPUs are free to order. A Talos pool on a raw vGPU name has no path here, since Talos workers carry no vGPU guest driver; it keeps its raw name until it moves to a worker OS that does. ComputePlane GPU groups have no path until the module accepts claim entries, so a cluster with them keeps raw names allowed. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes `claim` entries. A pool on one takes the pool path above. Quota keys move too. A raw GPU key is a quota key on a GPU resource name, such as `nvidia.com/AD102GL_L40S`. A claim-backed VM's pod still requests that name, so while a tenant migrates, its raw key covers both kinds of VM; after its last raw-name VM has moved, the key is replaced by `gpu.cozystack.io/*` keys. A tenant whose raw-name VMs use vGPU can order their claims only after it is entitled to vGPU (section 4), and within any class keys set above it. -When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. Preconditions: no Talos GPU pools in the cluster, ComputePlane GPU groups included (a hoster without a passthrough worker image keeps passthrough Talos pools and raw names), no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: +When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. Preconditions: no worker pools with raw-name `gpus` entries, ComputePlane GPU groups included, no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: - an admission policy refuses GPU entries (section 3) on every VMI created without a claim reference, on every path; - the same policy refuses every pod created in a tenant namespace that requests a GPU resource name, unless it is a virt-launcher pod created by KubeVirt's virt-controller service account with an owner reference to a VirtualMachineInstance. Tenants cannot forge such a pod, since no tenant role can create pods or VMIs, so app charts that copy free-form `resources` into pods cannot request GPUs either; @@ -291,8 +291,8 @@ The catalog is designed so later backends add `source` members and class fields, ## User-facing changes -- Tenants get `GPUClaim`, a GPU catalog with free capacity and remaining budget per class, `claim` entries in `vm-instance`, `claimSelector` entries for Kairos pools in `kubernetes-nodes`, and `claimTemplate` entries in both. Raw resource names keep working until a hoster forbids them. -- A held claim is billed while held, whether a VM runs on it or not. An autoscaled Kairos pool on `claimSelector` holds, and pays for, every claim its selector matches, which is at least `maxReplicas`; under `forbidden` nothing runs on GPUs without a claim. A tenant who wants to pay only while running uses `claimTemplate`, and gets the refusal at start instead of at order time. +- Tenants get `GPUClaim`, a GPU catalog with free capacity and remaining budget per class, `claim` entries in `vm-instance`, `claimSelector` entries for worker pools in `kubernetes-nodes`, and `claimTemplate` entries in both. Raw resource names keep working until a hoster forbids them. +- A held claim is billed while held, whether a VM runs on it or not. An autoscaled pool on `claimSelector` holds, and pays for, every claim its selector matches, which is at least `maxReplicas`; under `forbidden` nothing runs on GPUs without a claim. A tenant who wants to pay only while running uses `claimTemplate`, and gets the refusal at start instead of at order time. - Hosters get `GPUClass`, `GPUNode`, GPU budgets in `resourceQuotas` and the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout, `onCordon` on classes, the `gpu.resourceNames` list and `gpu.rawResourceNames`; an external billing system gets a usage record (`Workload`) per held claim. ## Upgrade and rollback compatibility @@ -301,7 +301,7 @@ The change is additive. Existing `gpus` entries render as before, and the existi Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, and the tenants' GPU limits, with four exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only limits were `gpu.cozystack.io/*` keys lose them (rules below), and tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through `GPUClaim`, so they lose their GPUs and run on the plain worker image. A child tenant that declares its own quotas no longer shares its parent's GPU limit, because a raw key is carved out of the parent's pool rather than capping the subtree. -The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `claim` and `claimTemplate` entries of VMs become raw `name` entries and those VMs restart. `claimSelector` and `claimTemplate` entries are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUClaim` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the GPU controller's `WorkloadMonitor` objects, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. +The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `claim` and `claimTemplate` entries of VMs and Talos pools become raw `name` entries, and those VMs restart and those pools roll. `claimSelector` and `claimTemplate` entries are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUClaim` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the GPU controller's `WorkloadMonitor` objects, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. Three rules hold throughout: @@ -333,8 +333,8 @@ Each case gives what is seen and what to do. - A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has (section 2). An `Always` or `Once` VM then starts on the new node once its claims have moved or been rebound; a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. - A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts it is refused there, and the claim moves or goes to `Lost`. - A layout change does not apply → the node shows `LayoutPending`; drain it. A VM with a raw `name` entry on the node holds the change until it stops. -- A Talos GPU pool, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a Kairos pool with `claimSelector`, or allow raw names again while the cluster has Talos GPU pools. -- A pool with `os: talos` and a `claimSelector` or `claimTemplate` fails to render → Talos pools take GPUs only through `name` entries; use a Kairos pool for claims. +- A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a `claimSelector` or a `claimTemplate`, or allow raw names again while the cluster has such pools. +- A Talos pool with a `claimSelector` or `claimTemplate` of a vGPU class is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. - A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. - A pool worker is refused → no matching claim is free, because of a surge machine during a rollout, claims still `Pending` or `Lost`, matching claims only on nodes that are not `Ready`, or another pool or a VM using the matching claims; wait, add claims, give each pool its own labels, or lower `maxReplicas`. @@ -342,7 +342,7 @@ Each case gives what is seen and what to do. - Unit tests for the binder and both webhooks: capacity with raw-name usage, budgets over the ancestor walk, the phase table, every move and refusal in sections 2 and 3, concurrent starts, and the pod pin after a LiveUpdate sync. - Webhook tests for cordons, templates and NICs: under `Move` a claim moves at VMI creation and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity the unscheduled VMI is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI creation, refused when no node fits, a `Stay` claim on a cordoned node pins the VMI when it holds all of the VM's claims and refuses it when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only template claims gets one node with room for all of them, and under `Stay` a single claim waits; a `claimTemplate` claim is created, refused when it does not fit, deleted with its VMI, deleted when its VMI is refused later in admission, reused on a repeated call, and on a dry run neither created nor attached nor moved; a VM with a GPU claim and `nvidia.com/hostdev` passes. -- Chart tests for `kubernetes-nodes`: a `claimSelector` or `claimTemplate` with `os: talos` fails to render. +- Admission tests for `KubernetesNodes`: a `claimSelector` or `claimTemplate` of a vGPU class on a Talos pool is refused, and one of a passthrough class is accepted. - Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held claims and refuses the next template claim, and `remainingBudget` is 0 without the entitlement. - Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. - Controller tests for the class metrics: capacity, bound units and the `Lost` count match the claims, and none of them appears in `GPUClass` status. @@ -351,14 +351,14 @@ Each case gives what is seen and what to do. - Controller tests for the usage record: present exactly while `Bound` or `Attached`, independent of tenant labels, not reported twice, and kept through reconciles of its `WorkloadMonitor` and of the `Workload` controller. - Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, `WorkloadMonitor`, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. - E2E with passthrough GPUs on the GPU runner: order, attach, stop, confirm another tenant cannot take the GPU, start again, and check the usage record. -- Manual on SR-IOV vGPU hosts: both modes of one model in one cluster, profiles across reboots, a layout change through a drain, and adoption from `default` and from `vgpu`, on Ubuntu and Kairos hosts. +- Manual on SR-IOV vGPU hosts: both modes of one model in one cluster, profiles across reboots, a layout change through a drain, and adoption from `default` and from `vgpu`, on Ubuntu hosts. ## Rollout 1. `GPUClass`, `GPUClaim`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, both webhooks, the policies, the option sources, `claim` and `claimTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRDs ship and finalizers and usage records still run. Before tenants order vGPU claims, the `gpu.vgpu` flag is set from the root tenant down. -2. `claimSelector` and `claimTemplate` in `kubernetes-nodes`, after cozystack/cozystack#4911. -3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, and `vgpu-vms` adds GPU guest images for VMs with `claim` entries. A cluster with Talos GPU pools stays on raw names and has no hard hold guarantee. -4. The host layer: the node layout, profile writes, node labels, the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, and Kairos vGPU hosts (the bundle on Kairos and a vGPU Manager image for it). +2. `claimSelector` and `claimTemplate` in `kubernetes-nodes`, for Talos pools with passthrough classes. +3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, and `vgpu-vms` adds GPU guest images for VMs with `claim` entries. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. +4. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. 5. Budgets through #48, only if its gate takes over subtree caps for GPU keys. 6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by ordering new claims. From af105e9027ea993b060a7021845bdc7625703988 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:18:39 +0300 Subject: [PATCH 09/32] docs(design-proposals): give GPUClass the tolerations binding reads A claim is bound when it is ordered, before any VM or pod exists, so the taint rule had no tolerations to check against, and a dedicated taint on GPU nodes took them out of binding and free even for VMs that tolerate it. The class now names the taints of its nodes that its VMs tolerate, the binder and free read them, and the launcher pod gets them with the pin. The new record supersedes the taint record and keeps its three skipped taints. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 14 +++++--- ...e-skips-exactly-three-kubernetes-taints.md | 2 +- ...y-to-a-node-that-cannot-serve-the-claim.md | 2 +- ...s-carries-the-tolerations-binding-reads.md | 33 +++++++++++++++++++ 4 files changed, 45 insertions(+), 6 deletions(-) create mode 100644 design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 9c762c02..c900af8e 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -23,6 +23,7 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Decisions +- [0010. `GPUClass` carries the tolerations that binding and `free` read](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md) - [0009. `GPUClass.mode` never changes, and a pool's `claimSelector` names its class](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md) - [0008. The GPU controller gets its node-label and plugin-restart rights with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) - [0007. Whoever can write a VM orders its template claims, while held claims stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md) @@ -92,6 +93,10 @@ spec: onCordon: Move # Move | Stay; can change nodeSelector: # nodes carrying this profile gpu.cozystack.io/10de-26b9: L40S-24C + tolerations: # taints of these nodes the class's VMs tolerate; can change + - key: dedicated + value: gpu + effect: NoSchedule source: devicePlugin: resourceName: nvidia.com/NVIDIA_L40S-24C @@ -137,7 +142,7 @@ status: A class maps an offer to a backend through a `source` union and to the nodes that serve it through `nodeSelector`. The first release has one source, `devicePlugin`; `dra` is reserved for section 8. Two classes may share a resource name when their node selectors do not overlap, which lets a host layer that cannot tell two offers apart by name tell them apart by node. A node that matches two classes with the same resource name gets a condition on both classes and on its `GPUNode`, which names the two classes, and is not used for binding. Class names are DNS labels, since they become part of quota keys. -A class referenced by held claims cannot disappear or change its meaning. A finalizer keeps a deleted class until its last `Bound`, `Attached` or `Lost` claim is gone, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `vendor`, `model`, `profile`, `memory`, `maxCount` and the kind of `source` cannot change while the finalizer is present. `mode` never changes, held claims or not, so a class name always stands for one mode, and whatever names a class, such as a worker pool's selector, knows the mode from the name ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). `resourceName` and `nodeSelector` can change, for changes in the host layer (section 6) and for moving the offer to other nodes. +A class referenced by held claims cannot disappear or change its meaning. A finalizer keeps a deleted class until its last `Bound`, `Attached` or `Lost` claim is gone, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `vendor`, `model`, `profile`, `memory`, `maxCount` and the kind of `source` cannot change while the finalizer is present. `mode` never changes, held claims or not, so a class name stands for one mode for the class's lifetime, and whatever names a class, such as a worker pool's selector, knows the mode from the name ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). `resourceName`, `nodeSelector` and `tolerations` can change, for changes in the host layer (section 6) and for moving the offer to other nodes. `tolerations` lists the taints of the class's nodes that its VMs tolerate, such as a taint that reserves GPU nodes for GPU workloads; a claim is bound before any VM exists, so the class is the only place the binder can read them from, and the virt-launcher pod of a VMI that uses the class gets them when it is created (section 3). A `GPUClaim` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate claims. How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 6), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPU claims of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. Admission refuses a claim above `maxCount`. The spec is immutable, so a change is a new order. @@ -154,7 +159,7 @@ A tenant sees a class, a count, a phase and, once there is one, a node, never wh ### 2. Binding and capacity -A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`. The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included ([0005](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md)). A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI creation (section 3): it may take an unattached claim off a node that is not `Ready`, because no VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. +A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included. A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI creation (section 3): it may take an unattached claim off a node that is not `Ready`, because no VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. Admission refuses an order that cannot be met at that moment, by capacity or by budget (section 4), with a message naming the class, the count and the limit it hit: a budget of the tenant or an ancestor, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. @@ -199,7 +204,7 @@ gpus: pool: inference ``` -The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named claims. For a pool worker it takes the claim the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the claim, so two VMIs can never hold one claim, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A claim still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template claims, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template claims included. This runs after the other refusal checks below and after the template claims pass their budget and entitlement checks, so every claim in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a claim of it, and each claim keeps its hold. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims still to place, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered, and the VMI is refused. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each claim and its node. A move that does not finish within the admission timeout refuses the VMI the same way; claims already moved keep their new node, and KubeVirt's retry continues from there. Template claims are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the claim's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. +The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named claims. For a pool worker it takes the claim the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the claim, so two VMIs can never hold one claim, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A claim still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template claims, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template claims included. This runs after the other refusal checks below and after the template claims pass their budget and entitlement checks, so every claim in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a claim of it, and each claim keeps its hold. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims still to place, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered, and the VMI is refused. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each claim and its node. A move that does not finish within the admission timeout refuses the VMI the same way; claims already moved keep their new node, and KubeVirt's retry continues from there. Template claims are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the claim's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. It also adds the `tolerations` of the classes of the VMI's claims to the pod, so the scheduler accepts the node the binder chose. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim: @@ -335,6 +340,7 @@ Each case gives what is seen and what to do. - A layout change does not apply → the node shows `LayoutPending`; drain it. A VM with a raw `name` entry on the node holds the change until it stops. - A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a `claimSelector` or a `claimTemplate`, or allow raw names again while the cluster has such pools. - A Talos pool with a `claimSelector` or `claimTemplate` of a vGPU class is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. +- A class shows `free: 0` while its nodes have GPUs to spare → the nodes carry a `NoSchedule` or `NoExecute` taint the class does not tolerate; add it to the class's `tolerations`, or remove the taint. - A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. - A pool worker is refused → no matching claim is free, because of a surge machine during a rollout, claims still `Pending` or `Lost`, matching claims only on nodes that are not `Ready`, or another pool or a VM using the matching claims; wait, add claims, give each pool its own labels, or lower `maxReplicas`. @@ -346,7 +352,7 @@ Each case gives what is seen and what to do. - Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held claims and refuses the next template claim, and `remainingBudget` is 0 without the entitlement. - Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. - Controller tests for the class metrics: capacity, bound units and the `Lost` count match the claims, and none of them appears in `GPUClass` status. -- Binder tests: a node with a `NoSchedule` taint that virt-launcher pods do not tolerate gets no binds, moves or gathers and its units leave `free`, while the Fence Agents Remediation taint alone does not make a node cordoned. +- Binder tests: a node with a `NoSchedule` taint that the class does not tolerate gets no binds, moves or gathers and its units leave `free`, a node whose taint the class tolerates is bound to and counts in `free`, a set of two classes skips a node only one of them tolerates, while the Fence Agents Remediation taint alone does not make a node cordoned. - Admission tests for the reserved label of the GPU usage monitor, refused on any object in a tenant namespace on create and on an update that adds it or changes its value, including a virt-launcher pod and a DataVolume that get it from a VM template, and allowed on an update that keeps or drops it. - Controller tests for the usage record: present exactly while `Bound` or `Attached`, independent of tenant labels, not reported twice, and kept through reconciles of its `WorkloadMonitor` and of the `Workload` controller. - Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, `WorkloadMonitor`, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. diff --git a/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md index f8c28592..49574ff3 100644 --- a/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md +++ b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md @@ -2,7 +2,7 @@ - **Number:** `0005` - **Date:** `2026-10-20` -- **Status:** Accepted +- **Status:** Superseded by [`0010`](./0010-gpuclass-carries-the-tolerations-binding-reads.md) - **Deciders:** `@lexfrei` - **Proposal:** [`../README.md`](../README.md) - **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) diff --git a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md index 303e0e19..7fd366fe 100644 --- a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md +++ b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md @@ -14,7 +14,7 @@ Section 3 pins a VMI to a cordoned node when a claim of a `Stay` class is there ## Decision -The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint virt-launcher pods do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (sections 2 and 3): a new VMI is refused when the node is gone, no longer serves the class, or has lost capacity the claim does not fit under, and otherwise its set is gathered onto one node with room for all of it; a VMI already waiting there unscheduled is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The binder at admission and the controller's background pass read the pin through one function, so admission never pins a VMI that the next pass deletes. +The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint its classes do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (sections 2 and 3): a new VMI is refused when the node is gone, no longer serves the class, or has lost capacity the claim does not fit under, and otherwise its set is gathered onto one node with room for all of it; a VMI already waiting there unscheduled is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The binder at admission and the controller's background pass read the pin through one function, so admission never pins a VMI that the next pass deletes. ## Why not the alternatives diff --git a/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md b/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md new file mode 100644 index 00000000..6a9baf1c --- /dev/null +++ b/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md @@ -0,0 +1,33 @@ +# 0010. `GPUClass` carries the tolerations that binding and `free` read + +- **Number:** `0010` +- **Date:** `2026-10-23` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) +- **Implemented in:** not yet + +## Context + +Section 2 kept binds, moves, rebinds and gathers off nodes with a `NoSchedule` or `NoExecute` taint "that virt-launcher pods do not tolerate", and [0005](./0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md) fixed which taints that rule skips. @kvaps pointed out in review that a claim is bound when it is ordered, before any VM or pod exists, so there are no tolerations to check a taint against. The implementation checked against none, so a dedicated taint on GPU nodes, the usual way to keep other workloads off them, took every such node out of binding and out of `free`, even for VMs that tolerate it. + +## Decision + +`GPUClass` has a `tolerations` field, in the shape of a pod's, which the hoster sets to the taints of the class's nodes that its VMs tolerate. A node counts for binding, moves, rebinds, gathers, capacity, `free` and `maxFreePerNode` of a class only if the class tolerates each of its `NoSchedule` and `NoExecute` taints, except the three that 0005 skips, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, for the reasons 0005 gives. A set of claims of several classes takes a node only when every class of the set tolerates its taints. The virt-launcher pod of a VMI gets the tolerations of its claims' classes when it is created. `tolerations` can change while claims are held, like `nodeSelector`. + +## Why not the alternatives + +- **Check taints against no tolerations, as the implementation did.** A dedicated taint takes the node out of the offer, which is the setup a hoster with dedicated GPU nodes uses. +- **Apply the taint rule only at gather time, against a concrete VMI**, the other option in the [review](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700). Binding and `free` would then ignore taints, so a claim could be bound onto a node no VM of the class can schedule to, and `free` would count units no VM can reach, until a VMI starts and the gather has to move the claim anyway. +- **Read the tolerations from the `vm-instance` chart's defaults.** They describe VMs, not the class, and a claim bound for a worker pool or a VM written directly would not follow them. + +## Consequences + +- A hoster with tainted GPU nodes lists the taint on every class of those nodes. Without it the class shows no free units there, and a failure case of the proposal says so. +- The class's tolerations reach the pod, so a tenant cannot get a GPU VM onto a tainted node without a class that tolerates the taint. +- A change of `tolerations` moves nothing by itself: as with any taint, a bound claim keeps its node, and only new binds, moves and gathers read the new list. + +## Revisit if + +Kubernetes lets a reservation be scheduled ahead of its pod, so tolerations can be checked against the pod that will use the GPU, for example through DRA (section 8). From 630315222723b6a8dd6c999e1be1e04134df4f15 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:19:30 +0300 Subject: [PATCH 10/32] docs(design-proposals): let a GPUClaim own its usage record The GPU usage records sat under a monitor whose selector had to match nothing, which took an admission policy on every write of every object in every tenant namespace, on every cluster. An ownerReference from the record to its claim ties the two directly: the Workload controller leaves such a record to its owner, garbage collection removes it with the claim, and the monitor, the reserved label and the policy go away. The record that kept the policy is dropped. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 13 ++++--- ...y-where-gpu-operator-serves-gpus-to-vms.md | 2 +- ...l-reservation-is-always-on-and-ratchets.md | 35 ------------------- .../0011-a-gpuclaim-owns-its-usage-record.md | 35 +++++++++++++++++++ 4 files changed, 42 insertions(+), 43 deletions(-) delete mode 100644 design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md create mode 100644 design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index c900af8e..d3216f0f 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -23,6 +23,7 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Decisions +- [0011. A `GPUClaim` owns its usage record, and no monitor or reserved label guards it](./decisions/0011-a-gpuclaim-owns-its-usage-record.md) - [0010. `GPUClass` carries the tolerations that binding and `free` read](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md) - [0009. `GPUClass.mode` never changes, and a pool's `claimSelector` names its class](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md) - [0008. The GPU controller gets its node-label and plugin-restart rights with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) @@ -30,7 +31,6 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra - [0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md) - [0005. The taint rule skips exactly the three taints that mirror a cordon or readiness](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md) - [0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts](./decisions/0004-gpunode-health-is-kubelet-allocatability.md) -- [0003. The usage-monitor label reservation is always on, and it ratchets](./decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md) - [0002. GPUClaim works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md) - [0001. A claim whose class resource name KubeVirt does not permit is refused at VMI admission](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md) @@ -241,7 +241,7 @@ A parent with `total: 4` caps everything below it at four GPUs, each vGPU counti ### 5. Usage accounting, catalog and licensing -Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads. Each `GPUClaim` gets one with `kind: gpu-claim` and resources `gpu.cozystack.io/: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 8), next to the records Cozystack already keeps for CPU, memory and storage. The GPU controller writes the record itself, under a `WorkloadMonitor` it owns in the tenant's namespace, whose selector matches no object, so tenant labels cannot drop a claim from the usage records. The selector is a reserved label key that admission refuses on every object in a tenant namespace, on create and on an update that adds it or changes its value, whoever makes the request and whether or not GPUs are on, so no new object can match it, not even a virt-launcher pod or a DataVolume that inherits the label from a VM template; an update that keeps or drops a label set before the policy existed passes ([0003](./decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md)). Each record carries the `workloads.cozystack.io/monitor` label naming that monitor: the `Workload` controller deletes a record only when that label is missing or names a monitor that is gone, and deletes nothing by selector. Like other `Workload` records it has no time fields: a held interval is the record's lifetime, which a billing system observes by watching and reconciles by sampling. A virt-launcher pod does not report GPUs its claims cover a second time. +Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads. Each `GPUClaim` gets one with `kind: gpu-claim` and resources `gpu.cozystack.io/: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 8), next to the records Cozystack already keeps for CPU, memory and storage. The GPU controller writes the record itself, with its `GPUClaim` as the record's owner through an ownerReference, and with no `WorkloadMonitor` and no label selector, so no tenant label can add a record or drop one ([0011](./decisions/0011-a-gpuclaim-owns-its-usage-record.md)). The `Workload` controller, which today deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, leaves a record owned by a `GPUClaim` to its owner: the GPU controller deletes it when the claim leaves `Bound` and `Attached`, and garbage collection deletes it with the claim. Like other `Workload` records it has no time fields: a held interval is the record's lifetime, which a billing system observes by watching and reconciles by sampling. A virt-launcher pod does not report GPUs its claims cover a second time. The catalog is the list of `GPUClass` objects with `displayName`, model, mode, profile, memory, `maxCount`, `status.free` and `status.maxFreePerNode`. Every authenticated user reads it, because a hoster publishes its offers and a tenant names a class when ordering, including from `kubectl` or GitOps without a portal. So the status holds only `free` and `maxFreePerNode`, and its conditions, such as `LostCapacity`, name no nodes and carry no counts. The class's capacity, bound units and `Lost` count show the hoster's inventory and load, and they are not in the status. The controller exports them as metrics that platform monitoring scrapes. Per node, a hoster reads device occupancy and health from `GPUNode`, and lists bound claims with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write. A tenant can still watch `free` and `maxFreePerNode` over time and see how busy a class is. The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest order, leaving out a field that is absent because it sets no limit. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. A `gpuclaim` source lists a tenant's own claims. The existing `gpu` source keeps serving raw names while they are allowed. @@ -306,7 +306,7 @@ The change is additive. Existing `gpus` entries render as before, and the existi Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, and the tenants' GPU limits, with four exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only limits were `gpu.cozystack.io/*` keys lose them (rules below), and tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through `GPUClaim`, so they lose their GPUs and run on the plain worker image. A child tenant that declares its own quotas no longer shares its parent's GPU limit, because a raw key is carved out of the parent's pool rather than capping the subtree. -The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `claim` and `claimTemplate` entries of VMs and Talos pools become raw `name` entries, and those VMs restart and those pools roll. `claimSelector` and `claimTemplate` entries are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUClaim` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the GPU controller's `WorkloadMonitor` objects, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. +The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `claim` and `claimTemplate` entries of VMs and Talos pools become raw `name` entries, and those VMs restart and those pools roll. `claimSelector` and `claimTemplate` entries are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUClaim` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. Three rules hold throughout: @@ -317,7 +317,7 @@ Three rules hold throughout: ## Security - Tenant access. Tenants with the `admin` role or above manage `GPUClaim` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. Tenants cannot read `GPUNode`. -- What a tenant controls. A tenant never supplies a resource name or a node through a claim reference: both come from `GPUClass` and from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. Admission refuses setting the reserved label of the GPU usage monitor on any object in a tenant namespace (section 5). A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes claims only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. +- What a tenant controls. A tenant never supplies a resource name or a node through a claim reference: both come from `GPUClass` and from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes claims only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. - Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. - Node agent. It reads sysfs and kubelet's pod-resources socket, writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can write only its own node's `GPUNode`. - Webhook. For `claimTemplate` entries it creates `GPUClaim` objects in tenant namespaces, only of the class and count the template names. It does not check who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template claims, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). @@ -353,9 +353,8 @@ Each case gives what is seen and what to do. - Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. - Controller tests for the class metrics: capacity, bound units and the `Lost` count match the claims, and none of them appears in `GPUClass` status. - Binder tests: a node with a `NoSchedule` taint that the class does not tolerate gets no binds, moves or gathers and its units leave `free`, a node whose taint the class tolerates is bound to and counts in `free`, a set of two classes skips a node only one of them tolerates, while the Fence Agents Remediation taint alone does not make a node cordoned. -- Admission tests for the reserved label of the GPU usage monitor, refused on any object in a tenant namespace on create and on an update that adds it or changes its value, including a virt-launcher pod and a DataVolume that get it from a VM template, and allowed on an update that keeps or drops it. -- Controller tests for the usage record: present exactly while `Bound` or `Attached`, independent of tenant labels, not reported twice, and kept through reconciles of its `WorkloadMonitor` and of the `Workload` controller. -- Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, `WorkloadMonitor`, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. +- Controller tests for the usage record: present exactly while `Bound` or `Attached`, owned by its `GPUClaim`, independent of tenant labels, not reported twice, kept through reconciles of the `Workload` controller although it has no monitor label, and gone with its claim. +- Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, the `Workload` controller, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. - E2E with passthrough GPUs on the GPU runner: order, attach, stop, confirm another tenant cannot take the GPU, start again, and check the usage record. - Manual on SR-IOV vGPU hosts: both modes of one model in one cluster, profiles across reboots, a layout change through a drain, and adoption from `default` and from `vgpu`, on Ubuntu hosts. diff --git a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md index d2ba606f..7a596473 100644 --- a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md +++ b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md @@ -16,7 +16,7 @@ The `vm-instance` chart renders only the reference, on the VMI template, and lea ## Decision -The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the binder, the VMI and virt-launcher webhooks, the controller's watches on KubeVirt objects, the GPU admission policies other than the label reservation of [0003](./0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md), the metrics scrape, and the tenant admin's write access to `GPUClaim`. Elsewhere: +The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the binder, the VMI and virt-launcher webhooks, the controller's watches on KubeVirt objects, the GPU admission policies, the metrics scrape, and the tenant admin's write access to `GPUClaim`. Elsewhere: - `vm-instance` refuses to render `claim` and `claimTemplate` entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today. - The platform refuses `gpu.rawResourceNames: forbidden`, because the policy that enforces it does not render there. diff --git a/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md b/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md deleted file mode 100644 index 9ab2cb76..00000000 --- a/design-proposals/gpudevice/decisions/0003-the-usage-monitor-label-reservation-is-always-on-and-ratchets.md +++ /dev/null @@ -1,35 +0,0 @@ -# 0003. The usage-monitor label reservation is always on, and it ratchets - -- **Number:** `0003` -- **Date:** `2026-10-20` -- **Status:** Accepted -- **Deciders:** `@lexfrei` -- **Proposal:** [`../README.md`](../README.md) -- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** [`cozystack/cozystack#5008`](https://github.com/cozystack/cozystack/pull/5008); refusing a changed value: not yet - -## Context - -Section 5 keeps the GPU usage records under a `WorkloadMonitor` whose selector is a reserved label key, `gpu.cozystack.io/usage-monitor`, and has admission refuse that key on every object in a tenant namespace, on create and update, so no object can match the monitor and no tenant can shape the GPU usage records. - -Two cases were left open. First, the GPU admission policies render only where GPUs serve VMs ([0002](./0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). Had the reservation followed them, a tenant could label its objects on a cluster without GPUs, and they would match the monitor the day the hoster turns GPUs on. Second, an object can already carry the key when the policy first renders, from before the upgrade that brought it. A policy that refuses every write of an object carrying the key would refuse every update of that object that keeps the label, including the one in which a controller removes its finalizer, so its deletion would hang until someone drops the label by hand. - -## Decision - -The reservation renders on every cluster where the `ValidatingAdmissionPolicy` API exists, whether or not GPUs are on, and it ratchets: it refuses a create that carries the key and an update that adds the key or changes its value, and it allows an update that keeps the value or drops the key. The rule lives in `packages/system/cozystack-basics/templates/gpu-usage-monitor-label-policy.yaml`, with its cases in `internal/gpu/policytest/usagemonitor_test.go`. - -## Why not the alternatives - -- **Render it with the other GPU policies.** That leaves the window above open, so the guarantee of section 5 would hold only for labels written after GPUs were enabled. -- **Refuse the key on every create and update, as section 5 has it.** It blocks every update of an object that predates the policy and carries the key, finalizer removal included, so controllers that know nothing about the label stall on it until a person removes the label. -- **Strip the key at admission instead of refusing it.** A mutating webhook on every object in every tenant namespace makes the controller's availability a dependency of every tenant write, for a label no legitimate writer sets. - -## Consequences - -- No new object in a tenant namespace can match the GPU usage monitor, on any cluster. -- An object that carried the key before the policy existed keeps it until someone removes it, and until then, if its value is `true`, the monitor's selector matches it. The policy cannot see such objects; the hoster finds them by listing objects with the label. -- The policy matches every create and update of every namespaced object in every tenant namespace, whether or not the cluster uses GPUs, so its expression reads only metadata. - -## Revisit if - -Cozystack gets a way to reserve label keys for its own controllers across tenant namespaces, so each reserved key no longer needs its own policy. diff --git a/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md b/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md new file mode 100644 index 00000000..f18377c3 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md @@ -0,0 +1,35 @@ +# 0011. A `GPUClaim` owns its usage record, and no monitor or reserved label guards it + +- **Number:** `0011` +- **Date:** `2026-10-23` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) +- **Implemented in:** not yet + +## Context + +Section 5 kept each claim's `Workload` usage record under a `WorkloadMonitor` that the GPU controller owned in the tenant's namespace. The `Workload` controller deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, so the record needed a monitor, and the monitor needed a selector. To keep tenant objects from matching that selector, the selector was a reserved label key, and an admission policy refused it on every create and update of every object in every tenant namespace, on every cluster, whether it used GPUs or not, with a ratchet for objects that carried the key before the policy existed. Record 0003 held that rule; it was withdrawn before the proposal merged, and this record replaces it. + +@kvaps asked in review why a policy on every tenant write protects a monitor whose selector must match nothing, when the record could have an explicit owner, and which object owns the GPU record, since monitor-made records are also removed by ownerReference garbage collection. + +## Decision + +Each GPU usage record carries an ownerReference to its `GPUClaim` and no monitor label. The GPU controller keeps it while the claim holds GPUs, as section 5 defines, and garbage collection removes it with the claim. The `Workload` controller leaves a record owned by a `GPUClaim` to its owner instead of deleting it for a missing monitor. There is no GPU `WorkloadMonitor`, no reserved label key and no admission policy for it. + +## Why not the alternatives + +- **Keep the monitor and reserve its label key, with a ratcheting policy**, as 0003 had it and the [review](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) questioned. The policy runs on every write in every tenant namespace on every cluster to protect one selector, and objects labelled before it existed still match the monitor until someone removes the label. It answers a problem the monitor created. +- **Keep the monitor with a selector nothing can match by construction, such as a label value tenants cannot write.** Kubernetes has no label value that a writer of an object cannot set, so this needs the same policy. +- **Write the record with no owner and no monitor label.** The `Workload` controller deletes such a record as soon as it sees it, and nothing would remove a record whose claim was deleted while the GPU controller was down. + +## Consequences + +- No admission policy matches tenant writes for usage records, and a cluster without GPUs carries nothing for them. +- The `Workload` controller gets a rule for one owner kind. Another kind of record that wants the same treatment needs its own case there. +- A billing system that lists records by monitor finds GPU records by `kind: gpu-claim` instead. + +## Revisit if + +The `Workload` controller moves to ownerReferences for every record, so the monitor label is no longer how it decides that a record is orphaned. From 805404a840eb112a61feb44c3eb0f13481b17749 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:22:15 +0300 Subject: [PATCH 11/32] docs(design-proposals): place GPU claims behind a launcher pod gate The VMI webhook moved claims through the binder, created and bound template claims and waited for the binder inside a fail-closed admission call, which the implementation could only do with a synchronous request field and a ten-second wait, and a slow move became a refused VMI under KubeVirt's backoff. The webhook now only checks and records. A pod webhook holds the virt-launcher pod at a scheduling gate, and the controller creates the template claims, gathers the set, pins the pod and releases the gate in one update. KubeVirt neither touches the gate nor times out a VMI whose pod waits, so a set that does not fit yet waits instead of being refused. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 51 ++++++++++++------- ...y-to-a-node-that-cannot-serve-the-claim.md | 2 +- ...n-write-a-vm-orders-its-template-claims.md | 2 +- ...s-host-layer-rights-with-the-host-layer.md | 1 + ...and-records-and-the-controller-prepares.md | 39 ++++++++++++++ 5 files changed, 74 insertions(+), 21 deletions(-) create mode 100644 design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index d3216f0f..ada72ff1 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -23,6 +23,7 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Decisions +- [0012. The VMI webhook only checks and records claims, and the controller places them while the launcher pod waits at a scheduling gate](./decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md) - [0011. A `GPUClaim` owns its usage record, and no monitor or reserved label guards it](./decisions/0011-a-gpuclaim-owns-its-usage-record.md) - [0010. `GPUClass` carries the tolerations that binding and `free` read](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md) - [0009. `GPUClass.mode` never changes, and a pool's `claimSelector` names its class](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md) @@ -159,7 +160,7 @@ A tenant sees a class, a count, a phase and, once there is one, a node, never wh ### 2. Binding and capacity -A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included. A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI creation (section 3): it may take an unattached claim off a node that is not `Ready`, because no VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. +A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included. A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI start (section 3): it may take a claim attached only to the starting VMI off a node that is not `Ready`, because no other VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. Admission refuses an order that cannot be met at that moment, by capacity or by budget (section 4), with a message naming the class, the count and the limit it hit: a budget of the tenant or an ancestor, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. @@ -176,16 +177,16 @@ A `dra` claim before its first start is the exception: it is `Bound` without a n A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 deletes a final VMI before it creates the VM's next one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)), so a new VMI never finds its claim attached to the old one. -A claim keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted, the controller also deletes the claim's unscheduled VMI, so the claim becomes unattached and moves. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules of section 3. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI creation, when one is needed, refuses such a set (section 3). +A claim keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer matches the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules of section 3. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set (section 3). A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity when the agent reports a device missing or unhealthy for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no claim. Then: - the node and the affected classes get a `LostCapacity` condition; - unattached claims that no longer fit move to another node of their class, or to `Lost` when none has room; -- on a `NotReady` node, attached claims go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI creation is moving off the node is the exception: it finishes the move, or loses its attachment and follows these rules when the gather fails; -- a VM started on such a node after the condition is set is refused (section 3), and the controller deletes the unscheduled VMI of a claim that no longer fits, so its claim becomes unattached and can move. What the VM does next follows its `runStrategy` for a deleted VMI: `Always` and `Once` create a new VMI and start once the claim has moved or been rebound from `Lost`, while `Manual` and `RerunOnFailure` stay stopped until the tenant starts the VM (Failure cases); unlike a cordon, the node cannot serve the claim, so waiting is no option. A VM restarted earlier, on a node that is not `Ready`, is gathered at creation instead (section 3). +- on a `NotReady` node, attached claims go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI start is moving off the node is the exception: it finishes the move; +- a VM started after the condition is set has its claims gathered off the node before its pod is released (section 3), and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. What the VM does next follows its `runStrategy` for a deleted VMI: `Always` and `Once` create a new VMI and start once the claim has moved or been rebound from `Lost`, while `Manual` and `RerunOnFailure` stay stopped until the tenant starts the VM (Failure cases); unlike a cordon, the node cannot serve the claim, so waiting is no option. -Claims in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a claim whose VMI is still live waits until that VMI is final or removed (above). The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` claim already counts. +Claims in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a claim whose VMI is still live on the lost node waits until that VMI is final or removed (above), while a claim of a VMI whose pod is still gated is rebound for that VMI. The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` claim already counts. ### 3. Attaching @@ -204,13 +205,11 @@ gpus: pool: inference ``` -The chart renders only the reference, on the VMI template. A mutating webhook on VirtualMachineInstance creation resolves it. For a VM it takes the named claims. For a pool worker it takes the claim the worker held before, unless a VM now names it, its node is not `Ready`, or its node has such a taint (section 2). Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment on the claim, so two VMIs can never hold one claim, and fills in the class's resource name. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. A claim still on a cordoned node under `Move` is first moved by the binder when another node has room, and the pod is pinned to the new node. A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way: for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are on different nodes, on a node that is not `Ready`, on a node with such a taint (section 2), or on a node with no room for the VMI's template claims, the webhook has the binder move them onto one `Ready` node that is not cordoned, has no pending layout change and has room for all of them, template claims included. This runs after the other refusal checks below and after the template claims pass their budget and entitlement checks, so every claim in the set is unattached or already attached to this VMI. The webhook records the attachment of the whole set before the move, so no other VMI or background move can take a claim of it, and each claim keeps its hold. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims still to place, there is no gather, even if the node is also not `Ready`: the VMI is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered, and the VMI is refused. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). With no node that fits, including when no node serves all the set's classes, the VMI is refused, the attachments are cleared, and the message names each claim and its node. A move that does not finish within the admission timeout refuses the VMI the same way; claims already moved keep their new node, and KubeVirt's retry continues from there. Template claims are created after this step, on the node of the set. A second webhook on virt-launcher pod creation pins the pod to the claim's node with a required node affinity on the node's name, as a `matchFields` requirement on `metadata.name` added to every node selector term, or as the only term when the pod has none, not with `nodeName`, which would let kubelet start the pod on a cordoned node. It also adds the `tolerations` of the classes of the VMI's claims to the pod, so the scheduler accepts the node the binder chose. The pin goes on the pod because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. +The chart renders only the reference, on the VMI template, with the reference label `gpu.cozystack.io/claim-reference`, which KubeVirt copies from the VMI to its pods. A mutating webhook on VirtualMachineInstance creation resolves the references and checks what needs no placement; it binds, moves and creates no claim, and waits for nothing. For a VM it takes the named claims. For a pool worker it takes the claim the worker held before, unless a VM now names it, its node is not `Ready`, or its node has a taint the class does not tolerate (section 2). Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook runs the refusal checks below and records the attachment of each held claim to this VMI, so two VMIs can never hold one claim and no background move can take one. It writes one `gpus` entry per GPU unit, held or from a template, with the class's resource name. These entries have to be complete at admission: KubeVirt renders the launcher pod's resource requests from the VMI, and nothing but KubeVirt itself can change a VMI's spec afterwards, so the number of template claims is fixed when the VMI is admitted. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. The GPU controller places the claims after admission, while the launcher pod waits behind a scheduling gate ([Placing a VMI's claims](#placing-a-vmis-claims)). The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim: -- is `Pending`, `Lost`, being deleted, or attached to another VMI; -- is bound to a node that is gone, no longer matches the class, has `LostCapacity` the claim does not fit under, or has a pending layout change (section 6); -- cannot be brought, with the VMI's other claims and template claims, onto one node with room for all of them (above), for example because a claim of a `Stay` class is on a cordoned node while other claims need gathering; +- does not exist, is being deleted, or is attached to another VMI; - is a vGPU claim next to vGPU claims of another class, or brings the VMI's GPUs of its class above `maxCount`; - is of a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). @@ -218,12 +217,24 @@ A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and usage record, under the `gpu.cozystack.io/claim-protection` finalizer until no VMI uses it, and is not attached again. -A `claimTemplate` entry names a class and a count instead of a held claim. At VMI creation the webhook creates a `GPUClaim` from it, named after the VMI and the entry's index and marked as a template claim by the `gpu.cozystack.io/template-claim` label. A repeated call or a recreated VMI finds the same claim and reuses it when it carries that mark, has no owner yet and matches the template; a claim of that name still being deleted, or still owned by the VM's previous VMI, refuses the VMI, and KubeVirt retries. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name refuses the VMI, and the message names it. The webhook has the binder bind the claim under the same capacity and budget checks as any order, within the admission timeout, and attaches it. The node for template claims is picked with room for all of the VMI's claims, held and template: the node of the gathered set when the VMI has held claims, and otherwise one node with room for all its template claims. The VMI is refused if no node has room for all of them. A bind that does not fit or does not finish in time refuses the VMI, and KubeVirt retries. On a dry run the webhook creates, attaches and moves nothing, so it declares `sideEffects: NoneOnDryRun`. The VMI has no UID yet at that point, so the controller sets the VMI as the claim's owner once the VMI exists, and deletes a template claim whose VMI never appears after the same grace period as a stale attachment. The controller deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives and is billed only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any claim, but only while the VMI exists: an order that does not fit is refused when the VM or worker starts, not ahead of time. A template counts as a claim reference under `forbidden`. It lets an autoscaled pool pay for running workers only. +A `claimTemplate` entry names a class and a count instead of a held claim. At VMI creation the webhook checks it against the class, `maxCount` and the vGPU entitlement, and refuses the VMI when they do not allow it. Once the launcher pod waits at its gate, the controller creates a `GPUClaim` from the entry, named after the VMI and the entry's index, marked as a template claim by the `gpu.cozystack.io/template-claim` label, and owned by the VMI. A claim of that name that carries the mark and is owned by this VMI is reused, so a repeated pass creates nothing twice; one still being deleted, or still owned by the VM's previous VMI, is waited for. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name is never taken, and the pod stays gated with a condition naming it. The controller binds the template claims under the same capacity and budget checks as any order, on the node of the gathered set when the VMI has held claims, and otherwise on one node with room for all of them, and attaches them. On a dry run the webhook records nothing, so it declares `sideEffects: NoneOnDryRun`. The controller deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives, and is in the usage records, only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any claim, but only while the VMI exists: an order that does not fit keeps the VM or worker waiting at start, not refused ahead of time. A template counts as a claim reference under `forbidden`. It lets an autoscaled pool pay for running workers only. A `vm-instance` entry has exactly one of `name`, `claim` or `claimTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `claimSelector` or `claimTemplate`. A list holds only `name` entries or only references, and a pool has at most one `claimSelector` or one `claimTemplate`, never both, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `claim` and `claimTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to claims is separate work. A pool states no mode of its own: its `claimSelector` or `claimTemplate` names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class; a hoster does not reuse a deleted class's name for the other mode ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes claims of passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a `claimSelector` or `claimTemplate` that names a vGPU class on a Talos pool, before any machine is created. Worker pools on another OS, such as the Kairos pools of `kairos-vgpu-workers`, take vGPU classes as that proposal describes. A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools on `name` entries keep working as today. +#### Placing a VMI's claims + +A launcher pod waits for its GPUs behind a scheduling gate, so nothing is refused for not being ready yet. A mutating webhook on the creation of a virt-launcher pod adds the gate `gpu.cozystack.io/claims` and the `tolerations` of the classes of the VMI's claims. It matches only pods labelled `kubevirt.io: virt-launcher` that carry the reference label, adds the gate only when it is not there yet, since a repeated call would otherwise fail on a duplicate gate, and fails closed: a pod that reached the scheduler ungated would run unpinned. It also gates the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on the node of the claims. Migration target pods are left alone: KubeVirt deletes a target pod that stays pending for 15 minutes, a VM with a passthrough GPU or a VF does not live-migrate, and Cozystack does not enable the `VGPULiveMigration` gate that mdev vGPUs would need. A set that already has a `WaitForFirstConsumer` volume bound on a node stays on that node. KubeVirt never touches scheduling gates and has no timeout for a VMI whose pod waits, so the VMI stays in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated` ([`lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go)), and the VM's own start backoff never begins. + +For a gated pod the controller creates the VMI's template claims under the usual budget and capacity checks, then brings the set onto one node. Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are `Pending` or `Lost`, on different nodes, on a node that is gone, no longer matches a class, has lost capacity or has a pending layout change, on a node that is not `Ready`, on a node with a taint one of their classes does not tolerate (section 2), on a cordoned node under `Move`, or on a node with no room for the VMI's template claims, the controller binds or moves them onto one `Ready` node that is not cordoned, has no pending layout change, is tolerated by every class of the set and has room for all of them, template claims included. Every held claim of the set is attached to this VMI by then, so no other VMI or background move can take one, and each claim keeps its hold while it moves. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims, there is no gather, even if the node is also not `Ready`: the pod is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). + +Once the set is on one node, the controller pins the pod and removes the gate in one update. The pin is a `matchFields` requirement on `metadata.name` appended to every required node selector term of the pod, or the single term when it has none. Kubernetes lets a gated pod only narrow its scheduling, so a term cannot be added next to existing ones, and KubeVirt usually sets a required term already, for host-model CPUs. The pin is not `nodeName`, which would let kubelet start the pod on a cordoned node, and it goes on the pod rather than the VMI because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. + +When the set cannot be placed, because no node has room, a budget is exhausted, or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, and the claims and the VMI carry a condition naming the reason. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules (below and section 2). After the deletion the VMI becomes `Failed`, and the VM's `runStrategy` decides what follows: `Always` and `RerunOnFailure` start again after KubeVirt's start-failure backoff, with the VM shown as `CrashLoopBackOff`; a `Once` VM keeps its failed VMI, so it stays down and also shows `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it (Failure cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. + +A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way while the pod is gated. Once the pod is pinned, under `Move`, for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). + ### 4. Quotas A tenant's GPU budget is declared in `resourceQuotas`, per class as `gpu.cozystack.io/: ` and in total as `gpu.cozystack.io/total: `; the class name `total` is reserved. These keys are not rendered into the namespace `ResourceQuota`, since `GPUClaim` is not a pod, and a tenant with only GPU keys gets no `ResourceQuota` or `LimitRange`. That is a change to the tenant chart, which today renders both for any `resourceQuotas` key. @@ -297,7 +308,7 @@ The catalog is designed so later backends add `source` members and class fields, ## User-facing changes - Tenants get `GPUClaim`, a GPU catalog with free capacity and remaining budget per class, `claim` entries in `vm-instance`, `claimSelector` entries for worker pools in `kubernetes-nodes`, and `claimTemplate` entries in both. Raw resource names keep working until a hoster forbids them. -- A held claim is billed while held, whether a VM runs on it or not. An autoscaled pool on `claimSelector` holds, and pays for, every claim its selector matches, which is at least `maxReplicas`; under `forbidden` nothing runs on GPUs without a claim. A tenant who wants to pay only while running uses `claimTemplate`, and gets the refusal at start instead of at order time. +- A held claim is billed while held, whether a VM runs on it or not. An autoscaled pool on `claimSelector` holds, and pays for, every claim its selector matches, which is at least `maxReplicas`; under `forbidden` nothing runs on GPUs without a claim. A tenant who wants to pay only while running uses `claimTemplate`, and finds out at start, not at order time, whether the claim fits. - Hosters get `GPUClass`, `GPUNode`, GPU budgets in `resourceQuotas` and the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout, `onCordon` on classes, the `gpu.resourceNames` list and `gpu.rawResourceNames`; an external billing system gets a usage record (`Workload`) per held claim. ## Upgrade and rollback compatibility @@ -320,8 +331,8 @@ Three rules hold throughout: - What a tenant controls. A tenant never supplies a resource name or a node through a claim reference: both come from `GPUClass` and from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes claims only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. - Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. - Node agent. It reads sysfs and kubelet's pod-resources socket, writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can write only its own node's `GPUNode`. -- Webhook. For `claimTemplate` entries it creates `GPUClaim` objects in tenant namespaces, only of the class and count the template names. It does not check who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template claims, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). -- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. It also writes one ConfigMap, the GPU settings in `cozy-system`. +- Template claims. For `claimTemplate` entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the template names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template claims, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). +- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Step 1 also gives it `update` and `delete` on pods in tenant namespaces. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod there, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses (section 3). It also writes one ConfigMap, the GPU settings in `cozy-system`. - Visibility. A tenant sees the node name of its own claims and, through the remaining budget, its siblings' GPU count under a shared parent budget (section 5). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. ## Failure and edge cases @@ -330,13 +341,15 @@ Each case gives what is seen and what to do. - A tenant's order is refused at create → the message names the class, the count and the limit, or, for a vGPU class, that the tenant is not entitled to vGPU; order a smaller count, another class, or ask for more budget. - A claim stays `Pending` after an order → its condition names the shortage; it binds when capacity or budget frees up, or the tenant deletes it. -- A VM with claims does not start → the webhook message names each claim and its node and the reason. When the reason is that the VM's claims, template claims included, cannot be brought onto one node, either no node can take all of them or a claim of a `Stay` class is on a cordoned node while others need gathering. Then uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. A refusal for the admission timeout usually clears on KubeVirt's retry. +- A VM with claims is refused at start → the webhook refuses the VMI for a reference problem, such as a claim that does not exist, is being deleted or is attached to another VMI, and the VirtualMachine shows the message in a `FailedCreate` event, naming each claim and the reason; fix the reference. +- A VM with claims stays `Starting` → its VMI is in `Scheduling` with its pod gated, and the claims and the VMI carry a condition naming the reason: no node can take the whole set, template claims included, a budget is exhausted, or a claim of a `Stay` class is on a cordoned node while others need gathering. The VM starts by itself once the set can be placed. To get there sooner, uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. +- The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. An `Always` or `RerunOnFailure` VM shows `CrashLoopBackOff` and starts again after a delay that grows with each failure up to five minutes, and the count resets once a VMI runs; a `Once` VM keeps its failed VMI and stays down, also shown as `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it. The binder avoids that node for the set for ten minutes or until the node changes (section 3). Free CPU or memory on GPU nodes if this repeats. - A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. -- A cordoned node is deleted while a drained VM's VMI still waits for it → the controller deletes that unscheduled VMI, the claim moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. -- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost` (section 2); once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached claims onto a `Ready` node with room (section 3) and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node (all in section 3). After a `Node` deletion the VM starts on a retry instead, once the controller has moved the claims. Only when no node fits is the VMI refused, and then it is not created at all. KubeVirt v1.9.0 retries the creation with backoff for `Always`, which deletes the failed VMI and starts a new one, and for `RerunOnFailure`, whose failed VMI leaves a start request that stays until a VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)); such a VM starts once a node has room for its set. The retry goes through the VM controller's rate-limited queue, whose per-item backoff grows to 1000 seconds, so the start can come up to about 17 minutes after room appears. `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. -- A node loses capacity while a VM's VMI waits for it unscheduled → the controller deletes that VMI so its claims can move; an `Always` or `Once` VM starts again once its claims have moved or been rebound, a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. +- A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. +- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost` (section 2); once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached claims onto a `Ready` node with room (section 3) and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node (all in section 3). After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set (section 3). `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. +- A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; an `Always` or `Once` VM starts again once its claims have moved or been rebound, a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. - A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has (section 2). An `Always` or `Once` VM then starts on the new node once its claims have moved or been rebound; a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. -- A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts it is refused there, and the claim moves or goes to `Lost`. +- A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts, the controller moves the claim off that node before it releases the pod, or the claim goes to `Lost` and the pod stays gated until it is rebound. - A layout change does not apply → the node shows `LayoutPending`; drain it. A VM with a raw `name` entry on the node holds the change until it stops. - A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a `claimSelector` or a `claimTemplate`, or allow raw names again while the cluster has such pools. - A Talos pool with a `claimSelector` or `claimTemplate` of a vGPU class is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. @@ -347,7 +360,7 @@ Each case gives what is seen and what to do. ## Testing - Unit tests for the binder and both webhooks: capacity with raw-name usage, budgets over the ancestor walk, the phase table, every move and refusal in sections 2 and 3, concurrent starts, and the pod pin after a LiveUpdate sync. -- Webhook tests for cordons, templates and NICs: under `Move` a claim moves at VMI creation and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity the unscheduled VMI is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI creation, refused when no node fits, a `Stay` claim on a cordoned node pins the VMI when it holds all of the VM's claims and refuses it when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only template claims gets one node with room for all of them, and under `Stay` a single claim waits; a `claimTemplate` claim is created, refused when it does not fit, deleted with its VMI, deleted when its VMI is refused later in admission, reused on a repeated call, and on a dry run neither created nor attached nor moved; a VM with a GPU claim and `nvidia.com/hostdev` passes. +- Webhook and controller tests for cordons, templates and NICs: the webhook binds, moves and creates no claim, under `Move` a claim moves at VMI start and the pod gets a required node affinity and the classes' tolerations, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity the unscheduled VMI is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only template claims gets one node with room for all of them, and under `Stay` a single claim waits; a `claimTemplate` entry is refused at admission when the budget or the entitlement does not allow it, its claim is created by the controller once the VMI exists and owned by it, deleted with its VMI, reused on a repeated pass, and a dry run records nothing; a VM with a GPU claim and `nvidia.com/hostdev` passes. - Admission tests for `KubernetesNodes`: a `claimSelector` or `claimTemplate` of a vGPU class on a Talos pool is refused, and one of a passthrough class is accepted. - Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held claims and refuses the next template claim, and `remainingBudget` is 0 without the entitlement. - Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. diff --git a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md index 7fd366fe..5ef332c0 100644 --- a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md +++ b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md @@ -14,7 +14,7 @@ Section 3 pins a VMI to a cordoned node when a claim of a `Stay` class is there ## Decision -The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint its classes do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (sections 2 and 3): a new VMI is refused when the node is gone, no longer serves the class, or has lost capacity the claim does not fit under, and otherwise its set is gathered onto one node with room for all of it; a VMI already waiting there unscheduled is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The binder at admission and the controller's background pass read the pin through one function, so admission never pins a VMI that the next pass deletes. +The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint its classes do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (sections 2 and 3): a new VMI's set is gathered onto one node with room for all of it while its launcher pod waits at the gate, or the pod stays gated when no node fits; a VMI whose pod is already pinned there is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The gather at VMI start and the controller's background pass read the pin through one function, so a VMI is never pinned where the next pass deletes it. ## Why not the alternatives diff --git a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md index 39f58d25..06b2c22b 100644 --- a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md +++ b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md @@ -16,7 +16,7 @@ In Cozystack more identities can write a `vm-instance` than can write a `GPUClai ## Decision -A `claimTemplate` entry is part of the VM, so whoever can write the VM can order its template claims, and whoever can start it makes them exist and enter the usage records for as long as the VMI runs. The webhook applies the tenant's budgets, the vGPU entitlement and capacity to every template claim, as to any order, and does not check the identity behind the VM. Ordering a held `GPUClaim` directly stays with tenant admins and above. +A `claimTemplate` entry is part of the VM, so whoever can write the VM can order its template claims, and whoever can start it makes them exist and enter the usage records for as long as the VMI runs. The webhook checks the vGPU entitlement for every template claim at VMI admission, the controller checks budgets and capacity when it creates and binds the claim, as for any order, and neither checks the identity behind the VM. Ordering a held `GPUClaim` directly stays with tenant admins and above. ## Why not the alternatives diff --git a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md index 5a77174a..0837667b 100644 --- a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md +++ b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md @@ -23,6 +23,7 @@ Of the three, Rollout step 1 grants the controller only the VMI deletion. The no ## Consequences +- The `update` and `delete` rights on tenant pods that the launcher pod gate of section 3 needs are a separate grant of step 1, not one of the three. - Between step 1 and step 4 the controller cannot change any node, so the GPU node labels that classes match are the ones the hoster writes. - The rights and their policy are reviewed in step 4, next to the code that uses them. diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md new file mode 100644 index 00000000..243c5d50 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -0,0 +1,39 @@ +# 0012. The VMI webhook only checks and records claims, and the controller places them while the launcher pod waits at a scheduling gate + +- **Number:** `0012` +- **Date:** `2026-10-23` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) +- **Implemented in:** not yet + +## Context + +Section 3 had the mutating webhook on VirtualMachineInstance creation do all of the start's GPU work inside the admission call, failing closed: record the attachments, have the binder move the VMI's claims onto one node, and create and bind its template claims, all within the admission timeout, refusing the VMI when a move or a bind did not finish in time. The implementation in [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) and [`#5030`](https://github.com/cozystack/cozystack/pull/5030) needed a synchronous handshake for that, a `status.bindRequest` on the claim that the webhook wrote and then waited up to ten seconds for the binder to answer, which the proposal did not describe. A refused VMI went back to KubeVirt's VM controller, whose per-item backoff grows to 1000 seconds, so a VM could start many minutes after room appeared. + +@kvaps asked in review of [#99](https://github.com/cozystack/community/pull/99#pullrequestreview-5587787354) for the webhook to check, record and pin, and for the controller to do the rest, and turned that into a request in the [next review](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700). + +## Decision + +The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the held claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. A second mutating webhook adds the scheduling gate `gpu.cozystack.io/claims` and the classes' tolerations to the virt-launcher pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, with a condition on the claims and the VMI. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. `status.bindRequest` is dropped. + +## Why not the alternatives + +- **Keep the work in the webhook.** A fail-closed admission call that writes several objects and waits on another controller is slow under load, needs its own request protocol between the two, and turns every slow move into a refused VMI and a KubeVirt backoff. +- **Refuse the VMI whenever a gather is needed and let the controller move the claims before KubeVirt's retry.** It drops the handshake but keeps the backoff, so the start still comes up to the backoff limit after the claims are ready, and the tenant sees a refusal for a VM that will start. +- **Hold the VMI instead of the pod**, through a KubeVirt start strategy or a VM-level hold. KubeVirt v1.9.0 offers none that keeps the pod from being scheduled: `startStrategy: Paused` still schedules the pod and only pauses the guest. A scheduling gate is the hook Kubernetes provides for this, and KubeVirt neither touches it nor times it out. +- **Write the pin into the VMI's node affinity.** Under `vmRolloutStrategy: LiveUpdate` KubeVirt rewrites a VMI's affinity from the VM template, and it never changes a launcher pod's affinity, so the pod is the place that keeps the pin. + +## Consequences + +- The webhook is a pure check plus the attachment record, so its latency no longer depends on the binder, and a dry run only skips the record. +- Template claims are created after the VMI exists, so the controller sets their owner at creation, and a template claim no longer needs a grace period for a VMI that never appears. +- A set that cannot be placed is no longer an admission error. The VM shows `Starting`, the VMI waits in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated`, and the reason is a condition on the claims and the VMI. The VM starts as soon as the set fits, with no KubeVirt backoff in between. +- The pod webhook is a fail-closed dependency of every start of a VM with claim references, like the VMI webhook. +- The cluster autoscaler does not scale up for a gated pod, and after the pin a new node cannot help that pod, so GPU node scale-up stays with the hoster. +- A pin cannot be undone, so a node that refuses a pinned pod costs one failed VMI and KubeVirt's start-failure backoff, which the rejected-node rule keeps from repeating. + +## Revisit if + +KubeVirt offers a hook between VMI creation and launcher pod scheduling that a controller can hold, so the pod no longer needs to be held on the Kubernetes side. From d4f7efcdb86665a4ac8d830340517a5cb0f23935 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:22:39 +0300 Subject: [PATCH 12/32] docs(design-proposals): describe the order webhook and the tamper trust Security named two fail-closed webhooks and said an outage blocks only VM starts, while the validating webhook on GPUClaim creation also fails closed and blocks new orders. Template claims skip that webhook and rely on the tamper policy, which lets the controller manager, system:masters and every service account in cozy-system and kube-system through unchecked, so that boundary is stated with what it allows. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index ada72ff1..694ca148 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -213,7 +213,7 @@ The webhook refuses to start a VMI, and names each claim and its node, when a re - is a vGPU claim next to vGPU claims of another class, or brings the VMI's GPUs of its class above `maxCount`; - is of a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). -A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are the resource names of `GPUClass` objects and the names in the hoster's list `gpu.resourceNames`, which defaults to the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. The list also takes every resource name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin, as the agent reports it, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU claim for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its claims unattached, so they move by the rules of section 2. Both webhooks see only VMIs and pods carrying the chart's reference label, `gpu.cozystack.io/claim-reference`, next to the `gpu.cozystack.io/claims` annotation that lists the references, and fail closed, so an outage blocks the start of VMs with claim references and nothing else. +A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are the resource names of `GPUClass` objects and the names in the hoster's list `gpu.resourceNames`, which defaults to the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. The list also takes every resource name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin, as the agent reports it, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU claim for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its claims unattached, so they move by the rules of section 2. Both webhooks see only VMIs and pods carrying the chart's reference label, `gpu.cozystack.io/claim-reference`, next to the `gpu.cozystack.io/claims` annotation that lists the references, and fail closed, so their outage blocks the start of VMs with claim references and no other VM or pod; the third GPU webhook, on orders, is described under Security. A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and usage record, under the `gpu.cozystack.io/claim-protection` finalizer until no VMI uses it, and is not attached again. @@ -264,7 +264,7 @@ vGPU needs a license, and passthrough needs one only for graphics. Compute (C-se The host layer does not depend on the host OS: it needs the vGPU host driver, `vfio-pci` and the device plugin, wherever they run. Which OSes Cozystack supports as hosts of the management cluster is decided outside this proposal. Today vGPU hosts run Ubuntu with `bundles.system.variant: isp-full-generic`, where the vGPU host driver and the patched plugin run. `docs/gpu-vgpu.md` does not recommend Talos for vGPU, so this phase does not cover Talos hosts; they keep passthrough through the `default` variant as today. NVIDIA AI Enterprise lists vGPU for Compute on Linux with KVM as supported only through specific hypervisor vendors and, on Ubuntu and RHEL with KVM, for single-node hosts only ([support matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/support/support-matrix-8/8.2.html)), so support for such a host is a matter between the hoster and NVIDIA. -The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ships as its own phase (Rollout, step 4), and it is replaceable. The API above it depends on four things only: each class's `resourceName` is advertised on the nodes its `nodeSelector` picks, one name never covers two different offers on one node, `GPUNode` reports devices and their health, and node labels mark each node's mode and profiles. `GPUClass`, `GPUClaim`, `GPUNode`, quotas, both webhooks, usage records and migration use no plugin-specific name, discovery format or configuration; the agent maps whatever device IDs a plugin reports to PCI addresses through sysfs. A different plugin, or a different version of the one chosen here, changes only this phase and the `resourceName` of affected classes, which can change while claims are held. Tenants keep their claims and `claim` entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to `GPUClaim` first. +The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ships as its own phase (Rollout, step 4), and it is replaceable. The API above it depends on four things only: each class's `resourceName` is advertised on the nodes its `nodeSelector` picks, one name never covers two different offers on one node, `GPUNode` reports devices and their health, and node labels mark each node's mode and profiles. `GPUClass`, `GPUClaim`, `GPUNode`, quotas, the three GPU webhooks, usage records and migration use no plugin-specific name, discovery format or configuration; the agent maps whatever device IDs a plugin reports to PCI addresses through sysfs. A different plugin, or a different version of the one chosen here, changes only this phase and the `resourceName` of affected classes, which can change while claims are held. Tenants keep their claims and `claim` entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to `GPUClaim` first. Until the phase ships, a cluster keeps the modes its variant gives it today, one per model for models whose whole GPU and VFs share a PCI ID, and vGPU classes keep the experimental status `docs/gpu-vgpu.md` gives the `vgpu` variant: the hoster writes profiles and node labels by hand. On clusters without a layout, which includes the `default` variant and Talos hosts, node labels stay hand-written. @@ -331,6 +331,8 @@ Three rules hold throughout: - What a tenant controls. A tenant never supplies a resource name or a node through a claim reference: both come from `GPUClass` and from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes claims only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. - Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. - Node agent. It reads sysfs and kubelet's pod-resources socket, writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can write only its own node's `GPUNode`. +- Webhooks. Three GPU webhooks fail closed. The VMI and virt-launcher pod webhooks see only objects carrying the reference label (section 3), so their outage blocks the start of VMs with claim references. The pod webhook has to fail closed, because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, the budgets and the vGPU entitlement (sections 2 and 4), so its outage also blocks new orders, while held claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity and budgets when it creates and binds the claim. +- Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `gpu.cozystack.io/template-claim` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so a held claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. - Template claims. For `claimTemplate` entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the template names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template claims, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). - Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Step 1 also gives it `update` and `delete` on pods in tenant namespaces. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod there, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses (section 3). It also writes one ConfigMap, the GPU settings in `cozy-system`. - Visibility. A tenant sees the node name of its own claims and, through the remaining budget, its siblings' GPU count under a shared parent budget (section 5). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. @@ -359,8 +361,8 @@ Each case gives what is seen and what to do. ## Testing -- Unit tests for the binder and both webhooks: capacity with raw-name usage, budgets over the ancestor walk, the phase table, every move and refusal in sections 2 and 3, concurrent starts, and the pod pin after a LiveUpdate sync. -- Webhook and controller tests for cordons, templates and NICs: the webhook binds, moves and creates no claim, under `Move` a claim moves at VMI start and the pod gets a required node affinity and the classes' tolerations, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity the unscheduled VMI is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only template claims gets one node with room for all of them, and under `Stay` a single claim waits; a `claimTemplate` entry is refused at admission when the budget or the entitlement does not allow it, its claim is created by the controller once the VMI exists and owned by it, deleted with its VMI, reused on a repeated pass, and a dry run records nothing; a VM with a GPU claim and `nvidia.com/hostdev` passes. +- Unit tests for the binder and the three GPU webhooks: capacity with raw-name usage, budgets over the ancestor walk, the phase table, every move and refusal in sections 2 and 3, concurrent starts, the gate added once on a repeated call and on the temporary `WaitForFirstConsumer` pod but not on a migration target, the pin appended to every existing node selector term and the gate removed in the same update, and the pod pin after a LiveUpdate sync. +- Webhook and controller tests for cordons, templates and NICs: the webhook binds, moves and creates no claim, a set that cannot be placed keeps the pod gated with a condition and is released once it can, a pinned pod that the node refuses is deleted and that node is avoided for the set, under `Move` a claim moves at VMI start and the pod gets a required node affinity and the classes' tolerations, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity a VMI with a pinned pod is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, a gated VMI is not deleted and its claims are gathered, a cordon or `NotReady` on the pinned node does not count as a refusal, so the refused-pod deletion does not fire, a `Once` or `Manual` VM stays stopped after a refused pod is deleted, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only template claims gets one node with room for all of them, and under `Stay` a single claim waits; a `claimTemplate` entry is refused at admission when the class, `maxCount` or the entitlement does not allow it, its claim is created by the controller once the pod waits at the gate, under the budget and capacity checks, and owned by the VMI, deleted with its VMI, reused on a repeated pass, and a dry run records nothing; a VM with a GPU claim and `nvidia.com/hostdev` passes. - Admission tests for `KubernetesNodes`: a `claimSelector` or `claimTemplate` of a vGPU class on a Talos pool is refused, and one of a passthrough class is accepted. - Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held claims and refuses the next template claim, and `remainingBudget` is 0 without the entitlement. - Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. @@ -373,7 +375,7 @@ Each case gives what is seen and what to do. ## Rollout -1. `GPUClass`, `GPUClaim`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, both webhooks, the policies, the option sources, `claim` and `claimTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRDs ship and finalizers and usage records still run. Before tenants order vGPU claims, the `gpu.vgpu` flag is set from the root tenant down. +1. `GPUClass`, `GPUClaim`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, the three GPU webhooks, the policies, the option sources, `claim` and `claimTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRDs ship and finalizers and usage records still run. Before tenants order vGPU claims, the `gpu.vgpu` flag is set from the root tenant down. 2. `claimSelector` and `claimTemplate` in `kubernetes-nodes`, for Talos pools with passthrough classes. 3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, and `vgpu-vms` adds GPU guest images for VMs with `claim` entries. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. 4. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. From 95c104d7aca2f3dfe4f65a5ae2955cdce2bbb580 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:23:04 +0300 Subject: [PATCH 13/32] docs(design-proposals): align the GPUClaim records with the code Several records said "not yet" for decisions the step 1 pull requests implement, which makes them misleading, so their headers now name those pull requests. Two places in the text described something the code deliberately does not: the hoster's GPU resource name list does not default to the KubeVirt table but is always unioned with it, the class names and the agent's names, and the platform does not check the preconditions of forbidden. The text now says so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 4 ++-- ...im-outside-permittedhostdevices-is-refused-at-admission.md | 2 +- ...m-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md | 2 +- .../0004-gpunode-health-is-kubelet-allocatability.md | 2 +- ...y-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md | 2 +- .../0007-whoever-can-write-a-vm-orders-its-template-claims.md | 2 +- ...ntroller-gets-its-host-layer-rights-with-the-host-layer.md | 2 +- 7 files changed, 8 insertions(+), 8 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 694ca148..9f22225f 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -213,7 +213,7 @@ The webhook refuses to start a VMI, and names each claim and its node, when a re - is a vGPU claim next to vGPU claims of another class, or brings the VMI's GPUs of its class above `maxCount`; - is of a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). -A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are the resource names of `GPUClass` objects and the names in the hoster's list `gpu.resourceNames`, which defaults to the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. The list also takes every resource name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin, as the agent reports it, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU claim for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its claims unattached, so they move by the rules of section 2. Both webhooks see only VMIs and pods carrying the chart's reference label, `gpu.cozystack.io/claim-reference`, next to the `gpu.cozystack.io/claims` annotation that lists the references, and fail closed, so their outage blocks the start of VMs with claim references and no other VM or pod; the third GPU webhook, on orders, is described under Security. +A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are always the union of four sets: the hoster's list `gpu.resourceNames`, empty unless the hoster sets it; the resource names of `GPUClass` objects; the names the node agent reports in `GPUNode` `status.resourceNames`, which include every name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin; and the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. None of them replaces another, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU claim for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its claims unattached, so they move by the rules of section 2. A claim that is `Pending` or `Lost`, or bound to a node that cannot serve it, is not refused: the controller binds or moves it before it releases the pod. The VMI and launcher pod webhooks see only VMIs and pods carrying the reference label, next to the `gpu.cozystack.io/claims` annotation that lists the references, and fail closed, so their outage blocks the start of VMs with claim references and no other VM or pod; the third GPU webhook, on orders, is described under Security. A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and usage record, under the `gpu.cozystack.io/claim-protection` finalizer until no VMI uses it, and is not attached again. @@ -288,7 +288,7 @@ A tenant moves a VM by stopping it, ordering claims of the matching class, repla Quota keys move too. A raw GPU key is a quota key on a GPU resource name, such as `nvidia.com/AD102GL_L40S`. A claim-backed VM's pod still requests that name, so while a tenant migrates, its raw key covers both kinds of VM; after its last raw-name VM has moved, the key is replaced by `gpu.cozystack.io/*` keys. A tenant whose raw-name VMs use vGPU can order their claims only after it is entitled to vGPU (section 4), and within any class keys set above it. -When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. Preconditions: no worker pools with raw-name `gpus` entries, ComputePlane GPU groups included, no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: +When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. The preconditions are the hoster's checklist, and the platform does not verify them: no worker pools with raw-name `gpus` entries, ComputePlane GPU groups included, no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: - an admission policy refuses GPU entries (section 3) on every VMI created without a claim reference, on every path; - the same policy refuses every pod created in a tenant namespace that requests a GPU resource name, unless it is a virt-launcher pod created by KubeVirt's virt-controller service account with an owner reference to a VirtualMachineInstance. Tenants cannot forge such a pod, since no tenant role can create pods or VMIs, so app charts that copy free-form `resources` into pods cannot request GPUs either; diff --git a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md index b8e24b06..a7faf6f9 100644 --- a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md +++ b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md @@ -6,7 +6,7 @@ - **Deciders:** `@lexfrei` - **Proposal:** [`../README.md`](../README.md) - **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** not yet +- **Implemented in:** [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (the class condition), [`cozystack/cozystack#5030`](https://github.com/cozystack/cozystack/pull/5030) (the refusal at VMI admission) ## Context diff --git a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md index 7a596473..e6971e57 100644 --- a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md +++ b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md @@ -6,7 +6,7 @@ - **Deciders:** `@lexfrei` - **Proposal:** [`../README.md`](../README.md) - **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** [`cozystack/cozystack#5008`](https://github.com/cozystack/cozystack/pull/5008) (the gate and the policies under it); the rest: not yet +- **Implemented in:** [`cozystack/cozystack#5008`](https://github.com/cozystack/cozystack/pull/5008) (the gate and the policies under it), [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (finalizers and usage records on every cluster), [`cozystack/cozystack#5030`](https://github.com/cozystack/cozystack/pull/5030) (the webhooks, the `vm-instance` refusal and the refusal of `forbidden`) ## Context diff --git a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md index cd43c7be..3e8d5517 100644 --- a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md +++ b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md @@ -6,7 +6,7 @@ - **Deciders:** `@lexfrei` - **Proposal:** [`../README.md`](../README.md) - **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) (the fields); the node agent: not yet +- **Implemented in:** [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) (the fields), [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (the node agent) ## Context diff --git a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md index 5ef332c0..6a498005 100644 --- a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md +++ b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md @@ -6,7 +6,7 @@ - **Deciders:** `@lexfrei` - **Proposal:** [`../README.md`](../README.md) - **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** not yet +- **Implemented in:** [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013); the gather after admission of [0012](./0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md): not yet ## Context diff --git a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md index 06b2c22b..b3100ad4 100644 --- a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md +++ b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md @@ -6,7 +6,7 @@ - **Deciders:** `@lexfrei` - **Proposal:** [`../README.md`](../README.md) - **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** not yet +- **Implemented in:** [`cozystack/cozystack#5030`](https://github.com/cozystack/cozystack/pull/5030); the controller-side creation and capacity check of [0012](./0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md): not yet ## Context diff --git a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md index 0837667b..c7f29dc9 100644 --- a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md +++ b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md @@ -6,7 +6,7 @@ - **Deciders:** `@lexfrei` - **Proposal:** [`../README.md`](../README.md) - **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) -- **Implemented in:** not yet +- **Implemented in:** [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (the VMI deletion right only) ## Context From 173fc654888e671143343bbde10380c23e0a7684 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 02:23:15 +0300 Subject: [PATCH 14/32] docs(design-proposals): state per run strategy how KubeVirt replaces a VMI The proposal said KubeVirt deletes a final VMI before it creates the next one without saying for which run strategies. A VMI is named after its VM, so the next one never coexists with the old one: Always and RerunOnFailure delete the final VMI first, Manual stops it on the start request, and Once creates no next VMI. The statement that L40S VFs report the PF's PCI ID had no source; it now states what was observed. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 9f22225f..5d133d65 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -41,7 +41,7 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra The `cozystack.gpu-operator` package has three variants, selected by `bundles.iaas.gpuOperatorVariant` ([`docs/gpu-vgpu.md`](https://github.com/cozystack/cozystack/blob/main/docs/gpu-vgpu.md)). `default` binds whole GPUs to `vfio-pci` for VMs, `vgpu` runs the NVIDIA vGPU host driver, and `container` serves GPUs to pods. For the two VM variants the platform writes the `HostDevices` feature gate and, when it has entries, a `permittedHostDevices` table into the KubeVirt CR (`packages/core/platform/templates/bundles/iaas.yaml`). Passthrough GPUs are advertised by NVIDIA's sandbox device plugin. -How NVIDIA's vGPU host driver exposes vGPUs depends on the GPU, the host OS and the driver branch. Older setups use mediated devices (mdev). SR-IOV GPUs on current releases, such as Ubuntu 24.04 with the vGPU 17 or newer branch that Cozystack's guide uses, get VFs, and each VF gets a profile when `current_vgpu_type` is written in its sysfs. KubeVirt v1.9, which Cozystack bundles, advertises such VFs through virt-handler's own device plugin ([kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890)), matched by PCI ID; on L40S every VF reports the PF's ID. On GPU Operator nodes nothing in Cozystack writes VF profiles; the `cozystack-nvidia-vgpu-restore` boot unit from cozystack/ansible-cozystack#70 covers only hosts with a host-installed driver. +How NVIDIA's vGPU host driver exposes vGPUs depends on the GPU, the host OS and the driver branch. Older setups use mediated devices (mdev). SR-IOV GPUs on current releases, such as Ubuntu 24.04 with the vGPU 17 or newer branch that Cozystack's guide uses, get VFs, and each VF gets a profile when `current_vgpu_type` is written in its sysfs. KubeVirt v1.9, which Cozystack bundles, advertises such VFs through virt-handler's own device plugin ([kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890)), matched by PCI ID. Observed on L40S with SR-IOV vGPU: the VF shows `10de:26b9` in the guest, the PF's ID, and `permittedHostDevices` selects VFs by `10DE:26B9`. On GPU Operator nodes nothing in Cozystack writes VF profiles; the `cozystack-nvidia-vgpu-restore` boot unit from cozystack/ansible-cozystack#70 covers only hosts with a host-installed driver. Both consumers take GPUs the same way: `vm-instance` and `kubernetes-nodes` read `gpus` as a list of `{name: }`, one entry per GPU. The deprecated `gn1.*` instance types carry GPU entries of their own in KubeVirt's shape, `{name: gpu1, deviceName: nvidia.com/A400}`, and neither default `permittedHostDevices` table lists that name. Tenant quotas become a namespace `ResourceQuota`, which refuses a GPU only when virt-launcher's pod is created, and the usage records (`Workload`) show a GPU only while a VM runs. @@ -175,7 +175,7 @@ The phases decide what a claim holds and costs: A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 8). -A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 deletes a final VMI before it creates the VM's next one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)), so a new VMI never finds its claim attached to the old one. +A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: under `Always` and `RerunOnFailure` the VM controller deletes a final VMI before it starts the next one, under `Manual` a start request first stops the final VMI and then starts a new one, and under `Once` no next VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. A claim keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer matches the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules of section 3. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set (section 3). From f30daf0cf876d697a51cd3acce6268bbcda17deb Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 03:44:49 +0300 Subject: [PATCH 15/32] docs(design-proposals): limit class tolerations to what pods admit A GPUClass toleration is copied into every virt-launcher pod of the class, so a toleration that pod validation refuses would make each of those pods unadmittable. The API therefore admits only the Exists and Equal operators and pod-valid fields. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../0010-gpuclass-carries-the-tolerations-binding-reads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md b/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md index 6a9baf1c..dbf6c285 100644 --- a/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md +++ b/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md @@ -14,7 +14,7 @@ Section 2 kept binds, moves, rebinds and gathers off nodes with a `NoSchedule` o ## Decision -`GPUClass` has a `tolerations` field, in the shape of a pod's, which the hoster sets to the taints of the class's nodes that its VMs tolerate. A node counts for binding, moves, rebinds, gathers, capacity, `free` and `maxFreePerNode` of a class only if the class tolerates each of its `NoSchedule` and `NoExecute` taints, except the three that 0005 skips, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, for the reasons 0005 gives. A set of claims of several classes takes a node only when every class of the set tolerates its taints. The virt-launcher pod of a VMI gets the tolerations of its claims' classes when it is created. `tolerations` can change while claims are held, like `nodeSelector`. +`GPUClass` has a `tolerations` field, in the shape of a pod's with the operators `Exists` and `Equal` only, which the hoster sets to the taints of the class's nodes that its VMs tolerate. A node counts for binding, moves, rebinds, gathers, capacity, `free` and `maxFreePerNode` of a class only if the class tolerates each of its `NoSchedule` and `NoExecute` taints, except the three that 0005 skips, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, for the reasons 0005 gives. A set of claims of several classes takes a node only when every class of the set tolerates its taints. The virt-launcher pod of a VMI gets the tolerations of its claims' classes when it is created. The class admits only tolerations that pod validation admits, so a class can never produce a launcher pod the API server refuses. `tolerations` can change while claims are held, like `nodeSelector`. ## Why not the alternatives From f46d2206ef4266c934cfd890f74c6e26f5298c1b Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 23 Oct 2026 04:02:45 +0300 Subject: [PATCH 16/32] docs(design-proposals): settle how a waiting set reports and remembers A VMI cannot carry a GPU condition, since KubeVirt's admission guards writes to it, so the reason a set waits reaches the VMI as a Warning event while the claims keep the condition. A set placed after a WaitForFirstConsumer volume bound is limited to the nodes that the volume's node affinity allows. A set of template claims only loses its claims with the failed VMI, so the controller keeps its rejected nodes in memory against the VM, and a restart forgets them. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 5d133d65..8e93fd15 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -225,13 +225,13 @@ A pool states no mode of its own: its `claimSelector` or `claimTemplate` names t #### Placing a VMI's claims -A launcher pod waits for its GPUs behind a scheduling gate, so nothing is refused for not being ready yet. A mutating webhook on the creation of a virt-launcher pod adds the gate `gpu.cozystack.io/claims` and the `tolerations` of the classes of the VMI's claims. It matches only pods labelled `kubevirt.io: virt-launcher` that carry the reference label, adds the gate only when it is not there yet, since a repeated call would otherwise fail on a duplicate gate, and fails closed: a pod that reached the scheduler ungated would run unpinned. It also gates the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on the node of the claims. Migration target pods are left alone: KubeVirt deletes a target pod that stays pending for 15 minutes, a VM with a passthrough GPU or a VF does not live-migrate, and Cozystack does not enable the `VGPULiveMigration` gate that mdev vGPUs would need. A set that already has a `WaitForFirstConsumer` volume bound on a node stays on that node. KubeVirt never touches scheduling gates and has no timeout for a VMI whose pod waits, so the VMI stays in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated` ([`lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go)), and the VM's own start backoff never begins. +A launcher pod waits for its GPUs behind a scheduling gate, so nothing is refused for not being ready yet. A mutating webhook on the creation of a virt-launcher pod adds the gate `gpu.cozystack.io/claims` and the `tolerations` of the classes of the VMI's claims. It matches only pods labelled `kubevirt.io: virt-launcher` that carry the reference label, adds the gate only when it is not there yet, since a repeated call would otherwise fail on a duplicate gate, and fails closed: a pod that reached the scheduler ungated would run unpinned. It also gates the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on the node of the claims. Migration target pods are left alone: KubeVirt deletes a target pod that stays pending for 15 minutes, a VM with a passthrough GPU or a VF does not live-migrate, and Cozystack does not enable the `VGPULiveMigration` gate that mdev vGPUs would need. A set that already has a `WaitForFirstConsumer` volume bound on a node stays on that node: the controller places it only where the node affinity of the VMI's bound volumes allows. KubeVirt never touches scheduling gates and has no timeout for a VMI whose pod waits, so the VMI stays in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated` ([`lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go)), and the VM's own start backoff never begins. For a gated pod the controller creates the VMI's template claims under the usual budget and capacity checks, then brings the set onto one node. Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are `Pending` or `Lost`, on different nodes, on a node that is gone, no longer matches a class, has lost capacity or has a pending layout change, on a node that is not `Ready`, on a node with a taint one of their classes does not tolerate (section 2), on a cordoned node under `Move`, or on a node with no room for the VMI's template claims, the controller binds or moves them onto one `Ready` node that is not cordoned, has no pending layout change, is tolerated by every class of the set and has room for all of them, template claims included. Every held claim of the set is attached to this VMI by then, so no other VMI or background move can take one, and each claim keeps its hold while it moves. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims, there is no gather, even if the node is also not `Ready`: the pod is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). Once the set is on one node, the controller pins the pod and removes the gate in one update. The pin is a `matchFields` requirement on `metadata.name` appended to every required node selector term of the pod, or the single term when it has none. Kubernetes lets a gated pod only narrow its scheduling, so a term cannot be added next to existing ones, and KubeVirt usually sets a required term already, for host-model CPUs. The pin is not `nodeName`, which would let kubelet start the pod on a cordoned node, and it goes on the pod rather than the VMI because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. -When the set cannot be placed, because no node has room, a budget is exhausted, or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, and the claims and the VMI carry a condition naming the reason. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules (below and section 2). After the deletion the VMI becomes `Failed`, and the VM's `runStrategy` decides what follows: `Always` and `RerunOnFailure` start again after KubeVirt's start-failure backoff, with the VM shown as `CrashLoopBackOff`; a `Once` VM keeps its failed VMI, so it stays down and also shows `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it (Failure cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. +When the set cannot be placed, because no node has room, a budget is exhausted, or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. The VMI gets no condition because KubeVirt's admission guards writes to a VMI. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules (below and section 2). After the deletion the VMI becomes `Failed`, and the VM's `runStrategy` decides what follows: `Always` and `RerunOnFailure` start again after KubeVirt's start-failure backoff, with the VM shown as `CrashLoopBackOff`; a `Once` VM keeps its failed VMI, so it stays down and also shows `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it (Failure cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. A set of template claims only loses its claims with the failed VMI, so for it the controller keeps the record in memory against the VM, and a controller restart forgets it. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way while the pod is gated. Once the pod is pinned, under `Move`, for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). @@ -344,7 +344,7 @@ Each case gives what is seen and what to do. - A tenant's order is refused at create → the message names the class, the count and the limit, or, for a vGPU class, that the tenant is not entitled to vGPU; order a smaller count, another class, or ask for more budget. - A claim stays `Pending` after an order → its condition names the shortage; it binds when capacity or budget frees up, or the tenant deletes it. - A VM with claims is refused at start → the webhook refuses the VMI for a reference problem, such as a claim that does not exist, is being deleted or is attached to another VMI, and the VirtualMachine shows the message in a `FailedCreate` event, naming each claim and the reason; fix the reference. -- A VM with claims stays `Starting` → its VMI is in `Scheduling` with its pod gated, and the claims and the VMI carry a condition naming the reason: no node can take the whole set, template claims included, a budget is exhausted, or a claim of a `Stay` class is on a cordoned node while others need gathering. The VM starts by itself once the set can be placed. To get there sooner, uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. +- A VM with claims stays `Starting` → its VMI is in `Scheduling` with its pod gated, the claims carry a condition and the VMI a `Warning` event naming the reason: no node can take the whole set, template claims included, a budget is exhausted, or a claim of a `Stay` class is on a cordoned node while others need gathering. The VM starts by itself once the set can be placed. To get there sooner, uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. - The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. An `Always` or `RerunOnFailure` VM shows `CrashLoopBackOff` and starts again after a delay that grows with each failure up to five minutes, and the count resets once a VMI runs; a `Once` VM keeps its failed VMI and stays down, also shown as `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it. The binder avoids that node for the set for ten minutes or until the node changes (section 3). Free CPU or memory on GPU nodes if this repeats. - A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. - A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. From 5fd6ef0988563e6598a15e87f67ebca652fa9e38 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Mon, 26 Oct 2026 10:30:16 +0300 Subject: [PATCH 17/32] docs(design-proposals): align gpudevice text with the controller code The controller's pod update and delete rights sit in its ClusterRole, like its write on Secrets, because tenant namespaces appear at runtime, so the Security section and DR 0008 now scope them to every namespace. DR 0004 described allocatable as summed over healthy functions, with an unhealthy function alone counting as lost capacity. The controller sums allocatable over every function carrying the resource name, caps it at the kubelet's node allocatable, and the agent counts an IOMMU group device once, on its lowest-addressed GPU function. DR 0012 now states, as the README does, that rejected nodes of a template-only set live in controller memory and do not survive a restart. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 2 +- .../decisions/0004-gpunode-health-is-kubelet-allocatability.md | 2 +- ...controller-gets-its-host-layer-rights-with-the-host-layer.md | 2 +- ...mi-webhook-checks-and-records-and-the-controller-prepares.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 8e93fd15..e1e9859d 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -334,7 +334,7 @@ Three rules hold throughout: - Webhooks. Three GPU webhooks fail closed. The VMI and virt-launcher pod webhooks see only objects carrying the reference label (section 3), so their outage blocks the start of VMs with claim references. The pod webhook has to fail closed, because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, the budgets and the vGPU entitlement (sections 2 and 4), so its outage also blocks new orders, while held claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity and budgets when it creates and binds the claim. - Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `gpu.cozystack.io/template-claim` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so a held claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. - Template claims. For `claimTemplate` entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the template names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template claims, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). -- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Step 1 also gives it `update` and `delete` on pods in tenant namespaces. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod there, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses (section 3). It also writes one ConfigMap, the GPU settings in `cozy-system`. +- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Step 1 also gives it `update` and `delete` on pods in every namespace, cluster-wide like its write on Secrets, because tenant namespaces appear at runtime. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses (section 3). It also writes one ConfigMap, the GPU settings in `cozy-system`. - Visibility. A tenant sees the node name of its own claims and, through the remaining budget, its siblings' GPU count under a shared parent budget (section 5). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. ## Failure and edge cases diff --git a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md index 3e8d5517..d8478290 100644 --- a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md +++ b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md @@ -21,7 +21,7 @@ Not every GPU device ID maps to a PCI function. The GPU Operator's device plugin ## Decision - A function is healthy when every plugin device on it is allocatable, and unhealthy when a pod that has not finished holds one of its devices that is not. A function no plugin device maps to has no health. -- Each function reports `allocatable`, the number of its plugin devices kubelet lists as allocatable: 1 for a VF or a GPU passed through whole, N for an mdev parent. The controller sums it over healthy functions, and a node has lost capacity for a resource name when it holds more units than that sum, or when a function is unhealthy. +- Each function reports `allocatable`, the number of its plugin devices kubelet lists as allocatable: 1 for a VF or a GPU passed through whole, N for an mdev parent. A plugin device that names a whole IOMMU group counts once, on the group's lowest-addressed GPU function, and 0 on its other functions. The controller sums it over the functions that carry the resource name and caps the sum at the node's allocatable count from kubelet, which alone counts for a held name no function carries. A node has lost capacity for a resource name when it holds more units than that. - `GPUNode.status.resourceNames` lists every resource name on the node whose devices are GPUs, including names whose device IDs are NVML UUIDs the agent cannot place under a function. The names join the cluster's GPU resource names of section 3. - A function serving several pods, such as an mdev parent, names one of them in `usedBy`, the smallest by namespace and name, so the report is stable. diff --git a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md index c7f29dc9..1a8bfc72 100644 --- a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md +++ b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md @@ -23,7 +23,7 @@ Of the three, Rollout step 1 grants the controller only the VMI deletion. The no ## Consequences -- The `update` and `delete` rights on tenant pods that the launcher pod gate of section 3 needs are a separate grant of step 1, not one of the three. +- The `update` and `delete` rights on pods in every namespace that the launcher pod gate of section 3 needs are a separate grant of step 1, not one of the three. - Between step 1 and step 4 the controller cannot change any node, so the GPU node labels that classes match are the ones the hoster writes. - The rights and their policy are reviewed in step 4, next to the code that uses them. diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md index 243c5d50..516e85fa 100644 --- a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -16,7 +16,7 @@ Section 3 had the mutating webhook on VirtualMachineInstance creation do all of ## Decision -The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the held claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. A second mutating webhook adds the scheduling gate `gpu.cozystack.io/claims` and the classes' tolerations to the virt-launcher pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, with a condition on the claims and the VMI. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. `status.bindRequest` is dropped. +The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the held claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. A second mutating webhook adds the scheduling gate `gpu.cozystack.io/claims` and the classes' tolerations to the virt-launcher pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, with a condition on the claims and the VMI. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. For a set made only of template claims, the controller keeps its rejected nodes in memory against the VM, and a controller restart forgets them. `status.bindRequest` is dropped. ## Why not the alternatives From 021b4c3ed0d8143acf5b4b8467af1d753d905af3 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Mon, 26 Oct 2026 13:04:12 +0300 Subject: [PATCH 18/32] docs(design-proposals): give the VMI a Warning event, not a condition, in 0012 KubeVirt's admission guards VMI writes, so the reason reaches the VMI as an event, as the README already states. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- ...-webhook-checks-and-records-and-the-controller-prepares.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md index 516e85fa..048526a8 100644 --- a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -16,7 +16,7 @@ Section 3 had the mutating webhook on VirtualMachineInstance creation do all of ## Decision -The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the held claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. A second mutating webhook adds the scheduling gate `gpu.cozystack.io/claims` and the classes' tolerations to the virt-launcher pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, with a condition on the claims and the VMI. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. For a set made only of template claims, the controller keeps its rejected nodes in memory against the VM, and a controller restart forgets them. `status.bindRequest` is dropped. +The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the held claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. A second mutating webhook adds the scheduling gate `gpu.cozystack.io/claims` and the classes' tolerations to the virt-launcher pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. For a set made only of template claims, the controller keeps its rejected nodes in memory against the VM, and a controller restart forgets them. `status.bindRequest` is dropped. ## Why not the alternatives @@ -29,7 +29,7 @@ The VMI webhook resolves the claim references, runs the checks that need no plac - The webhook is a pure check plus the attachment record, so its latency no longer depends on the binder, and a dry run only skips the record. - Template claims are created after the VMI exists, so the controller sets their owner at creation, and a template claim no longer needs a grace period for a VMI that never appears. -- A set that cannot be placed is no longer an admission error. The VM shows `Starting`, the VMI waits in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated`, and the reason is a condition on the claims and the VMI. The VM starts as soon as the set fits, with no KubeVirt backoff in between. +- A set that cannot be placed is no longer an admission error. The VM shows `Starting`, the VMI waits in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated`, and the claims carry a condition naming the reason while the VMI gets a `Warning` event with the same reason. The VM starts as soon as the set fits, with no KubeVirt backoff in between. - The pod webhook is a fail-closed dependency of every start of a VM with claim references, like the VMI webhook. - The cluster autoscaler does not scale up for a gated pod, and after the pin a new node cannot help that pod, so GPU node scale-up stays with the hoster. - A pin cannot be undone, so a node that refuses a pinned pod costs one failed VMI and KubeVirt's start-failure backoff, which the rejected-node rule keeps from repeating. From 694ecebd4c63de5d62a3121521373d0b7ad40786 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Mon, 26 Oct 2026 13:34:35 +0300 Subject: [PATCH 19/32] docs(design-proposals): admission reads past the controller cache The GPU admission webhooks read the KubeVirt table, GPUClass, GPUClaim, GPUNode, namespaces and VMIs from the API server, not from the controller's informer cache. The controller runs two replicas that all serve admission while only the leader binds, so a replica whose cache lags could admit what should be refused. Decision 0001 still said the permitted table is read through the cache. Correct it and give the reason. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- ...laim-outside-permittedhostdevices-is-refused-at-admission.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md index a7faf6f9..9698e44b 100644 --- a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md +++ b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md @@ -28,7 +28,7 @@ The VMI webhook reads the effective table from the KubeVirt CR, the resource nam - A tenant learns at start, from the webhook message, that the offer is not usable yet, and the claim stays unattached and moves by the usual rules. - The webhook and the controller condition read the same lists of the table, so the condition predicts the refusal. KubeVirt's own check also covers `usb` entries, which no GPU class uses. -- Every VMI admission with claim references reads the KubeVirt CR, through the controller's cache. +- Every VMI admission with claim references reads the KubeVirt CR from the API server, not from the controller's cache: the replicas serve admission and a lagging cache could admit what should be refused. ## Revisit if From 5eacb2c1cd0e98e4347ff01d6045f6ae698ac2fa Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 18:32:23 +0300 Subject: [PATCH 20/32] docs(design-proposals): drop Status and Date from the GPUDevice header Proposal status and dates now come from the PR and Git history. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index e1e9859d..b6950143 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -2,8 +2,6 @@ - **Title:** `GPUClaim: a GPU catalog, per-tenant GPU reservations for VMs and worker pools, and the host layer under them` - **Author(s):** `@lexfrei` -- **Date:** `2026-10-14` -- **Status:** Review ## Overview From 15612567b777374e2b1d190f820686ded4f7c689 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 18:32:31 +0300 Subject: [PATCH 21/32] docs(design-proposals): record the two-step GPU layout decisions No separate API group, no hold and no GPUNode in step 1, holding as a GPU-specific step 2 until DRA can carry it, and the vGPU flag as an entitlement apart from budgets. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- ...0013-the-gpu-kinds-live-in-cozystack-io.md | 39 ++++++++++++++++++ ...-step-1-names-a-class-and-holds-nothing.md | 41 +++++++++++++++++++ ...5-node-allocatable-is-the-gpu-inventory.md | 35 ++++++++++++++++ ...olding-is-step-2-and-stays-gpu-specific.md | 37 +++++++++++++++++ ...7-the-vgpu-entitlement-is-a-tenant-flag.md | 33 +++++++++++++++ 5 files changed, 185 insertions(+) create mode 100644 design-proposals/gpudevice/decisions/0013-the-gpu-kinds-live-in-cozystack-io.md create mode 100644 design-proposals/gpudevice/decisions/0014-step-1-names-a-class-and-holds-nothing.md create mode 100644 design-proposals/gpudevice/decisions/0015-node-allocatable-is-the-gpu-inventory.md create mode 100644 design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md create mode 100644 design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md diff --git a/design-proposals/gpudevice/decisions/0013-the-gpu-kinds-live-in-cozystack-io.md b/design-proposals/gpudevice/decisions/0013-the-gpu-kinds-live-in-cozystack-io.md new file mode 100644 index 00000000..96fafb86 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0013-the-gpu-kinds-live-in-cozystack-io.md @@ -0,0 +1,39 @@ +# 0013. The GPU kinds live in `cozystack.io`, with no group of their own + +- **Number:** `0013` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) +- **Implemented in:** not yet + +## Context + +The proposal introduced an API group, `gpu.cozystack.io`, for `GPUClass`, `GPUClaim` and `GPUNode`, and used the group name as the prefix of everything around them: the tenant quota keys `gpu.cozystack.io/` and `gpu.cozystack.io/total`, the node labels, the finalizers, the labels and annotations on VMIs and pods, and the scheduling gate. It gave no reason for a group of its own. cozystack/cozystack#4998, which would create it, was open. + +@lllamnyp pointed out in review that those keys are part of the API contract even though the API server does not enforce them, so the group decides their names too, and that upstream handled less interchangeable devices without a device-type group: distinct offers get distinct resource names, and where counts were not enough, the new group is named after the mechanism, `resource.k8s.io`, with class-scoped quota keys `.deviceclass.resource.k8s.io/devices`. He also pointed at `network.cozystack.io`, which cozystack/cozystack#3081 added on 2026-06-30 and cozystack/cozystack#3218 removed on 2026-07-16, as the case not to repeat with tenant quota keys involved. + +The `cozystack.io` group already held hoster-side kinds, `SchedulingClass` and `Package`, and tenant-namespace kinds, `Workload` and `WorkloadMonitor`. + +## Decision + +There is no `gpu.cozystack.io` group. `GPUClass` (step 1) and `GPUClaim` (step 2) are kinds of `cozystack.io/v1alpha1`, and there is no `GPUNode` ([0015](./0015-node-allocatable-is-the-gpu-inventory.md)). The budget of one class is `.gpuclass.cozystack.io/devices`, after DRA's form, and the budget over every class is `gpuclass.cozystack.io/devices`. Labels, annotations, finalizers and the scheduling gate are `cozystack.io/` keys with a `gpu` name, such as `cozystack.io/gpu-reference`, `cozystack.io/gpuclaim-protection` and the gate `cozystack.io/gpuclaims`; the vGPU entitlement stays on the existing `namespace.cozystack.io/` prefix of module labels. Section 1 of the proposal lists every name. + +`cozystack.io` then holds the hoster-side kinds (`SchedulingClass`, `Package`, `GPUClass`) and the tenant-namespace kinds the platform writes or a tenant admin owns (`Workload`, `WorkloadMonitor`, and in step 2 `GPUClaim`). + +## Why not the alternatives + +- **Keep `gpu.cozystack.io`.** Nothing in the design needs a group of its own: no separate versioning, no separate API server, no RBAC that a group boundary would make simpler. A CRD cannot move to another group in place, so a wrong group would cost every tenant's quota keys and objects a migration, as the `network.cozystack.io` removal did on a smaller scale. +- **Name the keys after the group, `cozystack.io/gpu-`.** A flat key cannot be told apart from a label-style name without parsing, and it does not follow the one upstream form readers already know. The DRA form puts the class in the DNS prefix, so a key without the prefix naturally means every class and no class name has to be reserved for a total. +- **Put the catalog in platform values instead of a kind.** Values can describe offers, but tenants and their GitOps tools could not read them, and nothing could carry a status with free units. + +## Consequences + +- cozystack/cozystack#4998 creates no new group; its CRDs move to `cozystack.io`, and every `gpu.cozystack.io` string in the GPU code is renamed before anything merges. +- The class name `total` is no longer reserved. +- A future GPU kind goes into `cozystack.io` too, unless a requirement appears that only a separate group can carry; then it is written down in a record first. + +## Revisit if + +GPU offers move to DRA `DeviceClass` objects, so the catalog and its quota keys become DRA's own, or a GPU kind needs versioning or serving apart from the rest of `cozystack.io`. diff --git a/design-proposals/gpudevice/decisions/0014-step-1-names-a-class-and-holds-nothing.md b/design-proposals/gpudevice/decisions/0014-step-1-names-a-class-and-holds-nothing.md new file mode 100644 index 00000000..c2962924 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0014-step-1-names-a-class-and-holds-nothing.md @@ -0,0 +1,41 @@ +# 0014. In step 1 a `gpus` entry names a class, budgets count declared entries, and usage is the virt-launcher pod's `Workload` + +- **Number:** `0014` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) +- **Implemented in:** not yet + +## Context + +Rollout step 1 had everything at once: `GPUClass`, `GPUClaim` as a tenant order, `GPUNode` and its agent, the binder, three fail-closed webhooks, the launcher pod gate, budgets with their own subtree rule, and usage records the GPU controller wrote per claim. @kvaps asked in [review](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) for the text to say which fields and phases exist in step 1 and what an entry means without the binder. @lllamnyp [went through](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) what the proposal adds above one resource name per offer: the catalog is hoster metadata, per-offer budgets argue for class-scoped keys, refusing an order that does not fit is what [#48](https://github.com/cozystack/community/pull/48) does for every kind by counting declared sizes, and the `Workload` records already sum virt-launcher requests, GPU resource names included. Only the hold needs a tenant object. + +The GPU budget rule of the former Quotas section diverged from the existing pool semantics and was meant to move to #48 later, so tenants would have migrated twice. + +## Decision + +Step 1 has no tenant object. A `vm-instance` or `kubernetes-nodes` `gpus` entry names a class and a count. The VMI webhook writes one KubeVirt `gpus` entry per unit with the class's resource name, after checking the class, `maxCount`, one vGPU class per VM, the vGPU entitlement and `permittedHostDevices`. A webhook on the virt-launcher pod adds the class's tolerations to the pod, since under `vmRolloutStrategy: LiveUpdate` KubeVirt resets a VMI's tolerations to the VM template's. Capacity is not checked at admission: a VM that finds no free unit waits in `Scheduling` like a VM short of CPU. A class is one resource name, unique among classes, and its nodes are the nodes that advertise it, so a class has no node selector. + +GPU budgets are ordinary `resourceQuotas` keys under the existing pool rules, and #48's gate checks them at write, counting `count` per class entry and `count` times `maxReplicas` per pool from the defaulted values. Until #48's gate is on, an interim check in the Cozystack API server does the same for GPU keys on `VMInstance` and `KubernetesNodes` writes, and supplies the pool's class entries as GPU usage to the declaration gate on Tenant writes, with the same inputs and pool arithmetic; it stands aside while #48's gate is switched on and is deleted in the release where that gate is on by default. + +Usage is the `Workload` the VM's `WorkloadMonitor` already keeps from the virt-launcher pod's requests. Step 1 writes no usage record of its own. + +## Why not the alternatives + +- **An order object in step 1, checked against budget and capacity at admission, without a hold.** Without the binder an order guarantees nothing at start, so the tenant would pay and wait for an object that only repeats what the VM's entry says. +- **Keep the GPU subtree rule until #48 takes over.** Tenants and hosters would learn one rule and then another, and #48 would have to adopt a GPU exception to keep verdicts the same. +- **Check capacity at VMI admission.** A refused VMI is left to the per-item backoff of KubeVirt's VM controller, which grows to 1000 seconds, for a shortage that may last seconds, while a pending pod starts as soon as a unit frees up. +- **Keep GPU-specific usage records next to the launcher pod's `Workload`.** The pod's record already carries the GPUs for exactly as long as they are used, and a second record would have to be kept from double counting. + +## Consequences + +- A stopped VM's class entries keep counting against the budget, as its CPU and memory do under #48, while its `Workload` shows no GPU. +- A billing system maps a resource name in a `Workload` record to a class through the catalog, and keeps the names it has seen when a class's `resourceName` changes. +- A vGPU class entry written by an unentitled tenant is accepted at write and refused at start, because the pool gate has no yes-or-no rule. +- Step 2 keeps the entry: it creates a template claim for it at start, so the same entry stays valid when holding ships. + +## Revisit if + +#48 does not ship or drops the pool semantics the interim check copies, or a hoster needs a capacity guarantee at start without holding. diff --git a/design-proposals/gpudevice/decisions/0015-node-allocatable-is-the-gpu-inventory.md b/design-proposals/gpudevice/decisions/0015-node-allocatable-is-the-gpu-inventory.md new file mode 100644 index 00000000..013f8daa --- /dev/null +++ b/design-proposals/gpudevice/decisions/0015-node-allocatable-is-the-gpu-inventory.md @@ -0,0 +1,35 @@ +# 0015. Node allocatable is the GPU inventory, and there is no `GPUNode` + +- **Number:** `0015` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) +- **Implemented in:** not yet + +## Context + +`GPUNode` was a cluster-scoped inventory kind per GPU node, written by a node agent from sysfs and kubelet's pod-resources API ([0004](./0004-gpunode-health-is-kubelet-allocatability.md)). It listed each PCI function with its health, an `allocatable` count, the pod using it, and the node's GPU resource names. Its readers were the binder, which judged lost capacity from it and capped its sums at kubelet's node allocatable anyway; the GPU resource names, which joined the name set of the raw-name policy; and a hoster tracing a physical device to a tenant. + +@lllamnyp asked in [review](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) whether a hoster-only inventory needs a kind when metrics, node allocatable or DRA's `ResourceSlice` can serve it. Step 1 no longer has a binder ([0014](./0014-step-1-names-a-class-and-holds-nothing.md)). + +## Decision + +There is no `GPUNode` and no inventory agent. A class's capacity, `free` and `maxFreePerNode` come from the `Node` objects' allocatable count of the class's resource and the requests of pods on those nodes. In step 2 a node has lost capacity for a class when its allocatable count of the class's resource stays below the units held there for longer than the grace period, or when it has been `NotReady` that long. GPU resource names are the union of the hoster's `gpu.resourceNames`, the classes' resource names and the GPU entries of `permittedHostDevices`. Use per tenant comes from the `Workload` records, per virt-launcher pod and resource name. The node agent of the host layer stays, writes VF profiles and restarts the device plugin on its own node. + +## Why not the alternatives + +- **Keep `GPUNode` for step 2's lost capacity.** 0004 already capped the agent's sum at kubelet's allocatable, and kubelet lists only the devices its plugin reports healthy, so the agent's health adds a per-function view of the same signal, not a different one. +- **Keep `GPUNode` for tracing a physical device to a tenant.** With a device plugin, kubelet picks the device when the pod starts, and the pod object does not say which one. The agent's `usedBy` was exact per VF or whole GPU but named only one pod per mdev parent, and a hoster replacing a failed card drains the node either way. +- **Expose the same data as metrics only.** It would still need the agent on every GPU node, privileged against kubelet's pod-resources socket, for a view no part of the design reads. + +## Consequences + +- A hoster cannot see from the API which VF or PCI function a VM uses, only which VM, class and count, and the node through the VM's launcher pod; the guest and the node's own tools show the device. +- A GPU name that only pods request, such as `nvidia.com/gpu` of the `container` variant, is a GPU resource name only if the hoster lists it in `gpu.resourceNames`. +- One privileged pod-resources reader fewer runs on every GPU node. + +## Revisit if + +The design needs a per-device view, for example exact-device allocation, NVLink sets or on-demand slicing; DRA's `ResourceSlice` is the first candidate to provide it. diff --git a/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md b/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md new file mode 100644 index 00000000..fe3ccefc --- /dev/null +++ b/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md @@ -0,0 +1,37 @@ +# 0016. Holding a GPU for a stopped VM is step 2, and stays GPU-specific until DRA can carry it + +- **Number:** `0016` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) +- **Implemented in:** not yet + +## Context + +The binder, moves, `onCordon`, `Lost` and rebind, the gather at VMI start, the launcher pod gate and pin, and `forbidden` all exist for one problem of six, keeping a GPU for a stopped VM, and for the capacity half of another, refusing an order that does not fit. @kvaps asked in [review](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) to split holding from the rest, and later [accepted one proposal](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) if the text says what exists in step 1 and gives step 2 its own section and record. @lllamnyp argued in [review](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) that keeping capacity for a stopped VM is not specific to GPUs, since a stopped VM releases its CPU and memory too, and that the hold should be proposed either as a general reservation or through DRA persistent claims, KubeVirt [VEP 344](https://github.com/kubevirt/enhancements/issues/344). + +When this was decided, VEP 344 was open and its implementation, kubevirt/kubevirt#17957, unmerged; the NVIDIA DRA driver served no vGPU (kubernetes-sigs/dra-driver-nvidia-gpu#1402); and the vGPU VF reached KubeVirt through a device plugin, the one of the host layer with NVIDIA/kubevirt-gpu-device-plugin#192. + +## Decision + +Holding is step 2 of the proposal, in a section of its own, and none of its kinds, fields or phases enter the API before it: `GPUClaim`, `onCordon` and the class finalizer, `claim` and `claimSelector` entries, the binder, moves, `Lost` and rebind, the gather, the launcher pod gate and pin, template claims for class entries, the claim order webhook and tamper policy, the usage records of held claims, and `gpu.rawResourceNames: forbidden`. + +Holding stays GPU-specific and does not become a generic reservation. CPU and memory are interchangeable across nodes, so a reservation of them is a count the scheduler can meet on any node with room. A GPU hold is a whole unit on a specific node that does not migrate, since KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, and that is the reason for the binder, `Lost`, the gather and `onCordon`. A DRA path does not exist today: VEP 344 is open and unimplemented, and the vGPU VF is handed out by a device plugin, not by a DRA driver. + +## Why not the alternatives + +- **Holding in step 1.** Step 1 answers five of the six problems without a hold, the third only for budgets, and the hold puts the most expensive part of the design into a v1alpha1 tenant API before any of it is used. +- **A generic reservation for stopped VMs.** For CPU and memory a per-tenant count would be enough, and none of the node binding would be needed; for GPUs a count without a node fails as soon as free units are spread one per node, as the cluster-wide reservation alternative of the proposal shows. One mechanism would carry the GPU machinery for resources that do not need it. +- **Wait for DRA.** No date can be given for VEP 344 or for vGPU in the NVIDIA DRA driver, and hosters sell vGPU first. + +## Consequences + +- Step 1 ships without a tenant object, and a GPU VM's next start can fail for a GPU someone else took, as it can today. +- Step 2's `GPUClaim` maps one to one onto a `ResourceClaim` for `dra` classes (section 10), so the tenant API does not change when holding moves to DRA. +- A cluster that keeps raw names allowed has no hard hold guarantee, also in step 2. + +## Revisit if + +VEP 344 is implemented and a DRA driver hands out vGPU VFs; then holding moves to DRA. diff --git a/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md b/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md new file mode 100644 index 00000000..cae6618f --- /dev/null +++ b/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md @@ -0,0 +1,33 @@ +# 0017. The vGPU entitlement is a Tenant flag stamped on the namespace, not a budget key + +- **Number:** `0017` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei, @kvaps` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) +- **Implemented in:** not yet + +## Context + +A vGPU needs a license, counted per unit, and a tenant that may use vGPU at all gets the vGPU guest driver and the license token. The proposal gave the yes-or-no part to a Tenant field, `gpu.vgpu`, which the tenant chart combines with the parent's value and stamps on the namespace as `namespace.cozystack.io/vgpu`. It lived inside the quota section. @kvaps [asked](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) whether a `gpu.cozystack.io/vgpu` budget key, 0 by default, would cover it, since it goes down the subtree like the other GPU keys and can cap the license pool. In Cozystack a missing quota key means no limit. He later [agreed](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) that the entitlement is a yes or no, not a quota, and that a Tenant field stamped as a namespace label is the pattern of `namespace.cozystack.io/etcd` and the other module labels, and asked for the flag in its own section with a record that says when to revisit it. + +`kairos-vgpu-workers` copies the hoster's license and registry Secrets into the namespaces of entitled tenants, and uses the same label to decide where. + +## Decision + +The vGPU entitlement stays the Tenant field `gpu.vgpu`, off by default, effective only when the parent is entitled, and stamped on the namespace as `namespace.cozystack.io/vgpu`, where only `"true"` means entitled. It has its own section in the proposal, outside the budgets. The VMI webhook refuses a vGPU class entry or claim without it at every start, and in step 2 the claim order webhook refuses a vGPU order without it. Budgets limit how many vGPUs an entitled tenant uses, through the class keys and the key over every class. + +## Why not the alternatives + +- **A vGPU budget key with 0 by default.** A missing key means no limit for every other key, so a tenant without the key would read as unlimited to anyone who knows the others, and this one key would mean the opposite. A cap on the license pool is a budget over the vGPU classes and can be added as one, apart from the entitlement. +- **Entitlement on the class.** A class is visible to every tenant and names an offer, not who may take it; per-tenant lists on a class would move tenant administration into the hoster's catalog. + +## Consequences + +- A hoster turns vGPU on from the root tenant down before tenants use vGPU classes. +- A write that names a vGPU class is not refused for the entitlement, only the start; the option source shows the class with a remaining budget of 0. + +## Revisit if + +`kairos-vgpu-workers` stops copying Secrets into tenant namespaces, for example once a hoster-side worker class reads the hoster's Secrets directly, so the flag guards nothing but the use of vGPU classes and a budget over those classes could replace it. From 083df0977f385784fb82c07fd145fad32da0e239 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 18:32:32 +0300 Subject: [PATCH 22/32] docs(design-proposals): split GPUDevice into a class step and a hold step Step 1 has no tenant object: a gpus entry names a GPUClass in cozystack.io, budgets are pool keys checked at write through #48 with an interim check until then, and usage is the launcher pod's Workload. Holding, with GPUClaim, the binder and the launcher pod gate, moves to its own step 2 section, and the vGPU flag gets its own section. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 509 +++++++++++++++++---------- 1 file changed, 324 insertions(+), 185 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index b6950143..1bac66ca 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -1,43 +1,48 @@ -# GPUClaim: GPUs as a tenant resource +# GPUs as a tenant resource -- **Title:** `GPUClaim: a GPU catalog, per-tenant GPU reservations for VMs and worker pools, and the host layer under them` +- **Title:** `GPU classes, per-class tenant budgets and the host layer under them, then GPU holds for stopped VMs` - **Author(s):** `@lexfrei` ## Overview -A tenant who wants a GPU today writes a device-plugin resource name such as `nvidia.com/AD102GL_L40S` into `gpus` of a VM or a worker pool and hopes one is free. The `gpu` option source shows how many of each name the nodes advertise and on which nodes, but nothing says which model, mode or profile a name stands for, nothing holds a GPU while the VM is off, and an order that does not fit is accepted and then sits in `Pending`. A hoster selling GPUs needs the opposite: a catalog of GPU models and modes, GPUs a tenant takes within a quota and keeps until they release them, the same handling for VMs and managed Kubernetes worker pools, and a usage record of who held what and when, which the hoster's external billing system reads. +A tenant who wants a GPU today writes a device-plugin resource name such as `nvidia.com/AD102GL_L40S` into `gpus` of a VM or a worker pool and hopes one is free. The `gpu` option source shows how many of each name the nodes advertise and on which nodes, but nothing says which model, mode or profile a name stands for, nothing holds a GPU while the VM is off, and an order that does not fit is accepted and then sits in `Pending`. A hoster selling GPUs needs the opposite: a catalog of GPU models and modes, budgets per offer, the same handling for VMs and managed Kubernetes worker pools, a usage record of who used what, which the hoster's external billing system reads, and, for some offers, GPUs a tenant keeps while its VM is stopped. -This proposal adds three kinds. `GPUClass` is the hoster's catalog: one entry per offer, such as a whole H100 in passthrough or an L40S-24C vGPU profile. `GPUClaim` is a tenant's GPU, or a set of GPUs on one node: ordered against a class, bound to a node with free capacity, and attached to a VM by name or to a worker pool by a label selector. `GPUNode` is the inventory: every physical GPU and VF on a node, its configuration, and which tenant uses it. The proposal also covers the host layer under them, as its own phase: passthrough and vGPU nodes in one cluster, declarative vGPU profiles, and the device plugin that advertises GPUs to KubeVirt. +The proposal comes in two steps. Step 1 has no tenant object. It adds a host layer that ends at one resource name per offer, `GPUClass` as the hoster's catalog in the existing `cozystack.io` group, `gpus` entries in `vm-instance` and `kubernetes-nodes` that name a class instead of a resource name, budgets per class that the tenant quota gate of [#48](https://github.com/cozystack/community/pull/48) checks when an application is written, and usage taken from the `Workload` records Cozystack already keeps for virt-launcher pods. Step 2 adds holding: `GPUClaim`, a tenant's GPU kept on a node while its VM is stopped, with the binder, moves, lost capacity and the launcher pod gate that a hold needs. The fields and phases of step 2 enter the API only in step 2. NVIDIA's vGPU stack changes quickly. Within the last few releases the driver branches, the way vGPUs are exposed (mediated devices or SR-IOV virtual functions), the KubeVirt device plugins and the licensing rules all changed. This proposal describes that stack as it was in October 2026. Expect it to be updated as NVIDIA and KubeVirt move, with a decision record under it for each change of course, and check the versions named here against your own before relying on them. ## Scope and related proposals -- [`kairos-vgpu-workers`](../kairos-vgpu-workers/README.md) gives managed Kubernetes worker pools on Kairos a worker image per GPU mode, with the vGPU guest driver or the data center driver, and delivers the vGPU license token to the workers. [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers VMs with passthrough and vGPU: the hoster adds GPU guest images to the golden image collection through `extraImages`, marked with a `gpu` field. Both are proposed alongside this one and use `GPUClaim` from the start, through `claim` entries and `claimSelector`; the links resolve once they merge. -- [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and section 2 rebinds their `Lost` claims. The link resolves once it merges. -- [IP addresses as a first-class resource](https://github.com/cozystack/community/pull/35) solves the same problem for addresses with a class, a tenant claim and a backend contract. -- [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared reservations the quota authority. A `GPUClaim` is a reservation in that sense, and section 4 hands GPU budgets over to it only if its gate takes over subtree caps for GPU keys. -- Host-side discovery of vGPU VFs grouped by profile is [NVIDIA/kubevirt-gpu-device-plugin#192](https://github.com/NVIDIA/kubevirt-gpu-device-plugin/pull/192), still open. Section 6 ships it downstream in the host layer, behind a boundary the rest of the design does not cross. +- [`kairos-vgpu-workers`](../kairos-vgpu-workers/README.md) gives managed Kubernetes worker pools on Kairos a worker image per GPU mode and delivers the vGPU license token to the workers. [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers GPU guest images for VMs. Both name a `GPUClass` in their `gpus` entries from step 1 on, and take held claims once step 2 ships; the links resolve once they merge. +- [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared sizes the quota authority and refuses an application that does not fit its pool when it is written. GPU budgets are ordinary keys of that gate (section 5). Until it ships, an interim check does the same for GPU keys only. +- [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and step 2 rebinds their `Lost` claims. The link resolves once it merges. +- [IP addresses as a first-class resource](https://github.com/cozystack/community/pull/35) solves a similar problem for addresses with a class, a tenant claim and a backend contract. +- Host-side discovery of vGPU VFs grouped by profile is [NVIDIA/kubevirt-gpu-device-plugin#192](https://github.com/NVIDIA/kubevirt-gpu-device-plugin/pull/192), still open. Section 2 ships it downstream in the host layer, behind a boundary the rest of the design does not cross. ## Decisions +- [0017. The vGPU entitlement is a Tenant flag stamped on the namespace, not a budget key](./decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md) +- [0016. Holding a GPU for a stopped VM is step 2, and stays GPU-specific until DRA can carry it](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md) +- [0015. Node allocatable is the GPU inventory, and there is no `GPUNode`](./decisions/0015-node-allocatable-is-the-gpu-inventory.md) +- [0014. In step 1 a `gpus` entry names a class, budgets count declared entries, and usage is the virt-launcher pod's `Workload`](./decisions/0014-step-1-names-a-class-and-holds-nothing.md) +- [0013. The GPU kinds live in `cozystack.io`, with no group of their own](./decisions/0013-the-gpu-kinds-live-in-cozystack-io.md) - [0012. The VMI webhook only checks and records claims, and the controller places them while the launcher pod waits at a scheduling gate](./decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md) - [0011. A `GPUClaim` owns its usage record, and no monitor or reserved label guards it](./decisions/0011-a-gpuclaim-owns-its-usage-record.md) -- [0010. `GPUClass` carries the tolerations that binding and `free` read](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md) -- [0009. `GPUClass.mode` never changes, and a pool's `claimSelector` names its class](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md) -- [0008. The GPU controller gets its node-label and plugin-restart rights with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) -- [0007. Whoever can write a VM orders its template claims, while held claims stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md) +- [0010. `GPUClass` carries the tolerations that `free`, binding and the launcher pod read](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md) +- [0009. `GPUClass.mode` never changes, and a pool names its class](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md) +- [0008. The GPU controller gets its node-label right with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) +- [0007. Whoever can write a VM uses the GPUs of its class entries, while held claims stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md) - [0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md) -- [0005. The taint rule skips exactly the three taints that mirror a cordon or readiness](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md) -- [0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts](./decisions/0004-gpunode-health-is-kubelet-allocatability.md) -- [0002. GPUClaim works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md) -- [0001. A claim whose class resource name KubeVirt does not permit is refused at VMI admission](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md) +- [0005. The taint rule skips exactly the three taints that mirror a cordon or readiness](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md), superseded by 0010 +- [0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts](./decisions/0004-gpunode-health-is-kubelet-allocatability.md), superseded by 0015 +- [0002. GPU classes serve VMs only where gpu-operator serves GPUs to VMs, while claim finalizers and usage records always run](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md) +- [0001. A class whose resource name KubeVirt does not permit is refused at VMI admission](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md) ## Context ### What exists -The `cozystack.gpu-operator` package has three variants, selected by `bundles.iaas.gpuOperatorVariant` ([`docs/gpu-vgpu.md`](https://github.com/cozystack/cozystack/blob/main/docs/gpu-vgpu.md)). `default` binds whole GPUs to `vfio-pci` for VMs, `vgpu` runs the NVIDIA vGPU host driver, and `container` serves GPUs to pods. For the two VM variants the platform writes the `HostDevices` feature gate and, when it has entries, a `permittedHostDevices` table into the KubeVirt CR (`packages/core/platform/templates/bundles/iaas.yaml`). Passthrough GPUs are advertised by NVIDIA's sandbox device plugin. +The `cozystack.gpu-operator` package has three variants, selected by `bundles.iaas.gpuOperatorVariant` (Cozystack's [GPU guide](https://cozystack.io/docs/virtualization/vgpu/)). `default` binds whole GPUs to `vfio-pci` for VMs, `vgpu` runs the NVIDIA vGPU host driver, and `container` serves GPUs to pods. For the two VM variants the platform writes the `HostDevices` feature gate and, when it has entries, a `permittedHostDevices` table into the KubeVirt CR (`packages/core/platform/templates/bundles/iaas.yaml`). Passthrough GPUs are advertised by NVIDIA's sandbox device plugin. How NVIDIA's vGPU host driver exposes vGPUs depends on the GPU, the host OS and the driver branch. Older setups use mediated devices (mdev). SR-IOV GPUs on current releases, such as Ubuntu 24.04 with the vGPU 17 or newer branch that Cozystack's guide uses, get VFs, and each VF gets a profile when `current_vgpu_type` is written in its sysfs. KubeVirt v1.9, which Cozystack bundles, advertises such VFs through virt-handler's own device plugin ([kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890)), matched by PCI ID. Observed on L40S with SR-IOV vGPU: the VF shows `10de:26b9` in the guest, the PF's ID, and `permittedHostDevices` selects VFs by `10DE:26B9`. On GPU Operator nodes nothing in Cozystack writes VF profiles; the `cozystack-nvidia-vgpu-restore` boot unit from cozystack/ansible-cozystack#70 covers only hosts with a host-installed driver. @@ -45,39 +50,92 @@ Both consumers take GPUs the same way: `vm-instance` and `kubernetes-nodes` read ### The problem -- A tenant sees resource names, not offers. The dashboard's `gpu` option source lists the names KubeVirt permits; they say nothing about mode, profile or price. +- A tenant sees resource names, not offers. The dashboard's `gpu` option source lists the names KubeVirt permits and how many of each are available on which nodes; they say nothing about mode, profile or price. - A GPU cannot be held. When a VM stops, its GPU goes back to the pool and the next start can fail. - An order that does not fit is accepted, and the tenant learns it from pod events. - Nobody can tell which tenant uses a physical GPU without reading pods on the node. - In the `vgpu` variant a node labelled for passthrough can serve whole GPUs of another model, but a model whose whole GPU and VFs share a PCI ID, such as L40S, is offered either whole or as vGPU, not both, because two device plugins would both take that ID. The limit is per GPU model, not per cluster. - VF profiles on GPU Operator nodes are written by hand after every reboot. +Step 1 answers the first, fifth and sixth problems, the third for budgets and the fourth per tenant and class. Step 2 answers the second, and the third for capacity, since an order is then checked against free units. + ## Goals -- A hoster publishes a catalog of GPU classes, each with a model, a mode (passthrough or vGPU) and, for vGPU, a profile. Tenants read the catalog with free capacity and their remaining budget before ordering. -- A tenant orders a number of GPUs of a class within their quota and holds them, attached or not, until they release them. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden` (section 7). That is the end state, not the default: `forbidden` needs every GPU worker pool on claims instead of raw names, ComputePlane GPU groups included, which take only raw names today. Until then such a cluster keeps raw names, and the hold is not guaranteed there. An order that cannot be satisfied is refused with a message naming the class. -- VMs and worker pools attach held GPUs the same way. A restarted VM gets the same held claim back, and a replaced worker gets one of its pool's claims back unless the tenant's own VM or another pool uses it. -- Every physical GPU and VF can be traced to its node, its configuration and the tenant using it. -- Held GPUs appear in usage records (`Workload`) by tenant, class and time, which an external billing system reads. Cozystack charges nothing and stores no prices. -- One cluster can have passthrough and vGPU nodes of the same GPU model, with vGPU profiles applied declaratively and kept across reboots. -- Existing `gpus` entries keep working, with a documented path to `GPUClaim` for VMs and for worker pools on passthrough GPUs. +Step 1: + +- One cluster can have passthrough and vGPU nodes of the same GPU model, with vGPU profiles applied declaratively and kept across reboots, and every offer advertised under a resource name of its own. +- A hoster publishes a catalog of GPU classes, each with a model, a mode (passthrough or vGPU) and, for vGPU, a profile. Tenants read the catalog with free capacity and their remaining budget. +- VMs and worker pools name a class, not a resource name, and a tenant's budget per class and in total is checked when the VM or pool is written, with a message naming the class and the limit. +- Usage records (`Workload`) show which tenant runs how many GPUs of which offer in which VM or worker, which an external billing system reads. Cozystack charges nothing and stores no prices. +- Existing `gpus` entries keep working, with a documented path to class entries. + +Step 2: + +- A tenant holds a number of GPUs of a class within its budget, attached or not, until it releases them, and a restarted VM gets its held claims back. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden`, which needs every GPU consumer on class or claim entries; until then it is not guaranteed. - The catalog can later describe MIG, other vendors and DRA-backed devices without changing the tenant API. ### Non-goals - Prices and charging. They live in an external billing system, keyed by class name. -- The vGPU guest driver, the license server and license tokens. The hoster provides them (section 5). +- The vGPU guest driver, the license server and license tokens. The hoster provides them (section 7). - The catalog UI. This proposal provides the data a portal reads. - GPUs for pods inside tenant clusters (gpu-operator in a tenant cluster, HAMi). -- Time-slicing one claim between tenants, and live migration of GPU VMs. +- Time-slicing one GPU between tenants, and live migration of GPU VMs. +- A per-device inventory naming the physical GPU or VF behind each VM ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). ## Design -### 1. Object model +Sections 1 to 8 are step 1. Section 9 is step 2 and is the only place that adds fields, kinds or phases for holding. + +### 1. API group and names + +The GPU kinds join the existing `cozystack.io` group, which already holds the hoster-side `SchedulingClass` and `Package` and the tenant-namespace `Workload` and `WorkloadMonitor`; there is no `gpu.cozystack.io` group ([0013](./decisions/0013-the-gpu-kinds-live-in-cozystack-io.md)). Step 1 adds one kind, `GPUClass`; step 2 adds `GPUClaim`. + +Every other name follows the group: + +| Name | Step | Use | +| --- | --- | --- | +| `.gpuclass.cozystack.io/devices` | 1 | budget of one class in `resourceQuotas` (section 5) | +| `gpuclass.cozystack.io/devices` | 1 | budget over every class | +| `cozystack.io/gpu-reference` | 1 | label on a VMI template with class or claim entries, which KubeVirt copies to the VMI and its pods | +| `cozystack.io/gpus` | 1 | annotation on that template listing the entries | +| `cozystack.io/gpu--` | 1 | host-layer node label carrying a model's mode or profile (section 2) | +| `cozystack.io/gpu-layout-pending` | 1 | node label marking a pending layout change (section 2) | +| `namespace.cozystack.io/vgpu` | 1 | namespace label carrying the vGPU entitlement, like the other module labels (section 6) | +| `cozystack.io/gpuclass-protection` | 2 | finalizer on a class in use (section 9) | +| `cozystack.io/gpuclaim-protection` | 2 | finalizer on a claim a VMI uses | +| `cozystack.io/gpuclaim-template` | 2 | label marking a claim the controller made for a class entry | +| `cozystack.io/gpuclaims` | 2 | scheduling gate on a launcher pod | + +The quota keys follow DRA's `.deviceclass.resource.k8s.io/devices`, so a class key never collides with a resource name a device plugin advertises, and a key without a class prefix counts every class. + +### 2. Host layer + +The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ends at one resource name per offer: a whole L40S in passthrough and an L40S-24C vGPU are two names, and one name never stands for two offers. That is its whole contract with the API above it. `GPUClass`, quotas, the GPU webhooks and usage records use no plugin-specific name, discovery format or configuration. A different plugin, or a different version of the one chosen here, changes only this layer and the `resourceName` of affected classes. Tenants keep their class entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to class entries first. It ships first and alone, with no tenant API. + +The host layer does not depend on the host OS: it needs the vGPU host driver, `vfio-pci` and the device plugin, wherever they run. Which OSes Cozystack supports as hosts of the management cluster is decided outside this proposal. Today vGPU hosts run Ubuntu with `bundles.system.variant: isp-full-generic`, where the vGPU host driver and the patched plugin run. Cozystack's GPU guide does not recommend Talos for vGPU, so this layer does not cover Talos hosts; they keep passthrough through the `default` variant as today. NVIDIA AI Enterprise lists vGPU for Compute on Linux with KVM as supported only through specific hypervisor vendors and, on Ubuntu and RHEL with KVM, for single-node hosts only ([support matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/support/support-matrix-8/8.2.html)), so support for such a host is a matter between the hoster and NVIDIA. + +Until the layer ships, a cluster keeps the modes its variant gives it today, one per model for models whose whole GPU and VFs share a PCI ID, and vGPU classes keep the experimental status the GPU guide gives the `vgpu` variant: the hoster writes profiles and node labels by hand. On clusters without a layout, which includes the `default` variant and Talos hosts, node labels stay hand-written. Where one resource name still covers two offers, as with KubeVirt's built-in plugin, which gives every vGPU profile of a model one name, the hoster can publish only one class for that name (section 3). + +The device plugin is the decision reviewers should look at first. Cozystack ships NVIDIA's kubevirt-gpu-device-plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, as the sandbox device plugin of the `vgpu` variant, until NVIDIA/kubevirt-gpu-device-plugin#192 merges upstream or upstream takes another approach. NVIDIA/kubevirt-gpu-device-plugin#192 adds discovery of vGPU VFs on the vendor-specific VFIO framework and advertises them with one resource name per profile, next to the passthrough names the plugin already advertises for GPUs bound to `vfio-pci`. One plugin then serves both modes of every model under distinct names, so the per-model limit of the problem list goes away. The branch is validated on hardware. Its upstream review has had no maintainer response since 2026-07-10, which is why it ships as a downstream patch rather than waiting. It reads profile names from sysfs `creatable_vgpu_types` first and falls back to NVML on a card whose VFs all carry a profile, so the platform runs the plugin as its own DaemonSet with the host driver's `libnvidia-ml.so.1` and the `/dev/nvidia*` nodes mounted, privileged; the GPU Operator's DaemonSet has no field for those mounts. The patch leaves when NVIDIA/kubevirt-gpu-device-plugin#192 merges upstream, in this or another form, or when the NVIDIA DRA driver serves vGPU VFs to KubeVirt (kubernetes-sigs/dra-driver-nvidia-gpu#1402); a merged NVIDIA/kubevirt-gpu-device-plugin#192 changes only this layer, while a move to DRA adds new `dra` classes (section 10). KubeVirt's built-in plugin was the other candidate (Alternatives). + +The layer adds a node layout to the `vgpu` variant, in platform values. For each node selector it names a mode and, for vGPU, the profile of each GPU model. Nodes outside every selector are passthrough, through the `defaultWorkload: vm-passthrough` the layout renders. With a layout: + +- The platform runs the patched plugin and renders the KubeVirt `permittedHostDevices` entries for the names it advertises, as entries with an external provider. Other vendors' entries, such as an FPGA, stay as written. +- A node agent writes the VF profiles at boot and when the layout changes. The controller sets each node's workload label and one label per GPU model, `cozystack.io/gpu--` (for L40S `cozystack.io/gpu-10de-26b9`), to the profile or to `passthrough`. +- The plugin does not pick up VFs that gain a profile after it started, so the node agent restarts the plugin pod on its own node after it has written profiles there, for example after a reboot. Running VMs are not affected. + +The controller gets the right to patch those node labels, restricted by a policy to the GPU workload label and `cozystack.io/gpu-*`, with this layer and not before ([0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)). + +The layout and mdev GPUs coexist: GPUs on the mdev path keep today's configuration through `vgpu-device-manager`, and the agent writes profiles only on nodes that use VFs. New deployments of SR-IOV GPUs on current host releases use VFs. + +A layout change on a node, of mode or of profile, waits until the node has no GPU VMs. The controller labels the node `cozystack.io/gpu-layout-pending`, and running VMs keep their GPUs until the hoster cordons and drains the node; the cordon is what keeps new GPU VMs off it, since nothing in step 1 places a launcher pod. A GPU VM created before Cozystack switched to `LiveMigrateIfPossible` needs one restart before the drain can evict it. A node whose hand-written profiles already match the layout is not a change. A `vgpu` cluster adopts the layout with vGPU selectors that cover its existing vGPU nodes, and labels those nodes `vm-vgpu` before setting it, so the layout's passthrough default does not change them in between. Its vGPU classes, raw `name` entries and raw quota keys move from the hand-written VF names to the plugin's per-profile names in one maintenance window, or the raw-name consumers move to class entries first. KubeVirt's built-in plugin and the patched plugin report different device IDs for the same VF, and kubelet would otherwise hand out VFs in use, so affected nodes are cordoned and free of GPU VMs until the patched plugin has registered there. + +### 3. GPU classes ```yaml # Hoster. One per offer. Cluster-scoped, readable by every authenticated user. -apiVersion: gpu.cozystack.io/v1alpha1 +apiVersion: cozystack.io/v1alpha1 kind: GPUClass metadata: name: l40s-24c @@ -85,14 +143,11 @@ spec: displayName: "NVIDIA L40S, 24 GB vGPU (compute)" vendor: nvidia model: L40S - mode: vgpu # passthrough | vgpu; never changes + mode: vgpu # passthrough | vgpu profile: L40S-24C # vgpu only memory: 24Gi maxCount: 2 # vGPUs of this class per VM - onCordon: Move # Move | Stay; can change - nodeSelector: # nodes carrying this profile - gpu.cozystack.io/10de-26b9: L40S-24C - tolerations: # taints of these nodes the class's VMs tolerate; can change + tolerations: # taints of this offer's nodes that its VMs tolerate - key: dedicated value: gpu effect: NoSchedule @@ -102,9 +157,105 @@ spec: status: free: 12 maxFreePerNode: 4 # most free units on one node +``` + +A class maps an offer to a backend through a `source` union. The first release has one source, `devicePlugin`; `dra` is reserved for section 10. A class is one resource name, and two classes cannot have the same one: a class whose `resourceName` another class already has gets a `ResourceNameConflict` condition and is not offered, so a budget on one class cannot be sidestepped through another, and a usage record's resource name leads to exactly one class. The nodes of a class are the nodes that advertise its resource name. Class names are DNS labels, since they become part of quota keys. + +`vendor`, `model`, `mode`, `profile`, `memory` and the kind of `source` never change, so a class name stands for one offer for the class's lifetime, and whatever names a class, a VM, a worker pool or a billing system, knows the offer from the name ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A different offer is a new class, and a hoster does not reuse a deleted class's name for another offer, since tenants' budgets and billing records are keyed by it. `displayName`, `maxCount`, `resourceName` and `tolerations` can change: `resourceName` for changes in the host layer, `tolerations` when the hoster retaints its GPU nodes. A change applies to VMs started after it; a running VM keeps the resource name it started with, and its launcher pod the tolerations. Deleting a class refuses the next start of every VM that names it and leaves running VMs alone. + +How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 2), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPUs of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. + +`tolerations` lists the taints of the class's nodes that its VMs tolerate, such as a taint that reserves GPU nodes for GPU workloads, in the shape of a pod's with the operators `Exists` and `Equal` only. The launcher pod of a VM that names the class gets them (section 4), and `free` reads them ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). + +`status.free` is the class's allocatable count summed over its nodes that are `Ready`, not cordoned, and have no `NoSchedule` or `NoExecute` taint the class does not tolerate, minus what pods on those nodes request of the resource name, and no node counts below zero; `status.maxFreePerNode` is the largest such figure on one node. A class whose `resourceName` the KubeVirt `permittedHostDevices` table leaves out gets a `ResourceNameNotPermitted` condition, so the hoster sees the gap before a tenant's VM is refused ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included ([0005](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md)). Both figures come from the `Node` and `Pod` objects alone, with no inventory kind ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). In step 1 they describe the moment, not a promise: nothing is held, so a unit counted free can be taken by the next VM to start. + +### 4. Class entries in VMs and worker pools + +VMs and worker pools name a class in `gpus`: + +```yaml +# vm-instance +gpus: +- className: l40s-24c + count: 2 +--- +# kubernetes-nodes: every worker of the pool gets the entry +gpus: +- className: h100-pt + count: 1 +``` + +A `vm-instance` or `kubernetes-nodes` entry has exactly one of `name` or `className`, and a list holds only `name` entries or only class entries. `count` defaults to 1. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` and `kubernetes-nodes` refuse class entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to class entries is separate work. + +The chart renders only the entries, on the VMI template, as the `cozystack.io/gpus` annotation next to the `cozystack.io/gpu-reference` label, which KubeVirt copies from the VMI to its pods. A mutating webhook on VirtualMachineInstance creation, which sees only VMIs carrying the label, resolves each entry to its class and writes one `gpus` entry per GPU unit with the class's resource name. These entries have to be complete at admission: KubeVirt renders the launcher pod's resource requests from the VMI, and nothing but KubeVirt itself can change a VMI's spec afterwards. The webhook refuses to start the VMI, and names the class, when an entry: + +- names a class that does not exist, is being deleted, or has a `ResourceNameConflict` condition; +- is a vGPU class next to another vGPU class, or brings the VMI's GPUs of its class above `maxCount`; +- names a vGPU class in a namespace without the vGPU entitlement (section 6); +- names a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` with no pod for good ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). + +A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are always the union of three sets: the hoster's list `gpu.resourceNames`, empty unless the hoster sets it; the resource names of `GPUClass` objects; and the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. None of them replaces another, so a vfio GPU missing from the table and a name that only pods request, which the hoster lists, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU class entry for GPUDirect RDMA. The webhook also refuses a labelled VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A second mutating webhook, on the creation of a virt-launcher pod that carries the reference label, adds the classes' `tolerations` to the pod, and to the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on a node the VM can use. They go on the pod and not on the VMI, because with `vmRolloutStrategy: LiveUpdate`, which Cozystack sets, KubeVirt v1.9.0 replaces a VMI's tolerations with the VM template's on every sync ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go), `handleTolerationsChangeRequest`). Both webhooks fail closed, so their outage blocks the start of VMs with class entries and no other VM. The VMI webhook reads the classes and the KubeVirt CR from the API server, not from a cache, since the controller's replicas serve admission and a lagging cache could admit what should be refused. Neither webhook writes any object besides its own patch, so in step 1 both declare `sideEffects: None`. + +Capacity is not checked at admission in step 1. A started VM whose class has no free unit on a node it can use waits in `Scheduling`, its launcher pod `Pending` with `Unschedulable` and the VM shown as `ErrorUnschedulable`, as a VM short of CPU or memory does today, and starts when a unit frees up; the catalog's `free` and `maxFreePerNode` tell a tenant beforehand whether that is likely. Refusing the VMI instead would leave the VM to the per-item backoff of KubeVirt's VM controller, which grows to 1000 seconds, for a shortage that may last seconds. + +A pool states no mode of its own: its entry names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a class entry that names a vGPU class on a Talos pool, before any machine is created. Worker pools on another OS, such as the Kairos pools of `kairos-vgpu-workers`, take vGPU classes as that proposal describes. Talos GPU pools on `name` entries keep working as today. + +### 5. Budgets + +A tenant's GPU budget is declared in `resourceQuotas`, per class as `.gpuclass.cozystack.io/devices: ` and over every class as `gpuclass.cozystack.io/devices: `, each vGPU counting as one. These keys are not resource names a pod requests, so they are not rendered into the namespace `ResourceQuota`, and a tenant with only GPU keys gets no `ResourceQuota` or `LimitRange`. That is a change to the tenant chart, which today renders both for any `resourceQuotas` key. + +GPU keys follow the same hierarchical pool rules as every other key, and #48 is what checks them, so tenants learn one rule and do not migrate when #48 ships: a `vm-instance` reserves `count` units of its class for each class entry, a `kubernetes-nodes` pool reserves `count` times `maxReplicas`, both evaluated from the defaulted values, and each also reserves the sum under the key without a class prefix. A write that would take a pool over a declared GPU key is refused when it is made, with a message naming the pool, the key, the limit and what is already reserved. A stopped VM keeps counting, as its CPU and memory do under #48: the budget limits what is declared, and the usage records (section 7) show what runs. A tenant with no GPU key anywhere above it is limited only by capacity, and for vGPU classes by its entitlement (section 6). + +Until #48's gate is on, an interim check in the Cozystack API server does this for GPU keys only. On a create or update of a `VMInstance` or a `KubernetesNodes`, it sums the class entries of every `VMInstance` and `KubernetesNodes` release in the pool, as above, with the object under admission at its new size, and refuses the write when a GPU key of the pool would be exceeded. On an update only the delta counts, so shrinking and no-op edits always pass. It reads the tenants' declared `resourceQuotas` and the applications' defaulted values, the inputs #48 reads, and applies the same pool arithmetic, so the switch to #48 changes no verdict. Like #48's gate it is a read-check-write without a transaction, so two writes at the same moment can both pass, and nothing evicts the excess. The interim check stands aside while #48's gate is switched on, and is deleted in the release where that gate is on by default, which needs `VMInstance` and `KubernetesNodes` to declare their class entries in their consumption. + +The tenant declaration gate validates GPU keys like any other: a child's declared GPU key may not exceed what its parent's pool has left. Today the gate reads what a pool uses from the `ResourceQuota`, which has no GPU keys, so until #48 the interim check also runs on Tenant writes and supplies the pool's class entries as its GPU usage. + +Pool membership follows the existing rules too, which #48 keeps. Until #48 ships, the runtime pool controller reads the rendered `ResourceQuota`, which has no GPU keys, so a tenant whose quotas hold only GPU keys is a pool root for the interim check and not for that controller; #48 removes the difference by reading declared values on both sides. Existing raw GPU keys, quota keys on a resource name such as `nvidia.com/AD102GL_L40S`, keep their meaning: they render into the `ResourceQuota` and limit launcher pods when they are created, whether the VM names the resource or a class of it (section 8). + +### 6. vGPU entitlement + +A vGPU class is used only by a tenant entitled to vGPU ([0017](./decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md)). The Tenant field `gpu.vgpu` grants it. The parent tenant's administrator sets it, or the hoster on a top-level tenant. A tenant is entitled only when the flag is set on it and its parent is entitled. The root tenant is rendered by the platform, not by the tenant chart, so its value comes from platform values. The flag is off by default everywhere, so the hoster turns it on for the root first. No tenant can grant itself vGPU, since its Tenant object lives in its parent's namespace. The tenant chart computes the effective value from the parent's, like its other namespace values, and stamps it on the namespace as the label `namespace.cozystack.io/vgpu`, the same pattern as `namespace.cozystack.io/etcd` and the other module labels; a change reaches grandchildren as each level re-renders. The chart always writes the label, so only the value `"true"` means entitled; any other value, or no label, means not entitled. + +Entitlement is separate from budget because a vGPU license is counted per unit, which is a budget, while getting the vGPU guest driver and the license token at all is yes or no. The GPU keys of section 5 then limit vGPU like any other GPU. The VMI webhook refuses a vGPU class entry without the entitlement at every start (section 4), so clearing the flag stops VMs on vGPU classes from starting again while running ones keep running, and the tenant's Kairos pools lose their vGPU entitlement as `kairos-vgpu-workers` describes. A write that names a vGPU class is not refused for the entitlement, since the pool gate of section 5 counts sizes and has no yes-or-no rule; the option source offers a vGPU class to an unentitled tenant with a remaining budget of 0 (section 7). Raw vGPU names in VMs stay as open as today until a raw key limits them or step 2's `forbidden` is set. + +### 7. Usage records, catalog and licensing + +Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads, and step 1 adds no record of its own. The `WorkloadMonitor` of a VM already sums the requests of its virt-launcher pod into a `Workload` per pod, extended resources included, and KubeVirt v1.9.0 puts one unit of the device's resource name into the pod's requests per GPU entry ([`renderresources.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/renderresources.go)). A VM on a class entry therefore shows its GPUs there under the class's resource name, for exactly as long as the pod exists, next to its CPU and memory. A worker pool's monitor selects its workers' launcher pods, so they are recorded the same way. The record names the VM's pod, not the node or the device ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). Since a resource name belongs to one class at a time (section 3), a billing system maps a record to a class through the catalog. A class whose `resourceName` changes appears under the new name from the next start of each VM, so a billing system keeps the names it has seen per class. Like other `Workload` records they have no time fields: a used interval is the pod's lifetime, which a billing system observes by watching and reconciles by sampling. + +The catalog is the list of `GPUClass` objects with `displayName`, model, mode, profile, memory, `maxCount`, `status.free` and `status.maxFreePerNode`. Every authenticated user reads it, because a hoster publishes its offers and a tenant names a class in a VM, including from `kubectl` or GitOps without a portal. So the status holds only `free` and `maxFreePerNode`, and its conditions name no nodes and carry no counts. The class's capacity and used units show the hoster's inventory and load, and they are not in the status; the controller exports them as metrics that platform monitoring scrapes. A tenant can still watch `free` and `maxFreePerNode` over time and see how busy a class is. + +The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest entry, leaving out a field that is absent because it sets no limit. The remaining budget is the pool figure of section 5, so a tenant under a shared parent budget can see how many GPUs its siblings declare, as a number, without names. This is accepted: the alternative is a budget the tenant can only discover by a refused write, which is the failure this proposal removes. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. The existing `gpu` source keeps serving raw names while they are allowed. + +vGPU needs a license, and passthrough needs one only for graphics. Compute (C-series) profiles are licensed through NVIDIA AI Enterprise, graphics (Q-series) profiles through NVIDIA vGPU software. The hoster provides the license server, the token and the guest drivers. In worker pools `kairos-vgpu-workers` delivers the token and the guest license settings. In VMs the hoster does, through its image or at boot, as `vgpu-vms` describes. The class's `mode` tells a portal and a tenant which case applies. + +### 8. Migration of existing `gpus` + +Raw `name` entries keep working, and nothing converts them automatically. A tenant moves a VM by replacing each `name` entry with a class entry of the matching class and restarting it; the VM requests the same resource name as before, so the move needs no free unit. A Talos GPU pool on a passthrough name moves by replacing its `name` entries with a class entry of the matching passthrough class, which rolls its workers. A Talos pool on a raw vGPU name has no path here, since Talos workers carry no vGPU guest driver; it keeps its raw name until it moves to a worker OS that does. ComputePlane GPU groups have no path until the module accepts class entries. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes class entries, and a pool on one takes the pool path above. + +Quota keys move too. A class entry's launcher pod requests the class's resource name, so while a tenant migrates, a raw key on that name limits both kinds of VM at pod creation, and a GPU key of section 5 limits only class entries, at write. After its last raw-name VM has moved, the raw key is replaced by GPU keys. A tenant whose raw-name VMs use vGPU can move them to class entries only after it is entitled to vGPU (section 6). + +Raw names stay allowed in step 1. Forbidding them is part of step 2, where it is what makes a hold hard (section 9). + +### 9. Step 2: holding GPUs + +Step 2 lets a tenant keep GPUs on a node while its VM is stopped, so the next start does not fail for a GPU someone else took. Holding stays GPU-specific and does not become a generic reservation ([0016](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md)): CPU and memory are interchangeable across nodes, while a GPU hold is a whole unit on a specific node that does not migrate, which is why it needs a binder, `Lost`, the gather and `onCordon`. DRA would be the general path, but KubeVirt [VEP 344](https://github.com/kubevirt/enhancements/issues/344) (VM-scoped persistent ResourceClaims) is open and unimplemented, and the vGPU VF is handed out by a device plugin (section 2), not by a DRA driver. Holding moves to DRA once VEP 344 is implemented and a DRA driver hands out vGPU VFs. + +Nothing in this section exists in step 1. Step 2 adds `GPUClaim`, the `onCordon` field and a finalizer on `GPUClass`, `claim` entries in `vm-instance`, `claimSelector` entries in `kubernetes-nodes`, the binder, the launcher pod gate and webhook, the claim order webhook, claim usage records and `gpu.rawResourceNames: forbidden`. + +#### Objects + +```yaml +# Hoster. Step 2 adds onCordon to GPUClass. +apiVersion: cozystack.io/v1alpha1 +kind: GPUClass +metadata: + name: l40s-24c +spec: + # ... the step 1 fields + onCordon: Move # Move | Stay; can change --- # Tenant. One per GPU or per set of GPUs used together on one node. -apiVersion: gpu.cozystack.io/v1alpha1 +apiVersion: cozystack.io/v1alpha1 kind: GPUClaim metadata: name: train-0 @@ -118,49 +269,26 @@ status: attachedTo: vmi: vm-instance-trainer boundAt: "2026-10-14T09:00:00Z" ---- -# Inventory. One per GPU node. Cluster-scoped, admin only. -apiVersion: gpu.cozystack.io/v1alpha1 -kind: GPUNode -metadata: - name: gpu-node-3 -status: - devices: - - pciAddress: "0000:41:00.0" - model: L40S - function: pf # pf | vf | whole - vfs: 16 - - pciAddress: "0000:41:00.4" - function: vf - profile: L40S-24C - healthy: true - allocatable: 1 # N for an mdev parent - usedBy: {namespace: tenant-acme, pod: virt-launcher-vm-instance-trainer-x7k2p} - resourceNames: [nvidia.com/NVIDIA_L40S-24C] ``` -A class maps an offer to a backend through a `source` union and to the nodes that serve it through `nodeSelector`. The first release has one source, `devicePlugin`; `dra` is reserved for section 8. Two classes may share a resource name when their node selectors do not overlap, which lets a host layer that cannot tell two offers apart by name tell them apart by node. A node that matches two classes with the same resource name gets a condition on both classes and on its `GPUNode`, which names the two classes, and is not used for binding. Class names are DNS labels, since they become part of quota keys. - -A class referenced by held claims cannot disappear or change its meaning. A finalizer keeps a deleted class until its last `Bound`, `Attached` or `Lost` claim is gone, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `vendor`, `model`, `profile`, `memory`, `maxCount` and the kind of `source` cannot change while the finalizer is present. `mode` never changes, held claims or not, so a class name stands for one mode for the class's lifetime, and whatever names a class, such as a worker pool's selector, knows the mode from the name ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). `resourceName`, `nodeSelector` and `tolerations` can change, for changes in the host layer (section 6) and for moving the offer to other nodes. `tolerations` lists the taints of the class's nodes that its VMs tolerate, such as a taint that reserves GPU nodes for GPU workloads; a claim is bound before any VM exists, so the class is the only place the binder can read them from, and the virt-launcher pod of a VMI that uses the class gets them when it is created (section 3). - -A `GPUClaim` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate claims. How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 6), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPU claims of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. Admission refuses a claim above `maxCount`. The spec is immutable, so a change is a new order. +A class is in use while any claim of it is `Bound`, `Attached` or `Lost`, and a class in use cannot disappear. A finalizer, `cozystack.io/gpuclass-protection`, keeps a deleted class while it is in use, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `maxCount` cannot change while the class is in use. -The inventory is written by a node agent from sysfs and from kubelet's pod-resources API, and goes away with its `Node`. That API carries no device health, so a function is healthy when kubelet lists its devices as allocatable, each function reports how many of them are, one for a VF or a whole GPU and several for an mdev parent, and `resourceNames` lists the node's GPU resource names, including those whose devices the agent cannot place under a PCI function ([0004](./decisions/0004-gpunode-health-is-kubelet-allocatability.md)). Ownership is exact for running GPUs: a `usedBy` pod leads to its VM, and the VM's claims name it in `status.attachedTo`; an mdev parent that serves several pods names one of them. A held but not running reservation is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. PCI addresses stay in `GPUNode`, which tenants cannot read. +A `GPUClaim` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate claims. Admission refuses a claim above the class's `maxCount`. The spec is immutable, so a change is a new order. A held claim is a `GPUClaim` a tenant orders and keeps, as opposed to a template claim (Attaching held claims); it holds GPUs while it is `Bound` or `Attached` (the phase table). A `Bound` claim that no VMI uses is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. What `Bound` guarantees depends on the class's source: | Source | A `Bound` claim holds | The physical device is picked by | | --- | --- | --- | | `devicePlugin` | `count` free units of the class on one node | kubelet, when the pod starts | -| `dra` (section 8) | one `ResourceClaim` within budget, and its devices once the claim is allocated at the first start | the scheduler, at allocation | +| `dra` (section 10) | one `ResourceClaim` within budget, and its devices once the claim is allocated at the first start | the scheduler, at allocation | A tenant sees a class, a count, a phase and, once there is one, a node, never which card it holds, so its API stays the same when an offer moves to a class of the other source. -### 2. Binding and capacity +#### Binding and capacity -A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node that matches its class and has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included. A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI start (section 3): it may take a claim attached only to the starting VMI off a node that is not `Ready`, because no other VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. +A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node of its class that has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, by the taint rule of section 3, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI start (Placing a VMI's claims): it may take a claim attached only to the starting VMI off a node that is not `Ready`, because no other VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. -Admission refuses an order that cannot be met at that moment, by capacity or by budget (section 4), with a message naming the class, the count and the limit it hit: a budget of the tenant or an ancestor, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. +Admission refuses an order that cannot be met at that moment, by capacity or by budget (Budgets and usage records of held claims), with a message naming the class, the count and the limit it hit: a budget of the pool, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. The phases decide what a claim holds and costs: @@ -171,24 +299,24 @@ The phases decide what a claim holds and costs: | `Attached` | yes, used by a VMI | yes | yes | | `Lost` | no | yes | no | -A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 8). +A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 10). A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: under `Always` and `RerunOnFailure` the VM controller deletes a final VMI before it starts the next one, under `Manual` a start request first stops the final VMI and then starts a new one, and under `Once` no next VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. -A claim keeps its node while the `Node` exists, matches the class, has not lost capacity, has no pending layout change (section 6) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer matches the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules of section 3. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set (section 3). +A claim keeps its node while the `Node` exists, serves the class, has not lost capacity, has no pending layout change (section 2) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer serves the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules below. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set. -A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity when the agent reports a device missing or unhealthy for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no claim. Then: +A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity for a class when its allocatable count of the class's resource stays below the units held there for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no claim ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). Then: -- the node and the affected classes get a `LostCapacity` condition; +- the affected classes get a `LostCapacity` condition; - unattached claims that no longer fit move to another node of their class, or to `Lost` when none has room; - on a `NotReady` node, attached claims go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI start is moving off the node is the exception: it finishes the move; -- a VM started after the condition is set has its claims gathered off the node before its pod is released (section 3), and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. What the VM does next follows its `runStrategy` for a deleted VMI: `Always` and `Once` create a new VMI and start once the claim has moved or been rebound from `Lost`, while `Manual` and `RerunOnFailure` stay stopped until the tenant starts the VM (Failure cases); unlike a cordon, the node cannot serve the claim, so waiting is no option. +- a VM started after the condition is set has its claims gathered off the node before its pod is released, and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. What the VM does next follows its `runStrategy` for a deleted VMI: `Always` and `Once` create a new VMI and start once the claim has moved or been rebound from `Lost`, while `Manual` and `RerunOnFailure` stay stopped until the tenant starts the VM (Failure cases); unlike a cordon, the node cannot serve the claim, so waiting is no option. Claims in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a claim whose VMI is still live on the lost node waits until that VMI is final or removed (above), while a claim of a VMI whose pod is still gated is rebound for that VMI. The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` claim already counts. -### 3. Attaching +#### Attaching held claims -VMs and worker pools reference claims in `gpus`: +VMs and worker pools reference held claims in `gpus`: ```yaml # vm-instance @@ -203,202 +331,213 @@ gpus: pool: inference ``` -The chart renders only the reference, on the VMI template, with the reference label `gpu.cozystack.io/claim-reference`, which KubeVirt copies from the VMI to its pods. A mutating webhook on VirtualMachineInstance creation resolves the references and checks what needs no placement; it binds, moves and creates no claim, and waits for nothing. For a VM it takes the named claims. For a pool worker it takes the claim the worker held before, unless a VM now names it, its node is not `Ready`, or its node has a taint the class does not tolerate (section 2). Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook runs the refusal checks below and records the attachment of each held claim to this VMI, so two VMIs can never hold one claim and no background move can take one. It writes one `gpus` entry per GPU unit, held or from a template, with the class's resource name. These entries have to be complete at admission: KubeVirt renders the launcher pod's resource requests from the VMI, and nothing but KubeVirt itself can change a VMI's spec afterwards, so the number of template claims is fixed when the VMI is admitted. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. The GPU controller places the claims after admission, while the launcher pod waits behind a scheduling gate ([Placing a VMI's claims](#placing-a-vmis-claims)). +A `vm-instance` entry has exactly one of `name`, `className` or `claim`, and a `kubernetes-nodes` entry exactly one of `name`, `className` or `claimSelector`. A list holds only `name` entries or only class and claim entries, a pool has at most one `claimSelector` and never next to a class entry, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The rules of section 4 for Talos pools, the vGPU entitlement and `permittedHostDevices` apply to the class of a claim as to a class entry. -The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim: +The VMI webhook of section 4 also resolves claim references, runs the checks that need no placement, and records the attachments; it binds, moves and creates no claim, and waits for nothing. For a VM it takes the named claims. For a pool worker it takes the claim the worker held before, unless a VM now names it, its node is not `Ready`, or its node has a taint the class does not tolerate. Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment of each held claim to this VMI, so two VMIs can never hold one claim and no background move can take one, and writes one `gpus` entry per GPU unit, held or from a class entry, with the class's resource name. The launcher pod webhook of section 4 adds the tolerations of the held claims' classes as it does for class entries. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. On a dry run the webhook records nothing, so in step 2 it declares `sideEffects: NoneOnDryRun`. The GPU controller places the claims after admission, while the launcher pod waits behind a scheduling gate (Placing a VMI's claims). -- does not exist, is being deleted, or is attached to another VMI; -- is a vGPU claim next to vGPU claims of another class, or brings the VMI's GPUs of its class above `maxCount`; -- is of a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). +The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim does not exist, is being deleted, or is attached to another VMI, besides the refusals of section 4 for its class. A refused VMI leaves its claims unattached, so they move by the rules above. A claim that is `Pending` or `Lost`, or bound to a node that cannot serve it, is not refused: the controller binds or moves it before it releases the pod. -A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are always the union of four sets: the hoster's list `gpu.resourceNames`, empty unless the hoster sets it; the resource names of `GPUClass` objects; the names the node agent reports in `GPUNode` `status.resourceNames`, which include every name advertised by a GPU device plugin the platform deploys, the sandbox or patched plugin and the GPU Operator's device plugin; and the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. None of them replaces another, so a vfio GPU missing from the table and a name that only pods request, such as `nvidia.com/gpu` of the `container` variant, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU claim for GPUDirect RDMA. The webhook also refuses a referencing VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A refused VMI leaves its claims unattached, so they move by the rules of section 2. A claim that is `Pending` or `Lost`, or bound to a node that cannot serve it, is not refused: the controller binds or moves it before it releases the pod. The VMI and launcher pod webhooks see only VMIs and pods carrying the reference label, next to the `gpu.cozystack.io/claims` annotation that lists the references, and fail closed, so their outage blocks the start of VMs with claim references and no other VM or pod; the third GPU webhook, on orders, is described under Security. +A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and usage record, under the `cozystack.io/gpuclaim-protection` finalizer until no VMI uses it, and is not attached again. -A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and usage record, under the `gpu.cozystack.io/claim-protection` finalizer until no VMI uses it, and is not attached again. +Once holding exists, a class entry needs a place on a node too, or a VM that names only a class could start on a unit another tenant holds. Once the launcher pod waits at its gate, the controller creates a `GPUClaim` from each class entry, named after the VMI and the entry's index, marked by the `cozystack.io/gpuclaim-template` label, and owned by the VMI. A claim of that name that carries the mark and is owned by this VMI is reused, so a repeated pass creates nothing twice; one still being deleted, or still owned by the VM's previous VMI, is waited for. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name is never taken, and the pod stays gated with a condition naming it. The controller binds these template claims under the same capacity checks as any order, on the node of the gathered set when the VMI has held claims, and otherwise on one node with room for all of them, and attaches them. It deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. A template claim counts against no budget of its own, since its class entry already counts (section 5), and it is not in the usage records, since its launcher pod is. A class entry counts as a claim reference under `forbidden`, and a pool on class entries pays for running workers only. -A `claimTemplate` entry names a class and a count instead of a held claim. At VMI creation the webhook checks it against the class, `maxCount` and the vGPU entitlement, and refuses the VMI when they do not allow it. Once the launcher pod waits at its gate, the controller creates a `GPUClaim` from the entry, named after the VMI and the entry's index, marked as a template claim by the `gpu.cozystack.io/template-claim` label, and owned by the VMI. A claim of that name that carries the mark and is owned by this VMI is reused, so a repeated pass creates nothing twice; one still being deleted, or still owned by the VM's previous VMI, is waited for. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name is never taken, and the pod stays gated with a condition naming it. The controller binds the template claims under the same capacity and budget checks as any order, on the node of the gathered set when the VMI has held claims, and otherwise on one node with room for all of them, and attaches them. On a dry run the webhook records nothing, so it declares `sideEffects: NoneOnDryRun`. The controller deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives, and is in the usage records, only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. Budgets, the vGPU entitlement (section 4), usage records and the refusals above apply to it as to any claim, but only while the VMI exists: an order that does not fit keeps the VM or worker waiting at start, not refused ahead of time. A template counts as a claim reference under `forbidden`. It lets an autoscaled pool pay for running workers only. - -A `vm-instance` entry has exactly one of `name`, `claim` or `claimTemplate`, and a `kubernetes-nodes` entry exactly one of `name`, `claimSelector` or `claimTemplate`. A list holds only `name` entries or only references, and a pool has at most one `claimSelector` or one `claimTemplate`, never both, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` refuses `claim` and `claimTemplate` entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to claims is separate work. - -A pool states no mode of its own: its `claimSelector` or `claimTemplate` names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class; a hoster does not reuse a deleted class's name for the other mode ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes claims of passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a `claimSelector` or `claimTemplate` that names a vGPU class on a Talos pool, before any machine is created. Worker pools on another OS, such as the Kairos pools of `kairos-vgpu-workers`, take vGPU classes as that proposal describes. A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. Talos GPU pools on `name` entries keep working as today. +A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. #### Placing a VMI's claims -A launcher pod waits for its GPUs behind a scheduling gate, so nothing is refused for not being ready yet. A mutating webhook on the creation of a virt-launcher pod adds the gate `gpu.cozystack.io/claims` and the `tolerations` of the classes of the VMI's claims. It matches only pods labelled `kubevirt.io: virt-launcher` that carry the reference label, adds the gate only when it is not there yet, since a repeated call would otherwise fail on a duplicate gate, and fails closed: a pod that reached the scheduler ungated would run unpinned. It also gates the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on the node of the claims. Migration target pods are left alone: KubeVirt deletes a target pod that stays pending for 15 minutes, a VM with a passthrough GPU or a VF does not live-migrate, and Cozystack does not enable the `VGPULiveMigration` gate that mdev vGPUs would need. A set that already has a `WaitForFirstConsumer` volume bound on a node stays on that node: the controller places it only where the node affinity of the VMI's bound volumes allows. KubeVirt never touches scheduling gates and has no timeout for a VMI whose pod waits, so the VMI stays in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated` ([`lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go)), and the VM's own start backoff never begins. +A launcher pod waits for its GPUs behind a scheduling gate, so nothing is refused for not being ready yet ([0012](./decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md)). The launcher pod webhook of section 4 also adds the gate `cozystack.io/gpuclaims` in step 2. It matches only pods labelled `kubevirt.io: virt-launcher` that carry the reference label, adds the gate only when it is not there yet, since a repeated call would otherwise fail on a duplicate gate, and fails closed: a pod that reached the scheduler ungated would run unpinned. It also gates the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on the node of the claims. Migration target pods are left alone: KubeVirt deletes a target pod that stays pending for 15 minutes, a VM with a passthrough GPU or a VF does not live-migrate, and Cozystack does not enable the `VGPULiveMigration` gate that mdev vGPUs would need. A set that already has a `WaitForFirstConsumer` volume bound on a node stays on that node: the controller places it only where the node affinity of the VMI's bound volumes allows. KubeVirt never touches scheduling gates and has no timeout for a VMI whose pod waits, so the VMI stays in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated` ([`lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go)), and the VM's own start backoff never begins. -For a gated pod the controller creates the VMI's template claims under the usual budget and capacity checks, then brings the set onto one node. Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are `Pending` or `Lost`, on different nodes, on a node that is gone, no longer matches a class, has lost capacity or has a pending layout change, on a node that is not `Ready`, on a node with a taint one of their classes does not tolerate (section 2), on a cordoned node under `Move`, or on a node with no room for the VMI's template claims, the controller binds or moves them onto one `Ready` node that is not cordoned, has no pending layout change, is tolerated by every class of the set and has room for all of them, template claims included. Every held claim of the set is attached to this VMI by then, so no other VMI or background move can take one, and each claim keeps its hold while it moves. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims, there is no gather, even if the node is also not `Ready`: the pod is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). +For a gated pod the controller creates the VMI's template claims, then brings the set onto one node. Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are `Pending` or `Lost`, on different nodes, on a node that is gone, no longer serves a class, has lost capacity or has a pending layout change, on a node that is not `Ready`, on a node with a taint one of their classes does not tolerate, on a cordoned node under `Move`, or on a node with no room for the VMI's template claims, the controller binds or moves them onto one `Ready` node that is not cordoned, has no pending layout change, is tolerated by every class of the set and has room for all of them, template claims included. Every held claim of the set is attached to this VMI by then, so no other VMI or background move can take one, and each claim keeps its hold while it moves. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims, there is no gather, even if the node is also not `Ready`: the pod is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). Once the set is on one node, the controller pins the pod and removes the gate in one update. The pin is a `matchFields` requirement on `metadata.name` appended to every required node selector term of the pod, or the single term when it has none. Kubernetes lets a gated pod only narrow its scheduling, so a term cannot be added next to existing ones, and KubeVirt usually sets a required term already, for host-model CPUs. The pin is not `nodeName`, which would let kubelet start the pod on a cordoned node, and it goes on the pod rather than the VMI because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. -When the set cannot be placed, because no node has room, a budget is exhausted, or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. The VMI gets no condition because KubeVirt's admission guards writes to a VMI. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules (below and section 2). After the deletion the VMI becomes `Failed`, and the VM's `runStrategy` decides what follows: `Always` and `RerunOnFailure` start again after KubeVirt's start-failure backoff, with the VM shown as `CrashLoopBackOff`; a `Once` VM keeps its failed VMI, so it stays down and also shows `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it (Failure cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. A set of template claims only loses its claims with the failed VMI, so for it the controller keeps the record in memory against the VM, and a controller restart forgets it. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. +When the set cannot be placed, because no node has room or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. The VMI gets no condition because KubeVirt's admission guards writes to a VMI. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules. After the deletion the VMI becomes `Failed`, and the VM's `runStrategy` decides what follows: `Always` and `RerunOnFailure` start again after KubeVirt's start-failure backoff, with the VM shown as `CrashLoopBackOff`; a `Once` VM keeps its failed VMI, so it stays down and also shows `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it (Failure cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. A set of template claims only loses its claims with the failed VMI, so for it the controller keeps the record in memory against the VM, and a controller restart forgets it. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way while the pod is gated. Once the pod is pinned, under `Move`, for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). -### 4. Quotas - -A tenant's GPU budget is declared in `resourceQuotas`, per class as `gpu.cozystack.io/: ` and in total as `gpu.cozystack.io/total: `; the class name `total` is reserved. These keys are not rendered into the namespace `ResourceQuota`, since `GPUClaim` is not a pod, and a tenant with only GPU keys gets no `ResourceQuota` or `LimitRange`. That is a change to the tenant chart, which today renders both for any `resourceQuotas` key. - -GPU budgets follow their own rule, not the pool and carve-out rules for other resources: every GPU key caps the whole subtree of the tenant that declares it. GPU keys stay out of the pool and carve-out model, so a GPU budget covers every descendant of the tenant that declares it. The tenant declaration gate skips `gpu.cozystack.io/*` keys, because under this rule a child's key may overlap its siblings'. - -- An order is refused if, for the tenant or any ancestor that declares the class key or `total`, the claims counting against the budget (table in section 2) in that ancestor's subtree would exceed the key. -- A child's key only adds a limit; it never lifts an ancestor's limit or a missing entitlement. -- A raw GPU key (section 7) on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has the class key or `total`. While the raw key exists, it also caps those classes' claims together with the raw-name VMs it covers, so an existing GPU limit is never bypassed. -- A tenant with no GPU key of either kind anywhere above it is limited only by capacity, for vGPU classes once it is entitled. -- A vGPU class is ordered only by a tenant entitled to vGPU. The Tenant field `gpu.vgpu` grants it. The parent tenant's administrator sets it, or the hoster on a top-level tenant. A tenant is entitled only when the flag is set on it and its parent is entitled. The root tenant is rendered by the platform, not by the tenant chart, so its value comes from platform values. The flag is off by default everywhere, so the hoster turns it on for the root first. No tenant can grant itself vGPU, since its Tenant object lives in its parent's namespace. The tenant chart computes the effective value from the parent's, like its other namespace values, and stamps it on the namespace as the label `namespace.cozystack.io/vgpu`; a change reaches grandchildren as each level re-renders. The chart always writes the label, so only the value `"true"` means entitled; any other value, or no label, means not entitled. Entitlement is separate from budget because a vGPU license is counted per unit, which is a budget, while getting the vGPU guest driver and the license token at all is yes or no. Class keys and `total` then limit vGPU like any other GPU. Clearing the flag refuses new vGPU orders, including the claim a `claimTemplate` would create at the next VMI start, so such a VM or worker no longer starts. Claims already held keep counting and can still move or be rebound, so a pool on `claimSelector` keeps its claims, and the tenant's Kairos pools lose their vGPU entitlement as `kairos-vgpu-workers` describes. Raw vGPU names in VMs stay as open as today until a raw key limits them or `forbidden` is set (section 7). -- GPU keys play no part in the existing pool membership and carve-outs. - -A parent with `total: 4` caps everything below it at four GPUs, each vGPU counting as one. A parent cannot reserve part of its budget for one child. #48 would add that, but as written it covers neither GPUs nor entitlement, so this section stays GPUClaim's own rule, and budgets move to #48 only if its gate takes over subtree caps for GPU keys. - -### 5. Usage accounting, catalog and licensing - -Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads. Each `GPUClaim` gets one with `kind: gpu-claim` and resources `gpu.cozystack.io/: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 8), next to the records Cozystack already keeps for CPU, memory and storage. The GPU controller writes the record itself, with its `GPUClaim` as the record's owner through an ownerReference, and with no `WorkloadMonitor` and no label selector, so no tenant label can add a record or drop one ([0011](./decisions/0011-a-gpuclaim-owns-its-usage-record.md)). The `Workload` controller, which today deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, leaves a record owned by a `GPUClaim` to its owner: the GPU controller deletes it when the claim leaves `Bound` and `Attached`, and garbage collection deletes it with the claim. Like other `Workload` records it has no time fields: a held interval is the record's lifetime, which a billing system observes by watching and reconciles by sampling. A virt-launcher pod does not report GPUs its claims cover a second time. - -The catalog is the list of `GPUClass` objects with `displayName`, model, mode, profile, memory, `maxCount`, `status.free` and `status.maxFreePerNode`. Every authenticated user reads it, because a hoster publishes its offers and a tenant names a class when ordering, including from `kubectl` or GitOps without a portal. So the status holds only `free` and `maxFreePerNode`, and its conditions, such as `LostCapacity`, name no nodes and carry no counts. The class's capacity, bound units and `Lost` count show the hoster's inventory and load, and they are not in the status. The controller exports them as metrics that platform monitoring scrapes. Per node, a hoster reads device occupancy and health from `GPUNode`, and lists bound claims with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write. A tenant can still watch `free` and `maxFreePerNode` over time and see how busy a class is. The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest order, leaving out a field that is absent because it sets no limit. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. A `gpuclaim` source lists a tenant's own claims. The existing `gpu` source keeps serving raw names while they are allowed. - -The remaining budget is computed over each ancestor's whole subtree, so a tenant under a shared parent `total` can see how many GPUs its siblings hold, as a number, without names. This is accepted: the alternative is a budget the tenant can only discover by a refused order, which is the failure this proposal removes. - -vGPU needs a license, and passthrough needs one only for graphics. Compute (C-series) profiles are licensed through NVIDIA AI Enterprise, graphics (Q-series) profiles through NVIDIA vGPU software. The hoster provides the license server, the token and the guest drivers. In worker pools `kairos-vgpu-workers` delivers the token and the guest license settings. In VMs the hoster does, through its image or at boot, as `vgpu-vms` describes. The class's `mode` tells a portal and a tenant which case applies. - -### 6. Host layer - -The host layer does not depend on the host OS: it needs the vGPU host driver, `vfio-pci` and the device plugin, wherever they run. Which OSes Cozystack supports as hosts of the management cluster is decided outside this proposal. Today vGPU hosts run Ubuntu with `bundles.system.variant: isp-full-generic`, where the vGPU host driver and the patched plugin run. `docs/gpu-vgpu.md` does not recommend Talos for vGPU, so this phase does not cover Talos hosts; they keep passthrough through the `default` variant as today. NVIDIA AI Enterprise lists vGPU for Compute on Linux with KVM as supported only through specific hypervisor vendors and, on Ubuntu and RHEL with KVM, for single-node hosts only ([support matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/support/support-matrix-8/8.2.html)), so support for such a host is a matter between the hoster and NVIDIA. - -The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ships as its own phase (Rollout, step 4), and it is replaceable. The API above it depends on four things only: each class's `resourceName` is advertised on the nodes its `nodeSelector` picks, one name never covers two different offers on one node, `GPUNode` reports devices and their health, and node labels mark each node's mode and profiles. `GPUClass`, `GPUClaim`, `GPUNode`, quotas, the three GPU webhooks, usage records and migration use no plugin-specific name, discovery format or configuration; the agent maps whatever device IDs a plugin reports to PCI addresses through sysfs. A different plugin, or a different version of the one chosen here, changes only this phase and the `resourceName` of affected classes, which can change while claims are held. Tenants keep their claims and `claim` entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to `GPUClaim` first. - -Until the phase ships, a cluster keeps the modes its variant gives it today, one per model for models whose whole GPU and VFs share a PCI ID, and vGPU classes keep the experimental status `docs/gpu-vgpu.md` gives the `vgpu` variant: the hoster writes profiles and node labels by hand. On clusters without a layout, which includes the `default` variant and Talos hosts, node labels stay hand-written. +#### Budgets and usage records of held claims -The device plugin is the decision reviewers should look at first. Cozystack ships NVIDIA's kubevirt-gpu-device-plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, as the sandbox device plugin of the `vgpu` variant, until NVIDIA/kubevirt-gpu-device-plugin#192 merges upstream or upstream takes another approach. NVIDIA/kubevirt-gpu-device-plugin#192 adds discovery of vGPU VFs on the vendor-specific VFIO framework and advertises them with one resource name per profile, next to the passthrough names the plugin already advertises for GPUs bound to `vfio-pci`. One plugin then serves both modes of every model under distinct names, so the per-model limit of the problem list goes away. The branch is validated on hardware. Its upstream review has had no maintainer response since 2026-07-10, which is why it ships as a downstream patch rather than waiting. On a card whose VFs all carry a profile, the plugin can read profile names only through NVML, so the platform runs it as its own DaemonSet with the host driver's `libnvidia-ml.so.1` and the `/dev/nvidia*` nodes mounted, privileged; the GPU Operator's DaemonSet has no field for those mounts. The patch leaves when NVIDIA/kubevirt-gpu-device-plugin#192 merges upstream, in this or another form, or when the NVIDIA DRA driver serves vGPU VFs to KubeVirt (kubernetes-sigs/dra-driver-nvidia-gpu#1402); a merged NVIDIA/kubevirt-gpu-device-plugin#192 changes only this phase, while a move to DRA adds new `dra` classes (Rollout, step 6). KubeVirt's built-in plugin was the other candidate (Alternatives). +A held claim counts against the same GPU keys as a class entry, from the moment it is `Bound` until it is deleted (the phase table), in the same pool figure as the class entries of section 5; a `claim` or `claimSelector` entry declares nothing of its own, since its claims already count. The validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, that pool figure and the vGPU entitlement. A raw GPU key on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has a GPU key covering the class, so a held claim, which no `ResourceQuota` sees, never bypasses an existing raw limit. Clearing the vGPU flag refuses new vGPU orders; claims already held keep counting and can still move or be rebound, so a pool on `claimSelector` keeps its claims. -The phase adds a node layout to the `vgpu` variant, in platform values. For each node selector it names a mode and, for vGPU, the profile of each GPU model. Nodes outside every selector are passthrough, through the `defaultWorkload: vm-passthrough` the layout renders. With a layout: +Each held claim gets a usage record with `kind: gpu-claim` and resources `.gpuclass.cozystack.io/devices: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 10). The GPU controller writes the record itself, with its `GPUClaim` as the record's owner through an ownerReference, and with no `WorkloadMonitor` and no label selector, so no tenant label can add a record or drop one ([0011](./decisions/0011-a-gpuclaim-owns-its-usage-record.md)). The `Workload` controller, which today deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, leaves a record owned by a `GPUClaim` to its owner: the GPU controller deletes it when the claim leaves `Bound` and `Attached`, and garbage collection deletes it with the claim. A virt-launcher pod's `Workload` does not report GPUs its held claims cover a second time. -- The platform runs the patched plugin and renders the KubeVirt `permittedHostDevices` entries for the names it advertises, as entries with an external provider. Other vendors' entries, such as an FPGA, stay as written. -- A node agent writes the VF profiles at boot and when the layout changes. The controller sets each node's workload label and one label per GPU model, `gpu.cozystack.io/-` (for L40S `gpu.cozystack.io/10de-26b9`), to the profile or to `passthrough`; class node selectors match these labels. -- The plugin does not pick up VFs that gain a profile after it started, so the controller restarts the plugin on a node when devices there become ready later, for example after a reboot. Running VMs are not affected. - -The layout and mdev GPUs coexist: GPUs on the mdev path keep today's configuration through `vgpu-device-manager`, and the agent writes profiles only on nodes that use VFs. New deployments of SR-IOV GPUs on current host releases use VFs. - -A layout change on a node, of mode or of profile, waits until the node has no GPU VMs. The node gets a `LayoutPending` condition: new binds and starts there are refused, unattached claims move away, and running VMs keep their GPUs until the hoster drains the node. A GPU VM created before Cozystack switched to `LiveMigrateIfPossible` needs one restart before the drain can evict it. A node whose hand-written profiles already match the layout is not a change. A `vgpu` cluster adopts the layout with vGPU selectors that cover its existing vGPU nodes, and labels those nodes `vm-vgpu` before setting it, so the layout's passthrough default does not change them in between. Its vGPU classes, raw `name` entries and raw quota keys move from the hand-written VF names to the plugin's per-profile names in one maintenance window, or the raw-name consumers move to `GPUClaim` first. KubeVirt's built-in plugin and the patched plugin report different device IDs for the same VF, and kubelet would otherwise hand out VFs in use, so affected nodes are cordoned and free of GPU VMs until the patched plugin has registered there. +The catalog's `free` and `maxFreePerNode` subtract held units from step 2 on. A hoster lists bound claims per node with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write, and the class's bound units and `Lost` count join its metrics. A `gpuclaim` option source lists a tenant's own claims. -### 7. Migration of existing `gpus` +#### Forbidding raw names -Raw `name` entries keep working, and nothing converts them automatically, because a conversion would bind GPUs a tenant never ordered and put them in the usage records. The binder counts their usage, so it never books a unit a raw-name VM holds. The reverse is not protected: the scheduler does not see a claim that is bound but not attached, so while raw names are allowed, a raw-name VM can take that unit. A reservation is a hard guarantee only with `gpu.rawResourceNames: forbidden`. +While raw names are allowed, the scheduler does not see a claim that is bound but not attached, so a raw-name VM can take that unit; the binder counts raw-name usage, so it never books a unit a raw-name VM holds, but not the reverse. A hold is a hard guarantee only with `gpu.rawResourceNames: forbidden`. -A tenant moves a VM by stopping it, ordering claims of the matching class, replacing each `name` entry with a `claim` entry and starting it. On a full pool the claim can only be ordered after the stop; with spare units, ordering first puts both in the usage records until the restart. A Talos GPU pool on a passthrough name moves by ordering claims of the matching passthrough class and replacing its `name` entries with a `claimSelector` or a `claimTemplate`, which rolls its workers; on a full cluster the pool scales down first, so its GPUs are free to order. A Talos pool on a raw vGPU name has no path here, since Talos workers carry no vGPU guest driver; it keeps its raw name until it moves to a worker OS that does. ComputePlane GPU groups have no path until the module accepts claim entries, so a cluster with them keeps raw names allowed. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes `claim` entries. A pool on one takes the pool path above. +When every consumer has moved to class or claim entries, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. The preconditions are the hoster's checklist, and the platform does not verify them: no worker pools with raw-name `gpus` entries, ComputePlane GPU groups included, no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: -Quota keys move too. A raw GPU key is a quota key on a GPU resource name, such as `nvidia.com/AD102GL_L40S`. A claim-backed VM's pod still requests that name, so while a tenant migrates, its raw key covers both kinds of VM; after its last raw-name VM has moved, the key is replaced by `gpu.cozystack.io/*` keys. A tenant whose raw-name VMs use vGPU can order their claims only after it is entitled to vGPU (section 4), and within any class keys set above it. - -When every consumer has moved, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. The preconditions are the hoster's checklist, and the platform does not verify them: no worker pools with raw-name `gpus` entries, ComputePlane GPU groups included, no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: - -- an admission policy refuses GPU entries (section 3) on every VMI created without a claim reference, on every path; +- an admission policy refuses GPU entries (section 4) on every VMI created without a class or claim reference, on every path; - the same policy refuses every pod created in a tenant namespace that requests a GPU resource name, unless it is a virt-launcher pod created by KubeVirt's virt-controller service account with an owner reference to a VirtualMachineInstance. Tenants cannot forge such a pod, since no tenant role can create pods or VMIs, so app charts that copy free-form `resources` into pods cannot request GPUs either; - tenant validation refuses raw GPU keys; - the `gpu` option source returns nothing, and the `instancetype` source hides instance types that carry GPU entries. -### 8. Extensions +The policy reads its settings from a ConfigMap the controller writes in `cozy-system`, so on a cluster with `forbidden` it refuses pod creation in tenant namespaces until the controller has published it once. + +### 10. Extensions The catalog is designed so later backends add `source` members and class fields, not tenant fields. -- DRA. Kubernetes DRA has been GA since 1.34, and KubeVirt v1.9 has `GPUsWithDRA` and `HostDevicesWithDRA` in beta. A `dra` source names a `DeviceClass` and selectors, and under it a `GPUClaim` maps one to one onto a `ResourceClaim` created at order time. The scheduler allocates the `ResourceClaim`, the GPU controller keeps it allocated through its own `reservedFor` entry, which the resourceclaim controller leaves in place, and the webhook renders the `ResourceClaim` reference into the VMI; the binder does not allocate. Three gaps remain. The NVIDIA DRA driver has no vGPU yet (kubernetes-sigs/dra-driver-nvidia-gpu#1402, milestone v0.6.0), and its passthrough is alpha. A `ResourceClaim` is allocated only when a pod first needs it, so nothing is held before the first boot: until then a `dra` claim is `Bound` without a node, counts against the budget, holds no GPU and stays out of the usage records, and the webhook attaches it like any `Bound` claim without pinning the pod to a node. DRA quota is per namespace, without subtree caps. KubeVirt VEP 344 (`persistWhenStopped`) holds a claim for a stopped VM from the VM side; a `dra` class and VEP 344 must not both manage one claim. When a DRA GPU driver is installed, the raw-name policy has to cover `ResourceClaim` references too. +- DRA. Kubernetes DRA has been GA since 1.34, and KubeVirt v1.9 has `GPUsWithDRA` and `HostDevicesWithDRA` in beta. A `dra` source names a `DeviceClass` and selectors. In step 1 a class entry of a `dra` class renders a `ResourceClaimTemplate` reference into the VMI, and the scheduler allocates per pod. In step 2 a `GPUClaim` maps one to one onto a `ResourceClaim` created at order time. The scheduler allocates the `ResourceClaim`, the GPU controller keeps it allocated through its own `reservedFor` entry, which the resourceclaim controller leaves in place, and the webhook renders the `ResourceClaim` reference into the VMI; the binder does not allocate. Three gaps remain. The NVIDIA DRA driver has no vGPU yet (kubernetes-sigs/dra-driver-nvidia-gpu#1402, milestone v0.6.0), and its passthrough is alpha. A `ResourceClaim` is allocated only when a pod first needs it, so nothing is held before the first boot: until then a `dra` claim is `Bound` without a node, counts against the budget, holds no GPU and stays out of the usage records, and the webhook attaches it like any `Bound` claim without pinning the pod to a node. DRA quota is per namespace, without pools. KubeVirt VEP 344 (`persistWhenStopped`) holds a claim for a stopped VM from the VM side; a `dra` class and VEP 344 must not both manage one claim. When a DRA GPU driver is installed, the raw-name policy has to cover `ResourceClaim` references too. - MIG-backed vGPU profiles get their own resource names from the patched plugin, so they are ordinary classes; with DRA, partitionable devices (beta in Kubernetes 1.36) describe MIG natively. - A non-NVIDIA GPU is a class with another `vendor`; Q, B and A series vGPU profiles are classes like C series. - NVLink sets that must be attached together are left to the DRA source. -- On-demand slicing. A hoster may let tenants take any size from a shared pool, from a fraction of a GPU, such as a MIG-backed vGPU on a 1/7 slice of an A100, up to several whole GPUs of one node, taken from GPUs the layout keeps out of MIG mode. The tenant contract stays a class, one per size, and a `count`, and a claim still lives on one node. The host layer and the binder change: the node agent creates the profile, and on MIG GPUs the MIG instance under it, when a claim binds instead of following a fixed layout, so the binder counts capacity from free GPU resources rather than from advertised names. Enabling MIG mode needs an idle GPU and, on Ampere, a GPU reset, so the layout enables it ahead of time. On Hopper and newer the mode does not survive a reboot, so the layout sets it again at boot. A plugin restart per bind, as section 6 does for late profiles, would leave the node's allocatable at zero until the plugin registers again on every bind, so the plugin needs to rediscover profiles without one, separate work on top of NVIDIA/kubevirt-gpu-device-plugin#192. +- On-demand slicing. A hoster may let tenants take any size from a shared pool, from a fraction of a GPU, such as a MIG-backed vGPU on a 1/7 slice of an A100, up to several whole GPUs of one node, taken from GPUs the layout keeps out of MIG mode. The tenant contract stays a class, one per size, and a `count`, and a claim still lives on one node. The host layer and the binder change: the node agent creates the profile, and on MIG GPUs the MIG instance under it, when a claim binds instead of following a fixed layout, so the binder counts capacity from free GPU resources rather than from advertised names. Enabling MIG mode needs an idle GPU and, on Ampere, a GPU reset, so the layout enables it ahead of time. On Hopper and newer the mode does not survive a reboot, so the layout sets it again at boot. A plugin restart per bind, as section 2 does for late profiles, would leave the node's allocatable at zero until the plugin registers again on every bind, so the plugin needs to rediscover profiles without one, separate work on top of NVIDIA/kubevirt-gpu-device-plugin#192. ## User-facing changes -- Tenants get `GPUClaim`, a GPU catalog with free capacity and remaining budget per class, `claim` entries in `vm-instance`, `claimSelector` entries for worker pools in `kubernetes-nodes`, and `claimTemplate` entries in both. Raw resource names keep working until a hoster forbids them. -- A held claim is billed while held, whether a VM runs on it or not. An autoscaled pool on `claimSelector` holds, and pays for, every claim its selector matches, which is at least `maxReplicas`; under `forbidden` nothing runs on GPUs without a claim. A tenant who wants to pay only while running uses `claimTemplate`, and finds out at start, not at order time, whether the claim fits. -- Hosters get `GPUClass`, `GPUNode`, GPU budgets in `resourceQuotas` and the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout, `onCordon` on classes, the `gpu.resourceNames` list and `gpu.rawResourceNames`; an external billing system gets a usage record (`Workload`) per held claim. +Step 1: + +- Tenants get a GPU catalog with free capacity and remaining budget per class, and `className` entries in `vm-instance` and `kubernetes-nodes`. Raw resource names keep working. +- A write of a VM or pool whose class entries do not fit a GPU budget is refused. A started VM that finds no free unit waits in `Scheduling` until one frees up. +- Hosters get `GPUClass`, GPU keys in `resourceQuotas`, the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout and the `gpu.resourceNames` list; an external billing system reads GPUs from the `Workload` records of virt-launcher pods. + +Step 2: + +- Tenants get `GPUClaim`, `claim` entries in `vm-instance` and `claimSelector` entries in `kubernetes-nodes`. A held claim is billed while it holds GPUs, `Bound` or `Attached`, whether a VM runs on it or not. An autoscaled pool on `claimSelector` holds, and pays for, every claim its selector matches, which is at least `maxReplicas`. A tenant who wants to pay only while running uses class entries, and finds out at start, not at order time, whether a unit is free. +- Hosters get `onCordon` on classes and `gpu.rawResourceNames`; an external billing system gets a usage record per held claim. ## Upgrade and rollback compatibility The change is additive. Existing `gpus` entries render as before, and the existing variants work as today until a layout is set. -Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, and the tenants' GPU limits, with four exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only limits were `gpu.cozystack.io/*` keys lose them (rules below), and tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through `GPUClaim`, so they lose their GPUs and run on the plain worker image. A child tenant that declares its own quotas no longer shares its parent's GPU limit, because a raw key is carved out of the parent's pool rather than capping the subtree. +Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, with these exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only GPU limits were GPU keys lose them, and on a cluster that ran step 2 with `forbidden`, tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through classes, so they lose their GPUs and run on the plain worker image. -The phases run in this order. The hoster removes the node layout and chooses the variant in a maintenance window. Raw names are allowed again together with the raw keys, since raw keys are refused while raw names are forbidden and a later quota does not evict a running pod. `claim` and `claimTemplate` entries of VMs and Talos pools become raw `name` entries, and those VMs restart and those pools roll. `claimSelector` and `claimTemplate` entries are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. The `GPUClaim` objects and the `gpu.cozystack.io/*` keys are deleted. Last, the components, the CRDs, the `gpu.vgpu` flag with its namespace label and the `gpu.cozystack.io/*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. +Step 2 rolls back first. Raw names are allowed again, since raw keys are refused while raw names are forbidden. `claim` entries of VMs and `claimSelector` entries of pools become class entries, and those VMs restart and those pools roll. The `GPUClaim` objects are deleted, and the step 2 components go. + +Step 1 rolls back next. The hoster removes the node layout and chooses the variant in a maintenance window. Class entries become raw `name` entries with the class's resource name, and GPU keys become raw keys on those names where the tenant wants a limit, and the remaining GPU keys are deleted. Class entries are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. Last, the components, the `GPUClass` CRD, the `gpu.vgpu` flag with its namespace label and the `cozystack.io/gpu-*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. Three rules hold throughout: -- Names move back by the window rule of section 6 in reverse: vGPU classes return to the hand-written VF names, and raw `name` entries and raw keys on the per-profile names move with them in the same window. -- Raw keys go through the existing tenant quota pools, not the subtree rule of section 4, so adding one changes how the tenant's other quotas are pooled. Raw keys therefore go only to tenants that already declare quotas other than `gpu.cozystack.io/*`, and raw names elsewhere are unlimited, as before this proposal. +- Names move back by the window rule of section 2 in reverse: vGPU classes return to the hand-written VF names, and raw `name` entries and raw keys on the per-profile names move with them in the same window. +- Raw keys render into the `ResourceQuota` and are checked at pod creation, not at write, so a tenant moved back to them is refused later than before. - A raw key also renders the tenant chart's default `LimitRange`, whose memory limit can sit below a VM's memory request, because KubeVirt sets virt-launcher limits from the namespace quota only when it limits `cpu` and `memory`, so a tenant given a raw key also declares `cpu` and `memory`. ## Security -- Tenant access. Tenants with the `admin` role or above manage `GPUClaim` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. Tenants cannot read `GPUNode`. -- What a tenant controls. A tenant never supplies a resource name or a node through a claim reference: both come from `GPUClass` and from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The webhook takes claims only from the VM's own namespace. A tenant super-admin who writes VirtualMachines directly is covered too: a referencing VMI with its own GPU entries is refused, and with `forbidden` any GPU entry without a claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. +Step 1: + +- Tenant access. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. +- What a tenant controls. A tenant never supplies a resource name through a class entry: it comes from `GPUClass`, which tenants cannot write. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. A tenant super-admin who writes VirtualMachines directly can still write raw GPU names, as today, until step 2's `forbidden`; a VMI with class entries and GPU entries of its own is refused. GPU keys are checked only when an application is written. A VirtualMachine written directly, with class entries in its annotation or a higher `count` than its application declares, is not counted against them, also under `forbidden`. That path closes when tenant super-admins lose write access to VirtualMachines, which #48 lists as a prerequisite of its phase 4. Until then a hoster who sells GPU budgets to sub-tenants does not give them the super-admin role. - Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. -- Node agent. It reads sysfs and kubelet's pod-resources socket, writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can write only its own node's `GPUNode`. -- Webhooks. Three GPU webhooks fail closed. The VMI and virt-launcher pod webhooks see only objects carrying the reference label (section 3), so their outage blocks the start of VMs with claim references. The pod webhook has to fail closed, because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, the budgets and the vGPU entitlement (sections 2 and 4), so its outage also blocks new orders, while held claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity and budgets when it creates and binds the claim. -- Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `gpu.cozystack.io/template-claim` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so a held claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. -- Template claims. For `claimTemplate` entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the template names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, orders its template claims, and anyone who can start it makes them exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). -- Controller. Besides its own `gpu.cozystack.io` kinds and the `Workload` records it already writes, it gains three rights, the first two only with the host layer (Rollout, step 4, [0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)): patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Step 1 also gives it `update` and `delete` on pods in every namespace, cluster-wide like its write on Secrets, because tenant namespaces appear at runtime. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses (section 3). It also writes one ConfigMap, the GPU settings in `cozy-system`. -- Visibility. A tenant sees the node name of its own claims and, through the remaining budget, its siblings' GPU count under a shared parent budget (section 5). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. +- Node agent. It reads sysfs and writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can delete only the device plugin pod of its own node. +- Webhooks. The VMI and virt-launcher pod webhooks fail closed and see only objects carrying the reference label, so their outage blocks the start of VMs with class entries. +- Controller. It reads nodes and pods to compute class status, and writes `GPUClass` status. With the host layer it gains patching node labels, restricted by a policy to the GPU workload label and `cozystack.io/gpu-*` ([0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)). +- Visibility. A tenant sees, through the remaining budget, its siblings' declared GPU count under a shared parent budget (section 7). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. + +Step 2 adds: + +- Tenant access. Tenants with the `admin` role or above manage `GPUClaim` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. +- What a tenant controls. A tenant never supplies a node through a claim reference: it comes from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. The webhook takes claims only from the VM's own namespace. With `forbidden` any GPU entry without a class or claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. +- Webhooks. Three GPU webhooks fail closed. The VMI and virt-launcher pod webhooks see only objects carrying the reference label, so their outage blocks the start of VMs with class or claim entries. The pod webhook has to fail closed, because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order, so its outage also blocks new orders, while held claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity when it creates and binds the claim. +- Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `cozystack.io/gpuclaim-template` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so a held claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. +- Template claims. For class entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the entry names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, uses the GPUs of its class entries, and anyone who can start it makes their claims exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). +- Controller. It gains deleting unscheduled VMIs that reference its claims, and `update` and `delete` on pods in every namespace, cluster-wide like its write on Secrets, because tenant namespaces appear at runtime. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses. It also writes one ConfigMap, the GPU settings in `cozy-system`. +- Visibility. A tenant sees the node name of its own claims. ## Failure and edge cases Each case gives what is seen and what to do. +Step 1: + +- A write of a VM or pool is refused → the message names the pool, the GPU key, the limit and what is already reserved; use a smaller `count`, another class, or ask for more budget. +- A VM with class entries is refused at start → the webhook refuses the VMI, and the VirtualMachine shows the message in a `FailedCreate` event, naming the class and the reason: the class does not exist or shares its resource name with another class, a second vGPU class, more than `maxCount`, no vGPU entitlement, a resource name KubeVirt does not permit, or GPU entries of the VMI's own; fix the entry, or ask the hoster. +- A VM with class entries shows `ErrorUnschedulable` → its launcher pod is `Pending` and `Unschedulable`: no node it can use has a free unit of the class. It starts by itself once one frees up. The catalog's `free` shows the shortage. +- A class shows `free: 0` while its nodes have GPUs to spare → the nodes carry a `NoSchedule` or `NoExecute` taint the class does not tolerate; add it to the class's `tolerations`, or remove the taint. +- A class has a `ResourceNameConflict` condition → another class has its resource name; delete one of them or point it at its own name. +- A class is deleted while VMs name it → running VMs keep running, and their next start is refused naming the class; the tenant moves them to another class. +- A layout change does not apply → the node carries the `cozystack.io/gpu-layout-pending` label; cordon and drain it. A GPU VM on the node holds the change until it stops or is evicted. +- A Talos pool with a vGPU class entry is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. + +Step 2: + - A tenant's order is refused at create → the message names the class, the count and the limit, or, for a vGPU class, that the tenant is not entitled to vGPU; order a smaller count, another class, or ask for more budget. - A claim stays `Pending` after an order → its condition names the shortage; it binds when capacity or budget frees up, or the tenant deletes it. - A VM with claims is refused at start → the webhook refuses the VMI for a reference problem, such as a claim that does not exist, is being deleted or is attached to another VMI, and the VirtualMachine shows the message in a `FailedCreate` event, naming each claim and the reason; fix the reference. -- A VM with claims stays `Starting` → its VMI is in `Scheduling` with its pod gated, the claims carry a condition and the VMI a `Warning` event naming the reason: no node can take the whole set, template claims included, a budget is exhausted, or a claim of a `Stay` class is on a cordoned node while others need gathering. The VM starts by itself once the set can be placed. To get there sooner, uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. -- The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. An `Always` or `RerunOnFailure` VM shows `CrashLoopBackOff` and starts again after a delay that grows with each failure up to five minutes, and the count resets once a VMI runs; a `Once` VM keeps its failed VMI and stays down, also shown as `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it. The binder avoids that node for the set for ten minutes or until the node changes (section 3). Free CPU or memory on GPU nodes if this repeats. +- A VM with claims stays `Starting` → its VMI is in `Scheduling` with its pod gated, the claims carry a condition and the VMI a `Warning` event naming the reason: no node can take the whole set, template claims included, or a claim of a `Stay` class is on a cordoned node while others need gathering. The VM starts by itself once the set can be placed. To get there sooner, uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. +- The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. An `Always` or `RerunOnFailure` VM shows `CrashLoopBackOff` and starts again after a delay that grows with each failure up to five minutes, and the count resets once a VMI runs; a `Once` VM keeps its failed VMI and stays down, also shown as `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it. The binder avoids that node for the set for ten minutes or until the node changes. Free CPU or memory on GPU nodes if this repeats. - A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. - A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. -- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost` (section 2); once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached claims onto a `Ready` node with room (section 3) and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node (all in section 3). After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set (section 3). `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. +- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. - A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; an `Always` or `Once` VM starts again once its claims have moved or been rebound, a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. -- A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has (section 2). An `Always` or `Once` VM then starts on the new node once its claims have moved or been rebound; a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. +- A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has. An `Always` or `Once` VM then starts on the new node once its claims have moved or been rebound; a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. - A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts, the controller moves the claim off that node before it releases the pod, or the claim goes to `Lost` and the pod stays gated until it is rebound. -- A layout change does not apply → the node shows `LayoutPending`; drain it. A VM with a raw `name` entry on the node holds the change until it stops. -- A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use a `claimSelector` or a `claimTemplate`, or allow raw names again while the cluster has such pools. -- A Talos pool with a `claimSelector` or `claimTemplate` of a vGPU class is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. -- A class shows `free: 0` while its nodes have GPUs to spare → the nodes carry a `NoSchedule` or `NoExecute` taint the class does not tolerate; add it to the class's `tolerations`, or remove the taint. +- A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use class entries or a `claimSelector`, or allow raw names again while the cluster has such pools. - A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. - A pool worker is refused → no matching claim is free, because of a surge machine during a rollout, claims still `Pending` or `Lost`, matching claims only on nodes that are not `Ready`, or another pool or a VM using the matching claims; wait, add claims, give each pool its own labels, or lower `maxReplicas`. ## Testing -- Unit tests for the binder and the three GPU webhooks: capacity with raw-name usage, budgets over the ancestor walk, the phase table, every move and refusal in sections 2 and 3, concurrent starts, the gate added once on a repeated call and on the temporary `WaitForFirstConsumer` pod but not on a migration target, the pin appended to every existing node selector term and the gate removed in the same update, and the pod pin after a LiveUpdate sync. -- Webhook and controller tests for cordons, templates and NICs: the webhook binds, moves and creates no claim, a set that cannot be placed keeps the pod gated with a condition and is released once it can, a pinned pod that the node refuses is deleted and that node is avoided for the set, under `Move` a claim moves at VMI start and the pod gets a required node affinity and the classes' tolerations, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity a VMI with a pinned pod is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, a gated VMI is not deleted and its claims are gathered, a cordon or `NotReady` on the pinned node does not count as a refusal, so the refused-pod deletion does not fire, a `Once` or `Manual` VM stays stopped after a refused pod is deleted, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only template claims gets one node with room for all of them, and under `Stay` a single claim waits; a `claimTemplate` entry is refused at admission when the class, `maxCount` or the entitlement does not allow it, its claim is created by the controller once the pod waits at the gate, under the budget and capacity checks, and owned by the VMI, deleted with its VMI, reused on a repeated pass, and a dry run records nothing; a VM with a GPU claim and `nvidia.com/hostdev` passes. -- Admission tests for `KubernetesNodes`: a `claimSelector` or `claimTemplate` of a vGPU class on a Talos pool is refused, and one of a passthrough class is accepted. -- Quota tests for vGPU: an entitled tenant with no class key orders, an unentitled one is refused, a child's flag without the parent's is refused, `total` caps vGPU, a cleared flag keeps held claims and refuses the next template claim, and `remainingBudget` is 0 without the entitlement. -- Admission tests for the schema rules, `maxReplicas`, class lifecycle, tenant finalizers, the agent and controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. -- Controller tests for the class metrics: capacity, bound units and the `Lost` count match the claims, and none of them appears in `GPUClass` status. -- Binder tests: a node with a `NoSchedule` taint that the class does not tolerate gets no binds, moves or gathers and its units leave `free`, a node whose taint the class tolerates is bound to and counts in `free`, a set of two classes skips a node only one of them tolerates, while the Fence Agents Remediation taint alone does not make a node cordoned. +Step 1: + +- Unit tests for the class status: `free` and `maxFreePerNode` from node allocatable and pod requests, a node with a `NoSchedule` taint the class does not tolerate left out and one whose taint it tolerates counted, the three skipped taints, the Fence Agents Remediation taint alone not making a node cordoned, and a second class with the same resource name getting `ResourceNameConflict`. +- Webhook tests for class entries: one `gpus` entry per unit with the class's resource name, the classes' tolerations on the launcher pod and on the temporary `WaitForFirstConsumer` pod and still there after a LiveUpdate sync of the VM, each refusal of section 4, a VM with a GPU class entry and `nvidia.com/hostdev` passing, and a dry run. +- Budget tests: the interim check refuses a `VMInstance` and a `KubernetesNodes` write over a class key or the total key, counts `count` times `maxReplicas` for a pool and the defaulted `maxReplicas` for a pool that does not set it, passes shrinking and no-op updates, refuses a child tenant's GPU key against the class entries already in the parent's pool, applies the pool rules like other keys, stands aside while #48's gate is on, and reaches the same verdicts as #48's evaluator on the same inputs. +- Admission tests for `KubernetesNodes`: a vGPU class entry on a Talos pool is refused, and a passthrough one is accepted. +- Entitlement tests: a child's flag without the parent's gives no entitlement, a vGPU class entry without it is refused at start, and `remainingBudget` is 0 without it. +- Usage tests: a VM on a class entry shows its GPUs in its `Workload` under the class's resource name while its launcher pod exists. +- Admission tests for the class schema: the frozen fields, `tolerations` operators, and the class name as a DNS label. +- Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. +- E2E with passthrough GPUs on the GPU runner: a VM on a class entry starts with the GPU, a write over budget is refused, and the `Workload` shows the GPU. +- Manual on SR-IOV vGPU hosts: both modes of one model in one cluster, profiles across reboots, a layout change through a drain, and adoption from `default` and from `vgpu`, on Ubuntu hosts. + +Step 2: + +- Unit tests for the binder and the three GPU webhooks: capacity with raw-name usage, budgets in the pool figure with class entries, the phase table, every move and refusal in section 9, concurrent starts, the gate added once on a repeated call and on the temporary `WaitForFirstConsumer` pod but not on a migration target, the pin appended to every existing node selector term and the gate removed in the same update, and the pod pin after a LiveUpdate sync. +- Webhook and controller tests for cordons, template claims and NICs: the webhook binds, moves and creates no claim, a set that cannot be placed keeps the pod gated with a condition and is released once it can, a pinned pod that the node refuses is deleted and that node is avoided for the set, under `Move` a claim moves at VMI start and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity a VMI with a pinned pod is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, a gated VMI is not deleted and its claims are gathered, a cordon or `NotReady` on the pinned node does not count as a refusal, a `Once` or `Manual` VM stays stopped after a refused pod is deleted, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only class entries gets one node with room for all of its template claims, and under `Stay` a single claim waits; a template claim is created by the controller once the pod waits at the gate, under the capacity checks, owned by the VMI, deleted with its VMI, reused on a repeated pass, counted against no budget of its own, and a dry run records nothing. +- Quota tests for held claims: a held claim counts from `Bound`, a `claim` entry declares nothing, a raw key blocks orders of its class until a GPU key covers it, and a cleared vGPU flag keeps held claims and refuses new orders. +- Admission tests for `maxReplicas`, class lifecycle while the class is in use, tenant finalizers, the controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no class or claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. - Controller tests for the usage record: present exactly while `Bound` or `Attached`, owned by its `GPUClaim`, independent of tenant labels, not reported twice, kept through reconciles of the `Workload` controller although it has no monitor label, and gone with its claim. -- Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, the `Workload` controller, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. - E2E with passthrough GPUs on the GPU runner: order, attach, stop, confirm another tenant cannot take the GPU, start again, and check the usage record. -- Manual on SR-IOV vGPU hosts: both modes of one model in one cluster, profiles across reboots, a layout change through a drain, and adoption from `default` and from `vgpu`, on Ubuntu hosts. ## Rollout -1. `GPUClass`, `GPUClaim`, `GPUNode` and its agent, the binder and its metrics with a scrape object for platform monitoring, the three GPU webhooks, the policies, the option sources, `claim` and `claimTemplate` entries in `vm-instance`, the `gpu.vgpu` Tenant flag, budgets and usage records, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRDs ship and finalizers and usage records still run. Before tenants order vGPU claims, the `gpu.vgpu` flag is set from the root tenant down. -2. `claimSelector` and `claimTemplate` in `kubernetes-nodes`, for Talos pools with passthrough classes. -3. The consumers, on top of steps 1 and 2: `kairos-vgpu-workers` adds Kairos GPU pools, and `vgpu-vms` adds GPU guest images for VMs with `claim` entries. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. -4. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. -5. Budgets through #48, only if its gate takes over subtree caps for GPU keys. -6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by ordering new claims. +Step 1: + +1. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. It needs no tenant API and can land first. +2. `GPUClass` with its status controller and metrics, class entries in `vm-instance` and `kubernetes-nodes` with the VMI webhook, the `gpu.vgpu` Tenant flag, GPU keys with the interim check, and the `gpuclass` option source, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRD ships and the charts refuse class entries. Before tenants use vGPU classes, the `gpu.vgpu` flag is set from the root tenant down. It works on any cluster where each offer it publishes has a resource name of its own, with or without item 1. +3. GPU keys through #48, and the interim check deleted, in the release where #48's gate is on by default. + +Step 2: + +4. Holding: `GPUClaim`, the binder and its metrics, the launcher pod webhook and gate, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. + +Then: + +5. The consumers: `kairos-vgpu-workers` adds Kairos GPU pools on class entries, and on `claimSelector` once step 2 ships, and `vgpu-vms` adds GPU guest images. +6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by naming it. ## Open questions -- Release on tenant suspension. A hoster may want a suspended tenant's GPUs released after a GPU-specific grace period; Cozystack has no platform-wide notion of suspension yet. -- Packing policy. Fullest-node packing is the default; hosters may need spreading across failure domains as a class option. +- Release on tenant suspension. A hoster may want a suspended tenant's held GPUs released after a GPU-specific grace period; Cozystack has no platform-wide notion of suspension yet. +- Packing policy. Fullest-node packing is the binder's default in step 2; hosters may need spreading across failure domains as a class option. +- Held claims under #48. A `GPUClaim` is not an application, so #48's oracle, which sums application releases, does not see it. Step 2 needs either `GPUClaim` counted next to applications in the pool figure, or the claim made an application kind with a declared consumption. ## Alternatives considered -DRA as the whole API, with `DeviceClass` as the catalog and `ResourceClaim` as the tenant object, or DRA first for passthrough with vGPU later. Rejected for now: a non-Pod `reservedFor` entry does keep a `ResourceClaim` allocated for a stopped VM, but the NVIDIA DRA driver does not serve vGPU yet and its passthrough is alpha, and vGPU is the case hosters sell first. DRA is the planned backend under `GPUClass` (section 8), mapped one to one onto `GPUClaim`, so tenants order claims of a `dra` class through the same API. +A `gpu.cozystack.io` group for the GPU kinds and their keys. Rejected ([0013](./decisions/0013-the-gpu-kinds-live-in-cozystack-io.md)): no requirement here needs a group of its own, and a CRD cannot move to another group in place. + +Holding in step 1, as earlier revisions had it. Rejected ([0016](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md)): step 1 answers five of the six problems without a hold, the third only for budgets, and the hold brings the binder, moves, lost capacity and the pod gate into a v1alpha1 tenant API. + +A generic reservation for stopped VMs that covers CPU, memory and GPUs. Rejected for now ([0016](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md)): CPU and memory are interchangeable across nodes, a GPU hold is a whole unit on one node that does not migrate. + +DRA as the whole API, with `DeviceClass` as the catalog and `ResourceClaim` as the tenant object, or DRA first for passthrough with vGPU later. Rejected for now: a non-Pod `reservedFor` entry does keep a `ResourceClaim` allocated for a stopped VM, but the NVIDIA DRA driver does not serve vGPU yet and its passthrough is alpha, and vGPU is the case hosters sell first. DRA is the planned backend under `GPUClass` (section 10). + +`GPUNode`, an inventory kind per node written by a node agent. Rejected ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)): node allocatable gives capacity and loss, and the `Workload` records give use per tenant. -A cluster-scoped object per reservation unit, like `IPAddress` in #35. Rejected for the device-plugin backend, where kubelet picks the physical device and such an object would promise a precision the backend does not have. `GPUNode` gives the per-claim view where it is real. +A cluster-scoped object per reservation unit, like `IPAddress` in #35. Rejected for the device-plugin backend, where kubelet picks the physical device and such an object would promise a precision the backend does not have. -Tenant quota on the resource name only. Stock `requests.` quotas need no new code, but they cannot hold a GPU for a stopped VM and still refuse orders only at pod creation. +Tenant quota on the resource name only. Stock `requests.` quotas need no new code, but they refuse only at pod creation and tie a budget to a name the host layer can change. -Placeholder pods that hold the claim until the VM starts. Rejected because preemption picks victims by priority, not by owner, so one tenant's VM could evict another tenant's placeholder. +Placeholder pods that hold a GPU until the VM starts. Rejected because preemption picks victims by priority, not by owner, so one tenant's VM could evict another tenant's placeholder. Class-level reservations for worker pools, without per-worker claims. Rejected because a pool would get a promise of capacity, not GPUs. A cluster-wide reservation per tenant and class, a count without a node. It needs no binding, moves or pod pin, and for single-GPU offers it would be enough. Rejected because the count holds only in sum. With free units spread one per node, a held set of two fits nowhere. A vGPU profile exists only on the nodes the layout gives it. A lost node silently shrinks every tenant's promise instead of showing which claims are `Lost`. -NVIDIA vgpu-device-manager as the profile writer. At v0.5.1 it writes mdev profiles only and has no path for `current_vgpu_type` on SR-IOV VFs, and the pull requests that added VF support were closed unmerged, NVIDIA/vgpu-device-manager#139 as not planned, and NVIDIA/vgpu-device-manager#195 after NVIDIA named gaps on the KubeVirt side and had no resources for that test round. The node agent writes VF profiles instead; nodes on the mdev path keep vgpu-device-manager (section 6). +NVIDIA vgpu-device-manager as the profile writer. At v0.5.1 it writes mdev profiles only and has no path for `current_vgpu_type` on SR-IOV VFs, and the pull requests that added VF support were closed unmerged, NVIDIA/vgpu-device-manager#139 as not planned, and NVIDIA/vgpu-device-manager#195 after NVIDIA named gaps on the KubeVirt side and had no resources for that test round. The node agent writes VF profiles instead; nodes on the mdev path keep vgpu-device-manager (section 2). -KubeVirt's built-in device plugin for VFs (kubevirt/kubevirt#16890), with the sandbox plugin for passthrough. It ships in KubeVirt v1.9 and needs no patch, but it gives every vGPU profile of a model one resource name, so classes tell profiles apart only by node. On GPUs whose whole GPU and VFs share a PCI ID it clashes with the sandbox plugin, so both modes of such a model need the built-in plugin for passthrough too, both modes share one name, and raw resource names would have to be forbidden on any cluster with vGPU nodes. Switching between it and the sandbox plugin is a maintenance window each time. Rejected in favour of the downstream patch, whose cost is carrying one patch until upstream settles and running the plugin privileged with NVML access. +KubeVirt's built-in device plugin for VFs (kubevirt/kubevirt#16890), with the sandbox plugin for passthrough. It ships in KubeVirt v1.9 and needs no patch, but it gives every vGPU profile of a model one resource name, which breaks the host layer's one name per offer. On GPUs whose whole GPU and VFs share a PCI ID it clashes with the sandbox plugin, so both modes of such a model need the built-in plugin for passthrough too, and both modes share one name. Switching between it and the sandbox plugin is a maintenance window each time. Rejected in favour of the downstream patch, whose cost is carrying one patch until upstream settles and running the plugin privileged with NVML access. From 76413eab51624f83ed70bb8206e4c7edc0315987 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 18:32:32 +0300 Subject: [PATCH 23/32] docs(design-proposals): align the GPUDevice records with the two steps Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- ...ttedhostdevices-is-refused-at-admission.md | 10 ++++---- ...y-where-gpu-operator-serves-gpus-to-vms.md | 24 +++++++++---------- ...punode-health-is-kubelet-allocatability.md | 2 +- ...e-skips-exactly-three-kubernetes-taints.md | 2 +- ...y-to-a-node-that-cannot-serve-the-claim.md | 8 +++---- ...n-write-a-vm-orders-its-template-claims.md | 12 +++++----- ...s-host-layer-rights-with-the-host-layer.md | 14 +++++------ ...is-immutable-and-a-pool-names-its-class.md | 10 ++++---- ...s-carries-the-tolerations-binding-reads.md | 6 ++--- .../0011-a-gpuclaim-owns-its-usage-record.md | 2 +- ...and-records-and-the-controller-prepares.md | 2 +- 11 files changed, 46 insertions(+), 46 deletions(-) diff --git a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md index 9698e44b..be5e919c 100644 --- a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md +++ b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md @@ -1,4 +1,4 @@ -# 0001. A claim whose class resource name KubeVirt does not permit is refused at VMI admission +# 0001. A class whose resource name KubeVirt does not permit is refused at VMI admission - **Number:** `0001` - **Date:** `2026-10-20` @@ -12,11 +12,11 @@ On the `default` and `vgpu` variants the platform writes a `permittedHostDevices` table into the KubeVirt CR whenever it has entries. The proposal assumed that KubeVirt refuses a GPU outside that table when the VMI is admitted. It does not. In KubeVirt v1.9.0 no admitter refuses a GPU or host-device name the table leaves out; the GraceIOVirtualization admitter reads the table only to apply its rules to names listed there, and the refusal is `validatePermittedHostDevices` in [`renderresources.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/renderresources.go), which runs when virt-controller renders the virt-launcher pod. A render error creates no pod, and the VMI stays `Pending` with `Synchronized=False`, reason `FailedCreate`, retried with backoff. It never reaches a final phase. -That defeats the attachment rules of section 2. A claim leaves `Attached` only when its VMI is final or removed, so a claim of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry. +A VM that names such a class would start and never run, with nothing telling the tenant why at start. With held claims (step 2) it is worse: a claim leaves `Attached` only when its VMI is final or removed, so a claim of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry. ## Decision -The VMI webhook reads the effective table from the KubeVirt CR, the resource names of its `pciHostDevices` and `mediatedDevices` entries, and refuses a VMI whose claim set includes a class whose `resourceName` is not in it, with a message naming the class and the name. An unset table means KubeVirt checks nothing, and then neither does the webhook. The controller also sets a `ResourceNameNotPermitted` condition on such a class, so the hoster sees the gap before a tenant hits it. +The VMI webhook reads the effective table from the KubeVirt CR, the resource names of its `pciHostDevices` and `mediatedDevices` entries, and refuses a VMI with a class entry or a claim of a class whose `resourceName` is not in it, with a message naming the class and the name. An unset table means KubeVirt checks nothing, and then neither does the webhook. The controller also sets a `ResourceNameNotPermitted` condition on such a class, so the hoster sees the gap before a tenant hits it. ## Why not the alternatives @@ -26,9 +26,9 @@ The VMI webhook reads the effective table from the KubeVirt CR, the resource nam ## Consequences -- A tenant learns at start, from the webhook message, that the offer is not usable yet, and the claim stays unattached and moves by the usual rules. +- A tenant learns at start, from the webhook message, that the offer is not usable yet; in step 2 the claim stays unattached and moves by the usual rules. - The webhook and the controller condition read the same lists of the table, so the condition predicts the refusal. KubeVirt's own check also covers `usb` entries, which no GPU class uses. -- Every VMI admission with claim references reads the KubeVirt CR from the API server, not from the controller's cache: the replicas serve admission and a lagging cache could admit what should be refused. +- Every VMI admission with class or claim entries reads the KubeVirt CR from the API server, not from the controller's cache: the replicas serve admission and a lagging cache could admit what should be refused. ## Revisit if diff --git a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md index e6971e57..c2938538 100644 --- a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md +++ b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md @@ -1,4 +1,4 @@ -# 0002. GPUClaim works only where gpu-operator serves GPUs to VMs, while its finalizers and usage records always run +# 0002. GPU classes serve VMs only where gpu-operator serves GPUs to VMs, while claim finalizers and usage records always run - **Number:** `0002` - **Date:** `2026-10-21` @@ -10,30 +10,30 @@ ## Context -The proposal rolls step 1 out "on the existing variants" and says nothing about a cluster where GPUs do not reach VMs: gpu-operator under the `container` variant, gpu-operator not enabled, or no iaas bundle. On such a cluster KubeVirt runs without the `HostDevices` feature gate and without a `permittedHostDevices` table, and nothing there resolves a claim reference. +The proposal rolled step 1 out "on the existing variants" and said nothing about a cluster where GPUs do not reach VMs: gpu-operator under the `container` variant, gpu-operator not enabled, or no iaas bundle. On such a cluster KubeVirt runs without the `HostDevices` feature gate and without a `permittedHostDevices` table, and nothing there resolves a claim reference. -The `vm-instance` chart renders only the reference, on the VMI template, and leaves it to the VMI webhook to fill in the GPUs (section 3). Without the webhook a `claim` or `claimTemplate` entry would boot the VM with no GPU and no error. The CRDs, on the other hand, ship with the controller on every cluster, so a `GPUClass` or a `GPUClaim` can exist on a cluster that has no placement running, and both carry finalizers. +The `vm-instance` and `kubernetes-nodes` charts render only the entries, on the VMI template, and leave it to the VMI webhook to fill in the GPUs (section 4). Without the webhook a class or claim entry would boot the VM with no GPU and no error. The CRDs, on the other hand, ship with the controller on every cluster, so a `GPUClass` or, in step 2, a `GPUClaim` can exist on a cluster that has no placement running, and in step 2 both carry finalizers. ## Decision -The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the binder, the VMI and virt-launcher webhooks, the controller's watches on KubeVirt objects, the GPU admission policies, the metrics scrape, and the tenant admin's write access to `GPUClaim`. Elsewhere: +The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the VMI and virt-launcher pod webhooks, the class status controller and its metrics scrape, and in step 2 the binder, the controller's watches on KubeVirt objects, the GPU admission policies and the tenant admin's write access to `GPUClaim`. Elsewhere: -- `vm-instance` refuses to render `claim` and `claimTemplate` entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today. -- The platform refuses `gpu.rawResourceNames: forbidden`, because the policy that enforces it does not render there. -- The controller still adds and removes the finalizers of `GPUClass` and `GPUClaim` and keeps the usage records of claims, so deleting a class, a claim or a tenant namespace never hangs. +- `vm-instance` and `kubernetes-nodes` refuse to render class and claim entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today. +- In step 2 the platform refuses `gpu.rawResourceNames: forbidden`, because the policy that enforces it does not render there. +- In step 2 the controller still adds and removes the finalizers of `GPUClass` and `GPUClaim` and keeps the usage records of claims, so deleting a class, a claim or a tenant namespace never hangs. ## Why not the alternatives -- **Run everything on every cluster.** The binder and the webhooks read KubeVirt objects, which do not exist without KubeVirt, and under the `container` variant the GPUs gpu-operator advertises are meant for pods, not VFIO devices a VM can take. A claim bound there would be held and recorded for a GPU no VM can use. +- **Run everything on every cluster.** The webhooks and the binder read KubeVirt objects, which do not exist without KubeVirt, and under the `container` variant the GPUs gpu-operator advertises are meant for pods, not VFIO devices a VM can take. A claim bound there would be held and recorded for a GPU no VM can use. - **Gate the finalizer paths with the rest.** A hoster who disables GPUs, or moves to the `container` variant, would then leave every existing class and claim stuck on its finalizer, and with it the tenant namespaces that hold claims. -- **Let `vm-instance` render claim references anyway and rely on the webhook.** On these clusters no webhook exists, so the failure is a VM that runs without the GPU it asked for, which the tenant may notice only from inside the guest. +- **Let the charts render class and claim entries anyway and rely on the webhook.** On these clusters no webhook exists, so the failure is a VM that runs without the GPU it asked for, which the tenant may notice only from inside the guest. ## Consequences -- On a cluster without GPUs for VMs, a tenant reads the catalog and its claims but cannot order, and a VM with claim references fails at render with a message that says GPUs for VMs are not served. +- On a cluster without GPUs for VMs, a tenant reads the catalog, and in step 2 its claims, but cannot use them, and a VM with class or claim entries fails at render with a message that says GPUs for VMs are not served. - Turning GPUs for VMs on writes one new cluster value, which every tenant release reads, so each upgrades once, and only on such clusters. -- The controller keeps a resync loop while a deleted claim waits for its VMI, because with the feature off nothing watches VMIs. +- In step 2 the controller keeps a resync loop while a deleted claim waits for its VMI, because with the feature off nothing watches VMIs. ## Revisit if -GPUs reach VMs through another path than gpu-operator's VM variants, for example a DRA driver installed on its own (section 8). +GPUs reach VMs through another path than gpu-operator's VM variants, for example a DRA driver installed on its own (section 10). diff --git a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md index d8478290..af4b6690 100644 --- a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md +++ b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md @@ -2,7 +2,7 @@ - **Number:** `0004` - **Date:** `2026-10-20` -- **Status:** Accepted +- **Status:** Superseded by [`0015`](./0015-node-allocatable-is-the-gpu-inventory.md) - **Deciders:** `@lexfrei` - **Proposal:** [`../README.md`](../README.md) - **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) diff --git a/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md index 49574ff3..727eda52 100644 --- a/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md +++ b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md @@ -10,7 +10,7 @@ ## Context -Section 2 keeps binds, moves, rebinds and gathers off nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and leaves out "the `node.kubernetes.io/*` taints Kubernetes sets itself for a cordon or a node that is not `Ready`", because the cordon and readiness rules already cover those nodes. Read as a prefix, that sentence would also skip the other well-known taints Kubernetes defines under `node.kubernetes.io/`: `memory-pressure`, `disk-pressure`, `pid-pressure`, `network-unavailable` and `out-of-service`. None of them mirrors a cordon or readiness, and a node carrying one of them as `NoSchedule` or `NoExecute` would still be bound to, though the scheduler keeps a virt-launcher pod that does not tolerate it off the node. +Section 2 kept binds, moves, rebinds and gathers off nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and left out "the `node.kubernetes.io/*` taints Kubernetes sets itself for a cordon or a node that is not `Ready`", because the cordon and readiness rules already cover those nodes. Read as a prefix, that sentence would also skip the other well-known taints Kubernetes defines under `node.kubernetes.io/`: `memory-pressure`, `disk-pressure`, `pid-pressure`, `network-unavailable` and `out-of-service`. None of them mirrors a cordon or readiness, and a node carrying one of them as `NoSchedule` or `NoExecute` would still be bound to, though the scheduler keeps a virt-launcher pod that does not tolerate it off the node. ## Decision diff --git a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md index 6a498005..ebc6c749 100644 --- a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md +++ b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md @@ -10,17 +10,17 @@ ## Context -Section 3 pins a VMI to a cordoned node when a claim of a `Stay` class is there and all of the VMI's claims are on that node, with no template claims: there is no gather, "even if the node is also not `Ready`", and the VMI waits. Section 2 has a `Stay` claim keep its node through a cordon, while lost capacity, a node that is gone and a node that no longer serves the class are hard reasons for a claim to leave. The proposal did not say which of the two wins when a pinned node also has a hard reason, so a node under maintenance that then died could hold a VM's claims, and keep them in the usage records, for as long as the dead node stayed registered. +Section 3 pinned a VMI to a cordoned node when a claim of a `Stay` class was there and all of the VMI's claims were on that node, with no template claims: there was no gather, "even if the node is also not `Ready`", and the VMI waited. Section 2 had a `Stay` claim keep its node through a cordon, while lost capacity, a node that is gone and a node that no longer serves the class were hard reasons for a claim to leave. The proposal did not say which of the two wins when a pinned node also has a hard reason, so a node under maintenance that then died could hold a VM's claims, and keep them in the usage records, for as long as the dead node stayed registered. ## Decision -The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint its classes do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (sections 2 and 3): a new VMI's set is gathered onto one node with room for all of it while its launcher pod waits at the gate, or the pod stays gated when no node fits; a VMI whose pod is already pinned there is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The gather at VMI start and the controller's background pass read the pin through one function, so a VMI is never pinned where the next pass deletes it. +The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint its classes do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (section 9): a new VMI's set is gathered onto one node with room for all of it while its launcher pod waits at the gate, or the pod stays gated when no node fits; a VMI whose pod is already pinned there is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The gather at VMI start and the controller's background pass read the pin through one function, so a VMI is never pinned where the next pass deletes it. ## Why not the alternatives - **The pin always wins.** A `Stay` class exists for a hoster who wants VMs back on the same hardware after maintenance. A dead node is not coming back to the same state on any known schedule, and the VM's claims would stay held and recorded against hardware that serves nothing. -- **Break the pin on `NotReady` alone.** It undoes the point of `Stay` for the most common maintenance step, a reboot, and section 2 already uses the lost-capacity grace period so that a short outage moves nothing. -- **Break the pin on any taint.** Apart from a gather at VMI creation, a taint never moves a bound claim; it only keeps new binds off the node (section 2). Breaking the pin on one would make a `Stay` claim leave in a case where a claim of a `Move` class on an uncordoned node stays. +- **Break the pin on `NotReady` alone.** It undoes the point of `Stay` for the most common maintenance step, a reboot, and step 2 already uses the lost-capacity grace period so that a short outage moves nothing. +- **Break the pin on any taint.** Apart from a gather at VMI creation, a taint never moves a bound claim; it only keeps new binds off the node (section 9). Breaking the pin on one would make a `Stay` claim leave in a case where a claim of a `Move` class on an uncordoned node stays. ## Consequences diff --git a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md index b3100ad4..c6e97493 100644 --- a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md +++ b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md @@ -1,4 +1,4 @@ -# 0007. Whoever can write a VM orders its template claims, while held claims stay with tenant admins +# 0007. Whoever can write a VM uses the GPUs of its class entries, while held claims stay with tenant admins - **Number:** `0007` - **Date:** `2026-10-20` @@ -10,23 +10,23 @@ ## Context -The Security section gives `GPUClaim` writes to tenants with the `admin` role or above, and has the webhook create `GPUClaim` objects for `claimTemplate` entries, only of the class and count the template names. It does not say who that webhook acts for. The VMI is created by KubeVirt's virt-controller, not by the person who wrote the VM, so by the time the webhook sees it the requester is KubeVirt and the identity behind the VM is gone. +The Security section gave `GPUClaim` writes to tenants with the `admin` role or above, and had the webhook create `GPUClaim` objects for `claimTemplate` entries, only of the class and count the template names. It did not say who that webhook acts for. The VMI is created by KubeVirt's virt-controller, not by the person who wrote the VM, so by the time the webhook sees it the requester is KubeVirt and the identity behind the VM is gone. In Cozystack more identities can write a `vm-instance` than can write a `GPUClaim`. The tenant role that tenant ServiceAccounts get grants every verb on `apps.cozystack.io`, which includes `vm-instance`, and the `use` role can start and restart a VM, which creates a new VMI. ## Decision -A `claimTemplate` entry is part of the VM, so whoever can write the VM can order its template claims, and whoever can start it makes them exist and enter the usage records for as long as the VMI runs. The webhook checks the vGPU entitlement for every template claim at VMI admission, the controller checks budgets and capacity when it creates and binds the claim, as for any order, and neither checks the identity behind the VM. Ordering a held `GPUClaim` directly stays with tenant admins and above. +A class entry is part of the VM, so whoever can write the VM uses its GPUs within the tenant's budget, as with the VM's CPU and memory, and whoever can start it makes them run and enter the usage records for as long as the VMI runs; in step 2 the start also makes the entry's template claims exist. The budget is checked when the VM is written (section 5), the vGPU entitlement at every VMI admission, and in step 2 capacity when the controller creates and binds the template claims; none of these checks the identity behind the VM. Ordering a held `GPUClaim` directly stays with tenant admins and above. ## Why not the alternatives - **Check the identity behind the VM at VMI admission.** The webhook sees KubeVirt's service account. The VM's author is recorded nowhere it could read reliably, and the identity that starts a VM can differ from the one that wrote it. -- **Refuse `claimTemplate` entries in `vm-instance` for identities that cannot write `GPUClaim`.** The aggregated API that serves `vm-instance` would need a per-field authorization step for one field of one kind, and the rule would still not cover a start by a `use` role holder, who changes no field at all. The cost is a judgement call, not a hard limit. +- **Refuse class entries in `vm-instance` for identities that cannot write `GPUClaim`.** The aggregated API that serves `vm-instance` would need a per-field authorization step for one field of one kind, and the rule would still not cover a start by a `use` role holder, who changes no field at all. The cost is a judgement call, not a hard limit. ## Consequences -- A tenant admin who hands out a tenant ServiceAccount also hands out the ability to order GPUs through template claims, up to the tenant's budget, and the `use` role lets its holder start VMs that already carry template claims. The budget, the class keys and the vGPU entitlement are the controls that limit both. -- A held claim, which is held and recorded whether or not anything runs on it, still needs a tenant admin. +- A tenant admin who hands out a tenant ServiceAccount also hands out the ability to use GPUs through class entries, up to the tenant's budget, and the `use` role lets its holder start VMs that already carry class entries. The budget, the class keys and the vGPU entitlement are the controls that limit both. +- A held claim, which holds GPUs and is recorded while it is `Bound` or `Attached` whether or not anything runs on it, still needs a tenant admin. ## Revisit if diff --git a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md index 1a8bfc72..a867ed58 100644 --- a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md +++ b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md @@ -1,4 +1,4 @@ -# 0008. The GPU controller gets its node-label and plugin-restart rights with the host layer, not before +# 0008. The GPU controller gets its node-label right with the host layer, not before - **Number:** `0008` - **Date:** `2026-10-20` @@ -10,22 +10,22 @@ ## Context -The Security section lists three rights the GPU controller gains beyond its own kinds and the usage records: patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Only the third is used by Rollout step 1. Node labels and plugin restarts belong to the node layout of section 6, which ships as Rollout step 4, and until then the hoster writes node labels by hand. +The Security section listed three rights the GPU controller gained beyond its own kinds and the usage records: patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Only the third was used by Rollout step 1. Node labels and plugin restarts belonged to the node layout of section 6, which was Rollout step 4, and until then the hoster wrote node labels by hand. ## Decision -Of the three, Rollout step 1 grants the controller only the VMI deletion. The node-label patch right, the policy that restricts it, and the right to restart the device plugin pod ship with the host layer, together with the code that uses them. +The node-label patch right and the policy that restricts it ship with the host layer, together with the code that uses them, and the VMI deletion right ships with step 2, which uses it. The device plugin restart is not the controller's: the node agent of the host layer restarts the plugin pod on its own node after writing profiles, under a policy that limits it to that pod. No GPU code gets a right before the code that uses it. ## Why not the alternatives -- **Grant all three in step 1, as the Security section lists them.** Every cluster would carry a write right on every `Node` that no code uses, and the policy that narrows it would have no writer to test against. +- **Grant all three with the first GPU code, as the Security section listed them.** Every cluster would carry a write right on every `Node` that no code uses, and the policy that narrows it would have no writer to test against. - **Grant the label right without its policy until the host layer ships.** That is a broader right than the proposal ever allowed. ## Consequences -- The `update` and `delete` rights on pods in every namespace that the launcher pod gate of section 3 needs are a separate grant of step 1, not one of the three. -- Between step 1 and step 4 the controller cannot change any node, so the GPU node labels that classes match are the ones the hoster writes. -- The rights and their policy are reviewed in step 4, next to the code that uses them. +- The `update` and `delete` rights on pods in every namespace that the launcher pod gate needs are a separate grant of step 2, not one of the three. +- Until the host layer ships, the controller cannot change any node, so the GPU node labels are the ones the hoster writes. +- The rights and their policy are reviewed with the host layer, next to the code that uses them. ## Revisit if diff --git a/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md b/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md index 7f664f51..cf7b345b 100644 --- a/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md +++ b/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md @@ -1,4 +1,4 @@ -# 0009. `GPUClass.mode` never changes, and a pool's `claimSelector` names its class +# 0009. `GPUClass.mode` never changes, and a pool names its class - **Number:** `0009` - **Date:** `2026-10-23` @@ -10,13 +10,13 @@ ## Context -Section 1 froze a class's `mode` only while its finalizer was present, that is while claims of the class were held. A worker pool selected claims by labels alone, and a selector's claims had to share a class anyway. The mode the pool's workers needed, passthrough or vGPU, came from a separate pool field, `kairos.gpuMode`, which the chart set as a `gpu.cozystack.io/mode` label on the worker VMI template, and the webhook refused a claim whose class mode differed from that label. The implementation accepted a pool reference on a VMI without the label, so the text and the code disagreed. +Section 1 froze a class's `mode` only while its finalizer was present, that is while claims of the class were `Bound`, `Attached` or `Lost`. A worker pool selected claims by labels alone, and a selector's claims had to share a class anyway. The mode the pool's workers needed, passthrough or vGPU, came from a separate pool field, `kairos.gpuMode`, which the chart set as a `gpu.cozystack.io/mode` label on the worker VMI template, and the webhook refused a claim whose class mode differed from that label. The implementation accepted a pool reference on a VMI without the label, so the text and the code disagreed. @kvaps asked in review of #99 and [#101](https://github.com/cozystack/community/pull/101#pullrequestreview-5616165668) for `mode` to be immutable always and for the selector to name the class, so that a pool does not state its mode at all. ## Decision -`GPUClass.mode` cannot change after the class is created, whether or not claims of it exist. A `claimSelector` carries `className` next to its label selector and matches only claims of that class, as a `claimTemplate` already names its class. A pool states no mode: the chart and admission read it from the named class. The mode label and the mode check at VMI admission are dropped. +`GPUClass.mode` cannot change after the class is created, whether or not claims of it exist, and neither can the other fields that define the offer: `vendor`, `model`, `profile`, `memory` and the kind of `source`. A pool's class entry names its class, and in step 2 a `claimSelector` carries `className` next to its label selector and matches only claims of that class. A pool states no mode: the chart and admission read it from the named class. The mode label and the mode check at VMI admission are dropped. ## Why not the alternatives @@ -26,8 +26,8 @@ Section 1 froze a class's `mode` only while its finalizer was present, that is w ## Consequences - A class name stands for one mode for the class's lifetime, so a pool's worker template follows from the class it names. -- A hoster who created a class in the wrong mode deletes it and creates another, once no claim of it is held. -- A selector cannot span classes, which section 3 already required in another form. +- A hoster who created a class in the wrong mode deletes it and creates another, once the class is not in use (section 9). +- A selector cannot span classes, which section 9 requires. - A pool rendered against a class keeps that class's worker image until it renders again, so a hoster who deletes a class does not create another class of the same name in the other mode. Nothing in the platform enforces this; the hoster documentation states it. ## Revisit if diff --git a/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md b/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md index dbf6c285..fae9d915 100644 --- a/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md +++ b/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md @@ -1,4 +1,4 @@ -# 0010. `GPUClass` carries the tolerations that binding and `free` read +# 0010. `GPUClass` carries the tolerations that `free`, binding and the launcher pod read - **Number:** `0010` - **Date:** `2026-10-23` @@ -14,7 +14,7 @@ Section 2 kept binds, moves, rebinds and gathers off nodes with a `NoSchedule` o ## Decision -`GPUClass` has a `tolerations` field, in the shape of a pod's with the operators `Exists` and `Equal` only, which the hoster sets to the taints of the class's nodes that its VMs tolerate. A node counts for binding, moves, rebinds, gathers, capacity, `free` and `maxFreePerNode` of a class only if the class tolerates each of its `NoSchedule` and `NoExecute` taints, except the three that 0005 skips, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, for the reasons 0005 gives. A set of claims of several classes takes a node only when every class of the set tolerates its taints. The virt-launcher pod of a VMI gets the tolerations of its claims' classes when it is created. The class admits only tolerations that pod validation admits, so a class can never produce a launcher pod the API server refuses. `tolerations` can change while claims are held, like `nodeSelector`. +`GPUClass` has a `tolerations` field, in the shape of a pod's with the operators `Exists` and `Equal` only, which the hoster sets to the taints of the class's nodes that its VMs tolerate. A node counts for a class's capacity, `free` and `maxFreePerNode`, and in step 2 for its binds, moves, rebinds and gathers, only if the class tolerates each of its `NoSchedule` and `NoExecute` taints, except the three that 0005 skips, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, for the reasons 0005 gives. A set of claims of several classes takes a node only when every class of the set tolerates its taints. The virt-launcher pod webhook adds the tolerations of the VMI's classes to the pod when it is created. The class admits only tolerations that pod validation admits, so a class can never produce a launcher pod the API server refuses. `tolerations` can change, also while the class is in use. ## Why not the alternatives @@ -30,4 +30,4 @@ Section 2 kept binds, moves, rebinds and gathers off nodes with a `NoSchedule` o ## Revisit if -Kubernetes lets a reservation be scheduled ahead of its pod, so tolerations can be checked against the pod that will use the GPU, for example through DRA (section 8). +Kubernetes lets a reservation be scheduled ahead of its pod, so tolerations can be checked against the pod that will use the GPU, for example through DRA (section 10). diff --git a/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md b/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md index f18377c3..a55181c8 100644 --- a/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md +++ b/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md @@ -16,7 +16,7 @@ Section 5 kept each claim's `Workload` usage record under a `WorkloadMonitor` th ## Decision -Each GPU usage record carries an ownerReference to its `GPUClaim` and no monitor label. The GPU controller keeps it while the claim holds GPUs, as section 5 defines, and garbage collection removes it with the claim. The `Workload` controller leaves a record owned by a `GPUClaim` to its owner instead of deleting it for a missing monitor. There is no GPU `WorkloadMonitor`, no reserved label key and no admission policy for it. +Each GPU usage record carries an ownerReference to its `GPUClaim` and no monitor label. The GPU controller keeps it while the claim holds GPUs, as section 9 defines, and garbage collection removes it with the claim. The `Workload` controller leaves a record owned by a `GPUClaim` to its owner instead of deleting it for a missing monitor. There is no GPU `WorkloadMonitor`, no reserved label key and no admission policy for it. ## Why not the alternatives diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md index 048526a8..c33876d4 100644 --- a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -16,7 +16,7 @@ Section 3 had the mutating webhook on VirtualMachineInstance creation do all of ## Decision -The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the held claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. A second mutating webhook adds the scheduling gate `gpu.cozystack.io/claims` and the classes' tolerations to the virt-launcher pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. For a set made only of template claims, the controller keeps its rejected nodes in memory against the VM, and a controller restart forgets them. `status.bindRequest` is dropped. +The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the held claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. The virt-launcher pod webhook, which from step 1 adds the classes' tolerations, also adds the scheduling gate `cozystack.io/gpuclaims` to the pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. For a set made only of template claims, the controller keeps its rejected nodes in memory against the VM, and a controller restart forgets them. `status.bindRequest` is dropped. ## Why not the alternatives From 542b5b9c81b77da05fb7226bea3457de97a76574 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 19:16:45 +0300 Subject: [PATCH 24/32] docs(design-proposals): trace step 1 GPU behaviour without a hold Without a binder, a drained, dead or failing node in step 1 leaves a VM on a class entry to its run strategy and to any node with a free unit. Say so, keep the raw-key cap on held claims and the layout rule of the step 2 binder, place the tolerations webhook in step 1, and test the class metrics and the two-class taint rule. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 21 +++++++++++-------- ...and-records-and-the-controller-prepares.md | 2 +- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 1bac66ca..a10ecaa1 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -230,7 +230,7 @@ vGPU needs a license, and passthrough needs one only for graphics. Compute (C-se ### 8. Migration of existing `gpus` -Raw `name` entries keep working, and nothing converts them automatically. A tenant moves a VM by replacing each `name` entry with a class entry of the matching class and restarting it; the VM requests the same resource name as before, so the move needs no free unit. A Talos GPU pool on a passthrough name moves by replacing its `name` entries with a class entry of the matching passthrough class, which rolls its workers. A Talos pool on a raw vGPU name has no path here, since Talos workers carry no vGPU guest driver; it keeps its raw name until it moves to a worker OS that does. ComputePlane GPU groups have no path until the module accepts class entries. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes class entries, and a pool on one takes the pool path above. +Raw `name` entries keep working, and nothing converts them automatically. A tenant moves a VM by replacing each `name` entry with a class entry of the matching class and restarting it; the VM requests the same resource name as before, so the move needs no extra unit. A Talos GPU pool on a passthrough name moves by replacing its `name` entries with a class entry of the matching passthrough class, which rolls its workers. A Talos pool on a raw vGPU name has no path here, since Talos workers carry no vGPU guest driver; it keeps its raw name until it moves to a worker OS that does. ComputePlane GPU groups have no path until the module accepts class entries. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes class entries, and a pool on one takes the pool path above. Quota keys move too. A class entry's launcher pod requests the class's resource name, so while a tenant migrates, a raw key on that name limits both kinds of VM at pod creation, and a GPU key of section 5 limits only class entries, at write. After its last raw-name VM has moved, the raw key is replaced by GPU keys. A tenant whose raw-name VMs use vGPU can move them to class entries only after it is entitled to vGPU (section 6). @@ -240,7 +240,7 @@ Raw names stay allowed in step 1. Forbidding them is part of step 2, where it is Step 2 lets a tenant keep GPUs on a node while its VM is stopped, so the next start does not fail for a GPU someone else took. Holding stays GPU-specific and does not become a generic reservation ([0016](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md)): CPU and memory are interchangeable across nodes, while a GPU hold is a whole unit on a specific node that does not migrate, which is why it needs a binder, `Lost`, the gather and `onCordon`. DRA would be the general path, but KubeVirt [VEP 344](https://github.com/kubevirt/enhancements/issues/344) (VM-scoped persistent ResourceClaims) is open and unimplemented, and the vGPU VF is handed out by a device plugin (section 2), not by a DRA driver. Holding moves to DRA once VEP 344 is implemented and a DRA driver hands out vGPU VFs. -Nothing in this section exists in step 1. Step 2 adds `GPUClaim`, the `onCordon` field and a finalizer on `GPUClass`, `claim` entries in `vm-instance`, `claimSelector` entries in `kubernetes-nodes`, the binder, the launcher pod gate and webhook, the claim order webhook, claim usage records and `gpu.rawResourceNames: forbidden`. +Nothing in this section exists in step 1. Step 2 adds `GPUClaim`, the `onCordon` field and a finalizer on `GPUClass`, `claim` entries in `vm-instance`, `claimSelector` entries in `kubernetes-nodes`, the binder, the scheduling gate in the launcher pod webhook, the claim order webhook, claim usage records and `gpu.rawResourceNames: forbidden`. #### Objects @@ -286,7 +286,7 @@ A tenant sees a class, a count, a phase and, once there is one, a node, never wh #### Binding and capacity -A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node of its class that has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, by the taint rule of section 3, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI start (Placing a VMI's claims): it may take a claim attached only to the starting VMI off a node that is not `Ready`, because no other VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. +A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node of its class that has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, nodes with a pending layout change (section 2), and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, by the taint rule of section 3, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI start (Placing a VMI's claims): it may take a claim attached only to the starting VMI off a node that is not `Ready`, because no other VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. Admission refuses an order that cannot be met at that moment, by capacity or by budget (Budgets and usage records of held claims), with a message naming the class, the count and the limit it hit: a budget of the pool, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. @@ -357,7 +357,7 @@ A claim that is already attached when its node is cordoned, to a VMI whose pod h #### Budgets and usage records of held claims -A held claim counts against the same GPU keys as a class entry, from the moment it is `Bound` until it is deleted (the phase table), in the same pool figure as the class entries of section 5; a `claim` or `claimSelector` entry declares nothing of its own, since its claims already count. The validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, that pool figure and the vGPU entitlement. A raw GPU key on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has a GPU key covering the class, so a held claim, which no `ResourceQuota` sees, never bypasses an existing raw limit. Clearing the vGPU flag refuses new vGPU orders; claims already held keep counting and can still move or be rebound, so a pool on `claimSelector` keeps its claims. +A held claim counts against the same GPU keys as a class entry, from the moment it is `Bound` until it is deleted (the phase table), in the same pool figure as the class entries of section 5; a `claim` or `claimSelector` entry declares nothing of its own, since its claims already count. The validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, that pool figure and the vGPU entitlement. A raw GPU key on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has a GPU key covering the class, so a held claim, which no `ResourceQuota` sees, never bypasses an existing raw limit. While the raw key exists, it also caps the claims of those classes together with the raw-name VMs it covers. Clearing the vGPU flag refuses new vGPU orders; held claims already ordered keep counting and can still move or be rebound, so a pool on `claimSelector` keeps its claims. Each held claim gets a usage record with `kind: gpu-claim` and resources `.gpuclass.cozystack.io/devices: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 10). The GPU controller writes the record itself, with its `GPUClaim` as the record's owner through an ownerReference, and with no `WorkloadMonitor` and no label selector, so no tenant label can add a record or drop one ([0011](./decisions/0011-a-gpuclaim-owns-its-usage-record.md)). The `Workload` controller, which today deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, leaves a record owned by a `GPUClaim` to its owner: the GPU controller deletes it when the claim leaves `Bound` and `Attached`, and garbage collection deletes it with the claim. A virt-launcher pod's `Workload` does not report GPUs its held claims cover a second time. @@ -431,7 +431,7 @@ Step 2 adds: - Tenant access. Tenants with the `admin` role or above manage `GPUClaim` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. - What a tenant controls. A tenant never supplies a node through a claim reference: it comes from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. The webhook takes claims only from the VM's own namespace. With `forbidden` any GPU entry without a class or claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. -- Webhooks. Three GPU webhooks fail closed. The VMI and virt-launcher pod webhooks see only objects carrying the reference label, so their outage blocks the start of VMs with class or claim entries. The pod webhook has to fail closed, because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order, so its outage also blocks new orders, while held claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity when it creates and binds the claim. +- Webhooks. Step 2 adds a third fail-closed GPU webhook, on orders, and the gate in the launcher pod webhook of step 1. The VMI and launcher pod webhooks still see only objects carrying the reference label, so their outage blocks the start of VMs with class or claim entries. The pod webhook has to fail closed, also because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order, so its outage also blocks new orders, while held claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity when it creates and binds the claim. - Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `cozystack.io/gpuclaim-template` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so a held claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. - Template claims. For class entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the entry names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, uses the GPUs of its class entries, and anyone who can start it makes their claims exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). - Controller. It gains deleting unscheduled VMIs that reference its claims, and `update` and `delete` on pods in every namespace, cluster-wide like its write on Secrets, because tenant namespaces appear at runtime. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses. It also writes one ConfigMap, the GPU settings in `cozy-system`. @@ -449,6 +449,9 @@ Step 1: - A class shows `free: 0` while its nodes have GPUs to spare → the nodes carry a `NoSchedule` or `NoExecute` taint the class does not tolerate; add it to the class's `tolerations`, or remove the taint. - A class has a `ResourceNameConflict` condition → another class has its resource name; delete one of them or point it at its own name. - A class is deleted while VMs name it → running VMs keep running, and their next start is refused naming the class; the tenant moves them to another class. +- A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. A VM whose `runStrategy` starts a new instance gets it on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. +- A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. A VM whose `runStrategy` starts a new instance then gets it on any node with a free unit of its class; `Once` and `Manual` VMs keep their failed instance and stay stopped until the tenant starts them. +- A GPU fails under a running VM on a class entry → the VM keeps running. Kubelet stops listing the device as allocatable, so the class's `free` drops, and the VM's next instance gets a unit on a node that still has one. - A layout change does not apply → the node carries the `cozystack.io/gpu-layout-pending` label; cordon and drain it. A GPU VM on the node holds the change until it stops or is evicted. - A Talos pool with a vGPU class entry is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. @@ -473,7 +476,7 @@ Step 2: Step 1: -- Unit tests for the class status: `free` and `maxFreePerNode` from node allocatable and pod requests, a node with a `NoSchedule` taint the class does not tolerate left out and one whose taint it tolerates counted, the three skipped taints, the Fence Agents Remediation taint alone not making a node cordoned, and a second class with the same resource name getting `ResourceNameConflict`. +- Unit tests for the class status: the capacity and used-unit metrics match the nodes and pods and none of them appears in `GPUClass` status, `free` and `maxFreePerNode` from node allocatable and pod requests, a node whose pods request more than its allocatable counting zero, a node with a `NoSchedule` taint the class does not tolerate left out and one whose taint it tolerates counted, the three skipped taints, the Fence Agents Remediation taint alone not making a node cordoned, and a second class with the same resource name getting `ResourceNameConflict`. - Webhook tests for class entries: one `gpus` entry per unit with the class's resource name, the classes' tolerations on the launcher pod and on the temporary `WaitForFirstConsumer` pod and still there after a LiveUpdate sync of the VM, each refusal of section 4, a VM with a GPU class entry and `nvidia.com/hostdev` passing, and a dry run. - Budget tests: the interim check refuses a `VMInstance` and a `KubernetesNodes` write over a class key or the total key, counts `count` times `maxReplicas` for a pool and the defaulted `maxReplicas` for a pool that does not set it, passes shrinking and no-op updates, refuses a child tenant's GPU key against the class entries already in the parent's pool, applies the pool rules like other keys, stands aside while #48's gate is on, and reaches the same verdicts as #48's evaluator on the same inputs. - Admission tests for `KubernetesNodes`: a vGPU class entry on a Talos pool is refused, and a passthrough one is accepted. @@ -486,7 +489,7 @@ Step 1: Step 2: -- Unit tests for the binder and the three GPU webhooks: capacity with raw-name usage, budgets in the pool figure with class entries, the phase table, every move and refusal in section 9, concurrent starts, the gate added once on a repeated call and on the temporary `WaitForFirstConsumer` pod but not on a migration target, the pin appended to every existing node selector term and the gate removed in the same update, and the pod pin after a LiveUpdate sync. +- Unit tests for the binder and the three GPU webhooks: capacity with raw-name usage, a set of two classes skipping a node only one of them tolerates, budgets in the pool figure with class entries, the phase table, every move and refusal in section 9, concurrent starts, the gate added once on a repeated call and on the temporary `WaitForFirstConsumer` pod but not on a migration target, the pin appended to every existing node selector term and the gate removed in the same update, and the pod pin after a LiveUpdate sync. - Webhook and controller tests for cordons, template claims and NICs: the webhook binds, moves and creates no claim, a set that cannot be placed keeps the pod gated with a condition and is released once it can, a pinned pod that the node refuses is deleted and that node is avoided for the set, under `Move` a claim moves at VMI start and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity a VMI with a pinned pod is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, a gated VMI is not deleted and its claims are gathered, a cordon or `NotReady` on the pinned node does not count as a refusal, a `Once` or `Manual` VM stays stopped after a refused pod is deleted, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only class entries gets one node with room for all of its template claims, and under `Stay` a single claim waits; a template claim is created by the controller once the pod waits at the gate, under the capacity checks, owned by the VMI, deleted with its VMI, reused on a repeated pass, counted against no budget of its own, and a dry run records nothing. - Quota tests for held claims: a held claim counts from `Bound`, a `claim` entry declares nothing, a raw key blocks orders of its class until a GPU key covers it, and a cleared vGPU flag keeps held claims and refuses new orders. - Admission tests for `maxReplicas`, class lifecycle while the class is in use, tenant finalizers, the controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no class or claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. @@ -498,12 +501,12 @@ Step 2: Step 1: 1. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. It needs no tenant API and can land first. -2. `GPUClass` with its status controller and metrics, class entries in `vm-instance` and `kubernetes-nodes` with the VMI webhook, the `gpu.vgpu` Tenant flag, GPU keys with the interim check, and the `gpuclass` option source, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRD ships and the charts refuse class entries. Before tenants use vGPU classes, the `gpu.vgpu` flag is set from the root tenant down. It works on any cluster where each offer it publishes has a resource name of its own, with or without item 1. +2. `GPUClass` with its status controller and metrics, class entries in `vm-instance` and `kubernetes-nodes` with the VMI webhook and the launcher pod webhook for tolerations, the `gpu.vgpu` Tenant flag, GPU keys with the interim check, and the `gpuclass` option source, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRD ships and the charts refuse class entries. Before tenants use vGPU classes, the `gpu.vgpu` flag is set from the root tenant down. It works on any cluster where each offer it publishes has a resource name of its own, with or without item 1. 3. GPU keys through #48, and the interim check deleted, in the release where #48's gate is on by default. Step 2: -4. Holding: `GPUClaim`, the binder and its metrics, the launcher pod webhook and gate, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. +4. Holding: `GPUClaim`, the binder and its metrics, the scheduling gate in the launcher pod webhook, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. Then: diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md index c33876d4..8a6ad5a4 100644 --- a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -30,7 +30,7 @@ The VMI webhook resolves the claim references, runs the checks that need no plac - The webhook is a pure check plus the attachment record, so its latency no longer depends on the binder, and a dry run only skips the record. - Template claims are created after the VMI exists, so the controller sets their owner at creation, and a template claim no longer needs a grace period for a VMI that never appears. - A set that cannot be placed is no longer an admission error. The VM shows `Starting`, the VMI waits in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated`, and the claims carry a condition naming the reason while the VMI gets a `Warning` event with the same reason. The VM starts as soon as the set fits, with no KubeVirt backoff in between. -- The pod webhook is a fail-closed dependency of every start of a VM with claim references, like the VMI webhook. +- The pod webhook is a fail-closed dependency of every start of a VM with class or claim entries, like the VMI webhook. - The cluster autoscaler does not scale up for a gated pod, and after the pin a new node cannot help that pod, so GPU node scale-up stays with the hoster. - A pin cannot be undone, so a node that refuses a pinned pod costs one failed VMI and KubeVirt's start-failure backoff, which the rejected-node rule keeps from repeating. From 1457bbd69e06952d71a2e573ff1f4bf6772882bf Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 19:32:55 +0300 Subject: [PATCH 25/32] docs(design-proposals): state GPU VM runStrategy outcomes once, from KubeVirt Whether KubeVirt creates a next VMI depends on how the previous one ended: a start-failure backoff applies only to a VMI that never ran, and a VMI the controller deletes goes Failed first. One table at the top of Failure and edge cases holds this, with KubeVirt v1.9.0 line references, and every case states an outcome only where its runStrategy produces one. Step 2 also leaves nodes with a pending layout change out of free. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 33 ++++++++++++++++++---------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index a10ecaa1..dc84831b 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -310,7 +310,7 @@ A drop in allocatable alone is not a loss, since kubelet reports zero after ever - the affected classes get a `LostCapacity` condition; - unattached claims that no longer fit move to another node of their class, or to `Lost` when none has room; - on a `NotReady` node, attached claims go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI start is moving off the node is the exception: it finishes the move; -- a VM started after the condition is set has its claims gathered off the node before its pod is released, and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. What the VM does next follows its `runStrategy` for a deleted VMI: `Always` and `Once` create a new VMI and start once the claim has moved or been rebound from `Lost`, while `Manual` and `RerunOnFailure` stay stopped until the tenant starts the VM (Failure cases); unlike a cordon, the node cannot serve the claim, so waiting is no option. +- a VM started after the condition is set has its claims gathered off the node before its pod is released, and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. Where the VM's `runStrategy` creates a next VMI after such a deletion (Failure and edge cases), it starts once the claim has moved or been rebound from `Lost`; unlike a cordon, the node cannot serve the claim, so waiting is no option. Claims in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a claim whose VMI is still live on the lost node waits until that VMI is final or removed (above), while a claim of a VMI whose pod is still gated is rebound for that VMI. The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` claim already counts. @@ -351,9 +351,9 @@ For a gated pod the controller creates the VMI's template claims, then brings th Once the set is on one node, the controller pins the pod and removes the gate in one update. The pin is a `matchFields` requirement on `metadata.name` appended to every required node selector term of the pod, or the single term when it has none. Kubernetes lets a gated pod only narrow its scheduling, so a term cannot be added next to existing ones, and KubeVirt usually sets a required term already, for host-model CPUs. The pin is not `nodeName`, which would let kubelet start the pod on a cordoned node, and it goes on the pod rather than the VMI because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. -When the set cannot be placed, because no node has room or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. The VMI gets no condition because KubeVirt's admission guards writes to a VMI. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules. After the deletion the VMI becomes `Failed`, and the VM's `runStrategy` decides what follows: `Always` and `RerunOnFailure` start again after KubeVirt's start-failure backoff, with the VM shown as `CrashLoopBackOff`; a `Once` VM keeps its failed VMI, so it stays down and also shows `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it (Failure cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. A set of template claims only loses its claims with the failed VMI, so for it the controller keeps the record in memory against the VM, and a controller restart forgets it. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. +When the set cannot be placed, because no node has room or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. The VMI gets no condition because KubeVirt's admission guards writes to a VMI. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules. After the deletion the VMI becomes `Failed`, and KubeVirt decides what happens next by the VM's `runStrategy` (Failure and edge cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. A set of template claims only loses its claims with the failed VMI, so for it the controller keeps the record in memory against the VM, and a controller restart forgets it. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. -A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way while the pod is gated. Once the pod is pinned, under `Move`, for a VM with `runStrategy` `Always` or `Once` the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure cases). +A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way while the pod is gated. Once the pod is pinned, under `Move`, for a VM whose `runStrategy` creates a next VMI after such a deletion, `Always` or `Once`, the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure and edge cases). #### Budgets and usage records of held claims @@ -361,7 +361,7 @@ A held claim counts against the same GPU keys as a class entry, from the moment Each held claim gets a usage record with `kind: gpu-claim` and resources `.gpuclass.cozystack.io/devices: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 10). The GPU controller writes the record itself, with its `GPUClaim` as the record's owner through an ownerReference, and with no `WorkloadMonitor` and no label selector, so no tenant label can add a record or drop one ([0011](./decisions/0011-a-gpuclaim-owns-its-usage-record.md)). The `Workload` controller, which today deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, leaves a record owned by a `GPUClaim` to its owner: the GPU controller deletes it when the claim leaves `Bound` and `Attached`, and garbage collection deletes it with the claim. A virt-launcher pod's `Workload` does not report GPUs its held claims cover a second time. -The catalog's `free` and `maxFreePerNode` subtract held units from step 2 on. A hoster lists bound claims per node with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write, and the class's bound units and `Lost` count join its metrics. A `gpuclaim` option source lists a tenant's own claims. +The catalog's `free` and `maxFreePerNode` subtract held units from step 2 on, and leave out nodes with a pending layout change, which the binder skips. A hoster lists bound claims per node with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write, and the class's bound units and `Lost` count join its metrics. A `gpuclaim` option source lists a tenant's own claims. #### Forbidding raw names @@ -441,6 +441,17 @@ Step 2 adds: Each case gives what is seen and what to do. +Some cases depend on whether KubeVirt creates a next VMI for the VM. In KubeVirt v1.9.0 that depends on the `runStrategy` and on how the VMI ended ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go), [`vmi/lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go), [`rest/lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/rest/lifecycle.go); line numbers at that tag): + +| `runStrategy` | VMI failed before it ever ran | VMI failed after it ran | Controller deleted an unscheduled VMI | +| --- | --- | --- | --- | +| `Always` | next VMI after the start-failure backoff (`vm.go` L987, L1025, L1580) | next VMI, no backoff (`vm.go` L987, L1539) | next VMI (`vm.go` L1025; the VMI goes `Failed` first, `vmi/lifecycle.go` L447, L456) | +| `RerunOnFailure` | next VMI after the start-failure backoff (`vm.go` L1050, L1074, L1580) | next VMI, no backoff (`vm.go` L1050, L1539) | none until the tenant starts the VM (`vm.go` L1050, L1070; `rest/lifecycle.go` L120) | +| `Once` | none (`vm.go` L1141) | none (`vm.go` L1141) | next VMI (`vm.go` L1141) | +| `Manual` | none until the tenant starts the VM (`vm.go` L1104, `rest/lifecycle.go` L120) | the same | the same | + +The start-failure backoff grows with each failure that happens before a VMI runs, up to five minutes, and resets once a VMI runs. A VMI that a drain evicts ends on its own, and [`vgpu-vms`](../vgpu-vms/README.md) gives what each `runStrategy` does then. + Step 1: - A write of a VM or pool is refused → the message names the pool, the GPU key, the limit and what is already reserved; use a smaller `count`, another class, or ask for more budget. @@ -449,8 +460,8 @@ Step 1: - A class shows `free: 0` while its nodes have GPUs to spare → the nodes carry a `NoSchedule` or `NoExecute` taint the class does not tolerate; add it to the class's `tolerations`, or remove the taint. - A class has a `ResourceNameConflict` condition → another class has its resource name; delete one of them or point it at its own name. - A class is deleted while VMs name it → running VMs keep running, and their next start is refused naming the class; the tenant moves them to another class. -- A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. A VM whose `runStrategy` starts a new instance gets it on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. -- A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. A VM whose `runStrategy` starts a new instance then gets it on any node with a free unit of its class; `Once` and `Manual` VMs keep their failed instance and stay stopped until the tenant starts them. +- A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. Where its `runStrategy` creates a new instance, it starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. +- A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. Where its `runStrategy` creates a next VMI after a failure, that VMI starts on any node with a free unit of its class. - A GPU fails under a running VM on a class entry → the VM keeps running. Kubelet stops listing the device as allocatable, so the class's `free` drops, and the VM's next instance gets a unit on a node that still has one. - A layout change does not apply → the node carries the `cozystack.io/gpu-layout-pending` label; cordon and drain it. A GPU VM on the node holds the change until it stops or is evicted. - A Talos pool with a vGPU class entry is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. @@ -461,12 +472,12 @@ Step 2: - A claim stays `Pending` after an order → its condition names the shortage; it binds when capacity or budget frees up, or the tenant deletes it. - A VM with claims is refused at start → the webhook refuses the VMI for a reference problem, such as a claim that does not exist, is being deleted or is attached to another VMI, and the VirtualMachine shows the message in a `FailedCreate` event, naming each claim and the reason; fix the reference. - A VM with claims stays `Starting` → its VMI is in `Scheduling` with its pod gated, the claims carry a condition and the VMI a `Warning` event naming the reason: no node can take the whole set, template claims included, or a claim of a `Stay` class is on a cordoned node while others need gathering. The VM starts by itself once the set can be placed. To get there sooner, uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. -- The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. An `Always` or `RerunOnFailure` VM shows `CrashLoopBackOff` and starts again after a delay that grows with each failure up to five minutes, and the count resets once a VMI runs; a `Once` VM keeps its failed VMI and stays down, also shown as `CrashLoopBackOff`, and a `Manual` VM stays stopped until the tenant starts it. The binder avoids that node for the set for ten minutes or until the node changes. Free CPU or memory on GPU nodes if this repeats. +- The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. Where its `runStrategy` creates a next VMI after a failure before running, that VMI waits for the start-failure backoff. The binder avoids that node for the set for ten minutes or until the node changes. Free CPU or memory on GPU nodes if this repeats. - A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. -- A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and what the VM does next follows its `runStrategy` for a VMI that was deleted, not one that finished: in KubeVirt v1.9.0 ([`pkg/virt-controller/watch/vm/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)) `Always` and `Once` create a new VMI when none exists, while `Manual` and `RerunOnFailure` wait for a start request, so the VM stays stopped until the tenant starts it. -- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of an `Always` or `RerunOnFailure` VM gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. `Once` and `Manual` VMs keep their failed VMI, so no new one is created: a `Once` VM stays stopped, and a `Manual` VM starts when the tenant starts it. The failed VMI of a `Once` or `Manual` VM leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. -- A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; an `Always` or `Once` VM starts again once its claims have moved or been rebound, a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. -- A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has. An `Always` or `Once` VM then starts on the new node once its claims have moved or been rebound; a `Manual` or `RerunOnFailure` VM stays stopped until the tenant starts it. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. +- A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and where the VM's `runStrategy` creates a next VMI after such a deletion, that VMI gets the moved claim. +- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of a VM whose `runStrategy` creates one after a VMI that ran fails gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. A VM whose `runStrategy` creates no next VMI keeps its failed VMI, which leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. +- A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; where the VM's `runStrategy` creates a next VMI after such a deletion, it starts once the claims have moved or been rebound. +- A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has. Where the VM's `runStrategy` creates a next VMI after such a deletion, it starts on the new node once the claims have moved or been rebound. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. - A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts, the controller moves the claim off that node before it releases the pod, or the claim goes to `Lost` and the pod stays gated until it is rebound. - A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use class entries or a `claimSelector`, or allow raw names again while the cluster has such pools. - A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. From a0b4e3e29d83216078cadd90b2ca6ed0830ed639 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 19:36:20 +0300 Subject: [PATCH 26/32] docs(design-proposals): restore CrashLoopBackOff in the runStrategy table A VMI that the controller deletes before it ran counts as a start failure, so an Always VM waits the backoff, and during the backoff the VM shows CrashLoopBackOff, as a Once VM does without retrying. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index dc84831b..a54200ff 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -445,12 +445,12 @@ Some cases depend on whether KubeVirt creates a next VMI for the VM. In KubeVirt | `runStrategy` | VMI failed before it ever ran | VMI failed after it ran | Controller deleted an unscheduled VMI | | --- | --- | --- | --- | -| `Always` | next VMI after the start-failure backoff (`vm.go` L987, L1025, L1580) | next VMI, no backoff (`vm.go` L987, L1539) | next VMI (`vm.go` L1025; the VMI goes `Failed` first, `vmi/lifecycle.go` L447, L456) | +| `Always` | next VMI after the start-failure backoff (`vm.go` L987, L1025, L1580) | next VMI, no backoff (`vm.go` L987, L1539) | next VMI after the start-failure backoff (`vm.go` L1025, L1580; the VMI goes `Failed` first, `vmi/lifecycle.go` L447, L456) | | `RerunOnFailure` | next VMI after the start-failure backoff (`vm.go` L1050, L1074, L1580) | next VMI, no backoff (`vm.go` L1050, L1539) | none until the tenant starts the VM (`vm.go` L1050, L1070; `rest/lifecycle.go` L120) | | `Once` | none (`vm.go` L1141) | none (`vm.go` L1141) | next VMI (`vm.go` L1141) | | `Manual` | none until the tenant starts the VM (`vm.go` L1104, `rest/lifecycle.go` L120) | the same | the same | -The start-failure backoff grows with each failure that happens before a VMI runs, up to five minutes, and resets once a VMI runs. A VMI that a drain evicts ends on its own, and [`vgpu-vms`](../vgpu-vms/README.md) gives what each `runStrategy` does then. +The start-failure backoff grows with each failure that happens before a VMI runs, up to five minutes, and resets once a VMI runs. During the backoff the VM shows `CrashLoopBackOff`, and so does a `Once` VM whose VMI failed before it ran, although it does not retry (`vm.go` L2526-L2546, L2572-L2575). A VMI that a drain evicts ends on its own, and [`vgpu-vms`](../vgpu-vms/README.md) gives what each `runStrategy` does then. Step 1: From ef1ded594dd3003f27f616151291699414a81612 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 19:38:59 +0300 Subject: [PATCH 27/32] docs(design-proposals): point GPU restart timing at the runStrategy table A refused pinned pod, or a VMI the controller deletes, costs a start-failure backoff only where the VM's runStrategy creates a next VMI. Record 0012 and the cases that restart a VM after its claims moved now refer to the runStrategy table for that, and the refused-VMI backoff is named as the VM controller's per-item backoff. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 8 ++++---- ...hook-checks-and-records-and-the-controller-prepares.md | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index a54200ff..c6eebc79 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -301,7 +301,7 @@ The phases decide what a claim holds and costs: A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 10). -A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: under `Always` and `RerunOnFailure` the VM controller deletes a final VMI before it starts the next one, under `Manual` a start request first stops the final VMI and then starts a new one, and under `Once` no next VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. +A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: under `Always` and `RerunOnFailure` the VM controller deletes a final VMI before it starts the next one, under `Manual` a start request first stops the final VMI and then starts a new one, and under `Once` the final VMI stays and no next VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. A claim keeps its node while the `Node` exists, serves the class, has not lost capacity, has no pending layout change (section 2) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer serves the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules below. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set. @@ -310,7 +310,7 @@ A drop in allocatable alone is not a loss, since kubelet reports zero after ever - the affected classes get a `LostCapacity` condition; - unattached claims that no longer fit move to another node of their class, or to `Lost` when none has room; - on a `NotReady` node, attached claims go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI start is moving off the node is the exception: it finishes the move; -- a VM started after the condition is set has its claims gathered off the node before its pod is released, and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. Where the VM's `runStrategy` creates a next VMI after such a deletion (Failure and edge cases), it starts once the claim has moved or been rebound from `Lost`; unlike a cordon, the node cannot serve the claim, so waiting is no option. +- a VM started after the condition is set has its claims gathered off the node before its pod is released, and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. Where the VM's `runStrategy` creates a next VMI after such a deletion (Failure and edge cases), it starts once the claim has moved or been rebound from `Lost` and any start-failure backoff the table gives has passed; unlike a cordon, the node cannot serve the claim, so waiting is no option. Claims in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a claim whose VMI is still live on the lost node waits until that VMI is final or removed (above), while a claim of a VMI whose pod is still gated is rebound for that VMI. The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` claim already counts. @@ -476,8 +476,8 @@ Step 2: - A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. - A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and where the VM's `runStrategy` creates a next VMI after such a deletion, that VMI gets the moved claim. - A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of a VM whose `runStrategy` creates one after a VMI that ran fails gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. A VM whose `runStrategy` creates no next VMI keeps its failed VMI, which leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. -- A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; where the VM's `runStrategy` creates a next VMI after such a deletion, it starts once the claims have moved or been rebound. -- A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has. Where the VM's `runStrategy` creates a next VMI after such a deletion, it starts on the new node once the claims have moved or been rebound. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. +- A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; where the VM's `runStrategy` creates a next VMI after such a deletion, it starts once the claims have moved or been rebound and any start-failure backoff the table gives has passed. +- A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has. Where the VM's `runStrategy` creates a next VMI after such a deletion, it starts on the new node once the claims have moved or been rebound and any start-failure backoff the table gives has passed. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. - A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts, the controller moves the claim off that node before it releases the pod, or the claim goes to `Lost` and the pod stays gated until it is rebound. - A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use class entries or a `claimSelector`, or allow raw names again while the cluster has such pools. - A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md index 8a6ad5a4..88de7cc6 100644 --- a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -20,8 +20,8 @@ The VMI webhook resolves the claim references, runs the checks that need no plac ## Why not the alternatives -- **Keep the work in the webhook.** A fail-closed admission call that writes several objects and waits on another controller is slow under load, needs its own request protocol between the two, and turns every slow move into a refused VMI and a KubeVirt backoff. -- **Refuse the VMI whenever a gather is needed and let the controller move the claims before KubeVirt's retry.** It drops the handshake but keeps the backoff, so the start still comes up to the backoff limit after the claims are ready, and the tenant sees a refusal for a VM that will start. +- **Keep the work in the webhook.** A fail-closed admission call that writes several objects and waits on another controller is slow under load, needs its own request protocol between the two, and turns every slow move into a refused VMI and the per-item backoff of KubeVirt's VM controller. +- **Refuse the VMI whenever a gather is needed and let the controller move the claims before KubeVirt's retry.** It drops the handshake but keeps the VM controller's per-item backoff, so the start still comes up to that backoff's limit after the claims are ready, and the tenant sees a refusal for a VM that will start. - **Hold the VMI instead of the pod**, through a KubeVirt start strategy or a VM-level hold. KubeVirt v1.9.0 offers none that keeps the pod from being scheduled: `startStrategy: Paused` still schedules the pod and only pauses the guest. A scheduling gate is the hook Kubernetes provides for this, and KubeVirt neither touches it nor times it out. - **Write the pin into the VMI's node affinity.** Under `vmRolloutStrategy: LiveUpdate` KubeVirt rewrites a VMI's affinity from the VM template, and it never changes a launcher pod's affinity, so the pod is the place that keeps the pin. @@ -29,10 +29,10 @@ The VMI webhook resolves the claim references, runs the checks that need no plac - The webhook is a pure check plus the attachment record, so its latency no longer depends on the binder, and a dry run only skips the record. - Template claims are created after the VMI exists, so the controller sets their owner at creation, and a template claim no longer needs a grace period for a VMI that never appears. -- A set that cannot be placed is no longer an admission error. The VM shows `Starting`, the VMI waits in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated`, and the claims carry a condition naming the reason while the VMI gets a `Warning` event with the same reason. The VM starts as soon as the set fits, with no KubeVirt backoff in between. +- A set that cannot be placed is no longer an admission error. The VM shows `Starting`, the VMI waits in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated`, and the claims carry a condition naming the reason while the VMI gets a `Warning` event with the same reason. The VM starts as soon as the set fits, with no per-item backoff of the VM controller in between. - The pod webhook is a fail-closed dependency of every start of a VM with class or claim entries, like the VMI webhook. - The cluster autoscaler does not scale up for a gated pod, and after the pin a new node cannot help that pod, so GPU node scale-up stays with the hoster. -- A pin cannot be undone, so a node that refuses a pinned pod costs one failed VMI and KubeVirt's start-failure backoff, which the rejected-node rule keeps from repeating. +- A pin cannot be undone, so a node that refuses a pinned pod costs one failed VMI. Where the VM's `runStrategy` creates a next VMI after it, that VMI also waits for KubeVirt's start-failure backoff (the `runStrategy` table in the proposal's Failure and edge cases). The rejected-node rule keeps this from repeating. ## Revisit if From 51938008b546c407b4ee63d19da7e46bd71c4486 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 19:44:42 +0300 Subject: [PATCH 28/32] docs(design-proposals): name the VMI situation wherever a runStrategy outcome appears Each sentence that states what a runStrategy does now names the situation of the runStrategy table it relies on, or points at the table, so no outcome is restated without its case. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 8 ++++---- ...hook-checks-and-records-and-the-controller-prepares.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index c6eebc79..b76c67d2 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -301,7 +301,7 @@ The phases decide what a claim holds and costs: A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 10). -A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: under `Always` and `RerunOnFailure` the VM controller deletes a final VMI before it starts the next one, under `Manual` a start request first stops the final VMI and then starts a new one, and under `Once` the final VMI stays and no next VMI is created ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. +A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: wherever a `runStrategy` creates a next VMI (the table in Failure and edge cases), the final VMI is gone first, and under `Manual` a start request first stops the final VMI and then starts a new one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. A claim keeps its node while the `Node` exists, serves the class, has not lost capacity, has no pending layout change (section 2) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer serves the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules below. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set. @@ -460,8 +460,8 @@ Step 1: - A class shows `free: 0` while its nodes have GPUs to spare → the nodes carry a `NoSchedule` or `NoExecute` taint the class does not tolerate; add it to the class's `tolerations`, or remove the taint. - A class has a `ResourceNameConflict` condition → another class has its resource name; delete one of them or point it at its own name. - A class is deleted while VMs name it → running VMs keep running, and their next start is refused naming the class; the tenant moves them to another class. -- A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. Where its `runStrategy` creates a new instance, it starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. -- A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. Where its `runStrategy` creates a next VMI after a failure, that VMI starts on any node with a free unit of its class. +- A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. Where its `runStrategy` creates a new instance after the eviction, as `vgpu-vms` gives it, it starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. +- A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. Where its `runStrategy` creates a next VMI after a VMI that ran fails, that VMI starts on any node with a free unit of its class. - A GPU fails under a running VM on a class entry → the VM keeps running. Kubelet stops listing the device as allocatable, so the class's `free` drops, and the VM's next instance gets a unit on a node that still has one. - A layout change does not apply → the node carries the `cozystack.io/gpu-layout-pending` label; cordon and drain it. A GPU VM on the node holds the change until it stops or is evicted. - A Talos pool with a vGPU class entry is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. @@ -475,7 +475,7 @@ Step 2: - The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. Where its `runStrategy` creates a next VMI after a failure before running, that VMI waits for the start-failure backoff. The binder avoids that node for the set for ten minutes or until the node changes. Free CPU or memory on GPU nodes if this repeats. - A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. - A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and where the VM's `runStrategy` creates a next VMI after such a deletion, that VMI gets the moved claim. -- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of a VM whose `runStrategy` creates one after a VMI that ran fails gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. A VM whose `runStrategy` creates no next VMI keeps its failed VMI, which leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. +- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of a VM whose `runStrategy` creates one after a VMI that ran fails gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. A VM whose `runStrategy` creates no next VMI after a VMI that ran fails keeps its failed VMI, which leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. - A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; where the VM's `runStrategy` creates a next VMI after such a deletion, it starts once the claims have moved or been rebound and any start-failure backoff the table gives has passed. - A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has. Where the VM's `runStrategy` creates a next VMI after such a deletion, it starts on the new node once the claims have moved or been rebound and any start-failure backoff the table gives has passed. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. - A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts, the controller moves the claim off that node before it releases the pod, or the claim goes to `Lost` and the pod stays gated until it is rebound. diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md index 88de7cc6..6751de02 100644 --- a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -32,7 +32,7 @@ The VMI webhook resolves the claim references, runs the checks that need no plac - A set that cannot be placed is no longer an admission error. The VM shows `Starting`, the VMI waits in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated`, and the claims carry a condition naming the reason while the VMI gets a `Warning` event with the same reason. The VM starts as soon as the set fits, with no per-item backoff of the VM controller in between. - The pod webhook is a fail-closed dependency of every start of a VM with class or claim entries, like the VMI webhook. - The cluster autoscaler does not scale up for a gated pod, and after the pin a new node cannot help that pod, so GPU node scale-up stays with the hoster. -- A pin cannot be undone, so a node that refuses a pinned pod costs one failed VMI. Where the VM's `runStrategy` creates a next VMI after it, that VMI also waits for KubeVirt's start-failure backoff (the `runStrategy` table in the proposal's Failure and edge cases). The rejected-node rule keeps this from repeating. +- A pin cannot be undone, so a node that refuses a pinned pod costs one failed VMI. Where the VM's `runStrategy` creates a next VMI after a VMI that failed before it ran, that VMI also waits for KubeVirt's start-failure backoff (the `runStrategy` table in the proposal's Failure and edge cases). The rejected-node rule keeps this from repeating. ## Revisit if From e6b4eb1b032c38f275130f6f89c23205476d9b78 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 19:57:09 +0300 Subject: [PATCH 29/32] docs(design-proposals): put GPU consumers in step 1 and close restart gaps The consumer proposals need only GPU classes, so the Rollout lists them in step 1, ahead of #48 and of holding. Restarts after an eviction are conditional on the runStrategy, a directly written VirtualMachine can also be missing from the usage records until super-admins lose that write access, and maxCount stays editable in step 1 while step 2 freezes it for a class in use. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index b76c67d2..a298e19b 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -161,7 +161,7 @@ status: A class maps an offer to a backend through a `source` union. The first release has one source, `devicePlugin`; `dra` is reserved for section 10. A class is one resource name, and two classes cannot have the same one: a class whose `resourceName` another class already has gets a `ResourceNameConflict` condition and is not offered, so a budget on one class cannot be sidestepped through another, and a usage record's resource name leads to exactly one class. The nodes of a class are the nodes that advertise its resource name. Class names are DNS labels, since they become part of quota keys. -`vendor`, `model`, `mode`, `profile`, `memory` and the kind of `source` never change, so a class name stands for one offer for the class's lifetime, and whatever names a class, a VM, a worker pool or a billing system, knows the offer from the name ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A different offer is a new class, and a hoster does not reuse a deleted class's name for another offer, since tenants' budgets and billing records are keyed by it. `displayName`, `maxCount`, `resourceName` and `tolerations` can change: `resourceName` for changes in the host layer, `tolerations` when the hoster retaints its GPU nodes. A change applies to VMs started after it; a running VM keeps the resource name it started with, and its launcher pod the tolerations. Deleting a class refuses the next start of every VM that names it and leaves running VMs alone. +`vendor`, `model`, `mode`, `profile`, `memory` and the kind of `source` never change, so a class name stands for one offer for the class's lifetime, and whatever names a class, a VM, a worker pool or a billing system, knows the offer from the name ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A different offer is a new class, and a hoster does not reuse a deleted class's name for another offer, since tenants' budgets and billing records are keyed by it. `displayName`, `maxCount`, `resourceName` and `tolerations` can change: `resourceName` for changes in the host layer, `tolerations` when the hoster retaints its GPU nodes. From step 2 on, `maxCount` changes only while the class is not in use (section 9). A change applies to VMs started after it; a running VM keeps the resource name it started with, and its launcher pod the tolerations. Deleting a class refuses the next start of every VM that names it and leaves running VMs alone. How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 2), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPUs of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. @@ -303,7 +303,7 @@ A `dra` claim before its first start is the exception: it is `Bound` without a n A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: wherever a `runStrategy` creates a next VMI (the table in Failure and edge cases), the final VMI is gone first, and under `Manual` a start request first stops the final VMI and then starts a new one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. -A claim keeps its node while the `Node` exists, serves the class, has not lost capacity, has no pending layout change (section 2) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer serves the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached, moves, and the VM's next VMI starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules below. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set. +A claim keeps its node while the `Node` exists, serves the class, has not lost capacity, has no pending layout change (section 2) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer serves the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached and moves, and where the VM's `runStrategy` creates a new instance after the eviction, as `vgpu-vms` gives it, that instance starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules below. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set. A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity for a class when its allocatable count of the class's resource stays below the units held there for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no claim ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). Then: @@ -351,7 +351,7 @@ For a gated pod the controller creates the VMI's template claims, then brings th Once the set is on one node, the controller pins the pod and removes the gate in one update. The pin is a `matchFields` requirement on `metadata.name` appended to every required node selector term of the pod, or the single term when it has none. Kubernetes lets a gated pod only narrow its scheduling, so a term cannot be added next to existing ones, and KubeVirt usually sets a required term already, for host-model CPUs. The pin is not `nodeName`, which would let kubelet start the pod on a cordoned node, and it goes on the pod rather than the VMI because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. -When the set cannot be placed, because no node has room or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. The VMI gets no condition because KubeVirt's admission guards writes to a VMI. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules. After the deletion the VMI becomes `Failed`, and KubeVirt decides what happens next by the VM's `runStrategy` (Failure and edge cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. A set of template claims only loses its claims with the failed VMI, so for it the controller keeps the record in memory against the VM, and a controller restart forgets it. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. +When the set cannot be placed, because no node has room or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. The VMI gets no condition because KubeVirt's admission guards writes to a VMI. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules. After the deletion the VMI becomes `Failed`, and KubeVirt decides what happens next by the VM's `runStrategy` for a VMI that failed before it ran (Failure and edge cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. A set of template claims only loses its claims with the failed VMI, so for it the controller keeps the record in memory against the VM, and a controller restart forgets it. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way while the pod is gated. Once the pod is pinned, under `Move`, for a VM whose `runStrategy` creates a next VMI after such a deletion, `Always` or `Once`, the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure and edge cases). @@ -420,7 +420,7 @@ Three rules hold throughout: Step 1: - Tenant access. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. -- What a tenant controls. A tenant never supplies a resource name through a class entry: it comes from `GPUClass`, which tenants cannot write. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. A tenant super-admin who writes VirtualMachines directly can still write raw GPU names, as today, until step 2's `forbidden`; a VMI with class entries and GPU entries of its own is refused. GPU keys are checked only when an application is written. A VirtualMachine written directly, with class entries in its annotation or a higher `count` than its application declares, is not counted against them, also under `forbidden`. That path closes when tenant super-admins lose write access to VirtualMachines, which #48 lists as a prerequisite of its phase 4. Until then a hoster who sells GPU budgets to sub-tenants does not give them the super-admin role. +- What a tenant controls. A tenant never supplies a resource name through a class entry: it comes from `GPUClass`, which tenants cannot write. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. A tenant super-admin who writes VirtualMachines directly can still write raw GPU names, as today, until step 2's `forbidden`; a VMI with class entries and GPU entries of its own is refused. GPU keys are checked only when an application is written. A VirtualMachine written directly, with class entries in its annotation or a higher `count` than its application declares, is not counted against them, also under `forbidden`. That path closes when tenant super-admins lose write access to VirtualMachines, which #48 requires before its phase 4. A directly written VirtualMachine can also be left out of the usage records, or recorded with sums other than what it uses. Until super-admins lose that write access, a hoster who sells GPU budgets or bills GPUs to sub-tenants does not give them the super-admin role. - Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. - Node agent. It reads sysfs and writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can delete only the device plugin pod of its own node. - Webhooks. The VMI and virt-launcher pod webhooks fail closed and see only objects carrying the reference label, so their outage blocks the start of VMs with class entries. @@ -433,7 +433,7 @@ Step 2 adds: - What a tenant controls. A tenant never supplies a node through a claim reference: it comes from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. The webhook takes claims only from the VM's own namespace. With `forbidden` any GPU entry without a class or claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. - Webhooks. Step 2 adds a third fail-closed GPU webhook, on orders, and the gate in the launcher pod webhook of step 1. The VMI and launcher pod webhooks still see only objects carrying the reference label, so their outage blocks the start of VMs with class or claim entries. The pod webhook has to fail closed, also because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order, so its outage also blocks new orders, while held claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity when it creates and binds the claim. - Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `cozystack.io/gpuclaim-template` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so a held claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. -- Template claims. For class entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the entry names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, uses the GPUs of its class entries, and anyone who can start it makes their claims exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). +- Template claims. For class entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the entry names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, uses the GPUs of its class entries, and anyone who can start it makes their claims exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). They are not in the usage records themselves, so the launcher pod's `Workload` carries them, with the gap that Security step 1 names for directly written VirtualMachines. - Controller. It gains deleting unscheduled VMIs that reference its claims, and `update` and `delete` on pods in every namespace, cluster-wide like its write on Secrets, because tenant namespaces appear at runtime. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses. It also writes one ConfigMap, the GPU settings in `cozy-system`. - Visibility. A tenant sees the node name of its own claims. @@ -461,8 +461,8 @@ Step 1: - A class has a `ResourceNameConflict` condition → another class has its resource name; delete one of them or point it at its own name. - A class is deleted while VMs name it → running VMs keep running, and their next start is refused naming the class; the tenant moves them to another class. - A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. Where its `runStrategy` creates a new instance after the eviction, as `vgpu-vms` gives it, it starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. -- A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. Where its `runStrategy` creates a next VMI after a VMI that ran fails, that VMI starts on any node with a free unit of its class. -- A GPU fails under a running VM on a class entry → the VM keeps running. Kubelet stops listing the device as allocatable, so the class's `free` drops, and the VM's next instance gets a unit on a node that still has one. +- A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. Where its `runStrategy` creates a next VMI after a VMI that ran fails, that VMI starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. +- A GPU fails under a running VM on a class entry → the VM keeps running. Kubelet stops listing the device as allocatable, so the class's `free` drops, and when the VM starts again, its next instance gets a unit on a node that still has one, or waits as `ErrorUnschedulable` while there is none. - A layout change does not apply → the node carries the `cozystack.io/gpu-layout-pending` label; cordon and drain it. A GPU VM on the node holds the change until it stops or is evicted. - A Talos pool with a vGPU class entry is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. @@ -473,7 +473,7 @@ Step 2: - A VM with claims is refused at start → the webhook refuses the VMI for a reference problem, such as a claim that does not exist, is being deleted or is attached to another VMI, and the VirtualMachine shows the message in a `FailedCreate` event, naming each claim and the reason; fix the reference. - A VM with claims stays `Starting` → its VMI is in `Scheduling` with its pod gated, the claims carry a condition and the VMI a `Warning` event naming the reason: no node can take the whole set, template claims included, or a claim of a `Stay` class is on a cordoned node while others need gathering. The VM starts by itself once the set can be placed. To get there sooner, uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. - The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. Where its `runStrategy` creates a next VMI after a failure before running, that VMI waits for the start-failure backoff. The binder avoids that node for the set for ten minutes or until the node changes. Free CPU or memory on GPU nodes if this repeats. -- A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and their VMs start there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. +- A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and a VM whose `runStrategy` creates a new instance after the eviction starts there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. - A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and where the VM's `runStrategy` creates a next VMI after such a deletion, that VMI gets the moved claim. - A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of a VM whose `runStrategy` creates one after a VMI that ran fails gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. A VM whose `runStrategy` creates no next VMI after a VMI that ran fails keeps its failed VMI, which leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. - A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; where the VM's `runStrategy` creates a next VMI after such a deletion, it starts once the claims have moved or been rebound and any start-failure backoff the table gives has passed. @@ -513,15 +513,15 @@ Step 1: 1. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. It needs no tenant API and can land first. 2. `GPUClass` with its status controller and metrics, class entries in `vm-instance` and `kubernetes-nodes` with the VMI webhook and the launcher pod webhook for tolerations, the `gpu.vgpu` Tenant flag, GPU keys with the interim check, and the `gpuclass` option source, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRD ships and the charts refuse class entries. Before tenants use vGPU classes, the `gpu.vgpu` flag is set from the root tenant down. It works on any cluster where each offer it publishes has a resource name of its own, with or without item 1. -3. GPU keys through #48, and the interim check deleted, in the release where #48's gate is on by default. +3. The consumers, on top of item 2 alone: `kairos-vgpu-workers` adds Kairos GPU pools on class entries, and `vgpu-vms` adds GPU guest images. Neither waits for item 4 or for step 2. +4. GPU keys through #48, and the interim check deleted, in the release where #48's gate is on by default. Step 2: -4. Holding: `GPUClaim`, the binder and its metrics, the scheduling gate in the launcher pod webhook, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. +5. Holding: `GPUClaim`, the binder and its metrics, the scheduling gate in the launcher pod webhook, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes and for the Kairos pools of `kairos-vgpu-workers`, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. Then: -5. The consumers: `kairos-vgpu-workers` adds Kairos GPU pools on class entries, and on `claimSelector` once step 2 ships, and `vgpu-vms` adds GPU guest images. 6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by naming it. ## Open questions From 9b3a4166aea05b84dd0caf436a78bc56e1826022 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 27 Oct 2026 20:48:02 +0300 Subject: [PATCH 30/32] docs(design-proposals): call tenant-ordered GPUClaims ordered claims "Held claim" as a noun collided with "holds GPUs", which only Bound and Attached claims do, and left the phase table, the refusal list and the usage-record sentences unclear about template claims. An ordered claim is a GPUClaim the tenant orders; a template claim holds GPUs in the same phases but has no budget or usage record of its own. A class being deleted keeps serving its Bound, Attached and Lost ordered claims, and refuses a Pending one, which would never be bound. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 56 +++++++++---------- ...ttedhostdevices-is-refused-at-admission.md | 2 +- ...y-where-gpu-operator-serves-gpus-to-vms.md | 2 +- ...n-write-a-vm-orders-its-template-claims.md | 6 +- ...is-immutable-and-a-pool-names-its-class.md | 2 +- ...and-records-and-the-controller-prepares.md | 2 +- ...olding-is-step-2-and-stays-gpu-specific.md | 2 +- 7 files changed, 36 insertions(+), 36 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index a298e19b..9cfbe5ef 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -13,7 +13,7 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Scope and related proposals -- [`kairos-vgpu-workers`](../kairos-vgpu-workers/README.md) gives managed Kubernetes worker pools on Kairos a worker image per GPU mode and delivers the vGPU license token to the workers. [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers GPU guest images for VMs. Both name a `GPUClass` in their `gpus` entries from step 1 on, and take held claims once step 2 ships; the links resolve once they merge. +- [`kairos-vgpu-workers`](../kairos-vgpu-workers/README.md) gives managed Kubernetes worker pools on Kairos a worker image per GPU mode and delivers the vGPU license token to the workers. [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers GPU guest images for VMs. Both name a `GPUClass` in their `gpus` entries from step 1 on, and take ordered claims once step 2 ships; the links resolve once they merge. - [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared sizes the quota authority and refuses an application that does not fit its pool when it is written. GPU budgets are ordinary keys of that gate (section 5). Until it ships, an interim check does the same for GPU keys only. - [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and step 2 rebinds their `Lost` claims. The link resolves once it merges. - [IP addresses as a first-class resource](https://github.com/cozystack/community/pull/35) solves a similar problem for addresses with a class, a tenant claim and a backend contract. @@ -31,7 +31,7 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra - [0010. `GPUClass` carries the tolerations that `free`, binding and the launcher pod read](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md) - [0009. `GPUClass.mode` never changes, and a pool names its class](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md) - [0008. The GPU controller gets its node-label right with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) -- [0007. Whoever can write a VM uses the GPUs of its class entries, while held claims stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md) +- [0007. Whoever can write a VM uses the GPUs of its class entries, while ordered claims stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md) - [0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md) - [0005. The taint rule skips exactly the three taints that mirror a cordon or readiness](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md), superseded by 0010 - [0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts](./decisions/0004-gpunode-health-is-kubelet-allocatability.md), superseded by 0015 @@ -71,7 +71,7 @@ Step 1: Step 2: -- A tenant holds a number of GPUs of a class within its budget, attached or not, until it releases them, and a restarted VM gets its held claims back. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden`, which needs every GPU consumer on class or claim entries; until then it is not guaranteed. +- A tenant holds a number of GPUs of a class within its budget, attached or not, until it releases them, and a restarted VM gets its ordered claims back. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden`, which needs every GPU consumer on class or claim entries; until then it is not guaranteed. - The catalog can later describe MIG, other vendors and DRA-backed devices without changing the tenant API. ### Non-goals @@ -273,7 +273,7 @@ status: A class is in use while any claim of it is `Bound`, `Attached` or `Lost`, and a class in use cannot disappear. A finalizer, `cozystack.io/gpuclass-protection`, keeps a deleted class while it is in use, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `maxCount` cannot change while the class is in use. -A `GPUClaim` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate claims. Admission refuses a claim above the class's `maxCount`. The spec is immutable, so a change is a new order. A held claim is a `GPUClaim` a tenant orders and keeps, as opposed to a template claim (Attaching held claims); it holds GPUs while it is `Bound` or `Attached` (the phase table). A `Bound` claim that no VMI uses is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. +A `GPUClaim` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate claims. Admission refuses a claim above the class's `maxCount`. The spec is immutable, so a change is a new order. An ordered claim is a `GPUClaim` a tenant orders and keeps, as opposed to a template claim (Attaching ordered claims); it holds GPUs while it is `Bound` or `Attached` (the phase table). A `Bound` claim that no VMI uses is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. What `Bound` guarantees depends on the class's source: @@ -286,9 +286,9 @@ A tenant sees a class, a count, a phase and, once there is one, a node, never wh #### Binding and capacity -A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node of its class that has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, nodes with a pending layout change (section 2), and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, by the taint rule of section 3, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI start (Placing a VMI's claims): it may take a claim attached only to the starting VMI off a node that is not `Ready`, because no other VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units held by bound claims, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. +A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node of its class that has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, nodes with a pending layout change (section 2), and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, by the taint rule of section 3, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI start (Placing a VMI's claims): it may take a claim attached only to the starting VMI off a node that is not `Ready`, because no other VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units that `Bound` and `Attached` claims hold, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. -Admission refuses an order that cannot be met at that moment, by capacity or by budget (Budgets and usage records of held claims), with a message naming the class, the count and the limit it hit: a budget of the pool, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. +Admission refuses an order that cannot be met at that moment, by capacity or by budget (Budgets and usage records of ordered claims), with a message naming the class, the count and the limit it hit: a budget of the pool, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. The phases decide what a claim holds and costs: @@ -299,7 +299,7 @@ The phases decide what a claim holds and costs: | `Attached` | yes, used by a VMI | yes | yes | | `Lost` | no | yes | no | -A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 10). +A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 10). A template claim holds GPUs in the same phases, but counts against no budget and is in no usage record of its own: its class entry and its launcher pod's `Workload` carry them (Attaching ordered claims). A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: wherever a `runStrategy` creates a next VMI (the table in Failure and edge cases), the final VMI is gone first, and under `Manual` a start request first stops the final VMI and then starts a new one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. @@ -309,14 +309,14 @@ A drop in allocatable alone is not a loss, since kubelet reports zero after ever - the affected classes get a `LostCapacity` condition; - unattached claims that no longer fit move to another node of their class, or to `Lost` when none has room; -- on a `NotReady` node, attached claims go to `Lost` as well, which drops them from the usage records, and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI start is moving off the node is the exception: it finishes the move; +- on a `NotReady` node, attached claims go to `Lost` as well; the ordered ones among them leave the usage records and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI start is moving off the node is the exception: it finishes the move; - a VM started after the condition is set has its claims gathered off the node before its pod is released, and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. Where the VM's `runStrategy` creates a next VMI after such a deletion (Failure and edge cases), it starts once the claim has moved or been rebound from `Lost` and any start-failure backoff the table gives has passed; unlike a cordon, the node cannot serve the claim, so waiting is no option. Claims in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a claim whose VMI is still live on the lost node waits until that VMI is final or removed (above), while a claim of a VMI whose pod is still gated is rebound for that VMI. The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` claim already counts. -#### Attaching held claims +#### Attaching ordered claims -VMs and worker pools reference held claims in `gpus`: +VMs and worker pools reference ordered claims in `gpus`: ```yaml # vm-instance @@ -333,13 +333,13 @@ gpus: A `vm-instance` entry has exactly one of `name`, `className` or `claim`, and a `kubernetes-nodes` entry exactly one of `name`, `className` or `claimSelector`. A list holds only `name` entries or only class and claim entries, a pool has at most one `claimSelector` and never next to a class entry, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The rules of section 4 for Talos pools, the vGPU entitlement and `permittedHostDevices` apply to the class of a claim as to a class entry. -The VMI webhook of section 4 also resolves claim references, runs the checks that need no placement, and records the attachments; it binds, moves and creates no claim, and waits for nothing. For a VM it takes the named claims. For a pool worker it takes the claim the worker held before, unless a VM now names it, its node is not `Ready`, or its node has a taint the class does not tolerate. Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment of each held claim to this VMI, so two VMIs can never hold one claim and no background move can take one, and writes one `gpus` entry per GPU unit, held or from a class entry, with the class's resource name. The launcher pod webhook of section 4 adds the tolerations of the held claims' classes as it does for class entries. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. On a dry run the webhook records nothing, so in step 2 it declares `sideEffects: NoneOnDryRun`. The GPU controller places the claims after admission, while the launcher pod waits behind a scheduling gate (Placing a VMI's claims). +The VMI webhook of section 4 also resolves claim references, runs the checks that need no placement, and records the attachments; it binds, moves and creates no claim, and waits for nothing. For a VM it takes the named claims. For a pool worker it takes the claim the worker had attached before, unless a VM now names it, its node is not `Ready`, or its node has a taint the class does not tolerate. Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment of each ordered claim to this VMI, so two VMIs can never hold one claim and no background move can take one, and writes one `gpus` entry per GPU unit, ordered or from a class entry, with the class's resource name. The launcher pod webhook of section 4 adds the tolerations of the ordered claims' classes as it does for class entries. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. On a dry run the webhook records nothing, so in step 2 it declares `sideEffects: NoneOnDryRun`. The GPU controller places the claims after admission, while the launcher pod waits behind a scheduling gate (Placing a VMI's claims). -The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim does not exist, is being deleted, or is attached to another VMI, besides the refusals of section 4 for its class. A refused VMI leaves its claims unattached, so they move by the rules above. A claim that is `Pending` or `Lost`, or bound to a node that cannot serve it, is not refused: the controller binds or moves it before it releases the pod. +The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim does not exist, is being deleted, or is attached to another VMI, besides the rules of section 4 that apply to the class of a claim (above); a class being deleted does not refuse an ordered claim that is `Bound`, `Attached` or `Lost`, which its finalizer keeps; a `Pending` claim of a deleted class is refused, since it is never bound. A refused VMI leaves its claims unattached, so they move by the rules above. Any other claim that is `Pending` or `Lost`, or bound to a node that cannot serve it, is not refused: the controller binds or moves it before it releases the pod. -A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and usage record, under the `cozystack.io/gpuclaim-protection` finalizer until no VMI uses it, and is not attached again. +A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and, for an ordered claim, its usage record, under the `cozystack.io/gpuclaim-protection` finalizer until no VMI uses it, and is not attached again. -Once holding exists, a class entry needs a place on a node too, or a VM that names only a class could start on a unit another tenant holds. Once the launcher pod waits at its gate, the controller creates a `GPUClaim` from each class entry, named after the VMI and the entry's index, marked by the `cozystack.io/gpuclaim-template` label, and owned by the VMI. A claim of that name that carries the mark and is owned by this VMI is reused, so a repeated pass creates nothing twice; one still being deleted, or still owned by the VM's previous VMI, is waited for. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name is never taken, and the pod stays gated with a condition naming it. The controller binds these template claims under the same capacity checks as any order, on the node of the gathered set when the VMI has held claims, and otherwise on one node with room for all of them, and attaches them. It deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. A template claim counts against no budget of its own, since its class entry already counts (section 5), and it is not in the usage records, since its launcher pod is. A class entry counts as a claim reference under `forbidden`, and a pool on class entries pays for running workers only. +Once holding exists, a class entry needs a place on a node too, or a VM that names only a class could start on a unit another tenant holds. Once the launcher pod waits at its gate, the controller creates a `GPUClaim` from each class entry, named after the VMI and the entry's index, marked by the `cozystack.io/gpuclaim-template` label, and owned by the VMI. A claim of that name that carries the mark and is owned by this VMI is reused, so a repeated pass creates nothing twice; one still being deleted, or still owned by the VM's previous VMI, is waited for. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name is never taken, and the pod stays gated with a condition naming it. The controller binds these template claims under the same capacity checks as any order, on the node of the gathered set when the VMI has ordered claims, and otherwise on one node with room for all of them, and attaches them. It deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. A template claim counts against no budget of its own, since its class entry already counts (section 5), and it is not in the usage records, since its launcher pod is. A class entry counts as a claim reference under `forbidden`, and a pool on class entries pays for running workers only. A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. @@ -347,7 +347,7 @@ A pool on `claimSelector` never runs more GPU workers than it has matching claim A launcher pod waits for its GPUs behind a scheduling gate, so nothing is refused for not being ready yet ([0012](./decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md)). The launcher pod webhook of section 4 also adds the gate `cozystack.io/gpuclaims` in step 2. It matches only pods labelled `kubevirt.io: virt-launcher` that carry the reference label, adds the gate only when it is not there yet, since a repeated call would otherwise fail on a duplicate gate, and fails closed: a pod that reached the scheduler ungated would run unpinned. It also gates the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on the node of the claims. Migration target pods are left alone: KubeVirt deletes a target pod that stays pending for 15 minutes, a VM with a passthrough GPU or a VF does not live-migrate, and Cozystack does not enable the `VGPULiveMigration` gate that mdev vGPUs would need. A set that already has a `WaitForFirstConsumer` volume bound on a node stays on that node: the controller places it only where the node affinity of the VMI's bound volumes allows. KubeVirt never touches scheduling gates and has no timeout for a VMI whose pod waits, so the VMI stays in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated` ([`lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go)), and the VM's own start backoff never begins. -For a gated pod the controller creates the VMI's template claims, then brings the set onto one node. Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are `Pending` or `Lost`, on different nodes, on a node that is gone, no longer serves a class, has lost capacity or has a pending layout change, on a node that is not `Ready`, on a node with a taint one of their classes does not tolerate, on a cordoned node under `Move`, or on a node with no room for the VMI's template claims, the controller binds or moves them onto one `Ready` node that is not cordoned, has no pending layout change, is tolerated by every class of the set and has room for all of them, template claims included. Every held claim of the set is attached to this VMI by then, so no other VMI or background move can take one, and each claim keeps its hold while it moves. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims, there is no gather, even if the node is also not `Ready`: the pod is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). +For a gated pod the controller creates the VMI's template claims, then brings the set onto one node. Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are `Pending` or `Lost`, on different nodes, on a node that is gone, no longer serves a class, has lost capacity or has a pending layout change, on a node that is not `Ready`, on a node with a taint one of their classes does not tolerate, on a cordoned node under `Move`, or on a node with no room for the VMI's template claims, the controller binds or moves them onto one `Ready` node that is not cordoned, has no pending layout change, is tolerated by every class of the set and has room for all of them, template claims included. Every ordered claim of the set is attached to this VMI by then, so no other VMI or background move can take one, and each `Bound` or `Attached` claim keeps its hold while it moves. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims, there is no gather, even if the node is also not `Ready`: the pod is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). Once the set is on one node, the controller pins the pod and removes the gate in one update. The pin is a `matchFields` requirement on `metadata.name` appended to every required node selector term of the pod, or the single term when it has none. Kubernetes lets a gated pod only narrow its scheduling, so a term cannot be added next to existing ones, and KubeVirt usually sets a required term already, for host-model CPUs. The pin is not `nodeName`, which would let kubelet start the pod on a cordoned node, and it goes on the pod rather than the VMI because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. @@ -355,13 +355,13 @@ When the set cannot be placed, because no node has room or a `Stay` claim sits o A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way while the pod is gated. Once the pod is pinned, under `Move`, for a VM whose `runStrategy` creates a next VMI after such a deletion, `Always` or `Once`, the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure and edge cases). -#### Budgets and usage records of held claims +#### Budgets and usage records of ordered claims -A held claim counts against the same GPU keys as a class entry, from the moment it is `Bound` until it is deleted (the phase table), in the same pool figure as the class entries of section 5; a `claim` or `claimSelector` entry declares nothing of its own, since its claims already count. The validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, that pool figure and the vGPU entitlement. A raw GPU key on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has a GPU key covering the class, so a held claim, which no `ResourceQuota` sees, never bypasses an existing raw limit. While the raw key exists, it also caps the claims of those classes together with the raw-name VMs it covers. Clearing the vGPU flag refuses new vGPU orders; held claims already ordered keep counting and can still move or be rebound, so a pool on `claimSelector` keeps its claims. +An ordered claim counts against the same GPU keys as a class entry, from the moment it is `Bound` until it is deleted (the phase table), in the same pool figure as the class entries of section 5; a `claim` or `claimSelector` entry declares nothing of its own, since its claims already count. The validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, that pool figure and the vGPU entitlement. A raw GPU key on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has a GPU key covering the class, so an ordered claim, which no `ResourceQuota` sees, never bypasses an existing raw limit. While the raw key exists, it also caps the claims of those classes together with the raw-name VMs it covers. Clearing the vGPU flag refuses new vGPU orders; ordered claims keep counting and can still move or be rebound, so a pool on `claimSelector` keeps its claims. -Each held claim gets a usage record with `kind: gpu-claim` and resources `.gpuclass.cozystack.io/devices: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 10). The GPU controller writes the record itself, with its `GPUClaim` as the record's owner through an ownerReference, and with no `WorkloadMonitor` and no label selector, so no tenant label can add a record or drop one ([0011](./decisions/0011-a-gpuclaim-owns-its-usage-record.md)). The `Workload` controller, which today deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, leaves a record owned by a `GPUClaim` to its owner: the GPU controller deletes it when the claim leaves `Bound` and `Attached`, and garbage collection deletes it with the claim. A virt-launcher pod's `Workload` does not report GPUs its held claims cover a second time. +Each ordered claim gets a usage record with `kind: gpu-claim` and resources `.gpuclass.cozystack.io/devices: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 10). The GPU controller writes the record itself, with its `GPUClaim` as the record's owner through an ownerReference, and with no `WorkloadMonitor` and no label selector, so no tenant label can add a record or drop one ([0011](./decisions/0011-a-gpuclaim-owns-its-usage-record.md)). The `Workload` controller, which today deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, leaves a record owned by a `GPUClaim` to its owner: the GPU controller deletes it when the claim leaves `Bound` and `Attached`, and garbage collection deletes it with the claim. A virt-launcher pod's `Workload` does not report GPUs its ordered claims cover a second time. -The catalog's `free` and `maxFreePerNode` subtract held units from step 2 on, and leave out nodes with a pending layout change, which the binder skips. A hoster lists bound claims per node with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write, and the class's bound units and `Lost` count join its metrics. A `gpuclaim` option source lists a tenant's own claims. +The catalog's `free` and `maxFreePerNode` subtract units that claims hold from step 2 on, and leave out nodes with a pending layout change, which the binder skips. A hoster lists bound claims per node with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write, and the class's bound units and `Lost` count join its metrics. A `gpuclaim` option source lists a tenant's own claims. #### Forbidding raw names @@ -396,8 +396,8 @@ Step 1: Step 2: -- Tenants get `GPUClaim`, `claim` entries in `vm-instance` and `claimSelector` entries in `kubernetes-nodes`. A held claim is billed while it holds GPUs, `Bound` or `Attached`, whether a VM runs on it or not. An autoscaled pool on `claimSelector` holds, and pays for, every claim its selector matches, which is at least `maxReplicas`. A tenant who wants to pay only while running uses class entries, and finds out at start, not at order time, whether a unit is free. -- Hosters get `onCordon` on classes and `gpu.rawResourceNames`; an external billing system gets a usage record per held claim. +- Tenants get `GPUClaim`, `claim` entries in `vm-instance` and `claimSelector` entries in `kubernetes-nodes`. An ordered claim is billed while it holds GPUs, `Bound` or `Attached`, whether a VM runs on it or not. An autoscaled pool on `claimSelector` holds, and pays for, every `Bound` or `Attached` claim its selector matches; admission keeps `maxReplicas` at or below the number of claims it matches. A tenant who wants to pay only while running uses class entries, and finds out at start, not at order time, whether a unit is free. +- Hosters get `onCordon` on classes and `gpu.rawResourceNames`; an external billing system gets a usage record per ordered claim. ## Upgrade and rollback compatibility @@ -430,9 +430,9 @@ Step 1: Step 2 adds: - Tenant access. Tenants with the `admin` role or above manage `GPUClaim` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. -- What a tenant controls. A tenant never supplies a node through a claim reference: it comes from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a held claim cannot leave the usage records. The webhook takes claims only from the VM's own namespace. With `forbidden` any GPU entry without a class or claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. -- Webhooks. Step 2 adds a third fail-closed GPU webhook, on orders, and the gate in the launcher pod webhook of step 1. The VMI and launcher pod webhooks still see only objects carrying the reference label, so their outage blocks the start of VMs with class or claim entries. The pod webhook has to fail closed, also because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order, so its outage also blocks new orders, while held claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity when it creates and binds the claim. -- Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `cozystack.io/gpuclaim-template` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so a held claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. +- What a tenant controls. A tenant never supplies a node through a claim reference: it comes from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a tenant cannot take an ordered claim out of the usage records. The webhook takes claims only from the VM's own namespace. With `forbidden` any GPU entry without a class or claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. +- Webhooks. Step 2 adds a third fail-closed GPU webhook, on orders, and the gate in the launcher pod webhook of step 1. The VMI and launcher pod webhooks still see only objects carrying the reference label, so their outage blocks the start of VMs with class or claim entries. The pod webhook has to fail closed, also because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order, so its outage also blocks new orders, while ordered claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity when it creates and binds the claim. +- Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `cozystack.io/gpuclaim-template` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so an ordered claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. - Template claims. For class entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the entry names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, uses the GPUs of its class entries, and anyone who can start it makes their claims exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). They are not in the usage records themselves, so the launcher pod's `Workload` carries them, with the gap that Security step 1 names for directly written VirtualMachines. - Controller. It gains deleting unscheduled VMIs that reference its claims, and `update` and `delete` on pods in every namespace, cluster-wide like its write on Secrets, because tenant namespaces appear at runtime. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses. It also writes one ConfigMap, the GPU settings in `cozy-system`. - Visibility. A tenant sees the node name of its own claims. @@ -475,10 +475,10 @@ Step 2: - The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. Where its `runStrategy` creates a next VMI after a failure before running, that VMI waits for the start-failure backoff. The binder avoids that node for the set for ten minutes or until the node changes. Free CPU or memory on GPU nodes if this repeats. - A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and a VM whose `runStrategy` creates a new instance after the eviction starts there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. - A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and where the VM's `runStrategy` creates a next VMI after such a deletion, that VMI gets the moved claim. -- A node dies but stays registered → after the grace period its attached claims go to `Lost` and leave the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of a VM whose `runStrategy` creates one after a VMI that ran fails gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's held claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. A VM whose `runStrategy` creates no next VMI after a VMI that ran fails keeps its failed VMI, which leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. +- A node dies but stays registered → after the grace period its attached claims go to `Lost`, the ordered ones among them leaving the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the ordered claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of a VM whose `runStrategy` creates one after a VMI that ran fails gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's ordered claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. A VM whose `runStrategy` creates no next VMI after a VMI that ran fails keeps its failed VMI, which leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. - A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; where the VM's `runStrategy` creates a next VMI after such a deletion, it starts once the claims have moved or been rebound and any start-failure backoff the table gives has passed. - A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has. Where the VM's `runStrategy` creates a next VMI after such a deletion, it starts on the new node once the claims have moved or been rebound and any start-failure backoff the table gives has passed. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. -- A GPU fails under a running VM → the claim stays `Attached` and in the usage records while the VM runs; when the VM restarts, the controller moves the claim off that node before it releases the pod, or the claim goes to `Lost` and the pod stays gated until it is rebound. +- A GPU fails under a running VM → the claim stays `Attached`, and an ordered one in the usage records, while the VM runs; when the VM restarts, the controller moves the claim off that node before it releases the pod, or the claim goes to `Lost` and the pod stays gated until it is rebound. - A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use class entries or a `claimSelector`, or allow raw names again while the cluster has such pools. - A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. - A pool worker is refused → no matching claim is free, because of a surge machine during a rollout, claims still `Pending` or `Lost`, matching claims only on nodes that are not `Ready`, or another pool or a VM using the matching claims; wait, add claims, give each pool its own labels, or lower `maxReplicas`. @@ -502,7 +502,7 @@ Step 2: - Unit tests for the binder and the three GPU webhooks: capacity with raw-name usage, a set of two classes skipping a node only one of them tolerates, budgets in the pool figure with class entries, the phase table, every move and refusal in section 9, concurrent starts, the gate added once on a repeated call and on the temporary `WaitForFirstConsumer` pod but not on a migration target, the pin appended to every existing node selector term and the gate removed in the same update, and the pod pin after a LiveUpdate sync. - Webhook and controller tests for cordons, template claims and NICs: the webhook binds, moves and creates no claim, a set that cannot be placed keeps the pod gated with a condition and is released once it can, a pinned pod that the node refuses is deleted and that node is avoided for the set, under `Move` a claim moves at VMI start and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity a VMI with a pinned pod is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, a gated VMI is not deleted and its claims are gathered, a cordon or `NotReady` on the pinned node does not count as a refusal, a `Once` or `Manual` VM stays stopped after a refused pod is deleted, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only class entries gets one node with room for all of its template claims, and under `Stay` a single claim waits; a template claim is created by the controller once the pod waits at the gate, under the capacity checks, owned by the VMI, deleted with its VMI, reused on a repeated pass, counted against no budget of its own, and a dry run records nothing. -- Quota tests for held claims: a held claim counts from `Bound`, a `claim` entry declares nothing, a raw key blocks orders of its class until a GPU key covers it, and a cleared vGPU flag keeps held claims and refuses new orders. +- Quota tests for ordered claims: an ordered claim counts from `Bound`, a `claim` entry declares nothing, a raw key blocks orders of its class until a GPU key covers it, and a cleared vGPU flag keeps ordered claims and refuses new orders. - Admission tests for `maxReplicas`, class lifecycle while the class is in use, tenant finalizers, the controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no class or claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. - Controller tests for the usage record: present exactly while `Bound` or `Attached`, owned by its `GPUClaim`, independent of tenant labels, not reported twice, kept through reconciles of the `Workload` controller although it has no monitor label, and gone with its claim. - E2E with passthrough GPUs on the GPU runner: order, attach, stop, confirm another tenant cannot take the GPU, start again, and check the usage record. @@ -528,7 +528,7 @@ Then: - Release on tenant suspension. A hoster may want a suspended tenant's held GPUs released after a GPU-specific grace period; Cozystack has no platform-wide notion of suspension yet. - Packing policy. Fullest-node packing is the binder's default in step 2; hosters may need spreading across failure domains as a class option. -- Held claims under #48. A `GPUClaim` is not an application, so #48's oracle, which sums application releases, does not see it. Step 2 needs either `GPUClaim` counted next to applications in the pool figure, or the claim made an application kind with a declared consumption. +- Ordered claims under #48. A `GPUClaim` is not an application, so #48's oracle, which sums application releases, does not see it. Step 2 needs either `GPUClaim` counted next to applications in the pool figure, or the claim made an application kind with a declared consumption. ## Alternatives considered diff --git a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md index be5e919c..82ff381c 100644 --- a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md +++ b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md @@ -12,7 +12,7 @@ On the `default` and `vgpu` variants the platform writes a `permittedHostDevices` table into the KubeVirt CR whenever it has entries. The proposal assumed that KubeVirt refuses a GPU outside that table when the VMI is admitted. It does not. In KubeVirt v1.9.0 no admitter refuses a GPU or host-device name the table leaves out; the GraceIOVirtualization admitter reads the table only to apply its rules to names listed there, and the refusal is `validatePermittedHostDevices` in [`renderresources.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/renderresources.go), which runs when virt-controller renders the virt-launcher pod. A render error creates no pod, and the VMI stays `Pending` with `Synchronized=False`, reason `FailedCreate`, retried with backoff. It never reaches a final phase. -A VM that names such a class would start and never run, with nothing telling the tenant why at start. With held claims (step 2) it is worse: a claim leaves `Attached` only when its VMI is final or removed, so a claim of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry. +A VM that names such a class would start and never run, with nothing telling the tenant why at start. With ordered claims (step 2) it is worse: a claim leaves `Attached` only when its VMI is final or removed, so a claim of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry. ## Decision diff --git a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md index c2938538..76b5daf6 100644 --- a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md +++ b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md @@ -20,7 +20,7 @@ The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.g - `vm-instance` and `kubernetes-nodes` refuse to render class and claim entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today. - In step 2 the platform refuses `gpu.rawResourceNames: forbidden`, because the policy that enforces it does not render there. -- In step 2 the controller still adds and removes the finalizers of `GPUClass` and `GPUClaim` and keeps the usage records of claims, so deleting a class, a claim or a tenant namespace never hangs. +- In step 2 the controller still adds and removes the finalizers of `GPUClass` and `GPUClaim` and keeps the usage records of ordered claims, so deleting a class, a claim or a tenant namespace never hangs. ## Why not the alternatives diff --git a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md index c6e97493..bf6544ad 100644 --- a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md +++ b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md @@ -1,4 +1,4 @@ -# 0007. Whoever can write a VM uses the GPUs of its class entries, while held claims stay with tenant admins +# 0007. Whoever can write a VM uses the GPUs of its class entries, while ordered claims stay with tenant admins - **Number:** `0007` - **Date:** `2026-10-20` @@ -16,7 +16,7 @@ In Cozystack more identities can write a `vm-instance` than can write a `GPUClai ## Decision -A class entry is part of the VM, so whoever can write the VM uses its GPUs within the tenant's budget, as with the VM's CPU and memory, and whoever can start it makes them run and enter the usage records for as long as the VMI runs; in step 2 the start also makes the entry's template claims exist. The budget is checked when the VM is written (section 5), the vGPU entitlement at every VMI admission, and in step 2 capacity when the controller creates and binds the template claims; none of these checks the identity behind the VM. Ordering a held `GPUClaim` directly stays with tenant admins and above. +A class entry is part of the VM, so whoever can write the VM uses its GPUs within the tenant's budget, as with the VM's CPU and memory, and whoever can start it makes them run and enter the usage records for as long as the VMI runs; in step 2 the start also makes the entry's template claims exist. The budget is checked when the VM is written (section 5), the vGPU entitlement at every VMI admission, and in step 2 capacity when the controller creates and binds the template claims; none of these checks the identity behind the VM. Ordering a `GPUClaim` directly stays with tenant admins and above. ## Why not the alternatives @@ -26,7 +26,7 @@ A class entry is part of the VM, so whoever can write the VM uses its GPUs withi ## Consequences - A tenant admin who hands out a tenant ServiceAccount also hands out the ability to use GPUs through class entries, up to the tenant's budget, and the `use` role lets its holder start VMs that already carry class entries. The budget, the class keys and the vGPU entitlement are the controls that limit both. -- A held claim, which holds GPUs and is recorded while it is `Bound` or `Attached` whether or not anything runs on it, still needs a tenant admin. +- An ordered claim, which holds GPUs and is recorded while it is `Bound` or `Attached` whether or not anything runs on it, still needs a tenant admin. ## Revisit if diff --git a/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md b/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md index cf7b345b..5cfbef8f 100644 --- a/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md +++ b/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md @@ -21,7 +21,7 @@ Section 1 froze a class's `mode` only while its finalizer was present, that is w ## Why not the alternatives - **Keep the pool's mode field and the label check.** Two fields say the same thing, one of them can be wrong, and the webhook has to refuse the mismatch at every worker start instead of the chart getting it right once. A VMI without the label was a third state nobody had defined. -- **Freeze `mode` only while claims exist, as section 1 had it.** A class without claims could switch mode, and a pool that names it would then render workers for the other mode, so the chart could not rely on the name. The hoster loses nothing by the stricter rule: a different mode is a new class with a new name, which a tenant has to order against anyway because budgets and prices are per class. +- **Freeze `mode` only while claims exist, as section 1 had it.** A class without claims could switch mode, and a pool that names it would then render workers for the other mode, so the chart could not rely on the name. The hoster loses nothing by the stricter rule: a different mode is a new class with a new name, which a tenant has to name anyway because budgets and prices are per class. ## Consequences diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md index 6751de02..97db84f4 100644 --- a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -16,7 +16,7 @@ Section 3 had the mutating webhook on VirtualMachineInstance creation do all of ## Decision -The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the held claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. The virt-launcher pod webhook, which from step 1 adds the classes' tolerations, also adds the scheduling gate `cozystack.io/gpuclaims` to the pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. For a set made only of template claims, the controller keeps its rejected nodes in memory against the VM, and a controller restart forgets them. `status.bindRequest` is dropped. +The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the ordered claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. The virt-launcher pod webhook, which from step 1 adds the classes' tolerations, also adds the scheduling gate `cozystack.io/gpuclaims` to the pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. For a set made only of template claims, the controller keeps its rejected nodes in memory against the VM, and a controller restart forgets them. `status.bindRequest` is dropped. ## Why not the alternatives diff --git a/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md b/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md index fe3ccefc..1b14d90d 100644 --- a/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md +++ b/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md @@ -16,7 +16,7 @@ When this was decided, VEP 344 was open and its implementation, kubevirt/kubevir ## Decision -Holding is step 2 of the proposal, in a section of its own, and none of its kinds, fields or phases enter the API before it: `GPUClaim`, `onCordon` and the class finalizer, `claim` and `claimSelector` entries, the binder, moves, `Lost` and rebind, the gather, the launcher pod gate and pin, template claims for class entries, the claim order webhook and tamper policy, the usage records of held claims, and `gpu.rawResourceNames: forbidden`. +Holding is step 2 of the proposal, in a section of its own, and none of its kinds, fields or phases enter the API before it: `GPUClaim`, `onCordon` and the class finalizer, `claim` and `claimSelector` entries, the binder, moves, `Lost` and rebind, the gather, the launcher pod gate and pin, template claims for class entries, the claim order webhook and tamper policy, the usage records of ordered claims, and `gpu.rawResourceNames: forbidden`. Holding stays GPU-specific and does not become a generic reservation. CPU and memory are interchangeable across nodes, so a reservation of them is a count the scheduler can meet on any node with room. A GPU hold is a whole unit on a specific node that does not migrate, since KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, and that is the reason for the binder, `Lost`, the gather and `onCordon`. A DRA path does not exist today: VEP 344 is open and unimplemented, and the vGPU VF is handed out by a device plugin, not by a DRA driver. From 5a2066602a3ce5d01fa6d167058ae4a96d48c9c7 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 29 Oct 2026 18:09:06 +0300 Subject: [PATCH 31/32] docs(design-proposals): deliver the vGPU license to each VM in step 1 A token baked into a shared guest image reaches every tenant that can clone it. GPUClass gains a license with the hoster's token Secret and the FeatureType; the GPU controller copies the Secret only into entitled namespaces, and the VMI webhook gives each VMI of a licensed vGPU class a Secret volume, so VMs and pool workers get the license the same way and rotation is a VMI restart. The vGPU flag's revisit condition now points at this copy, and links to the withdrawn Kairos GPU proposal point at the Kairos tenant pool proposal. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 55 ++++++++++++++----- ...7-the-vgpu-entitlement-is-a-tenant-flag.md | 4 +- ...ense-reaches-each-vm-as-a-secret-volume.md | 37 +++++++++++++ 3 files changed, 79 insertions(+), 17 deletions(-) create mode 100644 design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 9cfbe5ef..4939c10c 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -13,7 +13,8 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Scope and related proposals -- [`kairos-vgpu-workers`](../kairos-vgpu-workers/README.md) gives managed Kubernetes worker pools on Kairos a worker image per GPU mode and delivers the vGPU license token to the workers. [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers GPU guest images for VMs. Both name a `GPUClass` in their `gpus` entries from step 1 on, and take ordered claims once step 2 ships; the links resolve once they merge. +- [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers GPU guest images for VMs. It names a `GPUClass` in its `gpus` entries from step 1 on, and takes ordered claims once step 2 ships; the link resolves once it merges. +- [Kairos tenant worker pools](https://github.com/cozystack/community/pull/115) (cozystack/community#115) proposes worker pools on Kairos and defines no GPU pools. A future Kairos GPU pool takes GPUs through class entries, and through `claimSelector` once step 2 ships, under the rules of section 4 for pools on another worker OS, and its worker image installs the license from the `GPULICENSE` disk (section 6). - [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared sizes the quota authority and refuses an application that does not fit its pool when it is written. GPU budgets are ordinary keys of that gate (section 5). Until it ships, an interim check does the same for GPU keys only. - [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and step 2 rebinds their `Lost` claims. The link resolves once it merges. - [IP addresses as a first-class resource](https://github.com/cozystack/community/pull/35) solves a similar problem for addresses with a class, a tenant claim and a backend contract. @@ -21,6 +22,7 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Decisions +- [0018. A vGPU license reaches each VM as a Secret volume, copied only into entitled namespaces](./decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md) - [0017. The vGPU entitlement is a Tenant flag stamped on the namespace, not a budget key](./decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md) - [0016. Holding a GPU for a stopped VM is step 2, and stays GPU-specific until DRA can carry it](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md) - [0015. Node allocatable is the GPU inventory, and there is no `GPUNode`](./decisions/0015-node-allocatable-is-the-gpu-inventory.md) @@ -66,6 +68,7 @@ Step 1: - One cluster can have passthrough and vGPU nodes of the same GPU model, with vGPU profiles applied declaratively and kept across reboots, and every offer advertised under a resource name of its own. - A hoster publishes a catalog of GPU classes, each with a model, a mode (passthrough or vGPU) and, for vGPU, a profile. Tenants read the catalog with free capacity and their remaining budget. - VMs and worker pools name a class, not a resource name, and a tenant's budget per class and in total is checked when the VM or pool is written, with a message naming the class and the limit. +- A VM of a vGPU class gets its license token and license settings from the platform, in entitled tenants only, without a token in any shared image. - Usage records (`Workload`) show which tenant runs how many GPUs of which offer in which VM or worker, which an external billing system reads. Cozystack charges nothing and stores no prices. - Existing `gpus` entries keep working, with a documented path to class entries. @@ -77,7 +80,7 @@ Step 2: ### Non-goals - Prices and charging. They live in an external billing system, keyed by class name. -- The vGPU guest driver, the license server and license tokens. The hoster provides them (section 7). +- The vGPU guest driver and the license server. The hoster provides them and its license token, which the platform only delivers (section 6). - The catalog UI. This proposal provides the data a portal reads. - GPUs for pods inside tenant clusters (gpu-operator in a tenant cluster, HAMi). - Time-slicing one GPU between tenants, and live migration of GPU VMs. @@ -101,6 +104,9 @@ Every other name follows the group: | `cozystack.io/gpus` | 1 | annotation on that template listing the entries | | `cozystack.io/gpu--` | 1 | host-layer node label carrying a model's mode or profile (section 2) | | `cozystack.io/gpu-layout-pending` | 1 | node label marking a pending layout change (section 2) | +| `gpuclass--license` | 1 | name of a class's license copy in an entitled namespace (section 6) | +| `cozystack.io/gpuclass` | 1 | label on a license copy naming its class | +| `GPULICENSE` | 1 | volume label of the license disk a VMI of a licensed vGPU class gets (section 6) | | `namespace.cozystack.io/vgpu` | 1 | namespace label carrying the vGPU entitlement, like the other module labels (section 6) | | `cozystack.io/gpuclass-protection` | 2 | finalizer on a class in use (section 9) | | `cozystack.io/gpuclaim-protection` | 2 | finalizer on a claim a VMI uses | @@ -145,6 +151,9 @@ spec: model: L40S mode: vgpu # passthrough | vgpu profile: L40S-24C # vgpu only + license: # vgpu only; can change + secretName: nvidia-license-token # Secret in cozy-system + featureType: 4 # 1 by default memory: 24Gi maxCount: 2 # vGPUs of this class per VM tolerations: # taints of this offer's nodes that its VMs tolerate @@ -167,6 +176,8 @@ How many GPUs of a class one VM may take is the hoster's `maxCount` on the class `tolerations` lists the taints of the class's nodes that its VMs tolerate, such as a taint that reserves GPU nodes for GPU workloads, in the shape of a pod's with the operators `Exists` and `Equal` only. The launcher pod of a VM that names the class gets them (section 4), and `free` reads them ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). +`license` is how a VM of a vGPU class gets licensed (section 6). `secretName` names a Secret in `cozy-system` whose key `client_configuration_token.tok` holds the client configuration token the hoster downloads from its NVIDIA license server. `featureType` is the `FeatureType` the guest's license client reads from `gridd.conf`, `1` by default. NVIDIA's [client licensing guide](https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/configuring-nls-licensed-client.html) documents `1` for a VM with a vGPU, since "NVIDIA vGPU software automatically selects the correct type of license based on the vGPU type", and its `gridd.conf` template lists `4` as "NVIDIA vGPU for Compute"; `4` has been observed working on a C-series profile. The CRD admits `license` only on a class whose `mode` is `vgpu`, requires `secretName` as a DNS subdomain name, and takes `featureType` only as `1` or `4`. `license` can change, and a change reaches each VM at its next start. + `status.free` is the class's allocatable count summed over its nodes that are `Ready`, not cordoned, and have no `NoSchedule` or `NoExecute` taint the class does not tolerate, minus what pods on those nodes request of the resource name, and no node counts below zero; `status.maxFreePerNode` is the largest such figure on one node. A class whose `resourceName` the KubeVirt `permittedHostDevices` table leaves out gets a `ResourceNameNotPermitted` condition, so the hoster sees the gap before a tenant's VM is refused ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included ([0005](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md)). Both figures come from the `Node` and `Pod` objects alone, with no inventory kind ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). In step 1 they describe the moment, not a promise: nothing is held, so a unit counted free can be taken by the next VM to start. ### 4. Class entries in VMs and worker pools @@ -198,7 +209,7 @@ A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain Capacity is not checked at admission in step 1. A started VM whose class has no free unit on a node it can use waits in `Scheduling`, its launcher pod `Pending` with `Unschedulable` and the VM shown as `ErrorUnschedulable`, as a VM short of CPU or memory does today, and starts when a unit frees up; the catalog's `free` and `maxFreePerNode` tell a tenant beforehand whether that is likely. Refusing the VMI instead would leave the VM to the per-item backoff of KubeVirt's VM controller, which grows to 1000 seconds, for a shortage that may last seconds. -A pool states no mode of its own: its entry names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a class entry that names a vGPU class on a Talos pool, before any machine is created. Worker pools on another OS, such as the Kairos pools of `kairos-vgpu-workers`, take vGPU classes as that proposal describes. Talos GPU pools on `name` entries keep working as today. +A pool states no mode of its own: its entry names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a class entry that names a vGPU class on a Talos pool, before any machine is created. A pool on another worker OS takes a vGPU class only with a worker image that carries the vGPU guest driver, which is a requirement on future Kairos GPU pools; cozystack/community#115 defines none yet. Talos GPU pools on `name` entries keep working as today. ### 5. Budgets @@ -212,11 +223,21 @@ The tenant declaration gate validates GPU keys like any other: a child's declare Pool membership follows the existing rules too, which #48 keeps. Until #48 ships, the runtime pool controller reads the rendered `ResourceQuota`, which has no GPU keys, so a tenant whose quotas hold only GPU keys is a pool root for the interim check and not for that controller; #48 removes the difference by reading declared values on both sides. Existing raw GPU keys, quota keys on a resource name such as `nvidia.com/AD102GL_L40S`, keep their meaning: they render into the `ResourceQuota` and limit launcher pods when they are created, whether the VM names the resource or a class of it (section 8). -### 6. vGPU entitlement +### 6. vGPU entitlement and license delivery A vGPU class is used only by a tenant entitled to vGPU ([0017](./decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md)). The Tenant field `gpu.vgpu` grants it. The parent tenant's administrator sets it, or the hoster on a top-level tenant. A tenant is entitled only when the flag is set on it and its parent is entitled. The root tenant is rendered by the platform, not by the tenant chart, so its value comes from platform values. The flag is off by default everywhere, so the hoster turns it on for the root first. No tenant can grant itself vGPU, since its Tenant object lives in its parent's namespace. The tenant chart computes the effective value from the parent's, like its other namespace values, and stamps it on the namespace as the label `namespace.cozystack.io/vgpu`, the same pattern as `namespace.cozystack.io/etcd` and the other module labels; a change reaches grandchildren as each level re-renders. The chart always writes the label, so only the value `"true"` means entitled; any other value, or no label, means not entitled. -Entitlement is separate from budget because a vGPU license is counted per unit, which is a budget, while getting the vGPU guest driver and the license token at all is yes or no. The GPU keys of section 5 then limit vGPU like any other GPU. The VMI webhook refuses a vGPU class entry without the entitlement at every start (section 4), so clearing the flag stops VMs on vGPU classes from starting again while running ones keep running, and the tenant's Kairos pools lose their vGPU entitlement as `kairos-vgpu-workers` describes. A write that names a vGPU class is not refused for the entitlement, since the pool gate of section 5 counts sizes and has no yes-or-no rule; the option source offers a vGPU class to an unentitled tenant with a remaining budget of 0 (section 7). Raw vGPU names in VMs stay as open as today until a raw key limits them or step 2's `forbidden` is set. +Entitlement is separate from budget because a vGPU license is counted per unit, which is a budget, while getting the vGPU guest driver and the license token at all is yes or no. The GPU keys of section 5 then limit vGPU like any other GPU. The VMI webhook refuses a vGPU class entry without the entitlement at every start (section 4), so clearing the flag stops VMs on vGPU classes from starting again while running ones keep running, and the namespace's license copies are deleted (License delivery, below). A write that names a vGPU class is not refused for the entitlement, since the pool gate of section 5 counts sizes and has no yes-or-no rule; the option source offers a vGPU class to an unentitled tenant with a remaining budget of 0 (section 7). Raw vGPU names in VMs stay as open as today until a raw key limits them or step 2's `forbidden` is set. + +#### License delivery + +The platform hands the license of a vGPU class to each VM that uses the class, as a volume of that VM, not as part of an image ([0018](./decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md)). + +The GPU controller copies the hoster's Secret of each class with a `license` into every namespace labelled `namespace.cozystack.io/vgpu: "true"`, as a Secret named `gpuclass--license` and labelled `cozystack.io/gpuclass: `. A copy has two keys: `client_configuration_token.tok`, the token, and `gridd.conf`, which sets the class's `FeatureType`. The controller writes or refreshes a copy when a namespace gains the label, when a class gains or changes its `license`, and when the hoster's Secret changes. It deletes a namespace's copies when the label stops being `"true"`, and a class's copies when the class is deleted or drops its `license`. While the hoster's Secret is missing or has no `client_configuration_token.tok` key, the controller keeps the copies it has, writes no new ones, and sets a `LicenseSecretMissing` condition on the class. No copy is written into a namespace without the entitlement. + +When the VMI webhook of section 4 resolves a vGPU class whose `license` is set, it adds a KubeVirt `secret` volume for that class's copy, with the volume label `GPULICENSE`, and a disk for it after the VMI's other disks and without a `bootOrder`, so the boot order of a VM does not change. KubeVirt gives the guest the Secret's keys as files on a disk with that label ([`schema.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/staging/src/kubevirt.io/api/core/v1/schema.go), `SecretVolumeSource`). The guest mounts the disk by its label and puts the token under `/etc/nvidia/ClientConfigToken/` and `gridd.conf` under `/etc/nvidia/`; a GPU guest image for VMs and a worker image with the vGPU guest driver do this at boot, and this disk, its label and these paths are the contract those images follow. VMs and pool workers get the volume the same way, since both start as VMIs with class entries, and neither the `vm-instance` nor the `kubernetes-nodes` chart changes for the license. The volume stays on a running VMI: KubeVirt v1.9.0 rewrites a VMI's volumes from its VM only for declarative hotplug volumes and, under the `Migration` volume update strategy, for the PVC volumes it migrates ([`hotplug.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/storage/hotplug/hotplug.go), [`volume-migration.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/volume-migration/volume-migration.go)), and a `secret` volume is neither. Until the copy exists, kubelet does not start the launcher pod, so the VM waits rather than boots unlicensed: the VM shows `Starting`, its launcher pod stays in `ContainerCreating` with a `FailedMount` event naming the missing Secret, and while the hoster's Secret is missing or lacks the token key the class shows `LicenseSecretMissing`. + +Rotation is a VMI restart. The hoster replaces the token in its Secret, the controller refreshes every copy, and each VM reads the new token at its next start. Deleting a namespace's copies stops new starts there from getting a token, and the VMI webhook refuses them anyway without the entitlement; a running guest keeps the token on its disk until it stops, so a hoster who must cut a tenant off at once also revokes the token on its license server. ### 7. Usage records, catalog and licensing @@ -226,7 +247,7 @@ The catalog is the list of `GPUClass` objects with `displayName`, model, mode, p The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest entry, leaving out a field that is absent because it sets no limit. The remaining budget is the pool figure of section 5, so a tenant under a shared parent budget can see how many GPUs its siblings declare, as a number, without names. This is accepted: the alternative is a budget the tenant can only discover by a refused write, which is the failure this proposal removes. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. The existing `gpu` source keeps serving raw names while they are allowed. -vGPU needs a license, and passthrough needs one only for graphics. Compute (C-series) profiles are licensed through NVIDIA AI Enterprise, graphics (Q-series) profiles through NVIDIA vGPU software. The hoster provides the license server, the token and the guest drivers. In worker pools `kairos-vgpu-workers` delivers the token and the guest license settings. In VMs the hoster does, through its image or at boot, as `vgpu-vms` describes. The class's `mode` tells a portal and a tenant which case applies. +vGPU needs a license, and passthrough needs one only for graphics. Compute (C-series) profiles are licensed through NVIDIA AI Enterprise, graphics (Q-series) profiles through NVIDIA vGPU software. The hoster provides the license server, the token and the guest drivers, and the platform delivers the token and the `FeatureType` of a vGPU class to each VM and worker of an entitled tenant (section 6). A passthrough class carries no `license`, so a licensed passthrough GPU for graphics still gets its token through a private image the hoster gives the tenants it licenses, not through the shared golden images. The class's `mode` tells a portal and a tenant which case applies. ### 8. Migration of existing `gpus` @@ -333,7 +354,7 @@ gpus: A `vm-instance` entry has exactly one of `name`, `className` or `claim`, and a `kubernetes-nodes` entry exactly one of `name`, `className` or `claimSelector`. A list holds only `name` entries or only class and claim entries, a pool has at most one `claimSelector` and never next to a class entry, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The rules of section 4 for Talos pools, the vGPU entitlement and `permittedHostDevices` apply to the class of a claim as to a class entry. -The VMI webhook of section 4 also resolves claim references, runs the checks that need no placement, and records the attachments; it binds, moves and creates no claim, and waits for nothing. For a VM it takes the named claims. For a pool worker it takes the claim the worker had attached before, unless a VM now names it, its node is not `Ready`, or its node has a taint the class does not tolerate. Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment of each ordered claim to this VMI, so two VMIs can never hold one claim and no background move can take one, and writes one `gpus` entry per GPU unit, ordered or from a class entry, with the class's resource name. The launcher pod webhook of section 4 adds the tolerations of the ordered claims' classes as it does for class entries. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. On a dry run the webhook records nothing, so in step 2 it declares `sideEffects: NoneOnDryRun`. The GPU controller places the claims after admission, while the launcher pod waits behind a scheduling gate (Placing a VMI's claims). +The VMI webhook of section 4 also resolves claim references, runs the checks that need no placement, and records the attachments; it binds, moves and creates no claim, and waits for nothing. For a VM it takes the named claims. For a pool worker it takes the claim the worker had attached before, unless a VM now names it, its node is not `Ready`, or its node has a taint the class does not tolerate. Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment of each ordered claim to this VMI, so two VMIs can never hold one claim and no background move can take one, and writes one `gpus` entry per GPU unit, ordered or from a class entry, with the class's resource name. The launcher pod webhook of section 4 adds the tolerations of the ordered claims' classes as it does for class entries, and the VMI webhook adds the license volume of section 6 for a licensed vGPU class reached through an ordered claim, as for a class entry. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. On a dry run the webhook records nothing, so in step 2 it declares `sideEffects: NoneOnDryRun`. The GPU controller places the claims after admission, while the launcher pod waits behind a scheduling gate (Placing a VMI's claims). The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim does not exist, is being deleted, or is attached to another VMI, besides the rules of section 4 that apply to the class of a claim (above); a class being deleted does not refuse an ordered claim that is `Bound`, `Attached` or `Lost`, which its finalizer keeps; a `Pending` claim of a deleted class is refused, since it is never bound. A refused VMI leaves its claims unattached, so they move by the rules above. Any other claim that is `Pending` or `Lost`, or bound to a node that cannot serve it, is not refused: the controller binds or moves it before it releases the pod. @@ -392,7 +413,7 @@ Step 1: - Tenants get a GPU catalog with free capacity and remaining budget per class, and `className` entries in `vm-instance` and `kubernetes-nodes`. Raw resource names keep working. - A write of a VM or pool whose class entries do not fit a GPU budget is refused. A started VM that finds no free unit waits in `Scheduling` until one frees up. -- Hosters get `GPUClass`, GPU keys in `resourceQuotas`, the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout and the `gpu.resourceNames` list; an external billing system reads GPUs from the `Workload` records of virt-launcher pods. +- Hosters get `GPUClass` with a `license` for vGPU classes, which the platform delivers to VMs, GPU keys in `resourceQuotas`, the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout and the `gpu.resourceNames` list; an external billing system reads GPUs from the `Workload` records of virt-launcher pods. Step 2: @@ -403,11 +424,11 @@ Step 2: The change is additive. Existing `gpus` entries render as before, and the existing variants work as today until a layout is set. -Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, with these exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only GPU limits were GPU keys lose them, and on a cluster that ran step 2 with `forbidden`, tenants without the vGPU entitlement can use raw vGPU names again. Kairos GPU pools take GPUs only through classes, so they lose their GPUs and run on the plain worker image. +Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, with these exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only GPU limits were GPU keys lose them, and on a cluster that ran step 2 with `forbidden`, tenants without the vGPU entitlement can use raw vGPU names again. A worker pool that takes GPUs only through classes, as a future Kairos GPU pool would, loses its GPUs. A vGPU VM or worker that restarts after step 1 rolls back gets no `GPULICENSE` disk, so the hoster first puts the license token into private images it gives entitled tenants only, not into the shared golden images ([0018](./decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md)). Step 2 rolls back first. Raw names are allowed again, since raw keys are refused while raw names are forbidden. `claim` entries of VMs and `claimSelector` entries of pools become class entries, and those VMs restart and those pools roll. The `GPUClaim` objects are deleted, and the step 2 components go. -Step 1 rolls back next. The hoster removes the node layout and chooses the variant in a maintenance window. Class entries become raw `name` entries with the class's resource name, and GPU keys become raw keys on those names where the tenant wants a limit, and the remaining GPU keys are deleted. Class entries are removed from Kairos pools before the `gpu.vgpu` flag goes, since otherwise their new vGPU worker templates wait for a license copy. Last, the components, the `GPUClass` CRD, the `gpu.vgpu` flag with its namespace label and the `cozystack.io/gpu-*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. +Step 1 rolls back next. The hoster removes the node layout and chooses the variant in a maintenance window. Class entries become raw `name` entries with the class's resource name, and GPU keys become raw keys on those names where the tenant wants a limit, and the remaining GPU keys are deleted. After the class entries have become raw names, the hoster clears the `gpu.vgpu` flags before the components go, so the controller deletes the license copies in tenant namespaces; if the controller is already gone, the hoster deletes them by the `cozystack.io/gpuclass` label; guests that keep running keep the token on their disk until they stop. Last, the components, the `GPUClass` CRD, the `gpu.vgpu` flag with its namespace label and the `cozystack.io/gpu-*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. Three rules hold throughout: @@ -420,11 +441,11 @@ Three rules hold throughout: Step 1: - Tenant access. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. -- What a tenant controls. A tenant never supplies a resource name through a class entry: it comes from `GPUClass`, which tenants cannot write. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. A tenant super-admin who writes VirtualMachines directly can still write raw GPU names, as today, until step 2's `forbidden`; a VMI with class entries and GPU entries of its own is refused. GPU keys are checked only when an application is written. A VirtualMachine written directly, with class entries in its annotation or a higher `count` than its application declares, is not counted against them, also under `forbidden`. That path closes when tenant super-admins lose write access to VirtualMachines, which #48 requires before its phase 4. A directly written VirtualMachine can also be left out of the usage records, or recorded with sums other than what it uses. Until super-admins lose that write access, a hoster who sells GPU budgets or bills GPUs to sub-tenants does not give them the super-admin role. +- What a tenant controls. A tenant never supplies a resource name through a class entry: it comes from `GPUClass`, which tenants cannot write. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The license token reaches only the guests of entitled tenants, where the guest's root user can read it, which licensing a guest needs, and a tenant super-admin who writes a VirtualMachine directly can also mount its namespace's copy into it; a copy is never written into a namespace without the entitlement. A tenant super-admin who writes VirtualMachines directly can still write raw GPU names, as today, until step 2's `forbidden`; a VMI with class entries and GPU entries of its own is refused. GPU keys are checked only when an application is written. A VirtualMachine written directly, with class entries in its annotation or a higher `count` than its application declares, is not counted against them, also under `forbidden`. That path closes when tenant super-admins lose write access to VirtualMachines, which #48 requires before its phase 4. A directly written VirtualMachine can also be left out of the usage records, or recorded with sums other than what it uses. Until super-admins lose that write access, a hoster who sells GPU budgets or bills GPUs to sub-tenants does not give them the super-admin role. - Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. - Node agent. It reads sysfs and writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can delete only the device plugin pod of its own node. - Webhooks. The VMI and virt-launcher pod webhooks fail closed and see only objects carrying the reference label, so their outage blocks the start of VMs with class entries. -- Controller. It reads nodes and pods to compute class status, and writes `GPUClass` status. With the host layer it gains patching node labels, restricted by a policy to the GPU workload label and `cozystack.io/gpu-*` ([0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)). +- Controller. It reads nodes and pods to compute class status, and writes `GPUClass` status. It reads the license Secrets classes name in `cozy-system`, and creates, updates and deletes the `gpuclass--license` copies in entitled namespaces, through the cluster-wide write on Secrets it already holds. With the host layer it gains patching node labels, restricted by a policy to the GPU workload label and `cozystack.io/gpu-*` ([0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)). - Visibility. A tenant sees, through the remaining budget, its siblings' declared GPU count under a shared parent budget (section 7). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. Step 2 adds: @@ -464,6 +485,8 @@ Step 1: - A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. Where its `runStrategy` creates a next VMI after a VMI that ran fails, that VMI starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. - A GPU fails under a running VM on a class entry → the VM keeps running. Kubelet stops listing the device as allocatable, so the class's `free` drops, and when the VM starts again, its next instance gets a unit on a node that still has one, or waits as `ErrorUnschedulable` while there is none. - A layout change does not apply → the node carries the `cozystack.io/gpu-layout-pending` label; cordon and drain it. A GPU VM on the node holds the change until it stops or is evicted. +- A vGPU VM stays `Starting` with its launcher pod in `ContainerCreating` and a `FailedMount` event → the namespace has no license copy of the class yet; the class shows `LicenseSecretMissing` when the hoster's Secret is gone or has no `client_configuration_token.tok` key. Create or fix the hoster's Secret. +- A vGPU VM runs unlicensed → its class has no `license`, or the guest image does not install the files from the `GPULICENSE` disk; set the class's `license`, or fix the image. - A Talos pool with a vGPU class entry is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. Step 2: @@ -492,6 +515,7 @@ Step 1: - Budget tests: the interim check refuses a `VMInstance` and a `KubernetesNodes` write over a class key or the total key, counts `count` times `maxReplicas` for a pool and the defaulted `maxReplicas` for a pool that does not set it, passes shrinking and no-op updates, refuses a child tenant's GPU key against the class entries already in the parent's pool, applies the pool rules like other keys, stands aside while #48's gate is on, and reaches the same verdicts as #48's evaluator on the same inputs. - Admission tests for `KubernetesNodes`: a vGPU class entry on a Talos pool is refused, and a passthrough one is accepted. - Entitlement tests: a child's flag without the parent's gives no entitlement, a vGPU class entry without it is refused at start, and `remainingBudget` is 0 without it. +- License tests: a copy exists only in namespaces labelled `"true"` and only for classes with a `license`, carries the token and a `gridd.conf` with the class's `FeatureType`, is refreshed when the hoster's Secret changes, is deleted when the label goes, the class is deleted or drops its `license`, and is kept with `LicenseSecretMissing` while the hoster's Secret is missing; the CRD refuses `license` on a passthrough class and a `featureType` other than `1` and `4`, and defaults `featureType` to `1`; a hoster Secret without the token key gets no copy and sets `LicenseSecretMissing`; the VMI webhook adds the `secret` volume with the `GPULICENSE` label and its disk for a licensed vGPU class only, and the volume is still there after a LiveUpdate sync of the VM. - Usage tests: a VM on a class entry shows its GPUs in its `Workload` under the class's resource name while its launcher pod exists. - Admission tests for the class schema: the frozen fields, `tolerations` operators, and the class name as a DNS label. - Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. @@ -502,6 +526,7 @@ Step 2: - Unit tests for the binder and the three GPU webhooks: capacity with raw-name usage, a set of two classes skipping a node only one of them tolerates, budgets in the pool figure with class entries, the phase table, every move and refusal in section 9, concurrent starts, the gate added once on a repeated call and on the temporary `WaitForFirstConsumer` pod but not on a migration target, the pin appended to every existing node selector term and the gate removed in the same update, and the pod pin after a LiveUpdate sync. - Webhook and controller tests for cordons, template claims and NICs: the webhook binds, moves and creates no claim, a set that cannot be placed keeps the pod gated with a condition and is released once it can, a pinned pod that the node refuses is deleted and that node is avoided for the set, under `Move` a claim moves at VMI start and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity a VMI with a pinned pod is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, a gated VMI is not deleted and its claims are gathered, a cordon or `NotReady` on the pinned node does not count as a refusal, a `Once` or `Manual` VM stays stopped after a refused pod is deleted, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only class entries gets one node with room for all of its template claims, and under `Stay` a single claim waits; a template claim is created by the controller once the pod waits at the gate, under the capacity checks, owned by the VMI, deleted with its VMI, reused on a repeated pass, counted against no budget of its own, and a dry run records nothing. +- License tests for ordered claims: a VMI whose ordered claim is of a licensed vGPU class gets the `GPULICENSE` volume as a class entry does. - Quota tests for ordered claims: an ordered claim counts from `Bound`, a `claim` entry declares nothing, a raw key blocks orders of its class until a GPU key covers it, and a cleared vGPU flag keeps ordered claims and refuses new orders. - Admission tests for `maxReplicas`, class lifecycle while the class is in use, tenant finalizers, the controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no class or claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. - Controller tests for the usage record: present exactly while `Bound` or `Attached`, owned by its `GPUClaim`, independent of tenant labels, not reported twice, kept through reconciles of the `Workload` controller although it has no monitor label, and gone with its claim. @@ -512,13 +537,13 @@ Step 2: Step 1: 1. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. It needs no tenant API and can land first. -2. `GPUClass` with its status controller and metrics, class entries in `vm-instance` and `kubernetes-nodes` with the VMI webhook and the launcher pod webhook for tolerations, the `gpu.vgpu` Tenant flag, GPU keys with the interim check, and the `gpuclass` option source, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRD ships and the charts refuse class entries. Before tenants use vGPU classes, the `gpu.vgpu` flag is set from the root tenant down. It works on any cluster where each offer it publishes has a resource name of its own, with or without item 1. -3. The consumers, on top of item 2 alone: `kairos-vgpu-workers` adds Kairos GPU pools on class entries, and `vgpu-vms` adds GPU guest images. Neither waits for item 4 or for step 2. +2. `GPUClass` with its status controller and metrics, class entries in `vm-instance` and `kubernetes-nodes` with the VMI webhook and the launcher pod webhook for tolerations, the `gpu.vgpu` Tenant flag and license delivery, GPU keys with the interim check, and the `gpuclass` option source, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRD ships and the charts refuse class entries. Before tenants use vGPU classes, the `gpu.vgpu` flag is set from the root tenant down. It works on any cluster where each offer it publishes has a resource name of its own, with or without item 1. +3. The consumers, on top of item 2 alone: `vgpu-vms` adds GPU guest images, and Kairos GPU pools, once cozystack/community#115 or a successor adds them, take class entries. Neither waits for item 4 or for step 2. 4. GPU keys through #48, and the interim check deleted, in the release where #48's gate is on by default. Step 2: -5. Holding: `GPUClaim`, the binder and its metrics, the scheduling gate in the launcher pod webhook, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes and for the Kairos pools of `kairos-vgpu-workers`, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. +5. Holding: `GPUClaim`, the binder and its metrics, the scheduling gate in the launcher pod webhook, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes and for future Kairos GPU pools, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. Then: diff --git a/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md b/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md index cae6618f..92bfeed6 100644 --- a/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md +++ b/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md @@ -12,7 +12,7 @@ A vGPU needs a license, counted per unit, and a tenant that may use vGPU at all gets the vGPU guest driver and the license token. The proposal gave the yes-or-no part to a Tenant field, `gpu.vgpu`, which the tenant chart combines with the parent's value and stamps on the namespace as `namespace.cozystack.io/vgpu`. It lived inside the quota section. @kvaps [asked](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) whether a `gpu.cozystack.io/vgpu` budget key, 0 by default, would cover it, since it goes down the subtree like the other GPU keys and can cap the license pool. In Cozystack a missing quota key means no limit. He later [agreed](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) that the entitlement is a yes or no, not a quota, and that a Tenant field stamped as a namespace label is the pattern of `namespace.cozystack.io/etcd` and the other module labels, and asked for the flag in its own section with a record that says when to revisit it. -`kairos-vgpu-workers` copies the hoster's license and registry Secrets into the namespaces of entitled tenants, and uses the same label to decide where. +The license delivery of this proposal copies the hoster's license Secret into the namespaces of entitled tenants, and uses the same label to decide where (section 6). ## Decision @@ -30,4 +30,4 @@ The vGPU entitlement stays the Tenant field `gpu.vgpu`, off by default, effectiv ## Revisit if -`kairos-vgpu-workers` stops copying Secrets into tenant namespaces, for example once a hoster-side worker class reads the hoster's Secrets directly, so the flag guards nothing but the use of vGPU classes and a budget over those classes could replace it. +The platform stops copying license Secrets into tenant namespaces (License delivery in section 6), for example once a license reaches the guest without a copy per tenant, so the flag guards nothing but the use of vGPU classes and a budget over those classes could replace it. diff --git a/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md b/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md new file mode 100644 index 00000000..12446814 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md @@ -0,0 +1,37 @@ +# 0018. A vGPU license reaches each VM as a Secret volume, copied only into entitled namespaces + +- **Number:** `0018` +- **Date:** `2026-10-29` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** not yet + +## Context + +A vGPU guest needs two things before its driver licenses the GPU: the client configuration token the hoster downloads from its NVIDIA license server, and a `gridd.conf` with the `FeatureType` that fits the profile. The proposal left both to the hoster: in VMs through the guest image or at boot, and in Kairos worker pools through a separate proposal, which copied the hoster's Secrets into the namespaces of entitled tenants and put the token into each pool's bootstrap data. Rotating a token there meant replacing every vGPU machine. + +Golden images in `cozy-public` are open to every tenant: any tenant can clone one and boot it as root. A token baked into such an image is readable by every tenant, entitled or not. The license is a property of the offer, the class, and not of holding, so its delivery belongs to step 1. + +## Decision + +`GPUClass` carries a `license`, the name of the hoster's token Secret in `cozy-system` and the `FeatureType`, `1` by default or `4`, on vGPU classes only. NVIDIA's [client licensing guide](https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/configuring-nls-licensed-client.html) documents `1` for a VM with a vGPU, since "NVIDIA vGPU software automatically selects the correct type of license based on the vGPU type", and its `gridd.conf` template lists `4` as "NVIDIA vGPU for Compute"; `4` has been observed working on a C-series profile, and the field lets a hoster set it per class. The GPU controller copies each such Secret into every namespace whose `namespace.cozystack.io/vgpu` label is `"true"`, adding a `gridd.conf` with the class's `FeatureType`, and deletes the copies when the label, the class or its `license` goes. At VMI creation the VMI webhook adds a KubeVirt `secret` volume for the copy of the VMI's vGPU class, with the volume label `GPULICENSE`, and the guest installs the two files from that disk. VMs and pool workers work the same way, and neither chart changes for the license. Rotation is a VMI restart. + +## Why not the alternatives + +- **A token baked into a shared guest image.** Anyone who can clone the image reads the token, and every tenant can clone a golden image. Rotation means rebuilding the image and every disk cloned from it. +- **A license server reachable only from tenant networks, with the token in the image**, the way KMS serves Windows images. It limits where a leaked token can lease licenses, but the token still reaches every tenant that clones the image, and the network rule has to follow every tenant network. +- **The token in the VM's cloud-init or the pool's bootstrap data.** The data lives in values a tenant reads, and bootstrap data cannot change on a running node, so rotation replaces every machine. + +## Consequences + +- Only the guests of entitled tenants get the token, and they get it as a disk, so a tenant's root user inside its VM can read it; that is what licensing a guest needs. +- A guest image or a worker image with the vGPU guest driver mounts the `GPULICENSE` disk at boot; an image that does not boots unlicensed. +- The GPU controller writes Secrets in tenant namespaces, through the cluster-wide Secret write it already has. +- A running guest keeps its token until it stops, so revoking a tenant at once also needs the license server. +- A licensed passthrough GPU for graphics still gets its token from a private image the hoster gives the tenants it licenses, not from the shared golden images, since a passthrough class carries no `license`. + +## Revisit if + +KubeVirt or the NVIDIA stack delivers a license to a guest without a Secret in the tenant namespace, or hosters need licensed passthrough classes for graphics (`FeatureType` 2), which would turn the vGPU entitlement into an entitlement to licensed classes. Drop `featureType` if `1` proves sufficient on every C-series profile the platform ships. From 30da976d143cbed4907b508c008c8780d6c1a263 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 29 Oct 2026 18:48:46 +0300 Subject: [PATCH 32/32] docs(design-proposals): keep the license token off guest disks NVIDIA's license system documents no way to revoke a client configuration token, and a running guest renews its lease while the allocation serves the token. Revocation is therefore stated as what it is: clearing the entitlement takes effect at the next start, an immediate cut-off stops the VMs, and rotation moves to a new allocation and disables leasing on the old one. The guest contract now copies the token into a directory on a memory file system named by ClientConfigTokenPath, so a guest restarted without the license disk has no token left. The eviction outcomes per runStrategy and the live-migration limits of GPU VMs move from the GPU VMs proposal into the runStrategy table, which now covers a VMI that succeeded, and Scope and Rollout follow the licensing proposal. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- design-proposals/gpudevice/README.md | 44 ++++++++++--------- ...ense-reaches-each-vm-as-a-secret-volume.md | 8 ++-- 2 files changed, 28 insertions(+), 24 deletions(-) diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md index 4939c10c..a8ae5083 100644 --- a/design-proposals/gpudevice/README.md +++ b/design-proposals/gpudevice/README.md @@ -13,8 +13,8 @@ NVIDIA's vGPU stack changes quickly. Within the last few releases the driver bra ## Scope and related proposals -- [GPU virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) covers GPU guest images for VMs. It names a `GPUClass` in its `gpus` entries from step 1 on, and takes ordered claims once step 2 ships; the link resolves once it merges. -- [Kairos tenant worker pools](https://github.com/cozystack/community/pull/115) (cozystack/community#115) proposes worker pools on Kairos and defines no GPU pools. A future Kairos GPU pool takes GPUs through class entries, and through `claimSelector` once step 2 ships, under the rules of section 4 for pools on another worker OS, and its worker image installs the license from the `GPULICENSE` disk (section 6). +- [vGPU licensing for virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) builds on the license delivery of section 6: who the licensee is, how a guest image applies the license, how a guest reaches the license server, how license use maps to tenants, and rotation and revocation. The link resolves once it merges. +- [Kairos tenant worker pools](https://github.com/cozystack/community/pull/115) (cozystack/community#115) is where worker pools on Kairos are proposed. A Kairos GPU pool takes GPUs through class entries, and through `claimSelector` once step 2 ships, under the rules of section 4 for pools on another worker OS, and its worker image follows the `GPULICENSE` contract of section 6. - [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared sizes the quota authority and refuses an application that does not fit its pool when it is written. GPU budgets are ordinary keys of that gate (section 5). Until it ships, an interim check does the same for GPU keys only. - [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and step 2 rebinds their `Lost` claims. The link resolves once it merges. - [IP addresses as a first-class resource](https://github.com/cozystack/community/pull/35) solves a similar problem for addresses with a class, a tenant claim and a backend contract. @@ -68,7 +68,7 @@ Step 1: - One cluster can have passthrough and vGPU nodes of the same GPU model, with vGPU profiles applied declaratively and kept across reboots, and every offer advertised under a resource name of its own. - A hoster publishes a catalog of GPU classes, each with a model, a mode (passthrough or vGPU) and, for vGPU, a profile. Tenants read the catalog with free capacity and their remaining budget. - VMs and worker pools name a class, not a resource name, and a tenant's budget per class and in total is checked when the VM or pool is written, with a message naming the class and the limit. -- A VM of a vGPU class gets its license token and license settings from the platform, in entitled tenants only, without a token in any shared image. +- A VM of a vGPU class with a `license` gets its license token and license settings from the platform, in entitled tenants only, without a token in any shared image. - Usage records (`Workload`) show which tenant runs how many GPUs of which offer in which VM or worker, which an external billing system reads. Cozystack charges nothing and stores no prices. - Existing `gpus` entries keep working, with a documented path to class entries. @@ -209,7 +209,7 @@ A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain Capacity is not checked at admission in step 1. A started VM whose class has no free unit on a node it can use waits in `Scheduling`, its launcher pod `Pending` with `Unschedulable` and the VM shown as `ErrorUnschedulable`, as a VM short of CPU or memory does today, and starts when a unit frees up; the catalog's `free` and `maxFreePerNode` tell a tenant beforehand whether that is likely. Refusing the VMI instead would leave the VM to the per-item backoff of KubeVirt's VM controller, which grows to 1000 seconds, for a shortage that may last seconds. -A pool states no mode of its own: its entry names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a class entry that names a vGPU class on a Talos pool, before any machine is created. A pool on another worker OS takes a vGPU class only with a worker image that carries the vGPU guest driver, which is a requirement on future Kairos GPU pools; cozystack/community#115 defines none yet. Talos GPU pools on `name` entries keep working as today. +A pool states no mode of its own: its entry names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a class entry that names a vGPU class on a Talos pool, before any machine is created. A pool on another worker OS takes a vGPU class only with a worker image that carries the vGPU guest driver, which Kairos GPU pools have to meet. Talos GPU pools on `name` entries keep working as today. ### 5. Budgets @@ -235,9 +235,11 @@ The platform hands the license of a vGPU class to each VM that uses the class, a The GPU controller copies the hoster's Secret of each class with a `license` into every namespace labelled `namespace.cozystack.io/vgpu: "true"`, as a Secret named `gpuclass--license` and labelled `cozystack.io/gpuclass: `. A copy has two keys: `client_configuration_token.tok`, the token, and `gridd.conf`, which sets the class's `FeatureType`. The controller writes or refreshes a copy when a namespace gains the label, when a class gains or changes its `license`, and when the hoster's Secret changes. It deletes a namespace's copies when the label stops being `"true"`, and a class's copies when the class is deleted or drops its `license`. While the hoster's Secret is missing or has no `client_configuration_token.tok` key, the controller keeps the copies it has, writes no new ones, and sets a `LicenseSecretMissing` condition on the class. No copy is written into a namespace without the entitlement. -When the VMI webhook of section 4 resolves a vGPU class whose `license` is set, it adds a KubeVirt `secret` volume for that class's copy, with the volume label `GPULICENSE`, and a disk for it after the VMI's other disks and without a `bootOrder`, so the boot order of a VM does not change. KubeVirt gives the guest the Secret's keys as files on a disk with that label ([`schema.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/staging/src/kubevirt.io/api/core/v1/schema.go), `SecretVolumeSource`). The guest mounts the disk by its label and puts the token under `/etc/nvidia/ClientConfigToken/` and `gridd.conf` under `/etc/nvidia/`; a GPU guest image for VMs and a worker image with the vGPU guest driver do this at boot, and this disk, its label and these paths are the contract those images follow. VMs and pool workers get the volume the same way, since both start as VMIs with class entries, and neither the `vm-instance` nor the `kubernetes-nodes` chart changes for the license. The volume stays on a running VMI: KubeVirt v1.9.0 rewrites a VMI's volumes from its VM only for declarative hotplug volumes and, under the `Migration` volume update strategy, for the PVC volumes it migrates ([`hotplug.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/storage/hotplug/hotplug.go), [`volume-migration.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/volume-migration/volume-migration.go)), and a `secret` volume is neither. Until the copy exists, kubelet does not start the launcher pod, so the VM waits rather than boots unlicensed: the VM shows `Starting`, its launcher pod stays in `ContainerCreating` with a `FailedMount` event naming the missing Secret, and while the hoster's Secret is missing or lacks the token key the class shows `LicenseSecretMissing`. +When the VMI webhook of section 4 resolves a vGPU class whose `license` is set, it adds a KubeVirt `secret` volume for that class's copy, with the volume label `GPULICENSE`, and a disk for it after the VMI's other disks and without a `bootOrder`, so the boot order of a VM does not change. KubeVirt gives the guest the Secret's keys as files on a disk with that label ([`schema.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/staging/src/kubevirt.io/api/core/v1/schema.go), `SecretVolumeSource`). At every boot, before its license client starts, a Linux guest mounts the disk by its label, copies the token into a directory on a memory file system such as `/run`, and puts a `ClientConfigTokenPath` line naming that directory and the `FeatureType` line of the disk's `gridd.conf` into `/etc/nvidia/gridd.conf`. NVIDIA documents `ClientConfigTokenPath` for keeping the token outside the default `/etc/nvidia/ClientConfigToken/` and asks that the directory hold nothing but the token ([NLS user guide](https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html), section 5.1.3.2), so the guest copies the token instead of pointing at the disk, which also holds `gridd.conf`. The token never reaches a Linux guest's root disk, and a Linux guest restarted without the disk has none. A GPU guest image for VMs and a worker image with the vGPU guest driver do this, and this disk, its label and this handling of the token are the contract those images follow; `vgpu-vms` covers Windows guests, whose token folder is on the system disk. VMs and pool workers get the volume the same way, since both start as VMIs with class entries, and neither the `vm-instance` nor the `kubernetes-nodes` chart changes for the license. The volume stays on a running VMI: KubeVirt v1.9.0 rewrites a VMI's volumes from its VM only for declarative hotplug volumes and, under the `Migration` volume update strategy, for the PVC volumes it migrates ([`hotplug.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/storage/hotplug/hotplug.go), [`volume-migration.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/volume-migration/volume-migration.go)), and a `secret` volume is neither. Until the copy exists, kubelet does not start the launcher pod, so the VM waits rather than boots unlicensed: the VM shows `Starting`, its launcher pod stays in `ContainerCreating` with a `FailedMount` event naming the missing Secret, and while the hoster's Secret is missing or lacks the token key the class shows `LicenseSecretMissing`. -Rotation is a VMI restart. The hoster replaces the token in its Secret, the controller refreshes every copy, and each VM reads the new token at its next start. Deleting a namespace's copies stops new starts there from getting a token, and the VMI webhook refuses them anyway without the entitlement; a running guest keeps the token on its disk until it stops, so a hoster who must cut a tenant off at once also revokes the token on its license server. +Rotation is a VMI restart. A client configuration token names the license allocation that serves it, and NVIDIA's license system documents no way to revoke a token before the expiry chosen when it is generated, 12 years if none is chosen and never for a DLS token set not to expire ([NLS user guide](https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html), sections 4.4, 4.4.1 and 4.4.2). So the hoster rotates onto a new allocation or license server instance: it generates the new token there and replaces it in its Secret, the controller refreshes every copy, and each VM reads the new token at its next start. Once the VMs have restarted, the hoster disables leasing on the old allocation, which then serves no licenses (NLS user guide, section 4.6), and a guest still on the old token keeps its lease at most for the maximum lease duration set on that allocation (NLS user guide, section 4.7). + +Clearing a tenant's entitlement takes effect at each VM's next start. A running guest renews its lease while the allocation serves its token, and a client released on the license server can check a license out again while the token is served (NLS user guide, section 4.9.1), so to cut a tenant off at once the hoster or the parent's administrator also stops the tenant's vGPU VMs. A guest's root user can read the token and keep a copy, and only a rotation stops such a copy from working. One token serves every tenant of a class, and every entitled namespace gets a copy for every licensed class, so a token does not tell tenants apart and the platform cannot give one tenant a token of its own. ### 7. Usage records, catalog and licensing @@ -247,7 +249,7 @@ The catalog is the list of `GPUClass` objects with `displayName`, model, mode, p The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest entry, leaving out a field that is absent because it sets no limit. The remaining budget is the pool figure of section 5, so a tenant under a shared parent budget can see how many GPUs its siblings declare, as a number, without names. This is accepted: the alternative is a budget the tenant can only discover by a refused write, which is the failure this proposal removes. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. The existing `gpu` source keeps serving raw names while they are allowed. -vGPU needs a license, and passthrough needs one only for graphics. Compute (C-series) profiles are licensed through NVIDIA AI Enterprise, graphics (Q-series) profiles through NVIDIA vGPU software. The hoster provides the license server, the token and the guest drivers, and the platform delivers the token and the `FeatureType` of a vGPU class to each VM and worker of an entitled tenant (section 6). A passthrough class carries no `license`, so a licensed passthrough GPU for graphics still gets its token through a private image the hoster gives the tenants it licenses, not through the shared golden images. The class's `mode` tells a portal and a tenant which case applies. +vGPU needs a license, and passthrough needs one only for graphics. Compute (C-series) profiles are licensed through NVIDIA AI Enterprise, graphics (Q-series) profiles through NVIDIA vGPU software. The hoster provides the license server, the token and the guest drivers, and the platform delivers the token and the `FeatureType` of a vGPU class with a `license` to each VM and worker of an entitled tenant (section 6). A passthrough class carries no `license`, so a licensed passthrough GPU for graphics still gets its token through a private image the hoster gives the tenants it licenses, not through the shared golden images. The class's `mode` tells a portal and a tenant which case applies. ### 8. Migration of existing `gpus` @@ -324,7 +326,7 @@ A `dra` claim before its first start is the exception: it is `Bound` without a n A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: wherever a `runStrategy` creates a next VMI (the table in Failure and edge cases), the final VMI is gone first, and under `Manual` a start request first stops the final VMI and then starts a new one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. -A claim keeps its node while the `Node` exists, serves the class, has not lost capacity, has no pending layout change (section 2) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer serves the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached and moves, and where the VM's `runStrategy` creates a new instance after the eviction, as `vgpu-vms` gives it, that instance starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules below. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set. +A claim keeps its node while the `Node` exists, serves the class, has not lost capacity, has no pending layout change (section 2) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer serves the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached and moves, and where the VM's `runStrategy` creates a new instance after the eviction (the `runStrategy` table), that instance starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules below. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set. A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity for a class when its allocatable count of the class's resource stays below the units held there for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no claim ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). Then: @@ -424,11 +426,11 @@ Step 2: The change is additive. Existing `gpus` entries render as before, and the existing variants work as today until a layout is set. -Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, with these exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only GPU limits were GPU keys lose them, and on a cluster that ran step 2 with `forbidden`, tenants without the vGPU entitlement can use raw vGPU names again. A worker pool that takes GPUs only through classes, as a future Kairos GPU pool would, loses its GPUs. A vGPU VM or worker that restarts after step 1 rolls back gets no `GPULICENSE` disk, so the hoster first puts the license token into private images it gives entitled tenants only, not into the shared golden images ([0018](./decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md)). +Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, with these exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only GPU limits were GPU keys lose them, and on a cluster that ran step 2 with `forbidden`, tenants without the vGPU entitlement can use raw vGPU names again. A worker pool that takes GPUs only through classes, such as a Kairos GPU pool, loses its GPUs. A vGPU VM or worker that restarts after step 1 rolls back gets no `GPULICENSE` disk, so the hoster first puts the license token into private images it gives entitled tenants only, not into the shared golden images ([0018](./decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md)). Such a private image does not run the reference boot unit or script of the guest contract, since the boot unit points the license client at an empty token directory and the script empties the token folder at every boot. An existing VM or worker whose image follows the guest contract of section 6 keeps no token across boots, so it stays unlicensed after that restart until it is recreated from such a private image, or its tenant disables the reference boot unit or script, removes the `ClientConfigTokenPath` line from `gridd.conf` on Linux, and licenses the guest itself. Step 2 rolls back first. Raw names are allowed again, since raw keys are refused while raw names are forbidden. `claim` entries of VMs and `claimSelector` entries of pools become class entries, and those VMs restart and those pools roll. The `GPUClaim` objects are deleted, and the step 2 components go. -Step 1 rolls back next. The hoster removes the node layout and chooses the variant in a maintenance window. Class entries become raw `name` entries with the class's resource name, and GPU keys become raw keys on those names where the tenant wants a limit, and the remaining GPU keys are deleted. After the class entries have become raw names, the hoster clears the `gpu.vgpu` flags before the components go, so the controller deletes the license copies in tenant namespaces; if the controller is already gone, the hoster deletes them by the `cozystack.io/gpuclass` label; guests that keep running keep the token on their disk until they stop. Last, the components, the `GPUClass` CRD, the `gpu.vgpu` flag with its namespace label and the `cozystack.io/gpu-*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. +Step 1 rolls back next. The hoster removes the node layout and chooses the variant in a maintenance window. Class entries become raw `name` entries with the class's resource name, and GPU keys become raw keys on those names where the tenant wants a limit, and the remaining GPU keys are deleted. After the class entries have become raw names, the hoster clears the `gpu.vgpu` flags before the components go, so the controller deletes the license copies in tenant namespaces; if the controller is already gone, the hoster deletes them by the `cozystack.io/gpuclass` label; guests that keep running keep their lease while the allocation serves their token. Last, the components, the `GPUClass` CRD, the `gpu.vgpu` flag with its namespace label and the `cozystack.io/gpu-*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. Three rules hold throughout: @@ -464,14 +466,14 @@ Each case gives what is seen and what to do. Some cases depend on whether KubeVirt creates a next VMI for the VM. In KubeVirt v1.9.0 that depends on the `runStrategy` and on how the VMI ended ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go), [`vmi/lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go), [`rest/lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/rest/lifecycle.go); line numbers at that tag): -| `runStrategy` | VMI failed before it ever ran | VMI failed after it ran | Controller deleted an unscheduled VMI | -| --- | --- | --- | --- | -| `Always` | next VMI after the start-failure backoff (`vm.go` L987, L1025, L1580) | next VMI, no backoff (`vm.go` L987, L1539) | next VMI after the start-failure backoff (`vm.go` L1025, L1580; the VMI goes `Failed` first, `vmi/lifecycle.go` L447, L456) | -| `RerunOnFailure` | next VMI after the start-failure backoff (`vm.go` L1050, L1074, L1580) | next VMI, no backoff (`vm.go` L1050, L1539) | none until the tenant starts the VM (`vm.go` L1050, L1070; `rest/lifecycle.go` L120) | -| `Once` | none (`vm.go` L1141) | none (`vm.go` L1141) | next VMI (`vm.go` L1141) | -| `Manual` | none until the tenant starts the VM (`vm.go` L1104, `rest/lifecycle.go` L120) | the same | the same | +| `runStrategy` | VMI failed before it ever ran | VMI failed after it ran | VMI succeeded after it ran | Controller deleted an unscheduled VMI | +| --- | --- | --- | --- | --- | +| `Always` | next VMI after the start-failure backoff (`vm.go` L987, L1025, L1580) | next VMI, no backoff (`vm.go` L987, L1539) | next VMI, no backoff (`vm.go` L987, L1539) | next VMI after the start-failure backoff (`vm.go` L1025, L1580; the VMI goes `Failed` first, `vmi/lifecycle.go` L447, L456) | +| `RerunOnFailure` | next VMI after the start-failure backoff (`vm.go` L1050, L1074, L1580) | next VMI, no backoff (`vm.go` L1050, L1539) | none until the tenant starts the VM (`vm.go` L1050, L1059, L1070; `rest/lifecycle.go` L120) | none until the tenant starts the VM (`vm.go` L1050, L1070; `rest/lifecycle.go` L120) | +| `Once` | none (`vm.go` L1141) | none (`vm.go` L1141) | none (`vm.go` L1141) | next VMI (`vm.go` L1141) | +| `Manual` | none until the tenant starts the VM (`vm.go` L1104, `rest/lifecycle.go` L120) | the same | the same | the same | -The start-failure backoff grows with each failure that happens before a VMI runs, up to five minutes, and resets once a VMI runs. During the backoff the VM shows `CrashLoopBackOff`, and so does a `Once` VM whose VMI failed before it ran, although it does not retry (`vm.go` L2526-L2546, L2572-L2575). A VMI that a drain evicts ends on its own, and [`vgpu-vms`](../vgpu-vms/README.md) gives what each `runStrategy` does then. +The start-failure backoff grows with each failure that happens before a VMI runs, up to five minutes, and resets once a VMI runs. During the backoff the VM shows `CrashLoopBackOff`, and so does a `Once` VM whose VMI failed before it ran, although it does not retry (`vm.go` L2526-L2546, L2572-L2575). A drain evicts a GPU VM instead of migrating it: KubeVirt does not live-migrate a whole GPU in passthrough, KubeVirt v1.9 live-migrates one vGPU per VM only as an alpha feature behind the `VGPULiveMigration` gate and only for a resource name listed under `mediatedDevices` in `permittedHostDevices` ([`virt-handler/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-handler/vm.go), `isMdevGPU`), an SR-IOV vGPU is a VF under `pciHostDevices` and never migrates, and Cozystack does not enable the gate. An evicted VMI whose guest shuts down within its grace period ends `Succeeded`, one destroyed after it ends `Failed` while the domain has ACPI, the default, and the matching column applies. Step 1: @@ -481,12 +483,12 @@ Step 1: - A class shows `free: 0` while its nodes have GPUs to spare → the nodes carry a `NoSchedule` or `NoExecute` taint the class does not tolerate; add it to the class's `tolerations`, or remove the taint. - A class has a `ResourceNameConflict` condition → another class has its resource name; delete one of them or point it at its own name. - A class is deleted while VMs name it → running VMs keep running, and their next start is refused naming the class; the tenant moves them to another class. -- A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. Where its `runStrategy` creates a new instance after the eviction, as `vgpu-vms` gives it, it starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. +- A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. Where its `runStrategy` creates a new instance after the eviction (the `runStrategy` table), it starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. - A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. Where its `runStrategy` creates a next VMI after a VMI that ran fails, that VMI starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. - A GPU fails under a running VM on a class entry → the VM keeps running. Kubelet stops listing the device as allocatable, so the class's `free` drops, and when the VM starts again, its next instance gets a unit on a node that still has one, or waits as `ErrorUnschedulable` while there is none. - A layout change does not apply → the node carries the `cozystack.io/gpu-layout-pending` label; cordon and drain it. A GPU VM on the node holds the change until it stops or is evicted. - A vGPU VM stays `Starting` with its launcher pod in `ContainerCreating` and a `FailedMount` event → the namespace has no license copy of the class yet; the class shows `LicenseSecretMissing` when the hoster's Secret is gone or has no `client_configuration_token.tok` key. Create or fix the hoster's Secret. -- A vGPU VM runs unlicensed → its class has no `license`, or the guest image does not install the files from the `GPULICENSE` disk; set the class's `license`, or fix the image. +- A vGPU VM runs unlicensed → its class has no `license`, or the guest image does not follow the `GPULICENSE` contract of section 6; set the class's `license` unless the class is meant for the tenant's own license, or fix the image. - A Talos pool with a vGPU class entry is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. Step 2: @@ -538,12 +540,12 @@ Step 1: 1. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. It needs no tenant API and can land first. 2. `GPUClass` with its status controller and metrics, class entries in `vm-instance` and `kubernetes-nodes` with the VMI webhook and the launcher pod webhook for tolerations, the `gpu.vgpu` Tenant flag and license delivery, GPU keys with the interim check, and the `gpuclass` option source, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRD ships and the charts refuse class entries. Before tenants use vGPU classes, the `gpu.vgpu` flag is set from the root tenant down. It works on any cluster where each offer it publishes has a resource name of its own, with or without item 1. -3. The consumers, on top of item 2 alone: `vgpu-vms` adds GPU guest images, and Kairos GPU pools, once cozystack/community#115 or a successor adds them, take class entries. Neither waits for item 4 or for step 2. +3. The consumers, on top of item 2 alone: `vgpu-vms` adds the licensing guide and reference scripts for vGPU VMs, and Kairos GPU pools take class entries and follow the guest contract of section 6. Neither waits for item 4 or for step 2. 4. GPU keys through #48, and the interim check deleted, in the release where #48's gate is on by default. Step 2: -5. Holding: `GPUClaim`, the binder and its metrics, the scheduling gate in the launcher pod webhook, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes and for future Kairos GPU pools, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. +5. Holding: `GPUClaim`, the binder and its metrics, the scheduling gate in the launcher pod webhook, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes and for Kairos GPU pools, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. Then: diff --git a/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md b/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md index 12446814..04261448 100644 --- a/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md +++ b/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md @@ -16,7 +16,7 @@ Golden images in `cozy-public` are open to every tenant: any tenant can clone on ## Decision -`GPUClass` carries a `license`, the name of the hoster's token Secret in `cozy-system` and the `FeatureType`, `1` by default or `4`, on vGPU classes only. NVIDIA's [client licensing guide](https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/configuring-nls-licensed-client.html) documents `1` for a VM with a vGPU, since "NVIDIA vGPU software automatically selects the correct type of license based on the vGPU type", and its `gridd.conf` template lists `4` as "NVIDIA vGPU for Compute"; `4` has been observed working on a C-series profile, and the field lets a hoster set it per class. The GPU controller copies each such Secret into every namespace whose `namespace.cozystack.io/vgpu` label is `"true"`, adding a `gridd.conf` with the class's `FeatureType`, and deletes the copies when the label, the class or its `license` goes. At VMI creation the VMI webhook adds a KubeVirt `secret` volume for the copy of the VMI's vGPU class, with the volume label `GPULICENSE`, and the guest installs the two files from that disk. VMs and pool workers work the same way, and neither chart changes for the license. Rotation is a VMI restart. +`GPUClass` carries a `license`, the name of the hoster's token Secret in `cozy-system` and the `FeatureType`, `1` by default or `4`, on vGPU classes only. NVIDIA's [client licensing guide](https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/configuring-nls-licensed-client.html) documents `1` for a VM with a vGPU, since "NVIDIA vGPU software automatically selects the correct type of license based on the vGPU type", and its `gridd.conf` template lists `4` as "NVIDIA vGPU for Compute"; `4` has been observed working on a C-series profile, and the field lets a hoster set it per class. The GPU controller copies each such Secret into every namespace whose `namespace.cozystack.io/vgpu` label is `"true"`, adding a `gridd.conf` with the class's `FeatureType`, and deletes the copies when the label, the class or its `license` goes. At VMI creation the VMI webhook adds a KubeVirt `secret` volume for the copy of the VMI's vGPU class, with the volume label `GPULICENSE`, and at every boot a Linux guest copies the token from that disk into a directory on a memory file system and puts a `ClientConfigTokenPath` line naming that directory and the `FeatureType` into its `gridd.conf`, so the token stays off its root disk; `vgpu-vms` covers Windows guests, whose token folder is on the system disk. VMs and pool workers work the same way, and neither chart changes for the license. Rotation is a VMI restart. ## Why not the alternatives @@ -27,9 +27,11 @@ Golden images in `cozy-public` are open to every tenant: any tenant can clone on ## Consequences - Only the guests of entitled tenants get the token, and they get it as a disk, so a tenant's root user inside its VM can read it; that is what licensing a guest needs. -- A guest image or a worker image with the vGPU guest driver mounts the `GPULICENSE` disk at boot; an image that does not boots unlicensed. +- A guest image or a worker image with the vGPU guest driver mounts the `GPULICENSE` disk at boot; an image that does not ignores the disk and is licensed the way it was, or not at all. - The GPU controller writes Secrets in tenant namespaces, through the cluster-wide Secret write it already has. -- A running guest keeps its token until it stops, so revoking a tenant at once also needs the license server. +- A token names the allocation that serves it, and NVIDIA's license system documents no way to revoke one before its expiry, which defaults to 12 years. Rotation therefore moves to a new allocation or license server instance and disables leasing on the old one, after which a remaining guest keeps its lease at most for that allocation's maximum lease duration ([NLS user guide](https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html), sections 4.4, 4.6 and 4.7). +- Clearing the entitlement takes effect at each VM's next start. A running guest renews its lease while the allocation serves its token, and a released client can check a license out again while the token is served (NLS user guide, section 4.9.1), so an immediate cut-off also stops the tenant's vGPU VMs. +- Every entitled namespace gets a copy for every licensed class, so a token does not tell tenants apart, and the platform cannot give one tenant a token of its own. - A licensed passthrough GPU for graphics still gets its token from a private image the hoster gives the tenants it licenses, not from the shared golden images, since a passthrough class carries no `license`. ## Revisit if