diff --git a/design-proposals/gpudevice/README.md b/design-proposals/gpudevice/README.md new file mode 100644 index 00000000..a8ae5083 --- /dev/null +++ b/design-proposals/gpudevice/README.md @@ -0,0 +1,584 @@ +# GPUs as a tenant resource + +- **Title:** `GPU classes, per-class tenant budgets and the host layer under them, then GPU holds for stopped VMs` +- **Author(s):** `@lexfrei` + +## Overview + +A tenant who wants a GPU today writes a device-plugin resource name such as `nvidia.com/AD102GL_L40S` into `gpus` of a VM or a worker pool and hopes one is free. The `gpu` option source shows how many of each name the nodes advertise and on which nodes, but nothing says which model, mode or profile a name stands for, nothing holds a GPU while the VM is off, and an order that does not fit is accepted and then sits in `Pending`. A hoster selling GPUs needs the opposite: a catalog of GPU models and modes, budgets per offer, the same handling for VMs and managed Kubernetes worker pools, a usage record of who used what, which the hoster's external billing system reads, and, for some offers, GPUs a tenant keeps while its VM is stopped. + +The proposal comes in two steps. Step 1 has no tenant object. It adds a host layer that ends at one resource name per offer, `GPUClass` as the hoster's catalog in the existing `cozystack.io` group, `gpus` entries in `vm-instance` and `kubernetes-nodes` that name a class instead of a resource name, budgets per class that the tenant quota gate of [#48](https://github.com/cozystack/community/pull/48) checks when an application is written, and usage taken from the `Workload` records Cozystack already keeps for virt-launcher pods. Step 2 adds holding: `GPUClaim`, a tenant's GPU kept on a node while its VM is stopped, with the binder, moves, lost capacity and the launcher pod gate that a hold needs. The fields and phases of step 2 enter the API only in step 2. + +NVIDIA's vGPU stack changes quickly. Within the last few releases the driver branches, the way vGPUs are exposed (mediated devices or SR-IOV virtual functions), the KubeVirt device plugins and the licensing rules all changed. This proposal describes that stack as it was in October 2026. Expect it to be updated as NVIDIA and KubeVirt move, with a decision record under it for each change of course, and check the versions named here against your own before relying on them. + +## Scope and related proposals + +- [vGPU licensing for virtual machines](../vgpu-vms/README.md) (`vgpu-vms`) builds on the license delivery of section 6: who the licensee is, how a guest image applies the license, how a guest reaches the license server, how license use maps to tenants, and rotation and revocation. The link resolves once it merges. +- [Kairos tenant worker pools](https://github.com/cozystack/community/pull/115) (cozystack/community#115) is where worker pools on Kairos are proposed. A Kairos GPU pool takes GPUs through class entries, and through `claimSelector` once step 2 ships, under the rules of section 4 for pools on another worker OS, and its worker image follows the `GPULICENSE` contract of section 6. +- [Tenant quotas as reservation limits](https://github.com/cozystack/community/pull/48) makes declared sizes the quota authority and refuses an application that does not fit its pool when it is written. GPU budgets are ordinary keys of that gate (section 5). Until it ships, an interim check does the same for GPU keys only. +- [Node fencing](../node-fencing/README.md), proposed alongside this one, adds an optional package that fences a dead node through its BMC and taints it `out-of-service`, so KubeVirt fails its VMIs and step 2 rebinds their `Lost` claims. The link resolves once it merges. +- [IP addresses as a first-class resource](https://github.com/cozystack/community/pull/35) solves a similar problem for addresses with a class, a tenant claim and a backend contract. +- Host-side discovery of vGPU VFs grouped by profile is [NVIDIA/kubevirt-gpu-device-plugin#192](https://github.com/NVIDIA/kubevirt-gpu-device-plugin/pull/192), still open. Section 2 ships it downstream in the host layer, behind a boundary the rest of the design does not cross. + +## Decisions + +- [0018. A vGPU license reaches each VM as a Secret volume, copied only into entitled namespaces](./decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md) +- [0017. The vGPU entitlement is a Tenant flag stamped on the namespace, not a budget key](./decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md) +- [0016. Holding a GPU for a stopped VM is step 2, and stays GPU-specific until DRA can carry it](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md) +- [0015. Node allocatable is the GPU inventory, and there is no `GPUNode`](./decisions/0015-node-allocatable-is-the-gpu-inventory.md) +- [0014. In step 1 a `gpus` entry names a class, budgets count declared entries, and usage is the virt-launcher pod's `Workload`](./decisions/0014-step-1-names-a-class-and-holds-nothing.md) +- [0013. The GPU kinds live in `cozystack.io`, with no group of their own](./decisions/0013-the-gpu-kinds-live-in-cozystack-io.md) +- [0012. The VMI webhook only checks and records claims, and the controller places them while the launcher pod waits at a scheduling gate](./decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md) +- [0011. A `GPUClaim` owns its usage record, and no monitor or reserved label guards it](./decisions/0011-a-gpuclaim-owns-its-usage-record.md) +- [0010. `GPUClass` carries the tolerations that `free`, binding and the launcher pod read](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md) +- [0009. `GPUClass.mode` never changes, and a pool names its class](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md) +- [0008. The GPU controller gets its node-label right with the host layer, not before](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md) +- [0007. Whoever can write a VM uses the GPUs of its class entries, while ordered claims stay with tenant admins](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md) +- [0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md) +- [0005. The taint rule skips exactly the three taints that mirror a cordon or readiness](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md), superseded by 0010 +- [0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts](./decisions/0004-gpunode-health-is-kubelet-allocatability.md), superseded by 0015 +- [0002. GPU classes serve VMs only where gpu-operator serves GPUs to VMs, while claim finalizers and usage records always run](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md) +- [0001. A class whose resource name KubeVirt does not permit is refused at VMI admission](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md) + +## Context + +### What exists + +The `cozystack.gpu-operator` package has three variants, selected by `bundles.iaas.gpuOperatorVariant` (Cozystack's [GPU guide](https://cozystack.io/docs/virtualization/vgpu/)). `default` binds whole GPUs to `vfio-pci` for VMs, `vgpu` runs the NVIDIA vGPU host driver, and `container` serves GPUs to pods. For the two VM variants the platform writes the `HostDevices` feature gate and, when it has entries, a `permittedHostDevices` table into the KubeVirt CR (`packages/core/platform/templates/bundles/iaas.yaml`). Passthrough GPUs are advertised by NVIDIA's sandbox device plugin. + +How NVIDIA's vGPU host driver exposes vGPUs depends on the GPU, the host OS and the driver branch. Older setups use mediated devices (mdev). SR-IOV GPUs on current releases, such as Ubuntu 24.04 with the vGPU 17 or newer branch that Cozystack's guide uses, get VFs, and each VF gets a profile when `current_vgpu_type` is written in its sysfs. KubeVirt v1.9, which Cozystack bundles, advertises such VFs through virt-handler's own device plugin ([kubevirt/kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890)), matched by PCI ID. Observed on L40S with SR-IOV vGPU: the VF shows `10de:26b9` in the guest, the PF's ID, and `permittedHostDevices` selects VFs by `10DE:26B9`. On GPU Operator nodes nothing in Cozystack writes VF profiles; the `cozystack-nvidia-vgpu-restore` boot unit from cozystack/ansible-cozystack#70 covers only hosts with a host-installed driver. + +Both consumers take GPUs the same way: `vm-instance` and `kubernetes-nodes` read `gpus` as a list of `{name: }`, one entry per GPU. The deprecated `gn1.*` instance types carry GPU entries of their own in KubeVirt's shape, `{name: gpu1, deviceName: nvidia.com/A400}`, and neither default `permittedHostDevices` table lists that name. Tenant quotas become a namespace `ResourceQuota`, which refuses a GPU only when virt-launcher's pod is created, and the usage records (`Workload`) show a GPU only while a VM runs. + +### The problem + +- A tenant sees resource names, not offers. The dashboard's `gpu` option source lists the names KubeVirt permits and how many of each are available on which nodes; they say nothing about mode, profile or price. +- A GPU cannot be held. When a VM stops, its GPU goes back to the pool and the next start can fail. +- An order that does not fit is accepted, and the tenant learns it from pod events. +- Nobody can tell which tenant uses a physical GPU without reading pods on the node. +- In the `vgpu` variant a node labelled for passthrough can serve whole GPUs of another model, but a model whose whole GPU and VFs share a PCI ID, such as L40S, is offered either whole or as vGPU, not both, because two device plugins would both take that ID. The limit is per GPU model, not per cluster. +- VF profiles on GPU Operator nodes are written by hand after every reboot. + +Step 1 answers the first, fifth and sixth problems, the third for budgets and the fourth per tenant and class. Step 2 answers the second, and the third for capacity, since an order is then checked against free units. + +## Goals + +Step 1: + +- One cluster can have passthrough and vGPU nodes of the same GPU model, with vGPU profiles applied declaratively and kept across reboots, and every offer advertised under a resource name of its own. +- A hoster publishes a catalog of GPU classes, each with a model, a mode (passthrough or vGPU) and, for vGPU, a profile. Tenants read the catalog with free capacity and their remaining budget. +- VMs and worker pools name a class, not a resource name, and a tenant's budget per class and in total is checked when the VM or pool is written, with a message naming the class and the limit. +- A VM of a vGPU class with a `license` gets its license token and license settings from the platform, in entitled tenants only, without a token in any shared image. +- Usage records (`Workload`) show which tenant runs how many GPUs of which offer in which VM or worker, which an external billing system reads. Cozystack charges nothing and stores no prices. +- Existing `gpus` entries keep working, with a documented path to class entries. + +Step 2: + +- A tenant holds a number of GPUs of a class within its budget, attached or not, until it releases them, and a restarted VM gets its ordered claims back. The hold is a hard guarantee once `gpu.rawResourceNames` is `forbidden`, which needs every GPU consumer on class or claim entries; until then it is not guaranteed. +- The catalog can later describe MIG, other vendors and DRA-backed devices without changing the tenant API. + +### Non-goals + +- Prices and charging. They live in an external billing system, keyed by class name. +- The vGPU guest driver and the license server. The hoster provides them and its license token, which the platform only delivers (section 6). +- The catalog UI. This proposal provides the data a portal reads. +- GPUs for pods inside tenant clusters (gpu-operator in a tenant cluster, HAMi). +- Time-slicing one GPU between tenants, and live migration of GPU VMs. +- A per-device inventory naming the physical GPU or VF behind each VM ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). + +## Design + +Sections 1 to 8 are step 1. Section 9 is step 2 and is the only place that adds fields, kinds or phases for holding. + +### 1. API group and names + +The GPU kinds join the existing `cozystack.io` group, which already holds the hoster-side `SchedulingClass` and `Package` and the tenant-namespace `Workload` and `WorkloadMonitor`; there is no `gpu.cozystack.io` group ([0013](./decisions/0013-the-gpu-kinds-live-in-cozystack-io.md)). Step 1 adds one kind, `GPUClass`; step 2 adds `GPUClaim`. + +Every other name follows the group: + +| Name | Step | Use | +| --- | --- | --- | +| `.gpuclass.cozystack.io/devices` | 1 | budget of one class in `resourceQuotas` (section 5) | +| `gpuclass.cozystack.io/devices` | 1 | budget over every class | +| `cozystack.io/gpu-reference` | 1 | label on a VMI template with class or claim entries, which KubeVirt copies to the VMI and its pods | +| `cozystack.io/gpus` | 1 | annotation on that template listing the entries | +| `cozystack.io/gpu--` | 1 | host-layer node label carrying a model's mode or profile (section 2) | +| `cozystack.io/gpu-layout-pending` | 1 | node label marking a pending layout change (section 2) | +| `gpuclass--license` | 1 | name of a class's license copy in an entitled namespace (section 6) | +| `cozystack.io/gpuclass` | 1 | label on a license copy naming its class | +| `GPULICENSE` | 1 | volume label of the license disk a VMI of a licensed vGPU class gets (section 6) | +| `namespace.cozystack.io/vgpu` | 1 | namespace label carrying the vGPU entitlement, like the other module labels (section 6) | +| `cozystack.io/gpuclass-protection` | 2 | finalizer on a class in use (section 9) | +| `cozystack.io/gpuclaim-protection` | 2 | finalizer on a claim a VMI uses | +| `cozystack.io/gpuclaim-template` | 2 | label marking a claim the controller made for a class entry | +| `cozystack.io/gpuclaims` | 2 | scheduling gate on a launcher pod | + +The quota keys follow DRA's `.deviceclass.resource.k8s.io/devices`, so a class key never collides with a resource name a device plugin advertises, and a key without a class prefix counts every class. + +### 2. Host layer + +The host layer is everything between the GPUs on a node and the resource names classes point to: the host driver, VF profiles, node modes, and the device plugin that advertises GPUs to KubeVirt. It ends at one resource name per offer: a whole L40S in passthrough and an L40S-24C vGPU are two names, and one name never stands for two offers. That is its whole contract with the API above it. `GPUClass`, quotas, the GPU webhooks and usage records use no plugin-specific name, discovery format or configuration. A different plugin, or a different version of the one chosen here, changes only this layer and the `resourceName` of affected classes. Tenants keep their class entries; raw `name` entries and raw quota keys on a renamed resource move to the new name in the same maintenance window, or their consumers move to class entries first. It ships first and alone, with no tenant API. + +The host layer does not depend on the host OS: it needs the vGPU host driver, `vfio-pci` and the device plugin, wherever they run. Which OSes Cozystack supports as hosts of the management cluster is decided outside this proposal. Today vGPU hosts run Ubuntu with `bundles.system.variant: isp-full-generic`, where the vGPU host driver and the patched plugin run. Cozystack's GPU guide does not recommend Talos for vGPU, so this layer does not cover Talos hosts; they keep passthrough through the `default` variant as today. NVIDIA AI Enterprise lists vGPU for Compute on Linux with KVM as supported only through specific hypervisor vendors and, on Ubuntu and RHEL with KVM, for single-node hosts only ([support matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/support/support-matrix-8/8.2.html)), so support for such a host is a matter between the hoster and NVIDIA. + +Until the layer ships, a cluster keeps the modes its variant gives it today, one per model for models whose whole GPU and VFs share a PCI ID, and vGPU classes keep the experimental status the GPU guide gives the `vgpu` variant: the hoster writes profiles and node labels by hand. On clusters without a layout, which includes the `default` variant and Talos hosts, node labels stay hand-written. Where one resource name still covers two offers, as with KubeVirt's built-in plugin, which gives every vGPU profile of a model one name, the hoster can publish only one class for that name (section 3). + +The device plugin is the decision reviewers should look at first. Cozystack ships NVIDIA's kubevirt-gpu-device-plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream, as the sandbox device plugin of the `vgpu` variant, until NVIDIA/kubevirt-gpu-device-plugin#192 merges upstream or upstream takes another approach. NVIDIA/kubevirt-gpu-device-plugin#192 adds discovery of vGPU VFs on the vendor-specific VFIO framework and advertises them with one resource name per profile, next to the passthrough names the plugin already advertises for GPUs bound to `vfio-pci`. One plugin then serves both modes of every model under distinct names, so the per-model limit of the problem list goes away. The branch is validated on hardware. Its upstream review has had no maintainer response since 2026-07-10, which is why it ships as a downstream patch rather than waiting. It reads profile names from sysfs `creatable_vgpu_types` first and falls back to NVML on a card whose VFs all carry a profile, so the platform runs the plugin as its own DaemonSet with the host driver's `libnvidia-ml.so.1` and the `/dev/nvidia*` nodes mounted, privileged; the GPU Operator's DaemonSet has no field for those mounts. The patch leaves when NVIDIA/kubevirt-gpu-device-plugin#192 merges upstream, in this or another form, or when the NVIDIA DRA driver serves vGPU VFs to KubeVirt (kubernetes-sigs/dra-driver-nvidia-gpu#1402); a merged NVIDIA/kubevirt-gpu-device-plugin#192 changes only this layer, while a move to DRA adds new `dra` classes (section 10). KubeVirt's built-in plugin was the other candidate (Alternatives). + +The layer adds a node layout to the `vgpu` variant, in platform values. For each node selector it names a mode and, for vGPU, the profile of each GPU model. Nodes outside every selector are passthrough, through the `defaultWorkload: vm-passthrough` the layout renders. With a layout: + +- The platform runs the patched plugin and renders the KubeVirt `permittedHostDevices` entries for the names it advertises, as entries with an external provider. Other vendors' entries, such as an FPGA, stay as written. +- A node agent writes the VF profiles at boot and when the layout changes. The controller sets each node's workload label and one label per GPU model, `cozystack.io/gpu--` (for L40S `cozystack.io/gpu-10de-26b9`), to the profile or to `passthrough`. +- The plugin does not pick up VFs that gain a profile after it started, so the node agent restarts the plugin pod on its own node after it has written profiles there, for example after a reboot. Running VMs are not affected. + +The controller gets the right to patch those node labels, restricted by a policy to the GPU workload label and `cozystack.io/gpu-*`, with this layer and not before ([0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)). + +The layout and mdev GPUs coexist: GPUs on the mdev path keep today's configuration through `vgpu-device-manager`, and the agent writes profiles only on nodes that use VFs. New deployments of SR-IOV GPUs on current host releases use VFs. + +A layout change on a node, of mode or of profile, waits until the node has no GPU VMs. The controller labels the node `cozystack.io/gpu-layout-pending`, and running VMs keep their GPUs until the hoster cordons and drains the node; the cordon is what keeps new GPU VMs off it, since nothing in step 1 places a launcher pod. A GPU VM created before Cozystack switched to `LiveMigrateIfPossible` needs one restart before the drain can evict it. A node whose hand-written profiles already match the layout is not a change. A `vgpu` cluster adopts the layout with vGPU selectors that cover its existing vGPU nodes, and labels those nodes `vm-vgpu` before setting it, so the layout's passthrough default does not change them in between. Its vGPU classes, raw `name` entries and raw quota keys move from the hand-written VF names to the plugin's per-profile names in one maintenance window, or the raw-name consumers move to class entries first. KubeVirt's built-in plugin and the patched plugin report different device IDs for the same VF, and kubelet would otherwise hand out VFs in use, so affected nodes are cordoned and free of GPU VMs until the patched plugin has registered there. + +### 3. GPU classes + +```yaml +# Hoster. One per offer. Cluster-scoped, readable by every authenticated user. +apiVersion: cozystack.io/v1alpha1 +kind: GPUClass +metadata: + name: l40s-24c +spec: + displayName: "NVIDIA L40S, 24 GB vGPU (compute)" + vendor: nvidia + model: L40S + mode: vgpu # passthrough | vgpu + profile: L40S-24C # vgpu only + license: # vgpu only; can change + secretName: nvidia-license-token # Secret in cozy-system + featureType: 4 # 1 by default + memory: 24Gi + maxCount: 2 # vGPUs of this class per VM + tolerations: # taints of this offer's nodes that its VMs tolerate + - key: dedicated + value: gpu + effect: NoSchedule + source: + devicePlugin: + resourceName: nvidia.com/NVIDIA_L40S-24C +status: + free: 12 + maxFreePerNode: 4 # most free units on one node +``` + +A class maps an offer to a backend through a `source` union. The first release has one source, `devicePlugin`; `dra` is reserved for section 10. A class is one resource name, and two classes cannot have the same one: a class whose `resourceName` another class already has gets a `ResourceNameConflict` condition and is not offered, so a budget on one class cannot be sidestepped through another, and a usage record's resource name leads to exactly one class. The nodes of a class are the nodes that advertise its resource name. Class names are DNS labels, since they become part of quota keys. + +`vendor`, `model`, `mode`, `profile`, `memory` and the kind of `source` never change, so a class name stands for one offer for the class's lifetime, and whatever names a class, a VM, a worker pool or a billing system, knows the offer from the name ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A different offer is a new class, and a hoster does not reuse a deleted class's name for another offer, since tenants' budgets and billing records are keyed by it. `displayName`, `maxCount`, `resourceName` and `tolerations` can change: `resourceName` for changes in the host layer, `tolerations` when the hoster retaints its GPU nodes. From step 2 on, `maxCount` changes only while the class is not in use (section 9). A change applies to VMs started after it; a running VM keeps the resource name it started with, and its launcher pod the tolerations. Deleting a class refuses the next start of every VM that names it and leaves running VMs alone. + +How many GPUs of a class one VM may take is the hoster's `maxCount` on the class. For vGPU classes it follows NVIDIA's multi-vGPU matrix for the board and the profile series. Compute (C-series) profiles follow the [NVIDIA AI Enterprise matrix](https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/multi-vgpu-support.html), which allows several vGPU for Compute devices per VM on L40S under the Linux with KVM hypervisors it supports (section 2), so the `l40s-24c` class above sets `maxCount: 2`. Graphics (Q-series) profiles follow the vGPU 20 release notes for Linux with KVM, which allow several Q-series vGPUs per VM on L40S and offer no C-series profiles. For passthrough classes it is at most the GPUs per node. NVIDIA requires all vGPUs in one VM to share board type and series. Cozystack keeps it simpler and stricter: a VM takes vGPUs of one class only, although NVIDIA would also allow, say, A40-48Q next to A40-16Q. `maxCount` defaults to 1 for vGPU classes; on a passthrough class, unset means no limit beyond the GPUs of one node. + +`tolerations` lists the taints of the class's nodes that its VMs tolerate, such as a taint that reserves GPU nodes for GPU workloads, in the shape of a pod's with the operators `Exists` and `Equal` only. The launcher pod of a VM that names the class gets them (section 4), and `free` reads them ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). + +`license` is how a VM of a vGPU class gets licensed (section 6). `secretName` names a Secret in `cozy-system` whose key `client_configuration_token.tok` holds the client configuration token the hoster downloads from its NVIDIA license server. `featureType` is the `FeatureType` the guest's license client reads from `gridd.conf`, `1` by default. NVIDIA's [client licensing guide](https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/configuring-nls-licensed-client.html) documents `1` for a VM with a vGPU, since "NVIDIA vGPU software automatically selects the correct type of license based on the vGPU type", and its `gridd.conf` template lists `4` as "NVIDIA vGPU for Compute"; `4` has been observed working on a C-series profile. The CRD admits `license` only on a class whose `mode` is `vgpu`, requires `secretName` as a DNS subdomain name, and takes `featureType` only as `1` or `4`. `license` can change, and a change reaches each VM at its next start. + +`status.free` is the class's allocatable count summed over its nodes that are `Ready`, not cordoned, and have no `NoSchedule` or `NoExecute` taint the class does not tolerate, minus what pods on those nodes request of the resource name, and no node counts below zero; `status.maxFreePerNode` is the largest such figure on one node. A class whose `resourceName` the KubeVirt `permittedHostDevices` table leaves out gets a `ResourceNameNotPermitted` condition, so the hoster sees the gap before a tenant's VM is refused ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). The taint rule leaves out exactly the three taints Kubernetes sets itself for a cordon or a node that is not `Ready`, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, which the first two rules already cover; any other taint under that prefix counts, a pressure taint or `out-of-service` included ([0005](./decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md)). Both figures come from the `Node` and `Pod` objects alone, with no inventory kind ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). In step 1 they describe the moment, not a promise: nothing is held, so a unit counted free can be taken by the next VM to start. + +### 4. Class entries in VMs and worker pools + +VMs and worker pools name a class in `gpus`: + +```yaml +# vm-instance +gpus: +- className: l40s-24c + count: 2 +--- +# kubernetes-nodes: every worker of the pool gets the entry +gpus: +- className: h100-pt + count: 1 +``` + +A `vm-instance` or `kubernetes-nodes` entry has exactly one of `name` or `className`, and a list holds only `name` entries or only class entries. `count` defaults to 1. The new entries are KubeVirt-only; Proxmox pools keep refusing `gpus`. On a cluster where gpu-operator does not serve GPUs to VMs, `vm-instance` and `kubernetes-nodes` refuse class entries at render, since no webhook would resolve them ([0002](./decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md)). ComputePlane is not yet an equal consumer: the module does not let a tenant pick the worker OS, so its GPU groups are Talos pools on `name` entries, and moving them to class entries is separate work. + +The chart renders only the entries, on the VMI template, as the `cozystack.io/gpus` annotation next to the `cozystack.io/gpu-reference` label, which KubeVirt copies from the VMI to its pods. A mutating webhook on VirtualMachineInstance creation, which sees only VMIs carrying the label, resolves each entry to its class and writes one `gpus` entry per GPU unit with the class's resource name. These entries have to be complete at admission: KubeVirt renders the launcher pod's resource requests from the VMI, and nothing but KubeVirt itself can change a VMI's spec afterwards. The webhook refuses to start the VMI, and names the class, when an entry: + +- names a class that does not exist, is being deleted, or has a `ResourceNameConflict` condition; +- is a vGPU class next to another vGPU class, or brings the VMI's GPUs of its class above `maxCount`; +- names a vGPU class in a namespace without the vGPU entitlement (section 6); +- names a class whose `resourceName` is not in the KubeVirt `permittedHostDevices` table, when the table is set, since KubeVirt would otherwise accept the VMI and leave it `Pending` with no pod for good ([0001](./decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md)). + +A GPU entry is any `gpus` entry, any GPU resource name in the VMI's `spec.domain.resources`, and any `hostDevices` entry whose device name is a GPU resource name, on the VMI or from its instance type. GPU resource names are always the union of three sets: the hoster's list `gpu.resourceNames`, empty unless the hoster sets it; the resource names of `GPUClass` objects; and the NVIDIA (`10DE`) and mdev entries of the KubeVirt `permittedHostDevices` table. None of them replaces another, so a vfio GPU missing from the table and a name that only pods request, which the hoster lists, still count. A NIC is not a GPU entry: a ConnectX or BlueField VF, which NVIDIA's network operator advertises as `nvidia.com/hostdev`, can go next to a GPU class entry for GPUDirect RDMA. The webhook also refuses a labelled VMI that carries GPU entries of its own, from any source, including an instance type or a VM written directly by a tenant super-admin. A second mutating webhook, on the creation of a virt-launcher pod that carries the reference label, adds the classes' `tolerations` to the pod, and to the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on a node the VM can use. They go on the pod and not on the VMI, because with `vmRolloutStrategy: LiveUpdate`, which Cozystack sets, KubeVirt v1.9.0 replaces a VMI's tolerations with the VM template's on every sync ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go), `handleTolerationsChangeRequest`). Both webhooks fail closed, so their outage blocks the start of VMs with class entries and no other VM. The VMI webhook reads the classes and the KubeVirt CR from the API server, not from a cache, since the controller's replicas serve admission and a lagging cache could admit what should be refused. Neither webhook writes any object besides its own patch, so in step 1 both declare `sideEffects: None`. + +Capacity is not checked at admission in step 1. A started VM whose class has no free unit on a node it can use waits in `Scheduling`, its launcher pod `Pending` with `Unschedulable` and the VM shown as `ErrorUnschedulable`, as a VM short of CPU or memory does today, and starts when a unit frees up; the catalog's `free` and `maxFreePerNode` tell a tenant beforehand whether that is likely. Refusing the VMI instead would leave the VM to the per-item backoff of KubeVirt's VM controller, which grows to 1000 seconds, for a shortage that may last seconds. + +A pool states no mode of its own: its entry names the class, and the class's `mode` never changes, so the chart and admission on `KubernetesNodes` take the mode from the class ([0009](./decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md)). A Talos pool, which today is every pool, takes passthrough classes only: a passthrough GPU is what Talos GPU pools already take through `name` entries, while a vGPU class needs the vGPU guest driver, which Talos workers do not carry. Admission on `KubernetesNodes` refuses a class entry that names a vGPU class on a Talos pool, before any machine is created. A pool on another worker OS takes a vGPU class only with a worker image that carries the vGPU guest driver, which Kairos GPU pools have to meet. Talos GPU pools on `name` entries keep working as today. + +### 5. Budgets + +A tenant's GPU budget is declared in `resourceQuotas`, per class as `.gpuclass.cozystack.io/devices: ` and over every class as `gpuclass.cozystack.io/devices: `, each vGPU counting as one. These keys are not resource names a pod requests, so they are not rendered into the namespace `ResourceQuota`, and a tenant with only GPU keys gets no `ResourceQuota` or `LimitRange`. That is a change to the tenant chart, which today renders both for any `resourceQuotas` key. + +GPU keys follow the same hierarchical pool rules as every other key, and #48 is what checks them, so tenants learn one rule and do not migrate when #48 ships: a `vm-instance` reserves `count` units of its class for each class entry, a `kubernetes-nodes` pool reserves `count` times `maxReplicas`, both evaluated from the defaulted values, and each also reserves the sum under the key without a class prefix. A write that would take a pool over a declared GPU key is refused when it is made, with a message naming the pool, the key, the limit and what is already reserved. A stopped VM keeps counting, as its CPU and memory do under #48: the budget limits what is declared, and the usage records (section 7) show what runs. A tenant with no GPU key anywhere above it is limited only by capacity, and for vGPU classes by its entitlement (section 6). + +Until #48's gate is on, an interim check in the Cozystack API server does this for GPU keys only. On a create or update of a `VMInstance` or a `KubernetesNodes`, it sums the class entries of every `VMInstance` and `KubernetesNodes` release in the pool, as above, with the object under admission at its new size, and refuses the write when a GPU key of the pool would be exceeded. On an update only the delta counts, so shrinking and no-op edits always pass. It reads the tenants' declared `resourceQuotas` and the applications' defaulted values, the inputs #48 reads, and applies the same pool arithmetic, so the switch to #48 changes no verdict. Like #48's gate it is a read-check-write without a transaction, so two writes at the same moment can both pass, and nothing evicts the excess. The interim check stands aside while #48's gate is switched on, and is deleted in the release where that gate is on by default, which needs `VMInstance` and `KubernetesNodes` to declare their class entries in their consumption. + +The tenant declaration gate validates GPU keys like any other: a child's declared GPU key may not exceed what its parent's pool has left. Today the gate reads what a pool uses from the `ResourceQuota`, which has no GPU keys, so until #48 the interim check also runs on Tenant writes and supplies the pool's class entries as its GPU usage. + +Pool membership follows the existing rules too, which #48 keeps. Until #48 ships, the runtime pool controller reads the rendered `ResourceQuota`, which has no GPU keys, so a tenant whose quotas hold only GPU keys is a pool root for the interim check and not for that controller; #48 removes the difference by reading declared values on both sides. Existing raw GPU keys, quota keys on a resource name such as `nvidia.com/AD102GL_L40S`, keep their meaning: they render into the `ResourceQuota` and limit launcher pods when they are created, whether the VM names the resource or a class of it (section 8). + +### 6. vGPU entitlement and license delivery + +A vGPU class is used only by a tenant entitled to vGPU ([0017](./decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md)). The Tenant field `gpu.vgpu` grants it. The parent tenant's administrator sets it, or the hoster on a top-level tenant. A tenant is entitled only when the flag is set on it and its parent is entitled. The root tenant is rendered by the platform, not by the tenant chart, so its value comes from platform values. The flag is off by default everywhere, so the hoster turns it on for the root first. No tenant can grant itself vGPU, since its Tenant object lives in its parent's namespace. The tenant chart computes the effective value from the parent's, like its other namespace values, and stamps it on the namespace as the label `namespace.cozystack.io/vgpu`, the same pattern as `namespace.cozystack.io/etcd` and the other module labels; a change reaches grandchildren as each level re-renders. The chart always writes the label, so only the value `"true"` means entitled; any other value, or no label, means not entitled. + +Entitlement is separate from budget because a vGPU license is counted per unit, which is a budget, while getting the vGPU guest driver and the license token at all is yes or no. The GPU keys of section 5 then limit vGPU like any other GPU. The VMI webhook refuses a vGPU class entry without the entitlement at every start (section 4), so clearing the flag stops VMs on vGPU classes from starting again while running ones keep running, and the namespace's license copies are deleted (License delivery, below). A write that names a vGPU class is not refused for the entitlement, since the pool gate of section 5 counts sizes and has no yes-or-no rule; the option source offers a vGPU class to an unentitled tenant with a remaining budget of 0 (section 7). Raw vGPU names in VMs stay as open as today until a raw key limits them or step 2's `forbidden` is set. + +#### License delivery + +The platform hands the license of a vGPU class to each VM that uses the class, as a volume of that VM, not as part of an image ([0018](./decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md)). + +The GPU controller copies the hoster's Secret of each class with a `license` into every namespace labelled `namespace.cozystack.io/vgpu: "true"`, as a Secret named `gpuclass--license` and labelled `cozystack.io/gpuclass: `. A copy has two keys: `client_configuration_token.tok`, the token, and `gridd.conf`, which sets the class's `FeatureType`. The controller writes or refreshes a copy when a namespace gains the label, when a class gains or changes its `license`, and when the hoster's Secret changes. It deletes a namespace's copies when the label stops being `"true"`, and a class's copies when the class is deleted or drops its `license`. While the hoster's Secret is missing or has no `client_configuration_token.tok` key, the controller keeps the copies it has, writes no new ones, and sets a `LicenseSecretMissing` condition on the class. No copy is written into a namespace without the entitlement. + +When the VMI webhook of section 4 resolves a vGPU class whose `license` is set, it adds a KubeVirt `secret` volume for that class's copy, with the volume label `GPULICENSE`, and a disk for it after the VMI's other disks and without a `bootOrder`, so the boot order of a VM does not change. KubeVirt gives the guest the Secret's keys as files on a disk with that label ([`schema.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/staging/src/kubevirt.io/api/core/v1/schema.go), `SecretVolumeSource`). At every boot, before its license client starts, a Linux guest mounts the disk by its label, copies the token into a directory on a memory file system such as `/run`, and puts a `ClientConfigTokenPath` line naming that directory and the `FeatureType` line of the disk's `gridd.conf` into `/etc/nvidia/gridd.conf`. NVIDIA documents `ClientConfigTokenPath` for keeping the token outside the default `/etc/nvidia/ClientConfigToken/` and asks that the directory hold nothing but the token ([NLS user guide](https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html), section 5.1.3.2), so the guest copies the token instead of pointing at the disk, which also holds `gridd.conf`. The token never reaches a Linux guest's root disk, and a Linux guest restarted without the disk has none. A GPU guest image for VMs and a worker image with the vGPU guest driver do this, and this disk, its label and this handling of the token are the contract those images follow; `vgpu-vms` covers Windows guests, whose token folder is on the system disk. VMs and pool workers get the volume the same way, since both start as VMIs with class entries, and neither the `vm-instance` nor the `kubernetes-nodes` chart changes for the license. The volume stays on a running VMI: KubeVirt v1.9.0 rewrites a VMI's volumes from its VM only for declarative hotplug volumes and, under the `Migration` volume update strategy, for the PVC volumes it migrates ([`hotplug.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/storage/hotplug/hotplug.go), [`volume-migration.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/volume-migration/volume-migration.go)), and a `secret` volume is neither. Until the copy exists, kubelet does not start the launcher pod, so the VM waits rather than boots unlicensed: the VM shows `Starting`, its launcher pod stays in `ContainerCreating` with a `FailedMount` event naming the missing Secret, and while the hoster's Secret is missing or lacks the token key the class shows `LicenseSecretMissing`. + +Rotation is a VMI restart. A client configuration token names the license allocation that serves it, and NVIDIA's license system documents no way to revoke a token before the expiry chosen when it is generated, 12 years if none is chosen and never for a DLS token set not to expire ([NLS user guide](https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html), sections 4.4, 4.4.1 and 4.4.2). So the hoster rotates onto a new allocation or license server instance: it generates the new token there and replaces it in its Secret, the controller refreshes every copy, and each VM reads the new token at its next start. Once the VMs have restarted, the hoster disables leasing on the old allocation, which then serves no licenses (NLS user guide, section 4.6), and a guest still on the old token keeps its lease at most for the maximum lease duration set on that allocation (NLS user guide, section 4.7). + +Clearing a tenant's entitlement takes effect at each VM's next start. A running guest renews its lease while the allocation serves its token, and a client released on the license server can check a license out again while the token is served (NLS user guide, section 4.9.1), so to cut a tenant off at once the hoster or the parent's administrator also stops the tenant's vGPU VMs. A guest's root user can read the token and keep a copy, and only a rotation stops such a copy from working. One token serves every tenant of a class, and every entitled namespace gets a copy for every licensed class, so a token does not tell tenants apart and the platform cannot give one tenant a token of its own. + +### 7. Usage records, catalog and licensing + +Cozystack does not bill. It keeps usage records (`Workload`) that an external billing system reads, and step 1 adds no record of its own. The `WorkloadMonitor` of a VM already sums the requests of its virt-launcher pod into a `Workload` per pod, extended resources included, and KubeVirt v1.9.0 puts one unit of the device's resource name into the pod's requests per GPU entry ([`renderresources.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/renderresources.go)). A VM on a class entry therefore shows its GPUs there under the class's resource name, for exactly as long as the pod exists, next to its CPU and memory. A worker pool's monitor selects its workers' launcher pods, so they are recorded the same way. The record names the VM's pod, not the node or the device ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). Since a resource name belongs to one class at a time (section 3), a billing system maps a record to a class through the catalog. A class whose `resourceName` changes appears under the new name from the next start of each VM, so a billing system keeps the names it has seen per class. Like other `Workload` records they have no time fields: a used interval is the pod's lifetime, which a billing system observes by watching and reconciles by sampling. + +The catalog is the list of `GPUClass` objects with `displayName`, model, mode, profile, memory, `maxCount`, `status.free` and `status.maxFreePerNode`. Every authenticated user reads it, because a hoster publishes its offers and a tenant names a class in a VM, including from `kubectl` or GitOps without a portal. So the status holds only `free` and `maxFreePerNode`, and its conditions name no nodes and carry no counts. The class's capacity and used units show the hoster's inventory and load, and they are not in the status; the controller exports them as metrics that platform monitoring scrapes. A tenant can still watch `free` and `maxFreePerNode` over time and see how busy a class is. + +The dashboard reads the catalog through a new `gpuclass` option source, which also returns the tenant's remaining budget per class; `OptionItem` gains optional integer fields for `free`, `maxFreePerNode`, `maxCount` and `remainingBudget`, and a portal offers the smallest of `maxFreePerNode`, `remainingBudget` and `maxCount` as the largest entry, leaving out a field that is absent because it sets no limit. The remaining budget is the pool figure of section 5, so a tenant under a shared parent budget can see how many GPUs its siblings declare, as a number, without names. This is accepted: the alternative is a budget the tenant can only discover by a refused write, which is the failure this proposal removes. For a vGPU class and a tenant without the vGPU entitlement, `remainingBudget` is 0. The existing `gpu` source keeps serving raw names while they are allowed. + +vGPU needs a license, and passthrough needs one only for graphics. Compute (C-series) profiles are licensed through NVIDIA AI Enterprise, graphics (Q-series) profiles through NVIDIA vGPU software. The hoster provides the license server, the token and the guest drivers, and the platform delivers the token and the `FeatureType` of a vGPU class with a `license` to each VM and worker of an entitled tenant (section 6). A passthrough class carries no `license`, so a licensed passthrough GPU for graphics still gets its token through a private image the hoster gives the tenants it licenses, not through the shared golden images. The class's `mode` tells a portal and a tenant which case applies. + +### 8. Migration of existing `gpus` + +Raw `name` entries keep working, and nothing converts them automatically. A tenant moves a VM by replacing each `name` entry with a class entry of the matching class and restarting it; the VM requests the same resource name as before, so the move needs no extra unit. A Talos GPU pool on a passthrough name moves by replacing its `name` entries with a class entry of the matching passthrough class, which rolls its workers. A Talos pool on a raw vGPU name has no path here, since Talos workers carry no vGPU guest driver; it keeps its raw name until it moves to a worker OS that does. ComputePlane GPU groups have no path until the module accepts class entries. The `gn1.*` instance types get no class to migrate to: they are deprecated, and the name they request is in neither default `permittedHostDevices` table. A VM on one moves to an instance type without GPUs and takes class entries, and a pool on one takes the pool path above. + +Quota keys move too. A class entry's launcher pod requests the class's resource name, so while a tenant migrates, a raw key on that name limits both kinds of VM at pod creation, and a GPU key of section 5 limits only class entries, at write. After its last raw-name VM has moved, the raw key is replaced by GPU keys. A tenant whose raw-name VMs use vGPU can move them to class entries only after it is entitled to vGPU (section 6). + +Raw names stay allowed in step 1. Forbidding them is part of step 2, where it is what makes a hold hard (section 9). + +### 9. Step 2: holding GPUs + +Step 2 lets a tenant keep GPUs on a node while its VM is stopped, so the next start does not fail for a GPU someone else took. Holding stays GPU-specific and does not become a generic reservation ([0016](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md)): CPU and memory are interchangeable across nodes, while a GPU hold is a whole unit on a specific node that does not migrate, which is why it needs a binder, `Lost`, the gather and `onCordon`. DRA would be the general path, but KubeVirt [VEP 344](https://github.com/kubevirt/enhancements/issues/344) (VM-scoped persistent ResourceClaims) is open and unimplemented, and the vGPU VF is handed out by a device plugin (section 2), not by a DRA driver. Holding moves to DRA once VEP 344 is implemented and a DRA driver hands out vGPU VFs. + +Nothing in this section exists in step 1. Step 2 adds `GPUClaim`, the `onCordon` field and a finalizer on `GPUClass`, `claim` entries in `vm-instance`, `claimSelector` entries in `kubernetes-nodes`, the binder, the scheduling gate in the launcher pod webhook, the claim order webhook, claim usage records and `gpu.rawResourceNames: forbidden`. + +#### Objects + +```yaml +# Hoster. Step 2 adds onCordon to GPUClass. +apiVersion: cozystack.io/v1alpha1 +kind: GPUClass +metadata: + name: l40s-24c +spec: + # ... the step 1 fields + onCordon: Move # Move | Stay; can change +--- +# Tenant. One per GPU or per set of GPUs used together on one node. +apiVersion: cozystack.io/v1alpha1 +kind: GPUClaim +metadata: + name: train-0 + namespace: tenant-acme +spec: + className: l40s-24c + count: 1 # GPUs of the class, always on one node +status: + phase: Attached # Pending | Bound | Attached | Lost + nodeName: gpu-node-3 + attachedTo: + vmi: vm-instance-trainer + boundAt: "2026-10-14T09:00:00Z" +``` + +A class is in use while any claim of it is `Bound`, `Attached` or `Lost`, and a class in use cannot disappear. A finalizer, `cozystack.io/gpuclass-protection`, keeps a deleted class while it is in use, new orders against a deleted class are refused, a `Pending` order of a deleted class gets a condition and is never bound, and `maxCount` cannot change while the class is in use. + +A `GPUClaim` with `count: 2` is two GPUs on one node attached together; GPUs that move between VMs separately are separate claims. Admission refuses a claim above the class's `maxCount`. The spec is immutable, so a change is a new order. An ordered claim is a `GPUClaim` a tenant orders and keeps, as opposed to a template claim (Attaching ordered claims); it holds GPUs while it is `Bound` or `Attached` (the phase table). A `Bound` claim that no VMI uses is a count on a node, not a specific card, because with a device plugin kubelet picks the physical device when the pod starts. + +What `Bound` guarantees depends on the class's source: + +| Source | A `Bound` claim holds | The physical device is picked by | +| --- | --- | --- | +| `devicePlugin` | `count` free units of the class on one node | kubelet, when the pod starts | +| `dra` (section 10) | one `ResourceClaim` within budget, and its devices once the claim is allocated at the first start | the scheduler, at allocation | + +A tenant sees a class, a count, a phase and, once there is one, a node, never which card it holds, so its API stays the same when an offer moves to a class of the other source. + +#### Binding and capacity + +A controller in `cozystack-controller` binds each new `GPUClaim` to a schedulable node of its class that has `count` free units. A node is cordoned when its `spec.unschedulable` is `true`; taints do not count, including the `NoSchedule` taint Fence Agents Remediation sets on a node under remediation. New binds, moves, rebinds and gathers skip cordoned nodes, nodes that are not `Ready`, nodes with a pending layout change (section 2), and nodes with a `NoSchedule` or `NoExecute` taint that the class's `tolerations` do not tolerate, by the taint rule of section 3, and their units do not count in the class's capacity, `free` or `maxFreePerNode`; a set of claims of several classes takes a node only when every class of the set tolerates its taints ([0010](./decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md)). A claim already bound to a node that stops being `Ready` stays there until lost capacity moves it (below), so a short outage moves nothing. The one exception is the gather at VMI start (Placing a VMI's claims): it may take a claim attached only to the starting VMI off a node that is not `Ready`, because no other VMI uses that claim and the move happens once, for a VM that is starting now, so nothing flaps back when the node returns. Free capacity is the node's allocatable count of the class's resource, minus the units that `Bound` and `Attached` claims hold, minus the units pods hold beyond what their claims cover; the last term counts VMs that still use a raw resource name. By default the binder packs onto the fullest node, so whole nodes stay free for large sets. + +Admission refuses an order that cannot be met at that moment, by capacity or by budget (Budgets and usage records of ordered claims), with a message naming the class, the count and the limit it hit: a budget of the pool, the class's free units, or the free units on one node. Two orders admitted at the same moment can still exceed a limit together, so the binder checks capacity and budgets again before binding. An order that loses this race stays `Pending` with a condition naming the shortage, and is bound when units or budget free up. + +The phases decide what a claim holds and costs: + +| Phase | Holds GPUs | Counts against budget | In usage records | +| --- | --- | --- | --- | +| `Pending` | no | no | no | +| `Bound` | yes, on a node | yes | yes | +| `Attached` | yes, used by a VMI | yes | yes | +| `Lost` | no | yes | no | + +A `dra` claim before its first start is the exception: it is `Bound` without a node, holds no GPU and is not in the usage records (section 10). A template claim holds GPUs in the same phases, but counts against no budget and is in no usage record of its own: its class entry and its launcher pod's `Workload` carry them (Attaching ordered claims). + +A claim leaves `Attached` when its VMI reaches a final phase (`Succeeded` or `Failed`) or is removed from the API, whichever comes first. A deletion timestamp alone is not enough: on an isolated node the domain of a VMI being deleted can still hold the GPU, while a final VMI no longer runs its domain. KubeVirt v1.9.0 names a VMI after its VM, so it never creates a VM's next VMI while the previous one exists: wherever a `runStrategy` creates a next VMI (the table in Failure and edge cases), the final VMI is gone first, and under `Manual` a start request first stops the final VMI and then starts a new one ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go)). A new VMI therefore never finds its claim attached to the old one. + +A claim keeps its node while the `Node` exists, serves the class, has not lost capacity, has no pending layout change (section 2) and is not cordoned under `Move`; otherwise an unattached claim that no longer fits there moves to another node of its class, or to `Lost` when none has room. The claims one VM names move as a set: together to one node with room for all of them, together to `Lost` when they must leave and no node has room for all of them, and they are rebound together. A claim or set whose only reason to leave is a cordon is the exception: with no room elsewhere it stays where it is. When a claim's node is deleted or no longer serves the claim's class, the controller also deletes the claim's unscheduled VMI if its pod is already pinned, so the claim becomes unattached and moves; a VMI whose pod is still gated keeps waiting while its claims are gathered elsewhere. KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, so pinning adds no cost in normal operation. A cordon moves claims by the class's `onCordon` policy. With `Move`, the default, an unattached claim on a cordoned node moves to another node of its class when one has room, keeping its hold and its usage record, and carries a condition naming the cordon while it waits. Since cozystack/cozystack#4403 a drain evicts a `vm-instance` VM that cannot live-migrate, so its claim becomes unattached and moves, and where the VM's `runStrategy` creates a new instance after the eviction (the `runStrategy` table), that instance starts on the new node. Worker pool VMs use `evictionStrategy: External`, so a drain leaves them to Cluster API, and a replaced worker takes a claim by the pool rules below. With `Stay` the claim keeps its node and the VM waits for it, for a hoster who wants VMs back on the same hardware after maintenance. When the cordon is the only reason any claim of a set has to leave and the set mixes classes, the whole set stays in place, and its VM waits, while a claim of a `Stay` class sits on a cordoned node, because the set moves together; a gather at VMI start, when one is needed, cannot take such a set. + +A drop in allocatable alone is not a loss, since kubelet reports zero after every restart until the device plugin registers again. A node has lost capacity for a class when its allocatable count of the class's resource stays below the units held there for longer than a grace period, or when the `Node` has been `NotReady` that long without interruption; a node that is not `Ready` fits no claim ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)). Then: + +- the affected classes get a `LostCapacity` condition; +- unattached claims that no longer fit move to another node of their class, or to `Lost` when none has room; +- on a `NotReady` node, attached claims go to `Lost` as well; the ordered ones among them leave the usage records and are rebound once their VMIs reach a final phase or are removed from the API; if the node returns while the VMI still runs, the claim goes back to `Attached` there; a claim that a gather at VMI start is moving off the node is the exception: it finishes the move; +- a VM started after the condition is set has its claims gathered off the node before its pod is released, and the controller deletes the unscheduled VMI of a claim that no longer fits once that VMI's pod is already pinned, so its claim becomes unattached and can move; a VMI whose pod is still gated is not deleted, its claims are gathered instead. Where the VM's `runStrategy` creates a next VMI after such a deletion (Failure and edge cases), it starts once the claim has moved or been rebound from `Lost` and any start-failure backoff the table gives has passed; unlike a cordon, the node cannot serve the claim, so waiting is no option. + +Claims in `Lost` are rebound before new orders when capacity returns, a VM's set onto one node, without blocking smaller orders behind a large one; a claim whose VMI is still live on the lost node waits until that VMI is final or removed (above), while a claim of a VMI whose pod is still gated is rebound for that VMI. The controller's metrics show how many GPUs are waiting. A rebind never fails a budget check, since a `Lost` claim already counts. + +#### Attaching ordered claims + +VMs and worker pools reference ordered claims in `gpus`: + +```yaml +# vm-instance +gpus: +- claim: train-0 +--- +# kubernetes-nodes: each worker of the pool takes one matching claim +gpus: +- claimSelector: + className: l40s-24c + matchLabels: + pool: inference +``` + +A `vm-instance` entry has exactly one of `name`, `className` or `claim`, and a `kubernetes-nodes` entry exactly one of `name`, `className` or `claimSelector`. A list holds only `name` entries or only class and claim entries, a pool has at most one `claimSelector` and never next to a class entry, and a selector names its class in `className` and matches only claims of that class, which must share a `count`; the webhook checks the `count` at each attach, since claim labels can change after the pool is admitted. The rules of section 4 for Talos pools, the vGPU entitlement and `permittedHostDevices` apply to the class of a claim as to a class entry. + +The VMI webhook of section 4 also resolves claim references, runs the checks that need no placement, and records the attachments; it binds, moves and creates no claim, and waits for nothing. For a VM it takes the named claims. For a pool worker it takes the claim the worker had attached before, unless a VM now names it, its node is not `Ready`, or its node has a taint the class does not tolerate. Otherwise it takes one unattached `Bound` claim on a `Ready` node that matches the selector and that no VM names in a `claim` entry, preferring nodes that are not cordoned and have no such taint. Selectors are not checked against each other: when two pools match the same claim, the first worker to start takes it, so a tenant gives each pool its own labels. The webhook records the attachment of each ordered claim to this VMI, so two VMIs can never hold one claim and no background move can take one, and writes one `gpus` entry per GPU unit, ordered or from a class entry, with the class's resource name. The launcher pod webhook of section 4 adds the tolerations of the ordered claims' classes as it does for class entries, and the VMI webhook adds the license volume of section 6 for a licensed vGPU class reached through an ordered claim, as for a class entry. An attachment recorded for a VMI that is never created, because a later admission step refused it, is cleared by the controller after a grace period longer than the API server's request timeout. On a dry run the webhook records nothing, so in step 2 it declares `sideEffects: NoneOnDryRun`. The GPU controller places the claims after admission, while the launcher pod waits behind a scheduling gate (Placing a VMI's claims). + +The webhook refuses to start a VMI, and names each claim and its node, when a referenced claim does not exist, is being deleted, or is attached to another VMI, besides the rules of section 4 that apply to the class of a claim (above); a class being deleted does not refuse an ordered claim that is `Bound`, `Attached` or `Lost`, which its finalizer keeps; a `Pending` claim of a deleted class is refused, since it is never bound. A refused VMI leaves its claims unattached, so they move by the rules above. Any other claim that is `Pending` or `Lost`, or bound to a node that cannot serve it, is not refused: the controller binds or moves it before it releases the pod. + +A claim cannot disappear under a running VM: a deleted `GPUClaim` stays, with its binding and, for an ordered claim, its usage record, under the `cozystack.io/gpuclaim-protection` finalizer until no VMI uses it, and is not attached again. + +Once holding exists, a class entry needs a place on a node too, or a VM that names only a class could start on a unit another tenant holds. Once the launcher pod waits at its gate, the controller creates a `GPUClaim` from each class entry, named after the VMI and the entry's index, marked by the `cozystack.io/gpuclaim-template` label, and owned by the VMI. A claim of that name that carries the mark and is owned by this VMI is reused, so a repeated pass creates nothing twice; one still being deleted, or still owned by the VM's previous VMI, is waited for. Tenants cannot set the mark, so a tenant's own order is never taken or deleted this way; an unmarked claim that already has the name is never taken, and the pod stays gated with a condition naming it. The controller binds these template claims under the same capacity checks as any order, on the node of the gathered set when the VMI has ordered claims, and otherwise on one node with room for all of them, and attaches them. It deletes a template claim as soon as its VMI is final or removed, since KubeVirt can keep a finished VMI until the next start, so the claim lives only while its VMI runs, as a `ResourceClaim` made from a `ResourceClaimTemplate` lives only with its pod. A template claim counts against no budget of its own, since its class entry already counts (section 5), and it is not in the usage records, since its launcher pod is. A class entry counts as a claim reference under `forbidden`, and a pool on class entries pays for running workers only. + +A pool on `claimSelector` never runs more GPU workers than it has matching claims. Admission on `KubernetesNodes` refuses a `maxReplicas` above that number; the chart default is 10, so a pool with fewer claims sets it. A pool without spare claims rolls one worker at a time: surge machines are created at once, their VMIs are refused until an old worker frees a claim, and a surge machine that waits past `nodeStartupTimeout` is replaced by MachineHealthCheck. + +#### Placing a VMI's claims + +A launcher pod waits for its GPUs behind a scheduling gate, so nothing is refused for not being ready yet ([0012](./decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md)). The launcher pod webhook of section 4 also adds the gate `cozystack.io/gpuclaims` in step 2. It matches only pods labelled `kubevirt.io: virt-launcher` that carry the reference label, adds the gate only when it is not there yet, since a repeated call would otherwise fail on a duplicate gate, and fails closed: a pod that reached the scheduler ungated would run unpinned. It also gates the temporary pod KubeVirt creates to bind `WaitForFirstConsumer` volumes, so a local volume binds on the node of the claims. Migration target pods are left alone: KubeVirt deletes a target pod that stays pending for 15 minutes, a VM with a passthrough GPU or a VF does not live-migrate, and Cozystack does not enable the `VGPULiveMigration` gate that mdev vGPUs would need. A set that already has a `WaitForFirstConsumer` volume bound on a node stays on that node: the controller places it only where the node affinity of the VMI's bound volumes allows. KubeVirt never touches scheduling gates and has no timeout for a VMI whose pod waits, so the VMI stays in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated` ([`lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go)), and the VM's own start backoff never begins. + +For a gated pod the controller creates the VMI's template claims, then brings the set onto one node. Claims a VM ordered separately can sit on different nodes, for example two classes the binder placed apart. When the claims a VMI references are `Pending` or `Lost`, on different nodes, on a node that is gone, no longer serves a class, has lost capacity or has a pending layout change, on a node that is not `Ready`, on a node with a taint one of their classes does not tolerate, on a cordoned node under `Move`, or on a node with no room for the VMI's template claims, the controller binds or moves them onto one `Ready` node that is not cordoned, has no pending layout change, is tolerated by every class of the set and has room for all of them, template claims included. Every ordered claim of the set is attached to this VMI by then, so no other VMI or background move can take one, and each `Bound` or `Attached` claim keeps its hold while it moves. A claim of a `Stay` class on a cordoned node does not move. When all of the VMI's claims are on that node and the VMI has no template claims, there is no gather, even if the node is also not `Ready`: the pod is pinned there and waits. When some of its claims are elsewhere, or it has template claims, the set cannot be gathered. The pin holds through a node that is not `Ready` or has such a taint, and breaks once a claim of the set has lost capacity there, the node is gone, or it no longer serves a claim's class; the set then follows the rules for those cases ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)). + +Once the set is on one node, the controller pins the pod and removes the gate in one update. The pin is a `matchFields` requirement on `metadata.name` appended to every required node selector term of the pod, or the single term when it has none. Kubernetes lets a gated pod only narrow its scheduling, so a term cannot be added next to existing ones, and KubeVirt usually sets a required term already, for host-model CPUs. The pin is not `nodeName`, which would let kubelet start the pod on a cordoned node, and it goes on the pod rather than the VMI because, with `vmRolloutStrategy: LiveUpdate`, KubeVirt rewrites a VMI's node affinity from the VM template. + +When the set cannot be placed, because no node has room or a `Stay` claim sits on a cordoned node while others need gathering, the pod stays gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. The VMI gets no condition because KubeVirt's admission guards writes to a VMI. The controller releases the pod once the set becomes placeable. A pod that is already pinned cannot be gated again or unpinned. If the scheduler then refuses it on a node that is `Ready`, not cordoned and still serves the set, for lack of CPU, memory or a device the plugin no longer advertises, the controller deletes the launcher pod. A cordon, a node that is not `Ready` and a `Stay` pin do not count as such a refusal, and the pinned pod waits there, as above; a cordon under `Move` and lost capacity follow their own rules. After the deletion the VMI becomes `Failed`, and KubeVirt decides what happens next by the VM's `runStrategy` for a VMI that failed before it ran (Failure and edge cases). The controller records that node as rejected for that set, and the binder does not pick it for the set again until the node's allocatable, its taints or its `Ready` condition change, or ten minutes pass. A set of template claims only loses its claims with the failed VMI, so for it the controller keeps the record in memory against the VM, and a controller restart forgets it. Ten minutes outlasts a kubelet restart and the device plugin registering again, and keeps a node that the hoster has fixed from staying excluded for long. + +A claim that is already attached when its node is cordoned, to a VMI whose pod has not started, moves the same way while the pod is gated. Once the pod is pinned, under `Move`, for a VM whose `runStrategy` creates a next VMI after such a deletion, `Always` or `Once`, the controller deletes that unscheduled VMI and the VM's next VMI gets the moved claim; under `Manual` or `RerunOnFailure` the VMI waits on the cordoned node, since deleting it would leave the VM stopped (Failure and edge cases). + +#### Budgets and usage records of ordered claims + +An ordered claim counts against the same GPU keys as a class entry, from the moment it is `Bound` until it is deleted (the phase table), in the same pool figure as the class entries of section 5; a `claim` or `claimSelector` entry declares nothing of its own, since its claims already count. The validating webhook on `GPUClaim` creation checks each order against its class, `maxCount`, free capacity, that pool figure and the vGPU entitlement. A raw GPU key on the tenant or any ancestor for a class's resource name blocks orders of that class until the tenant that carries the raw key also has a GPU key covering the class, so an ordered claim, which no `ResourceQuota` sees, never bypasses an existing raw limit. While the raw key exists, it also caps the claims of those classes together with the raw-name VMs it covers. Clearing the vGPU flag refuses new vGPU orders; ordered claims keep counting and can still move or be rebound, so a pool on `claimSelector` keeps its claims. + +Each ordered claim gets a usage record with `kind: gpu-claim` and resources `.gpuclass.cozystack.io/devices: ` while it is `Bound` or `Attached`, a `dra` claim only once its `ResourceClaim` is allocated (section 10). The GPU controller writes the record itself, with its `GPUClaim` as the record's owner through an ownerReference, and with no `WorkloadMonitor` and no label selector, so no tenant label can add a record or drop one ([0011](./decisions/0011-a-gpuclaim-owns-its-usage-record.md)). The `Workload` controller, which today deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, leaves a record owned by a `GPUClaim` to its owner: the GPU controller deletes it when the claim leaves `Bound` and `Attached`, and garbage collection deletes it with the claim. A virt-launcher pod's `Workload` does not report GPUs its ordered claims cover a second time. + +The catalog's `free` and `maxFreePerNode` subtract units that claims hold from step 2 on, and leave out nodes with a pending layout change, which the binder skips. A hoster lists bound claims per node with a field selector on the `GPUClaim` field `status.nodeName`, which the CRD declares in `selectableFields` and tenants cannot write, and the class's bound units and `Lost` count join its metrics. A `gpuclaim` option source lists a tenant's own claims. + +#### Forbidding raw names + +While raw names are allowed, the scheduler does not see a claim that is bound but not attached, so a raw-name VM can take that unit; the binder counts raw-name usage, so it never books a unit a raw-name VM holds, but not the reverse. A hold is a hard guarantee only with `gpu.rawResourceNames: forbidden`. + +When every consumer has moved to class or claim entries, the hoster sets `gpu.rawResourceNames: forbidden`. The platform refuses `forbidden` on a cluster where gpu-operator does not serve GPUs to VMs, since the policy that enforces it renders only there. The preconditions are the hoster's checklist, and the platform does not verify them: no worker pools with raw-name `gpus` entries, ComputePlane GPU groups included, no `gn1.*` GPU instance types in use, and no raw GPU quota keys left. With `forbidden`: + +- an admission policy refuses GPU entries (section 4) on every VMI created without a class or claim reference, on every path; +- the same policy refuses every pod created in a tenant namespace that requests a GPU resource name, unless it is a virt-launcher pod created by KubeVirt's virt-controller service account with an owner reference to a VirtualMachineInstance. Tenants cannot forge such a pod, since no tenant role can create pods or VMIs, so app charts that copy free-form `resources` into pods cannot request GPUs either; +- tenant validation refuses raw GPU keys; +- the `gpu` option source returns nothing, and the `instancetype` source hides instance types that carry GPU entries. + +The policy reads its settings from a ConfigMap the controller writes in `cozy-system`, so on a cluster with `forbidden` it refuses pod creation in tenant namespaces until the controller has published it once. + +### 10. Extensions + +The catalog is designed so later backends add `source` members and class fields, not tenant fields. + +- DRA. Kubernetes DRA has been GA since 1.34, and KubeVirt v1.9 has `GPUsWithDRA` and `HostDevicesWithDRA` in beta. A `dra` source names a `DeviceClass` and selectors. In step 1 a class entry of a `dra` class renders a `ResourceClaimTemplate` reference into the VMI, and the scheduler allocates per pod. In step 2 a `GPUClaim` maps one to one onto a `ResourceClaim` created at order time. The scheduler allocates the `ResourceClaim`, the GPU controller keeps it allocated through its own `reservedFor` entry, which the resourceclaim controller leaves in place, and the webhook renders the `ResourceClaim` reference into the VMI; the binder does not allocate. Three gaps remain. The NVIDIA DRA driver has no vGPU yet (kubernetes-sigs/dra-driver-nvidia-gpu#1402, milestone v0.6.0), and its passthrough is alpha. A `ResourceClaim` is allocated only when a pod first needs it, so nothing is held before the first boot: until then a `dra` claim is `Bound` without a node, counts against the budget, holds no GPU and stays out of the usage records, and the webhook attaches it like any `Bound` claim without pinning the pod to a node. DRA quota is per namespace, without pools. KubeVirt VEP 344 (`persistWhenStopped`) holds a claim for a stopped VM from the VM side; a `dra` class and VEP 344 must not both manage one claim. When a DRA GPU driver is installed, the raw-name policy has to cover `ResourceClaim` references too. +- MIG-backed vGPU profiles get their own resource names from the patched plugin, so they are ordinary classes; with DRA, partitionable devices (beta in Kubernetes 1.36) describe MIG natively. +- A non-NVIDIA GPU is a class with another `vendor`; Q, B and A series vGPU profiles are classes like C series. +- NVLink sets that must be attached together are left to the DRA source. +- On-demand slicing. A hoster may let tenants take any size from a shared pool, from a fraction of a GPU, such as a MIG-backed vGPU on a 1/7 slice of an A100, up to several whole GPUs of one node, taken from GPUs the layout keeps out of MIG mode. The tenant contract stays a class, one per size, and a `count`, and a claim still lives on one node. The host layer and the binder change: the node agent creates the profile, and on MIG GPUs the MIG instance under it, when a claim binds instead of following a fixed layout, so the binder counts capacity from free GPU resources rather than from advertised names. Enabling MIG mode needs an idle GPU and, on Ampere, a GPU reset, so the layout enables it ahead of time. On Hopper and newer the mode does not survive a reboot, so the layout sets it again at boot. A plugin restart per bind, as section 2 does for late profiles, would leave the node's allocatable at zero until the plugin registers again on every bind, so the plugin needs to rediscover profiles without one, separate work on top of NVIDIA/kubevirt-gpu-device-plugin#192. + +## User-facing changes + +Step 1: + +- Tenants get a GPU catalog with free capacity and remaining budget per class, and `className` entries in `vm-instance` and `kubernetes-nodes`. Raw resource names keep working. +- A write of a VM or pool whose class entries do not fit a GPU budget is refused. A started VM that finds no free unit waits in `Scheduling` until one frees up. +- Hosters get `GPUClass` with a `license` for vGPU classes, which the platform delivers to VMs, GPU keys in `resourceQuotas`, the `gpu.vgpu` Tenant flag that opens vGPU classes to a tenant, the node layout and the `gpu.resourceNames` list; an external billing system reads GPUs from the `Workload` records of virt-launcher pods. + +Step 2: + +- Tenants get `GPUClaim`, `claim` entries in `vm-instance` and `claimSelector` entries in `kubernetes-nodes`. An ordered claim is billed while it holds GPUs, `Bound` or `Attached`, whether a VM runs on it or not. An autoscaled pool on `claimSelector` holds, and pays for, every `Bound` or `Attached` claim its selector matches; admission keeps `maxReplicas` at or below the number of claims it matches. A tenant who wants to pay only while running uses class entries, and finds out at start, not at order time, whether a unit is free. +- Hosters get `onCordon` on classes and `gpu.rawResourceNames`; an external billing system gets a usage record per ordered claim. + +## Upgrade and rollback compatibility + +The change is additive. Existing `gpus` entries render as before, and the existing variants work as today until a layout is set. + +Rollback is a hoster operation; its step-by-step runbook belongs to the documentation of the implementation. It keeps VMs and worker pools, which restart in the windows, with these exceptions. Choosing `default`, or one mode for a model whose whole GPU and VFs share a PCI ID, drops the other mode's devices and stops their VMs, a loss of service the hoster announces first. Tenants whose only GPU limits were GPU keys lose them, and on a cluster that ran step 2 with `forbidden`, tenants without the vGPU entitlement can use raw vGPU names again. A worker pool that takes GPUs only through classes, such as a Kairos GPU pool, loses its GPUs. A vGPU VM or worker that restarts after step 1 rolls back gets no `GPULICENSE` disk, so the hoster first puts the license token into private images it gives entitled tenants only, not into the shared golden images ([0018](./decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md)). Such a private image does not run the reference boot unit or script of the guest contract, since the boot unit points the license client at an empty token directory and the script empties the token folder at every boot. An existing VM or worker whose image follows the guest contract of section 6 keeps no token across boots, so it stays unlicensed after that restart until it is recreated from such a private image, or its tenant disables the reference boot unit or script, removes the `ClientConfigTokenPath` line from `gridd.conf` on Linux, and licenses the guest itself. + +Step 2 rolls back first. Raw names are allowed again, since raw keys are refused while raw names are forbidden. `claim` entries of VMs and `claimSelector` entries of pools become class entries, and those VMs restart and those pools roll. The `GPUClaim` objects are deleted, and the step 2 components go. + +Step 1 rolls back next. The hoster removes the node layout and chooses the variant in a maintenance window. Class entries become raw `name` entries with the class's resource name, and GPU keys become raw keys on those names where the tenant wants a limit, and the remaining GPU keys are deleted. After the class entries have become raw names, the hoster clears the `gpu.vgpu` flags before the components go, so the controller deletes the license copies in tenant namespaces; if the controller is already gone, the hoster deletes them by the `cozystack.io/gpuclass` label; guests that keep running keep their lease while the allocation serves their token. Last, the components, the `GPUClass` CRD, the `gpu.vgpu` flag with its namespace label and the `cozystack.io/gpu-*` node labels are removed. Each node's workload label stays on `vgpu`, since without the layout the variant's default would turn passthrough nodes into vGPU nodes; on `default` the former vGPU nodes are relabelled `vm-passthrough` in the first window. + +Three rules hold throughout: + +- Names move back by the window rule of section 2 in reverse: vGPU classes return to the hand-written VF names, and raw `name` entries and raw keys on the per-profile names move with them in the same window. +- Raw keys render into the `ResourceQuota` and are checked at pod creation, not at write, so a tenant moved back to them is refused later than before. +- A raw key also renders the tenant chart's default `LimitRange`, whose memory limit can sit below a VM's memory request, because KubeVirt sets virt-launcher limits from the namespace quota only when it limits `cpu` and `memory`, so a tenant given a raw key also declares `cpu` and `memory`. + +## Security + +Step 1: + +- Tenant access. Every authenticated user can read `GPUClass`, through a ClusterRoleBinding to `system:authenticated`, because tenant roles are bound per namespace and cannot grant a cluster-scoped kind. Only platform administrators write `GPUClass`. +- What a tenant controls. A tenant never supplies a resource name through a class entry: it comes from `GPUClass`, which tenants cannot write. A tenant cannot set its own `gpu.vgpu` flag, since its Tenant object lives in its parent's namespace. The license token reaches only the guests of entitled tenants, where the guest's root user can read it, which licensing a guest needs, and a tenant super-admin who writes a VirtualMachine directly can also mount its namespace's copy into it; a copy is never written into a namespace without the entitlement. A tenant super-admin who writes VirtualMachines directly can still write raw GPU names, as today, until step 2's `forbidden`; a VMI with class entries and GPU entries of its own is refused. GPU keys are checked only when an application is written. A VirtualMachine written directly, with class entries in its annotation or a higher `count` than its application declares, is not counted against them, also under `forbidden`. That path closes when tenant super-admins lose write access to VirtualMachines, which #48 requires before its phase 4. A directly written VirtualMachine can also be left out of the usage records, or recorded with sums other than what it uses. Until super-admins lose that write access, a hoster who sells GPU budgets or bills GPUs to sub-tenants does not give them the super-admin role. +- Device plugin. The patched plugin runs privileged with the host driver's NVML library and device nodes, like the vGPU manager does. +- Node agent. It reads sysfs and writes VF profiles on vGPU nodes from a privileged container, as the vGPU manager already does, and can delete only the device plugin pod of its own node. +- Webhooks. The VMI and virt-launcher pod webhooks fail closed and see only objects carrying the reference label, so their outage blocks the start of VMs with class entries. +- Controller. It reads nodes and pods to compute class status, and writes `GPUClass` status. It reads the license Secrets classes name in `cozy-system`, and creates, updates and deletes the `gpuclass--license` copies in entitled namespaces, through the cluster-wide write on Secrets it already holds. With the host layer it gains patching node labels, restricted by a policy to the GPU workload label and `cozystack.io/gpu-*` ([0008](./decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md)). +- Visibility. A tenant sees, through the remaining budget, its siblings' declared GPU count under a shared parent budget (section 7). Any authenticated user sees each class's `free`, `maxFreePerNode` and conditions, and can follow them over time. + +Step 2 adds: + +- Tenant access. Tenants with the `admin` role or above manage `GPUClaim` in their namespaces, on a cluster where gpu-operator serves GPUs to VMs; lower roles, and every role elsewhere, read it. +- What a tenant controls. A tenant never supplies a node through a claim reference: it comes from the claim's status, which tenants cannot write. Tenants cannot add or remove finalizers on claims, so a tenant cannot take an ordered claim out of the usage records. The webhook takes claims only from the VM's own namespace. With `forbidden` any GPU entry without a class or claim reference is refused, as is any other pod in a tenant namespace that requests a GPU resource name. +- Webhooks. Step 2 adds a third fail-closed GPU webhook, on orders, and the gate in the launcher pod webhook of step 1. The VMI and launcher pod webhooks still see only objects carrying the reference label, so their outage blocks the start of VMs with class or claim entries. The pod webhook has to fail closed, also because a pod created without the gate would reach the scheduler unpinned. A validating webhook on `GPUClaim` creation checks each order, so its outage also blocks new orders, while ordered claims and running VMs are not affected. A template claim skips it: the VMI webhook has checked the entry, and the controller checks capacity when it creates and binds the claim. +- Tamper policy and its trust boundary. An admission policy on `GPUClaim` creation and update refuses adding or removing a finalizer, except `orphan` and `foregroundDeletion`, which the API server and the garbage collector own, and setting, changing or removing the `cozystack.io/gpuclaim-template` label. It lets through, unchecked, requests from `system:kube-controller-manager`, from members of `system:masters`, and from every service account in `cozy-system` and `kube-system`. Anything running under a service account in those two namespaces can therefore create a claim marked as a template claim, which skips the order webhook, or remove a claim's finalizer, so an ordered claim leaves the usage records; write access to workloads in those namespaces is as sensitive as the GPU controller's own rights. +- Template claims. For class entries the controller creates `GPUClaim` objects in tenant namespaces, only of the class and count the entry names, after the VMI webhook has checked them. Neither checks who wrote the VM, so anyone who can write a VM, a tenant ServiceAccount included, uses the GPUs of its class entries, and anyone who can start it makes their claims exist, within the tenant's budgets and entitlement ([0007](./decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md)). They are not in the usage records themselves, so the launcher pod's `Workload` carries them, with the gap that Security step 1 names for directly written VirtualMachines. +- Controller. It gains deleting unscheduled VMIs that reference its claims, and `update` and `delete` on pods in every namespace, cluster-wide like its write on Secrets, because tenant namespaces appear at runtime. RBAC cannot narrow these to virt-launcher pods, so the right covers every pod, container images, labels and tolerations included, which is enough to run code under a tenant pod's service account; the controller uses it only to remove its own gate while appending the pin, and to delete a pinned launcher pod that its node refuses. It also writes one ConfigMap, the GPU settings in `cozy-system`. +- Visibility. A tenant sees the node name of its own claims. + +## Failure and edge cases + +Each case gives what is seen and what to do. + +Some cases depend on whether KubeVirt creates a next VMI for the VM. In KubeVirt v1.9.0 that depends on the `runStrategy` and on how the VMI ended ([`vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go), [`vmi/lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vmi/lifecycle.go), [`rest/lifecycle.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/rest/lifecycle.go); line numbers at that tag): + +| `runStrategy` | VMI failed before it ever ran | VMI failed after it ran | VMI succeeded after it ran | Controller deleted an unscheduled VMI | +| --- | --- | --- | --- | --- | +| `Always` | next VMI after the start-failure backoff (`vm.go` L987, L1025, L1580) | next VMI, no backoff (`vm.go` L987, L1539) | next VMI, no backoff (`vm.go` L987, L1539) | next VMI after the start-failure backoff (`vm.go` L1025, L1580; the VMI goes `Failed` first, `vmi/lifecycle.go` L447, L456) | +| `RerunOnFailure` | next VMI after the start-failure backoff (`vm.go` L1050, L1074, L1580) | next VMI, no backoff (`vm.go` L1050, L1539) | none until the tenant starts the VM (`vm.go` L1050, L1059, L1070; `rest/lifecycle.go` L120) | none until the tenant starts the VM (`vm.go` L1050, L1070; `rest/lifecycle.go` L120) | +| `Once` | none (`vm.go` L1141) | none (`vm.go` L1141) | none (`vm.go` L1141) | next VMI (`vm.go` L1141) | +| `Manual` | none until the tenant starts the VM (`vm.go` L1104, `rest/lifecycle.go` L120) | the same | the same | the same | + +The start-failure backoff grows with each failure that happens before a VMI runs, up to five minutes, and resets once a VMI runs. During the backoff the VM shows `CrashLoopBackOff`, and so does a `Once` VM whose VMI failed before it ran, although it does not retry (`vm.go` L2526-L2546, L2572-L2575). A drain evicts a GPU VM instead of migrating it: KubeVirt does not live-migrate a whole GPU in passthrough, KubeVirt v1.9 live-migrates one vGPU per VM only as an alpha feature behind the `VGPULiveMigration` gate and only for a resource name listed under `mediatedDevices` in `permittedHostDevices` ([`virt-handler/vm.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-handler/vm.go), `isMdevGPU`), an SR-IOV vGPU is a VF under `pciHostDevices` and never migrates, and Cozystack does not enable the gate. An evicted VMI whose guest shuts down within its grace period ends `Succeeded`, one destroyed after it ends `Failed` while the domain has ACPI, the default, and the matching column applies. + +Step 1: + +- A write of a VM or pool is refused → the message names the pool, the GPU key, the limit and what is already reserved; use a smaller `count`, another class, or ask for more budget. +- A VM with class entries is refused at start → the webhook refuses the VMI, and the VirtualMachine shows the message in a `FailedCreate` event, naming the class and the reason: the class does not exist or shares its resource name with another class, a second vGPU class, more than `maxCount`, no vGPU entitlement, a resource name KubeVirt does not permit, or GPU entries of the VMI's own; fix the entry, or ask the hoster. +- A VM with class entries shows `ErrorUnschedulable` → its launcher pod is `Pending` and `Unschedulable`: no node it can use has a free unit of the class. It starts by itself once one frees up. The catalog's `free` shows the shortage. +- A class shows `free: 0` while its nodes have GPUs to spare → the nodes carry a `NoSchedule` or `NoExecute` taint the class does not tolerate; add it to the class's `tolerations`, or remove the taint. +- A class has a `ResourceNameConflict` condition → another class has its resource name; delete one of them or point it at its own name. +- A class is deleted while VMs name it → running VMs keep running, and their next start is refused naming the class; the tenant moves them to another class. +- A node with VMs on class entries is cordoned → nothing moves: running VMs keep their GPUs, and new instances start elsewhere. When it is drained, `LiveMigrateIfPossible` evicts the `vm-instance` VMs, since a GPU VM does not live-migrate, and worker pool VMs, which use `evictionStrategy: External`, are left to Cluster API. Where its `runStrategy` creates a new instance after the eviction (the `runStrategy` table), it starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. Nothing holds the GPUs it used; uncordon the node or add capacity. +- A node dies while VMs on class entries run there → KubeVirt fails their VMIs only once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, as [Node fencing](../node-fencing/README.md) describes. Where its `runStrategy` creates a next VMI after a VMI that ran fails, that VMI starts on any node with a free unit of its class, or waits as `ErrorUnschedulable` while there is none. +- A GPU fails under a running VM on a class entry → the VM keeps running. Kubelet stops listing the device as allocatable, so the class's `free` drops, and when the VM starts again, its next instance gets a unit on a node that still has one, or waits as `ErrorUnschedulable` while there is none. +- A layout change does not apply → the node carries the `cozystack.io/gpu-layout-pending` label; cordon and drain it. A GPU VM on the node holds the change until it stops or is evicted. +- A vGPU VM stays `Starting` with its launcher pod in `ContainerCreating` and a `FailedMount` event → the namespace has no license copy of the class yet; the class shows `LicenseSecretMissing` when the hoster's Secret is gone or has no `client_configuration_token.tok` key. Create or fix the hoster's Secret. +- A vGPU VM runs unlicensed → its class has no `license`, or the guest image does not follow the `GPULICENSE` contract of section 6; set the class's `license` unless the class is meant for the tenant's own license, or fix the image. +- A Talos pool with a vGPU class entry is refused → Talos workers carry no vGPU guest driver; use a passthrough class, or a worker OS that carries the driver. + +Step 2: + +- A tenant's order is refused at create → the message names the class, the count and the limit, or, for a vGPU class, that the tenant is not entitled to vGPU; order a smaller count, another class, or ask for more budget. +- A claim stays `Pending` after an order → its condition names the shortage; it binds when capacity or budget frees up, or the tenant deletes it. +- A VM with claims is refused at start → the webhook refuses the VMI for a reference problem, such as a claim that does not exist, is being deleted or is attached to another VMI, and the VirtualMachine shows the message in a `FailedCreate` event, naming each claim and the reason; fix the reference. +- A VM with claims stays `Starting` → its VMI is in `Scheduling` with its pod gated, the claims carry a condition and the VMI a `Warning` event naming the reason: no node can take the whole set, template claims included, or a claim of a `Stay` class is on a cordoned node while others need gathering. The VM starts by itself once the set can be placed. To get there sooner, uncordon that node, free capacity on one node, drop a claim the VM does not need, or, within one class, replace them with one claim of the combined `count`. +- The pinned node refuses the launcher pod after the pin, for lack of CPU, memory or a device → the controller deletes the pod, the VMI fails, and a condition on the claims names the node and the reason. Where its `runStrategy` creates a next VMI after a failure before running, that VMI waits for the start-failure backoff. The binder avoids that node for the set for ten minutes or until the node changes. Free CPU or memory on GPU nodes if this repeats. +- A node is drained or cordoned → under `Move` its unattached claims move when another node of their class has room, and a VM whose `runStrategy` creates a new instance after the eviction starts there; with no room, or under `Stay`, the claims stay bound and in the usage records and their VMs wait; uncordon it, add capacity, or delete the node. +- A cordoned node is deleted while a drained VM's VMI still waits for it with its pod pinned → the controller deletes that unscheduled VMI (a VMI whose pod is still gated just has its claims gathered elsewhere), the claim moves to another node of its class or to `Lost`, and where the VM's `runStrategy` creates a next VMI after such a deletion, that VMI gets the moved claim. +- A node dies but stays registered → after the grace period its attached claims go to `Lost`, the ordered ones among them leaving the usage records, and its unattached ones move or go to `Lost`; once the node is tainted `out-of-service`, its `Node` deleted or its pods force-deleted, KubeVirt fails the VMIs and the ordered claims are rebound. If the taint or the force-delete comes before the grace period ends, the next VMI of a VM whose `runStrategy` creates one after a VMI that ran fails gathers its now unattached claims onto a `Ready` node with room and starts there at once. That holds for a VM's ordered claims. A pool worker takes only a spare claim of its pool, template claims follow their own placement, and a VMI pinned by the `Stay` rule waits on its cordoned node. After a `Node` deletion it starts the same way, once the controller has moved the claims. When no node fits, the new VMI waits in `Scheduling` with its pod gated until a node has room for its set. A VM whose `runStrategy` creates no next VMI after a VMI that ran fails keeps its failed VMI, which leaves its claims unattached, so after the grace period they move to another node of their class, or to `Lost` when none has room. When the hoster enables the optional `node-remediation` package of [Node fencing](../node-fencing/README.md), it sets the taint after Medik8s NodeHealthCheck and Fence Agents Remediation power the node `off` through its BMC; when the package is off or the node is not fenced, for example because the BMC lost power with the node, the manual procedure of that proposal applies. +- A node loses capacity while a VM's VMI waits for it with its pod already pinned → the controller deletes that VMI so its claims can move, while a VMI whose pod is still gated just has its claims gathered elsewhere; where the VM's `runStrategy` creates a next VMI after such a deletion, it starts once the claims have moved or been rebound and any start-failure backoff the table gives has passed. +- A cordoned node dies while a VM's VMI waits on it, pinned there by a claim of a `Stay` class → the pin holds while the node is `NotReady`, so the VMI keeps waiting and its claims stay bound and in the usage records. Once the node has been `NotReady` for the grace period it has lost capacity, and the pin breaks ([0006](./decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md)): the controller deletes the unscheduled VMI, and the VM's claims move together onto a `Ready` node with room for all of them, or go to `Lost` together, leaving the usage records, when no node has room, and are rebound together once one has. Where the VM's `runStrategy` creates a next VMI after such a deletion, it starts on the new node once the claims have moved or been rebound and any start-failure backoff the table gives has passed. A `Node` deleted before the grace period ends has the same effect at once. To keep the VM on that hardware, bring the node back within the grace period. +- A GPU fails under a running VM → the claim stays `Attached`, and an ordered one in the usage records, while the VM runs; when the VM restarts, the controller moves the claim off that node before it releases the pod, or the claim goes to `Lost` and the pod stays gated until it is rebound. +- A GPU pool on raw names, ComputePlane GPU groups included, is created after `gpu.rawResourceNames: forbidden` → its worker VMIs are refused by the raw-name policy; use class entries or a `claimSelector`, or allow raw names again while the cluster has such pools. +- A tenant's VM is `Pending` while its claim is `Bound` and in the usage records → a raw-name VM took the unit; this can happen only while raw names are allowed, and forbidding them prevents it. +- A pool worker is refused → no matching claim is free, because of a surge machine during a rollout, claims still `Pending` or `Lost`, matching claims only on nodes that are not `Ready`, or another pool or a VM using the matching claims; wait, add claims, give each pool its own labels, or lower `maxReplicas`. + +## Testing + +Step 1: + +- Unit tests for the class status: the capacity and used-unit metrics match the nodes and pods and none of them appears in `GPUClass` status, `free` and `maxFreePerNode` from node allocatable and pod requests, a node whose pods request more than its allocatable counting zero, a node with a `NoSchedule` taint the class does not tolerate left out and one whose taint it tolerates counted, the three skipped taints, the Fence Agents Remediation taint alone not making a node cordoned, and a second class with the same resource name getting `ResourceNameConflict`. +- Webhook tests for class entries: one `gpus` entry per unit with the class's resource name, the classes' tolerations on the launcher pod and on the temporary `WaitForFirstConsumer` pod and still there after a LiveUpdate sync of the VM, each refusal of section 4, a VM with a GPU class entry and `nvidia.com/hostdev` passing, and a dry run. +- Budget tests: the interim check refuses a `VMInstance` and a `KubernetesNodes` write over a class key or the total key, counts `count` times `maxReplicas` for a pool and the defaulted `maxReplicas` for a pool that does not set it, passes shrinking and no-op updates, refuses a child tenant's GPU key against the class entries already in the parent's pool, applies the pool rules like other keys, stands aside while #48's gate is on, and reaches the same verdicts as #48's evaluator on the same inputs. +- Admission tests for `KubernetesNodes`: a vGPU class entry on a Talos pool is refused, and a passthrough one is accepted. +- Entitlement tests: a child's flag without the parent's gives no entitlement, a vGPU class entry without it is refused at start, and `remainingBudget` is 0 without it. +- License tests: a copy exists only in namespaces labelled `"true"` and only for classes with a `license`, carries the token and a `gridd.conf` with the class's `FeatureType`, is refreshed when the hoster's Secret changes, is deleted when the label goes, the class is deleted or drops its `license`, and is kept with `LicenseSecretMissing` while the hoster's Secret is missing; the CRD refuses `license` on a passthrough class and a `featureType` other than `1` and `4`, and defaults `featureType` to `1`; a hoster Secret without the token key gets no copy and sets `LicenseSecretMissing`; the VMI webhook adds the `secret` volume with the `GPULICENSE` label and its disk for a licensed vGPU class only, and the volume is still there after a LiveUpdate sync of the VM. +- Usage tests: a VM on a class entry shows its GPUs in its `Workload` under the class's resource name while its launcher pod exists. +- Admission tests for the class schema: the frozen fields, `tolerations` operators, and the class name as a DNS label. +- Changed existing tests: cozy-lib and tenant chart quotas, `quota_test.go`, `providers_test.go`, and the GPU wiring tests in `bundles_gpu_kubevirt_wiring_test.yaml`. +- E2E with passthrough GPUs on the GPU runner: a VM on a class entry starts with the GPU, a write over budget is refused, and the `Workload` shows the GPU. +- Manual on SR-IOV vGPU hosts: both modes of one model in one cluster, profiles across reboots, a layout change through a drain, and adoption from `default` and from `vgpu`, on Ubuntu hosts. + +Step 2: + +- Unit tests for the binder and the three GPU webhooks: capacity with raw-name usage, a set of two classes skipping a node only one of them tolerates, budgets in the pool figure with class entries, the phase table, every move and refusal in section 9, concurrent starts, the gate added once on a repeated call and on the temporary `WaitForFirstConsumer` pod but not on a migration target, the pin appended to every existing node selector term and the gate removed in the same update, and the pod pin after a LiveUpdate sync. +- Webhook and controller tests for cordons, template claims and NICs: the webhook binds, moves and creates no claim, a set that cannot be placed keeps the pod gated with a condition and is released once it can, a pinned pod that the node refuses is deleted and that node is avoided for the set, under `Move` a claim moves at VMI start and the pod gets a required node affinity, an attached claim with an unstarted pod moves for `Always` and waits for `Manual`, on lost capacity a VMI with a pinned pod is deleted and an `Always` VM restarts while a `Manual` VM stays stopped, a gated VMI is not deleted and its claims are gathered, a cordon or `NotReady` on the pinned node does not count as a refusal, a `Once` or `Manual` VM stays stopped after a refused pod is deleted, the claims of one VM move, go to `Lost` and are rebound as a set, claims of two classes on different nodes, and a claim whose node has no room for the VMI's template claim, are moved onto one node at VMI start, left in place when no node fits, a `Stay` claim on a cordoned node pins the pod when it holds all of the VM's claims and blocks the gather when other claims need gathering, claims on a node that is not `Ready` are gathered, two VMIs whose sets share a claim do not both gather it, a VMI with only class entries gets one node with room for all of its template claims, and under `Stay` a single claim waits; a template claim is created by the controller once the pod waits at the gate, under the capacity checks, owned by the VMI, deleted with its VMI, reused on a repeated pass, counted against no budget of its own, and a dry run records nothing. +- License tests for ordered claims: a VMI whose ordered claim is of a licensed vGPU class gets the `GPULICENSE` volume as a class entry does. +- Quota tests for ordered claims: an ordered claim counts from `Bound`, a `claim` entry declares nothing, a raw key blocks orders of its class until a GPU key covers it, and a cleared vGPU flag keeps ordered claims and refuses new orders. +- Admission tests for `maxReplicas`, class lifecycle while the class is in use, tenant finalizers, the controller restrictions, and the raw-name policy and quota keys under `forbidden`, including a tenant super-admin whose direct Pod or VMI is refused by RBAC, whose VirtualMachine with a GPU entry and no class or claim reference gets its VMI refused, and a pod requesting a GPU resource name from any creator other than virt-controller. +- Controller tests for the usage record: present exactly while `Bound` or `Attached`, owned by its `GPUClaim`, independent of tenant labels, not reported twice, kept through reconciles of the `Workload` controller although it has no monitor label, and gone with its claim. +- E2E with passthrough GPUs on the GPU runner: order, attach, stop, confirm another tenant cannot take the GPU, start again, and check the usage record. + +## Rollout + +Step 1: + +1. The host layer: the node layout, profile writes, node labels, and the device plugin with NVIDIA/kubevirt-gpu-device-plugin#192 applied downstream. It needs no tenant API and can land first. +2. `GPUClass` with its status controller and metrics, class entries in `vm-instance` and `kubernetes-nodes` with the VMI webhook and the launcher pod webhook for tolerations, the `gpu.vgpu` Tenant flag and license delivery, GPU keys with the interim check, and the `gpuclass` option source, on the existing variants that serve GPUs to VMs, `default` and `vgpu`; elsewhere the CRD ships and the charts refuse class entries. Before tenants use vGPU classes, the `gpu.vgpu` flag is set from the root tenant down. It works on any cluster where each offer it publishes has a resource name of its own, with or without item 1. +3. The consumers, on top of item 2 alone: `vgpu-vms` adds the licensing guide and reference scripts for vGPU VMs, and Kairos GPU pools take class entries and follow the guest contract of section 6. Neither waits for item 4 or for step 2. +4. GPU keys through #48, and the interim check deleted, in the release where #48's gate is on by default. + +Step 2: + +5. Holding: `GPUClaim`, the binder and its metrics, the scheduling gate in the launcher pod webhook, the claim order webhook and tamper policy, template claims for class entries, `claim` entries in `vm-instance`, `claimSelector` in `kubernetes-nodes` for Talos pools with passthrough classes and for Kairos GPU pools, claim usage records, `onCordon`, and `gpu.rawResourceNames`. A cluster with GPU pools on raw names keeps raw names allowed and has no hard hold guarantee. + +Then: + +6. The upstream plugin in place of the downstream patch once NVIDIA/kubevirt-gpu-device-plugin#192, or what replaces it, ships upstream, and `dra` classes once NVIDIA's DRA driver serves vGPU to KubeVirt; an offer moves to a new class of the other source, since a class's kind of source cannot change. The plugin change touches only the host layer and the `resourceName` of affected classes, with raw-name consumers of a renamed resource moved in the same window. A `dra` class is a new class, so tenants move to it by naming it. + +## Open questions + +- Release on tenant suspension. A hoster may want a suspended tenant's held GPUs released after a GPU-specific grace period; Cozystack has no platform-wide notion of suspension yet. +- Packing policy. Fullest-node packing is the binder's default in step 2; hosters may need spreading across failure domains as a class option. +- Ordered claims under #48. A `GPUClaim` is not an application, so #48's oracle, which sums application releases, does not see it. Step 2 needs either `GPUClaim` counted next to applications in the pool figure, or the claim made an application kind with a declared consumption. + +## Alternatives considered + +A `gpu.cozystack.io` group for the GPU kinds and their keys. Rejected ([0013](./decisions/0013-the-gpu-kinds-live-in-cozystack-io.md)): no requirement here needs a group of its own, and a CRD cannot move to another group in place. + +Holding in step 1, as earlier revisions had it. Rejected ([0016](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md)): step 1 answers five of the six problems without a hold, the third only for budgets, and the hold brings the binder, moves, lost capacity and the pod gate into a v1alpha1 tenant API. + +A generic reservation for stopped VMs that covers CPU, memory and GPUs. Rejected for now ([0016](./decisions/0016-holding-is-step-2-and-stays-gpu-specific.md)): CPU and memory are interchangeable across nodes, a GPU hold is a whole unit on one node that does not migrate. + +DRA as the whole API, with `DeviceClass` as the catalog and `ResourceClaim` as the tenant object, or DRA first for passthrough with vGPU later. Rejected for now: a non-Pod `reservedFor` entry does keep a `ResourceClaim` allocated for a stopped VM, but the NVIDIA DRA driver does not serve vGPU yet and its passthrough is alpha, and vGPU is the case hosters sell first. DRA is the planned backend under `GPUClass` (section 10). + +`GPUNode`, an inventory kind per node written by a node agent. Rejected ([0015](./decisions/0015-node-allocatable-is-the-gpu-inventory.md)): node allocatable gives capacity and loss, and the `Workload` records give use per tenant. + +A cluster-scoped object per reservation unit, like `IPAddress` in #35. Rejected for the device-plugin backend, where kubelet picks the physical device and such an object would promise a precision the backend does not have. + +Tenant quota on the resource name only. Stock `requests.` quotas need no new code, but they refuse only at pod creation and tie a budget to a name the host layer can change. + +Placeholder pods that hold a GPU until the VM starts. Rejected because preemption picks victims by priority, not by owner, so one tenant's VM could evict another tenant's placeholder. + +Class-level reservations for worker pools, without per-worker claims. Rejected because a pool would get a promise of capacity, not GPUs. + +A cluster-wide reservation per tenant and class, a count without a node. It needs no binding, moves or pod pin, and for single-GPU offers it would be enough. Rejected because the count holds only in sum. With free units spread one per node, a held set of two fits nowhere. A vGPU profile exists only on the nodes the layout gives it. A lost node silently shrinks every tenant's promise instead of showing which claims are `Lost`. + +NVIDIA vgpu-device-manager as the profile writer. At v0.5.1 it writes mdev profiles only and has no path for `current_vgpu_type` on SR-IOV VFs, and the pull requests that added VF support were closed unmerged, NVIDIA/vgpu-device-manager#139 as not planned, and NVIDIA/vgpu-device-manager#195 after NVIDIA named gaps on the KubeVirt side and had no resources for that test round. The node agent writes VF profiles instead; nodes on the mdev path keep vgpu-device-manager (section 2). + +KubeVirt's built-in device plugin for VFs (kubevirt/kubevirt#16890), with the sandbox plugin for passthrough. It ships in KubeVirt v1.9 and needs no patch, but it gives every vGPU profile of a model one resource name, which breaks the host layer's one name per offer. On GPUs whose whole GPU and VFs share a PCI ID it clashes with the sandbox plugin, so both modes of such a model need the built-in plugin for passthrough too, and both modes share one name. Switching between it and the sandbox plugin is a maintenance window each time. Rejected in favour of the downstream patch, whose cost is carrying one patch until upstream settles and running the plugin privileged with NVML access. diff --git a/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md new file mode 100644 index 00000000..82ff381c --- /dev/null +++ b/design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md @@ -0,0 +1,35 @@ +# 0001. A class whose resource name KubeVirt does not permit is refused at VMI admission + +- **Number:** `0001` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (the class condition), [`cozystack/cozystack#5030`](https://github.com/cozystack/cozystack/pull/5030) (the refusal at VMI admission) + +## Context + +On the `default` and `vgpu` variants the platform writes a `permittedHostDevices` table into the KubeVirt CR whenever it has entries. The proposal assumed that KubeVirt refuses a GPU outside that table when the VMI is admitted. It does not. In KubeVirt v1.9.0 no admitter refuses a GPU or host-device name the table leaves out; the GraceIOVirtualization admitter reads the table only to apply its rules to names listed there, and the refusal is `validatePermittedHostDevices` in [`renderresources.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/renderresources.go), which runs when virt-controller renders the virt-launcher pod. A render error creates no pod, and the VMI stays `Pending` with `Synchronized=False`, reason `FailedCreate`, retried with backoff. It never reaches a final phase. + +A VM that names such a class would start and never run, with nothing telling the tenant why at start. With ordered claims (step 2) it is worse: a claim leaves `Attached` only when its VMI is final or removed, so a claim of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry. + +## Decision + +The VMI webhook reads the effective table from the KubeVirt CR, the resource names of its `pciHostDevices` and `mediatedDevices` entries, and refuses a VMI with a class entry or a claim of a class whose `resourceName` is not in it, with a message naming the class and the name. An unset table means KubeVirt checks nothing, and then neither does the webhook. The controller also sets a `ResourceNameNotPermitted` condition on such a class, so the hoster sees the gap before a tenant hits it. + +## Why not the alternatives + +- **Rely on KubeVirt's own refusal**, as the proposal did. Rejected on the mechanism above: the refusal comes after admission, as a `Pending` VMI that holds its claims, not as an error the tenant sees at start. +- **Treat a `Pending` VMI with `Synchronized=False` and `FailedCreate` as final, and collect it.** It would release the claims, but only after the tenant has started a VM that cannot run, and the reason would surface as a VMI condition rather than in the start request. `FailedCreate` is also the reason KubeVirt gives for other failures to render or create the pod, so the rule would have to parse messages to tell this one apart. +- **Refuse such a class when it is created.** A class may legitimately come before its table entry, and the table can change after the class exists, so a create-time check catches only one of the two paths. + +## Consequences + +- A tenant learns at start, from the webhook message, that the offer is not usable yet; in step 2 the claim stays unattached and moves by the usual rules. +- The webhook and the controller condition read the same lists of the table, so the condition predicts the refusal. KubeVirt's own check also covers `usb` entries, which no GPU class uses. +- Every VMI admission with class or claim entries reads the KubeVirt CR from the API server, not from the controller's cache: the replicas serve admission and a lagging cache could admit what should be refused. + +## Revisit if + +KubeVirt starts refusing a claim outside `permittedHostDevices` at VMI admission, or the table is dropped for DRA-backed devices. diff --git a/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md new file mode 100644 index 00000000..76b5daf6 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md @@ -0,0 +1,39 @@ +# 0002. GPU classes serve VMs only where gpu-operator serves GPUs to VMs, while claim finalizers and usage records always run + +- **Number:** `0002` +- **Date:** `2026-10-21` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#5008`](https://github.com/cozystack/cozystack/pull/5008) (the gate and the policies under it), [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (finalizers and usage records on every cluster), [`cozystack/cozystack#5030`](https://github.com/cozystack/cozystack/pull/5030) (the webhooks, the `vm-instance` refusal and the refusal of `forbidden`) + +## Context + +The proposal rolled step 1 out "on the existing variants" and said nothing about a cluster where GPUs do not reach VMs: gpu-operator under the `container` variant, gpu-operator not enabled, or no iaas bundle. On such a cluster KubeVirt runs without the `HostDevices` feature gate and without a `permittedHostDevices` table, and nothing there resolves a claim reference. + +The `vm-instance` and `kubernetes-nodes` charts render only the entries, on the VMI template, and leave it to the VMI webhook to fill in the GPUs (section 4). Without the webhook a class or claim entry would boot the VM with no GPU and no error. The CRDs, on the other hand, ship with the controller on every cluster, so a `GPUClass` or, in step 2, a `GPUClaim` can exist on a cluster that has no placement running, and in step 2 both carry finalizers. + +## Decision + +The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the VMI and virt-launcher pod webhooks, the class status controller and its metrics scrape, and in step 2 the binder, the controller's watches on KubeVirt objects, the GPU admission policies and the tenant admin's write access to `GPUClaim`. Elsewhere: + +- `vm-instance` and `kubernetes-nodes` refuse to render class and claim entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today. +- In step 2 the platform refuses `gpu.rawResourceNames: forbidden`, because the policy that enforces it does not render there. +- In step 2 the controller still adds and removes the finalizers of `GPUClass` and `GPUClaim` and keeps the usage records of ordered claims, so deleting a class, a claim or a tenant namespace never hangs. + +## Why not the alternatives + +- **Run everything on every cluster.** The webhooks and the binder read KubeVirt objects, which do not exist without KubeVirt, and under the `container` variant the GPUs gpu-operator advertises are meant for pods, not VFIO devices a VM can take. A claim bound there would be held and recorded for a GPU no VM can use. +- **Gate the finalizer paths with the rest.** A hoster who disables GPUs, or moves to the `container` variant, would then leave every existing class and claim stuck on its finalizer, and with it the tenant namespaces that hold claims. +- **Let the charts render class and claim entries anyway and rely on the webhook.** On these clusters no webhook exists, so the failure is a VM that runs without the GPU it asked for, which the tenant may notice only from inside the guest. + +## Consequences + +- On a cluster without GPUs for VMs, a tenant reads the catalog, and in step 2 its claims, but cannot use them, and a VM with class or claim entries fails at render with a message that says GPUs for VMs are not served. +- Turning GPUs for VMs on writes one new cluster value, which every tenant release reads, so each upgrades once, and only on such clusters. +- In step 2 the controller keeps a resync loop while a deleted claim waits for its VMI, because with the feature off nothing watches VMIs. + +## Revisit if + +GPUs reach VMs through another path than gpu-operator's VM variants, for example a DRA driver installed on its own (section 10). diff --git a/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md new file mode 100644 index 00000000..af4b6690 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md @@ -0,0 +1,42 @@ +# 0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts + +- **Number:** `0004` +- **Date:** `2026-10-20` +- **Status:** Superseded by [`0015`](./0015-node-allocatable-is-the-gpu-inventory.md) +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) (the fields), [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (the node agent) + +## Context + +Section 1 has the node agent write `GPUNode` from sysfs and kubelet's pod-resources API, with a `healthy` flag per PCI function, and section 2 decides lost capacity from devices the agent reports missing or unhealthy. Section 3 adds every resource name a GPU device plugin advertises to the GPU resource names, "as the agent reports it". Building the agent showed three gaps in that picture. + +The pod-resources API carries no device health. It lists the allocatable devices per resource name and the devices each pod holds, and kubelet lists as allocatable only the devices their plugin reports healthy. + +One PCI function is not always one unit. An mdev parent serves several plugin devices, so counting one healthy unit per function would make a node with more than one held mdev device look as if it had lost capacity. + +Not every GPU device ID maps to a PCI function. The GPU Operator's device plugin of the `container` variant reports NVML UUIDs, `GPU-…` or `MIG-…`, which sysfs cannot resolve, so the agent has no function to put such a GPU under, and section 3 still needs its resource name, such as `nvidia.com/gpu`. + +## Decision + +- A function is healthy when every plugin device on it is allocatable, and unhealthy when a pod that has not finished holds one of its devices that is not. A function no plugin device maps to has no health. +- Each function reports `allocatable`, the number of its plugin devices kubelet lists as allocatable: 1 for a VF or a GPU passed through whole, N for an mdev parent. A plugin device that names a whole IOMMU group counts once, on the group's lowest-addressed GPU function, and 0 on its other functions. The controller sums it over the functions that carry the resource name and caps the sum at the node's allocatable count from kubelet, which alone counts for a held name no function carries. A node has lost capacity for a resource name when it holds more units than that. +- `GPUNode.status.resourceNames` lists every resource name on the node whose devices are GPUs, including names whose device IDs are NVML UUIDs the agent cannot place under a function. The names join the cluster's GPU resource names of section 3. +- A function serving several pods, such as an mdev parent, names one of them in `usedBy`, the smallest by namespace and name, so the report is stable. + +## Why not the alternatives + +- **Read health from each device plugin's socket.** The agent would talk to every plugin the host layer runs, while kubelet already folds their health into what it lists as allocatable, and kubelet's view is the one the scheduler acts on. A judgement call, not a hard limit. +- **Count one unit per function.** Wrong for mdev parents, as above, and it would fire lost capacity on nodes that have lost nothing. +- **Leave GPUs the agent cannot place out of the report.** Section 3 then misses `nvidia.com/gpu`, so a VMI or a pod that names it would not count as carrying a GPU entry. + +## Consequences + +- An unhealthy device that no pod holds disappears from the allocatable list and looks the same as a missing one. Both lower `allocatable`, and both become lost capacity after the grace period once the node holds more units than remain. +- `usedBy` traces only one pod per function. Section 1's "ownership is exact for running GPUs" holds per function for VFs and whole GPUs, and for an mdev parent only through the claims' own `status.attachedTo`. +- The agent does not fill `profile` yet; writing and reading VF profiles is host-layer work (Rollout, step 4). + +## Revisit if + +kubelet's pod-resources API starts reporting device health, or a device plugin that Cozystack ships reports IDs the agent can neither resolve through sysfs nor recognise as NVML UUIDs. diff --git a/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md new file mode 100644 index 00000000..727eda52 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md @@ -0,0 +1,31 @@ +# 0005. The taint rule skips exactly the three taints that mirror a cordon or readiness + +- **Number:** `0005` +- **Date:** `2026-10-20` +- **Status:** Superseded by [`0010`](./0010-gpuclass-carries-the-tolerations-binding-reads.md) +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) + +## Context + +Section 2 kept binds, moves, rebinds and gathers off nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and left out "the `node.kubernetes.io/*` taints Kubernetes sets itself for a cordon or a node that is not `Ready`", because the cordon and readiness rules already cover those nodes. Read as a prefix, that sentence would also skip the other well-known taints Kubernetes defines under `node.kubernetes.io/`: `memory-pressure`, `disk-pressure`, `pid-pressure`, `network-unavailable` and `out-of-service`. None of them mirrors a cordon or readiness, and a node carrying one of them as `NoSchedule` or `NoExecute` would still be bound to, though the scheduler keeps a virt-launcher pod that does not tolerate it off the node. + +## Decision + +The rule skips exactly `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`. Every other `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate counts, under `node.kubernetes.io/` or not. The rule and its cases live in `internal/gpu/model/node.go` and `node_test.go`. + +## Why not the alternatives + +- **Skip the whole `node.kubernetes.io/` prefix.** It binds claims onto nodes that the scheduler then refuses, and keeps their units in `free`, which is the failure the taint rule exists to prevent. +- **Count the three as well.** Kubernetes sets `node.kubernetes.io/unschedulable` on every cordoned node, so a cordoned node would also read as tainted. A taint keeps a bound claim where it is, so under `Move` a cordon would never move a claim, and `onCordon` would stop meaning anything. + +## Consequences + +- A node under memory, disk or PID pressure, one whose network is unavailable, and one tainted `out-of-service` get no binds, moves or gathers while the taint holds and virt-launcher pods do not tolerate it, and their units leave `free` and `maxFreePerNode`. Outside a gather at VMI creation, a claim already bound there stays, as on any tainted node. +- A pressure taint comes and goes with the node's condition, so `free` on a busy class can drop and return without any claim moving. + +## Revisit if + +Kubernetes adds a taint of its own that mirrors a cordon or readiness, or Cozystack's virt-launcher pods start tolerating taints by default. diff --git a/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md new file mode 100644 index 00000000..ebc6c749 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md @@ -0,0 +1,32 @@ +# 0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim + +- **Number:** `0006` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013); the gather after admission of [0012](./0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md): not yet + +## Context + +Section 3 pinned a VMI to a cordoned node when a claim of a `Stay` class was there and all of the VMI's claims were on that node, with no template claims: there was no gather, "even if the node is also not `Ready`", and the VMI waited. Section 2 had a `Stay` claim keep its node through a cordon, while lost capacity, a node that is gone and a node that no longer serves the class were hard reasons for a claim to leave. The proposal did not say which of the two wins when a pinned node also has a hard reason, so a node under maintenance that then died could hold a VM's claims, and keep them in the usage records, for as long as the dead node stayed registered. + +## Decision + +The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint its classes do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (section 9): a new VMI's set is gathered onto one node with room for all of it while its launcher pod waits at the gate, or the pod stays gated when no node fits; a VMI whose pod is already pinned there is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The gather at VMI start and the controller's background pass read the pin through one function, so a VMI is never pinned where the next pass deletes it. + +## Why not the alternatives + +- **The pin always wins.** A `Stay` class exists for a hoster who wants VMs back on the same hardware after maintenance. A dead node is not coming back to the same state on any known schedule, and the VM's claims would stay held and recorded against hardware that serves nothing. +- **Break the pin on `NotReady` alone.** It undoes the point of `Stay` for the most common maintenance step, a reboot, and step 2 already uses the lost-capacity grace period so that a short outage moves nothing. +- **Break the pin on any taint.** Apart from a gather at VMI creation, a taint never moves a bound claim; it only keeps new binds off the node (section 9). Breaking the pin on one would make a `Stay` claim leave in a case where a claim of a `Move` class on an uncordoned node stays. + +## Consequences + +- A `Stay` VM waits through a reboot of its cordoned node and comes back on it, and after a node failure its claims move like any other VM's once the grace period has passed. +- A pinned VM waits on a dead node for the lost-capacity grace period, five minutes by default, before its claims can move. + +## Revisit if + +Hosters ask for a `Stay` policy that also outlasts a node failure, for example to wait for a repaired node rather than move off it. diff --git a/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md new file mode 100644 index 00000000..bf6544ad --- /dev/null +++ b/design-proposals/gpudevice/decisions/0007-whoever-can-write-a-vm-orders-its-template-claims.md @@ -0,0 +1,33 @@ +# 0007. Whoever can write a VM uses the GPUs of its class entries, while ordered claims stay with tenant admins + +- **Number:** `0007` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#5030`](https://github.com/cozystack/cozystack/pull/5030); the controller-side creation and capacity check of [0012](./0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md): not yet + +## Context + +The Security section gave `GPUClaim` writes to tenants with the `admin` role or above, and had the webhook create `GPUClaim` objects for `claimTemplate` entries, only of the class and count the template names. It did not say who that webhook acts for. The VMI is created by KubeVirt's virt-controller, not by the person who wrote the VM, so by the time the webhook sees it the requester is KubeVirt and the identity behind the VM is gone. + +In Cozystack more identities can write a `vm-instance` than can write a `GPUClaim`. The tenant role that tenant ServiceAccounts get grants every verb on `apps.cozystack.io`, which includes `vm-instance`, and the `use` role can start and restart a VM, which creates a new VMI. + +## Decision + +A class entry is part of the VM, so whoever can write the VM uses its GPUs within the tenant's budget, as with the VM's CPU and memory, and whoever can start it makes them run and enter the usage records for as long as the VMI runs; in step 2 the start also makes the entry's template claims exist. The budget is checked when the VM is written (section 5), the vGPU entitlement at every VMI admission, and in step 2 capacity when the controller creates and binds the template claims; none of these checks the identity behind the VM. Ordering a `GPUClaim` directly stays with tenant admins and above. + +## Why not the alternatives + +- **Check the identity behind the VM at VMI admission.** The webhook sees KubeVirt's service account. The VM's author is recorded nowhere it could read reliably, and the identity that starts a VM can differ from the one that wrote it. +- **Refuse class entries in `vm-instance` for identities that cannot write `GPUClaim`.** The aggregated API that serves `vm-instance` would need a per-field authorization step for one field of one kind, and the rule would still not cover a start by a `use` role holder, who changes no field at all. The cost is a judgement call, not a hard limit. + +## Consequences + +- A tenant admin who hands out a tenant ServiceAccount also hands out the ability to use GPUs through class entries, up to the tenant's budget, and the `use` role lets its holder start VMs that already carry class entries. The budget, the class keys and the vGPU entitlement are the controls that limit both. +- An ordered claim, which holds GPUs and is recorded while it is `Bound` or `Attached` whether or not anything runs on it, still needs a tenant admin. + +## Revisit if + +KubeVirt starts recording the VM's author on the VMI in a form admission can trust, or Cozystack gains per-field authorization on application kinds. diff --git a/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md new file mode 100644 index 00000000..a867ed58 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0008-the-controller-gets-its-host-layer-rights-with-the-host-layer.md @@ -0,0 +1,32 @@ +# 0008. The GPU controller gets its node-label right with the host layer, not before + +- **Number:** `0008` +- **Date:** `2026-10-20` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (the VMI deletion right only) + +## Context + +The Security section listed three rights the GPU controller gained beyond its own kinds and the usage records: patching node labels, restricted by a policy to the GPU workload label and `gpu.cozystack.io/*`; restarting the device plugin pod on a node; and deleting unscheduled VMIs that reference its claims. Only the third was used by Rollout step 1. Node labels and plugin restarts belonged to the node layout of section 6, which was Rollout step 4, and until then the hoster wrote node labels by hand. + +## Decision + +The node-label patch right and the policy that restricts it ship with the host layer, together with the code that uses them, and the VMI deletion right ships with step 2, which uses it. The device plugin restart is not the controller's: the node agent of the host layer restarts the plugin pod on its own node after writing profiles, under a policy that limits it to that pod. No GPU code gets a right before the code that uses it. + +## Why not the alternatives + +- **Grant all three with the first GPU code, as the Security section listed them.** Every cluster would carry a write right on every `Node` that no code uses, and the policy that narrows it would have no writer to test against. +- **Grant the label right without its policy until the host layer ships.** That is a broader right than the proposal ever allowed. + +## Consequences + +- The `update` and `delete` rights on pods in every namespace that the launcher pod gate needs are a separate grant of step 2, not one of the three. +- Until the host layer ships, the controller cannot change any node, so the GPU node labels are the ones the hoster writes. +- The rights and their policy are reviewed with the host layer, next to the code that uses them. + +## Revisit if + +The host layer is dropped or replaced by one that does not need the controller to write nodes. diff --git a/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md b/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md new file mode 100644 index 00000000..5cfbef8f --- /dev/null +++ b/design-proposals/gpudevice/decisions/0009-gpuclass-mode-is-immutable-and-a-pool-names-its-class.md @@ -0,0 +1,35 @@ +# 0009. `GPUClass.mode` never changes, and a pool names its class + +- **Number:** `0009` +- **Date:** `2026-10-23` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) +- **Implemented in:** not yet + +## Context + +Section 1 froze a class's `mode` only while its finalizer was present, that is while claims of the class were `Bound`, `Attached` or `Lost`. A worker pool selected claims by labels alone, and a selector's claims had to share a class anyway. The mode the pool's workers needed, passthrough or vGPU, came from a separate pool field, `kairos.gpuMode`, which the chart set as a `gpu.cozystack.io/mode` label on the worker VMI template, and the webhook refused a claim whose class mode differed from that label. The implementation accepted a pool reference on a VMI without the label, so the text and the code disagreed. + +@kvaps asked in review of #99 and [#101](https://github.com/cozystack/community/pull/101#pullrequestreview-5616165668) for `mode` to be immutable always and for the selector to name the class, so that a pool does not state its mode at all. + +## Decision + +`GPUClass.mode` cannot change after the class is created, whether or not claims of it exist, and neither can the other fields that define the offer: `vendor`, `model`, `profile`, `memory` and the kind of `source`. A pool's class entry names its class, and in step 2 a `claimSelector` carries `className` next to its label selector and matches only claims of that class. A pool states no mode: the chart and admission read it from the named class. The mode label and the mode check at VMI admission are dropped. + +## Why not the alternatives + +- **Keep the pool's mode field and the label check.** Two fields say the same thing, one of them can be wrong, and the webhook has to refuse the mismatch at every worker start instead of the chart getting it right once. A VMI without the label was a third state nobody had defined. +- **Freeze `mode` only while claims exist, as section 1 had it.** A class without claims could switch mode, and a pool that names it would then render workers for the other mode, so the chart could not rely on the name. The hoster loses nothing by the stricter rule: a different mode is a new class with a new name, which a tenant has to name anyway because budgets and prices are per class. + +## Consequences + +- A class name stands for one mode for the class's lifetime, so a pool's worker template follows from the class it names. +- A hoster who created a class in the wrong mode deletes it and creates another, once the class is not in use (section 9). +- A selector cannot span classes, which section 9 requires. +- A pool rendered against a class keeps that class's worker image until it renders again, so a hoster who deletes a class does not create another class of the same name in the other mode. Nothing in the platform enforces this; the hoster documentation states it. + +## Revisit if + +A class needs to serve both modes, for example if one resource name ever covers whole GPUs and vGPUs on the same node. diff --git a/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md b/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md new file mode 100644 index 00000000..fae9d915 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0010-gpuclass-carries-the-tolerations-binding-reads.md @@ -0,0 +1,33 @@ +# 0010. `GPUClass` carries the tolerations that `free`, binding and the launcher pod read + +- **Number:** `0010` +- **Date:** `2026-10-23` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) +- **Implemented in:** not yet + +## Context + +Section 2 kept binds, moves, rebinds and gathers off nodes with a `NoSchedule` or `NoExecute` taint "that virt-launcher pods do not tolerate", and [0005](./0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md) fixed which taints that rule skips. @kvaps pointed out in review that a claim is bound when it is ordered, before any VM or pod exists, so there are no tolerations to check a taint against. The implementation checked against none, so a dedicated taint on GPU nodes, the usual way to keep other workloads off them, took every such node out of binding and out of `free`, even for VMs that tolerate it. + +## Decision + +`GPUClass` has a `tolerations` field, in the shape of a pod's with the operators `Exists` and `Equal` only, which the hoster sets to the taints of the class's nodes that its VMs tolerate. A node counts for a class's capacity, `free` and `maxFreePerNode`, and in step 2 for its binds, moves, rebinds and gathers, only if the class tolerates each of its `NoSchedule` and `NoExecute` taints, except the three that 0005 skips, `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`, for the reasons 0005 gives. A set of claims of several classes takes a node only when every class of the set tolerates its taints. The virt-launcher pod webhook adds the tolerations of the VMI's classes to the pod when it is created. The class admits only tolerations that pod validation admits, so a class can never produce a launcher pod the API server refuses. `tolerations` can change, also while the class is in use. + +## Why not the alternatives + +- **Check taints against no tolerations, as the implementation did.** A dedicated taint takes the node out of the offer, which is the setup a hoster with dedicated GPU nodes uses. +- **Apply the taint rule only at gather time, against a concrete VMI**, the other option in the [review](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700). Binding and `free` would then ignore taints, so a claim could be bound onto a node no VM of the class can schedule to, and `free` would count units no VM can reach, until a VMI starts and the gather has to move the claim anyway. +- **Read the tolerations from the `vm-instance` chart's defaults.** They describe VMs, not the class, and a claim bound for a worker pool or a VM written directly would not follow them. + +## Consequences + +- A hoster with tainted GPU nodes lists the taint on every class of those nodes. Without it the class shows no free units there, and a failure case of the proposal says so. +- The class's tolerations reach the pod, so a tenant cannot get a GPU VM onto a tainted node without a class that tolerates the taint. +- A change of `tolerations` moves nothing by itself: as with any taint, a bound claim keeps its node, and only new binds, moves and gathers read the new list. + +## Revisit if + +Kubernetes lets a reservation be scheduled ahead of its pod, so tolerations can be checked against the pod that will use the GPU, for example through DRA (section 10). diff --git a/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md b/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md new file mode 100644 index 00000000..a55181c8 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0011-a-gpuclaim-owns-its-usage-record.md @@ -0,0 +1,35 @@ +# 0011. A `GPUClaim` owns its usage record, and no monitor or reserved label guards it + +- **Number:** `0011` +- **Date:** `2026-10-23` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) +- **Implemented in:** not yet + +## Context + +Section 5 kept each claim's `Workload` usage record under a `WorkloadMonitor` that the GPU controller owned in the tenant's namespace. The `Workload` controller deletes a record whose `workloads.cozystack.io/monitor` label is missing or names a monitor that is gone, so the record needed a monitor, and the monitor needed a selector. To keep tenant objects from matching that selector, the selector was a reserved label key, and an admission policy refused it on every create and update of every object in every tenant namespace, on every cluster, whether it used GPUs or not, with a ratchet for objects that carried the key before the policy existed. Record 0003 held that rule; it was withdrawn before the proposal merged, and this record replaces it. + +@kvaps asked in review why a policy on every tenant write protects a monitor whose selector must match nothing, when the record could have an explicit owner, and which object owns the GPU record, since monitor-made records are also removed by ownerReference garbage collection. + +## Decision + +Each GPU usage record carries an ownerReference to its `GPUClaim` and no monitor label. The GPU controller keeps it while the claim holds GPUs, as section 9 defines, and garbage collection removes it with the claim. The `Workload` controller leaves a record owned by a `GPUClaim` to its owner instead of deleting it for a missing monitor. There is no GPU `WorkloadMonitor`, no reserved label key and no admission policy for it. + +## Why not the alternatives + +- **Keep the monitor and reserve its label key, with a ratcheting policy**, as 0003 had it and the [review](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) questioned. The policy runs on every write in every tenant namespace on every cluster to protect one selector, and objects labelled before it existed still match the monitor until someone removes the label. It answers a problem the monitor created. +- **Keep the monitor with a selector nothing can match by construction, such as a label value tenants cannot write.** Kubernetes has no label value that a writer of an object cannot set, so this needs the same policy. +- **Write the record with no owner and no monitor label.** The `Workload` controller deletes such a record as soon as it sees it, and nothing would remove a record whose claim was deleted while the GPU controller was down. + +## Consequences + +- No admission policy matches tenant writes for usage records, and a cluster without GPUs carries nothing for them. +- The `Workload` controller gets a rule for one owner kind. Another kind of record that wants the same treatment needs its own case there. +- A billing system that lists records by monitor finds GPU records by `kind: gpu-claim` instead. + +## Revisit if + +The `Workload` controller moves to ownerReferences for every record, so the monitor label is no longer how it decides that a record is orphaned. diff --git a/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md new file mode 100644 index 00000000..97db84f4 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md @@ -0,0 +1,39 @@ +# 0012. The VMI webhook only checks and records claims, and the controller places them while the launcher pod waits at a scheduling gate + +- **Number:** `0012` +- **Date:** `2026-10-23` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) +- **Implemented in:** not yet + +## Context + +Section 3 had the mutating webhook on VirtualMachineInstance creation do all of the start's GPU work inside the admission call, failing closed: record the attachments, have the binder move the VMI's claims onto one node, and create and bind its template claims, all within the admission timeout, refusing the VMI when a move or a bind did not finish in time. The implementation in [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) and [`#5030`](https://github.com/cozystack/cozystack/pull/5030) needed a synchronous handshake for that, a `status.bindRequest` on the claim that the webhook wrote and then waited up to ten seconds for the binder to answer, which the proposal did not describe. A refused VMI went back to KubeVirt's VM controller, whose per-item backoff grows to 1000 seconds, so a VM could start many minutes after room appeared. + +@kvaps asked in review of [#99](https://github.com/cozystack/community/pull/99#pullrequestreview-5587787354) for the webhook to check, record and pin, and for the controller to do the rest, and turned that into a request in the [next review](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700). + +## Decision + +The VMI webhook resolves the claim references, runs the checks that need no placement, records the attachments of the ordered claims, and writes one `gpus` entry per GPU unit with the class's resource name, template units included, since KubeVirt renders the launcher pod's requests from the VMI and the VMI spec cannot change afterwards. It binds, moves and creates no claim and waits for nothing. The virt-launcher pod webhook, which from step 1 adds the classes' tolerations, also adds the scheduling gate `cozystack.io/gpuclaims` to the pod at creation, and to the temporary pod for `WaitForFirstConsumer` volumes, but not to migration target pods. The GPU controller then creates the template claims, gathers the set onto one node, and in one update appends a `metadata.name` requirement to every required node selector term of the pod and removes the gate. A set that cannot be placed keeps the pod gated, the claims carry a condition naming the reason, and the VMI gets a `Warning` event with the same reason. A pinned pod that its node refuses for lack of resources is deleted, and the node is avoided for that set for ten minutes or until it changes. For a set made only of template claims, the controller keeps its rejected nodes in memory against the VM, and a controller restart forgets them. `status.bindRequest` is dropped. + +## Why not the alternatives + +- **Keep the work in the webhook.** A fail-closed admission call that writes several objects and waits on another controller is slow under load, needs its own request protocol between the two, and turns every slow move into a refused VMI and the per-item backoff of KubeVirt's VM controller. +- **Refuse the VMI whenever a gather is needed and let the controller move the claims before KubeVirt's retry.** It drops the handshake but keeps the VM controller's per-item backoff, so the start still comes up to that backoff's limit after the claims are ready, and the tenant sees a refusal for a VM that will start. +- **Hold the VMI instead of the pod**, through a KubeVirt start strategy or a VM-level hold. KubeVirt v1.9.0 offers none that keeps the pod from being scheduled: `startStrategy: Paused` still schedules the pod and only pauses the guest. A scheduling gate is the hook Kubernetes provides for this, and KubeVirt neither touches it nor times it out. +- **Write the pin into the VMI's node affinity.** Under `vmRolloutStrategy: LiveUpdate` KubeVirt rewrites a VMI's affinity from the VM template, and it never changes a launcher pod's affinity, so the pod is the place that keeps the pin. + +## Consequences + +- The webhook is a pure check plus the attachment record, so its latency no longer depends on the binder, and a dry run only skips the record. +- Template claims are created after the VMI exists, so the controller sets their owner at creation, and a template claim no longer needs a grace period for a VMI that never appears. +- A set that cannot be placed is no longer an admission error. The VM shows `Starting`, the VMI waits in `Scheduling` with `PodScheduled=False`, reason `SchedulingGated`, and the claims carry a condition naming the reason while the VMI gets a `Warning` event with the same reason. The VM starts as soon as the set fits, with no per-item backoff of the VM controller in between. +- The pod webhook is a fail-closed dependency of every start of a VM with class or claim entries, like the VMI webhook. +- The cluster autoscaler does not scale up for a gated pod, and after the pin a new node cannot help that pod, so GPU node scale-up stays with the hoster. +- A pin cannot be undone, so a node that refuses a pinned pod costs one failed VMI. Where the VM's `runStrategy` creates a next VMI after a VMI that failed before it ran, that VMI also waits for KubeVirt's start-failure backoff (the `runStrategy` table in the proposal's Failure and edge cases). The rejected-node rule keeps this from repeating. + +## Revisit if + +KubeVirt offers a hook between VMI creation and launcher pod scheduling that a controller can hold, so the pod no longer needs to be held on the Kubernetes side. diff --git a/design-proposals/gpudevice/decisions/0013-the-gpu-kinds-live-in-cozystack-io.md b/design-proposals/gpudevice/decisions/0013-the-gpu-kinds-live-in-cozystack-io.md new file mode 100644 index 00000000..96fafb86 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0013-the-gpu-kinds-live-in-cozystack-io.md @@ -0,0 +1,39 @@ +# 0013. The GPU kinds live in `cozystack.io`, with no group of their own + +- **Number:** `0013` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) +- **Implemented in:** not yet + +## Context + +The proposal introduced an API group, `gpu.cozystack.io`, for `GPUClass`, `GPUClaim` and `GPUNode`, and used the group name as the prefix of everything around them: the tenant quota keys `gpu.cozystack.io/` and `gpu.cozystack.io/total`, the node labels, the finalizers, the labels and annotations on VMIs and pods, and the scheduling gate. It gave no reason for a group of its own. cozystack/cozystack#4998, which would create it, was open. + +@lllamnyp pointed out in review that those keys are part of the API contract even though the API server does not enforce them, so the group decides their names too, and that upstream handled less interchangeable devices without a device-type group: distinct offers get distinct resource names, and where counts were not enough, the new group is named after the mechanism, `resource.k8s.io`, with class-scoped quota keys `.deviceclass.resource.k8s.io/devices`. He also pointed at `network.cozystack.io`, which cozystack/cozystack#3081 added on 2026-06-30 and cozystack/cozystack#3218 removed on 2026-07-16, as the case not to repeat with tenant quota keys involved. + +The `cozystack.io` group already held hoster-side kinds, `SchedulingClass` and `Package`, and tenant-namespace kinds, `Workload` and `WorkloadMonitor`. + +## Decision + +There is no `gpu.cozystack.io` group. `GPUClass` (step 1) and `GPUClaim` (step 2) are kinds of `cozystack.io/v1alpha1`, and there is no `GPUNode` ([0015](./0015-node-allocatable-is-the-gpu-inventory.md)). The budget of one class is `.gpuclass.cozystack.io/devices`, after DRA's form, and the budget over every class is `gpuclass.cozystack.io/devices`. Labels, annotations, finalizers and the scheduling gate are `cozystack.io/` keys with a `gpu` name, such as `cozystack.io/gpu-reference`, `cozystack.io/gpuclaim-protection` and the gate `cozystack.io/gpuclaims`; the vGPU entitlement stays on the existing `namespace.cozystack.io/` prefix of module labels. Section 1 of the proposal lists every name. + +`cozystack.io` then holds the hoster-side kinds (`SchedulingClass`, `Package`, `GPUClass`) and the tenant-namespace kinds the platform writes or a tenant admin owns (`Workload`, `WorkloadMonitor`, and in step 2 `GPUClaim`). + +## Why not the alternatives + +- **Keep `gpu.cozystack.io`.** Nothing in the design needs a group of its own: no separate versioning, no separate API server, no RBAC that a group boundary would make simpler. A CRD cannot move to another group in place, so a wrong group would cost every tenant's quota keys and objects a migration, as the `network.cozystack.io` removal did on a smaller scale. +- **Name the keys after the group, `cozystack.io/gpu-`.** A flat key cannot be told apart from a label-style name without parsing, and it does not follow the one upstream form readers already know. The DRA form puts the class in the DNS prefix, so a key without the prefix naturally means every class and no class name has to be reserved for a total. +- **Put the catalog in platform values instead of a kind.** Values can describe offers, but tenants and their GitOps tools could not read them, and nothing could carry a status with free units. + +## Consequences + +- cozystack/cozystack#4998 creates no new group; its CRDs move to `cozystack.io`, and every `gpu.cozystack.io` string in the GPU code is renamed before anything merges. +- The class name `total` is no longer reserved. +- A future GPU kind goes into `cozystack.io` too, unless a requirement appears that only a separate group can carry; then it is written down in a record first. + +## Revisit if + +GPU offers move to DRA `DeviceClass` objects, so the catalog and its quota keys become DRA's own, or a GPU kind needs versioning or serving apart from the rest of `cozystack.io`. diff --git a/design-proposals/gpudevice/decisions/0014-step-1-names-a-class-and-holds-nothing.md b/design-proposals/gpudevice/decisions/0014-step-1-names-a-class-and-holds-nothing.md new file mode 100644 index 00000000..c2962924 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0014-step-1-names-a-class-and-holds-nothing.md @@ -0,0 +1,41 @@ +# 0014. In step 1 a `gpus` entry names a class, budgets count declared entries, and usage is the virt-launcher pod's `Workload` + +- **Number:** `0014` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) +- **Implemented in:** not yet + +## Context + +Rollout step 1 had everything at once: `GPUClass`, `GPUClaim` as a tenant order, `GPUNode` and its agent, the binder, three fail-closed webhooks, the launcher pod gate, budgets with their own subtree rule, and usage records the GPU controller wrote per claim. @kvaps asked in [review](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) for the text to say which fields and phases exist in step 1 and what an entry means without the binder. @lllamnyp [went through](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) what the proposal adds above one resource name per offer: the catalog is hoster metadata, per-offer budgets argue for class-scoped keys, refusing an order that does not fit is what [#48](https://github.com/cozystack/community/pull/48) does for every kind by counting declared sizes, and the `Workload` records already sum virt-launcher requests, GPU resource names included. Only the hold needs a tenant object. + +The GPU budget rule of the former Quotas section diverged from the existing pool semantics and was meant to move to #48 later, so tenants would have migrated twice. + +## Decision + +Step 1 has no tenant object. A `vm-instance` or `kubernetes-nodes` `gpus` entry names a class and a count. The VMI webhook writes one KubeVirt `gpus` entry per unit with the class's resource name, after checking the class, `maxCount`, one vGPU class per VM, the vGPU entitlement and `permittedHostDevices`. A webhook on the virt-launcher pod adds the class's tolerations to the pod, since under `vmRolloutStrategy: LiveUpdate` KubeVirt resets a VMI's tolerations to the VM template's. Capacity is not checked at admission: a VM that finds no free unit waits in `Scheduling` like a VM short of CPU. A class is one resource name, unique among classes, and its nodes are the nodes that advertise it, so a class has no node selector. + +GPU budgets are ordinary `resourceQuotas` keys under the existing pool rules, and #48's gate checks them at write, counting `count` per class entry and `count` times `maxReplicas` per pool from the defaulted values. Until #48's gate is on, an interim check in the Cozystack API server does the same for GPU keys on `VMInstance` and `KubernetesNodes` writes, and supplies the pool's class entries as GPU usage to the declaration gate on Tenant writes, with the same inputs and pool arithmetic; it stands aside while #48's gate is switched on and is deleted in the release where that gate is on by default. + +Usage is the `Workload` the VM's `WorkloadMonitor` already keeps from the virt-launcher pod's requests. Step 1 writes no usage record of its own. + +## Why not the alternatives + +- **An order object in step 1, checked against budget and capacity at admission, without a hold.** Without the binder an order guarantees nothing at start, so the tenant would pay and wait for an object that only repeats what the VM's entry says. +- **Keep the GPU subtree rule until #48 takes over.** Tenants and hosters would learn one rule and then another, and #48 would have to adopt a GPU exception to keep verdicts the same. +- **Check capacity at VMI admission.** A refused VMI is left to the per-item backoff of KubeVirt's VM controller, which grows to 1000 seconds, for a shortage that may last seconds, while a pending pod starts as soon as a unit frees up. +- **Keep GPU-specific usage records next to the launcher pod's `Workload`.** The pod's record already carries the GPUs for exactly as long as they are used, and a second record would have to be kept from double counting. + +## Consequences + +- A stopped VM's class entries keep counting against the budget, as its CPU and memory do under #48, while its `Workload` shows no GPU. +- A billing system maps a resource name in a `Workload` record to a class through the catalog, and keeps the names it has seen when a class's `resourceName` changes. +- A vGPU class entry written by an unentitled tenant is accepted at write and refused at start, because the pool gate has no yes-or-no rule. +- Step 2 keeps the entry: it creates a template claim for it at start, so the same entry stays valid when holding ships. + +## Revisit if + +#48 does not ship or drops the pool semantics the interim check copies, or a hoster needs a capacity guarantee at start without holding. diff --git a/design-proposals/gpudevice/decisions/0015-node-allocatable-is-the-gpu-inventory.md b/design-proposals/gpudevice/decisions/0015-node-allocatable-is-the-gpu-inventory.md new file mode 100644 index 00000000..013f8daa --- /dev/null +++ b/design-proposals/gpudevice/decisions/0015-node-allocatable-is-the-gpu-inventory.md @@ -0,0 +1,35 @@ +# 0015. Node allocatable is the GPU inventory, and there is no `GPUNode` + +- **Number:** `0015` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) +- **Implemented in:** not yet + +## Context + +`GPUNode` was a cluster-scoped inventory kind per GPU node, written by a node agent from sysfs and kubelet's pod-resources API ([0004](./0004-gpunode-health-is-kubelet-allocatability.md)). It listed each PCI function with its health, an `allocatable` count, the pod using it, and the node's GPU resource names. Its readers were the binder, which judged lost capacity from it and capped its sums at kubelet's node allocatable anyway; the GPU resource names, which joined the name set of the raw-name policy; and a hoster tracing a physical device to a tenant. + +@lllamnyp asked in [review](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) whether a hoster-only inventory needs a kind when metrics, node allocatable or DRA's `ResourceSlice` can serve it. Step 1 no longer has a binder ([0014](./0014-step-1-names-a-class-and-holds-nothing.md)). + +## Decision + +There is no `GPUNode` and no inventory agent. A class's capacity, `free` and `maxFreePerNode` come from the `Node` objects' allocatable count of the class's resource and the requests of pods on those nodes. In step 2 a node has lost capacity for a class when its allocatable count of the class's resource stays below the units held there for longer than the grace period, or when it has been `NotReady` that long. GPU resource names are the union of the hoster's `gpu.resourceNames`, the classes' resource names and the GPU entries of `permittedHostDevices`. Use per tenant comes from the `Workload` records, per virt-launcher pod and resource name. The node agent of the host layer stays, writes VF profiles and restarts the device plugin on its own node. + +## Why not the alternatives + +- **Keep `GPUNode` for step 2's lost capacity.** 0004 already capped the agent's sum at kubelet's allocatable, and kubelet lists only the devices its plugin reports healthy, so the agent's health adds a per-function view of the same signal, not a different one. +- **Keep `GPUNode` for tracing a physical device to a tenant.** With a device plugin, kubelet picks the device when the pod starts, and the pod object does not say which one. The agent's `usedBy` was exact per VF or whole GPU but named only one pod per mdev parent, and a hoster replacing a failed card drains the node either way. +- **Expose the same data as metrics only.** It would still need the agent on every GPU node, privileged against kubelet's pod-resources socket, for a view no part of the design reads. + +## Consequences + +- A hoster cannot see from the API which VF or PCI function a VM uses, only which VM, class and count, and the node through the VM's launcher pod; the guest and the node's own tools show the device. +- A GPU name that only pods request, such as `nvidia.com/gpu` of the `container` variant, is a GPU resource name only if the hoster lists it in `gpu.resourceNames`. +- One privileged pod-resources reader fewer runs on every GPU node. + +## Revisit if + +The design needs a per-device view, for example exact-device allocation, NVLink sets or on-demand slicing; DRA's `ResourceSlice` is the first candidate to provide it. diff --git a/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md b/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md new file mode 100644 index 00000000..1b14d90d --- /dev/null +++ b/design-proposals/gpudevice/decisions/0016-holding-is-step-2-and-stays-gpu-specific.md @@ -0,0 +1,37 @@ +# 0016. Holding a GPU for a stopped VM is step 2, and stays GPU-specific until DRA can carry it + +- **Number:** `0016` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) +- **Implemented in:** not yet + +## Context + +The binder, moves, `onCordon`, `Lost` and rebind, the gather at VMI start, the launcher pod gate and pin, and `forbidden` all exist for one problem of six, keeping a GPU for a stopped VM, and for the capacity half of another, refusing an order that does not fit. @kvaps asked in [review](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) to split holding from the rest, and later [accepted one proposal](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) if the text says what exists in step 1 and gives step 2 its own section and record. @lllamnyp argued in [review](https://github.com/cozystack/community/pull/99#issuecomment-6445311702) that keeping capacity for a stopped VM is not specific to GPUs, since a stopped VM releases its CPU and memory too, and that the hold should be proposed either as a general reservation or through DRA persistent claims, KubeVirt [VEP 344](https://github.com/kubevirt/enhancements/issues/344). + +When this was decided, VEP 344 was open and its implementation, kubevirt/kubevirt#17957, unmerged; the NVIDIA DRA driver served no vGPU (kubernetes-sigs/dra-driver-nvidia-gpu#1402); and the vGPU VF reached KubeVirt through a device plugin, the one of the host layer with NVIDIA/kubevirt-gpu-device-plugin#192. + +## Decision + +Holding is step 2 of the proposal, in a section of its own, and none of its kinds, fields or phases enter the API before it: `GPUClaim`, `onCordon` and the class finalizer, `claim` and `claimSelector` entries, the binder, moves, `Lost` and rebind, the gather, the launcher pod gate and pin, template claims for class entries, the claim order webhook and tamper policy, the usage records of ordered claims, and `gpu.rawResourceNames: forbidden`. + +Holding stays GPU-specific and does not become a generic reservation. CPU and memory are interchangeable across nodes, so a reservation of them is a count the scheduler can meet on any node with room. A GPU hold is a whole unit on a specific node that does not migrate, since KubeVirt cannot live-migrate a VM with a passthrough GPU or an SR-IOV VF, and that is the reason for the binder, `Lost`, the gather and `onCordon`. A DRA path does not exist today: VEP 344 is open and unimplemented, and the vGPU VF is handed out by a device plugin, not by a DRA driver. + +## Why not the alternatives + +- **Holding in step 1.** Step 1 answers five of the six problems without a hold, the third only for budgets, and the hold puts the most expensive part of the design into a v1alpha1 tenant API before any of it is used. +- **A generic reservation for stopped VMs.** For CPU and memory a per-tenant count would be enough, and none of the node binding would be needed; for GPUs a count without a node fails as soon as free units are spread one per node, as the cluster-wide reservation alternative of the proposal shows. One mechanism would carry the GPU machinery for resources that do not need it. +- **Wait for DRA.** No date can be given for VEP 344 or for vGPU in the NVIDIA DRA driver, and hosters sell vGPU first. + +## Consequences + +- Step 1 ships without a tenant object, and a GPU VM's next start can fail for a GPU someone else took, as it can today. +- Step 2's `GPUClaim` maps one to one onto a `ResourceClaim` for `dra` classes (section 10), so the tenant API does not change when holding moves to DRA. +- A cluster that keeps raw names allowed has no hard hold guarantee, also in step 2. + +## Revisit if + +VEP 344 is implemented and a DRA driver hands out vGPU VFs; then holding moves to DRA. diff --git a/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md b/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md new file mode 100644 index 00000000..92bfeed6 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0017-the-vgpu-entitlement-is-a-tenant-flag.md @@ -0,0 +1,33 @@ +# 0017. The vGPU entitlement is a Tenant flag stamped on the namespace, not a budget key + +- **Number:** `0017` +- **Date:** `2026-10-27` +- **Status:** Accepted +- **Deciders:** `@lexfrei, @kvaps` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) +- **Implemented in:** not yet + +## Context + +A vGPU needs a license, counted per unit, and a tenant that may use vGPU at all gets the vGPU guest driver and the license token. The proposal gave the yes-or-no part to a Tenant field, `gpu.vgpu`, which the tenant chart combines with the parent's value and stamps on the namespace as `namespace.cozystack.io/vgpu`. It lived inside the quota section. @kvaps [asked](https://github.com/cozystack/community/pull/99#pullrequestreview-5616164700) whether a `gpu.cozystack.io/vgpu` budget key, 0 by default, would cover it, since it goes down the subtree like the other GPU keys and can cap the license pool. In Cozystack a missing quota key means no limit. He later [agreed](https://github.com/cozystack/community/pull/99#pullrequestreview-5653244822) that the entitlement is a yes or no, not a quota, and that a Tenant field stamped as a namespace label is the pattern of `namespace.cozystack.io/etcd` and the other module labels, and asked for the flag in its own section with a record that says when to revisit it. + +The license delivery of this proposal copies the hoster's license Secret into the namespaces of entitled tenants, and uses the same label to decide where (section 6). + +## Decision + +The vGPU entitlement stays the Tenant field `gpu.vgpu`, off by default, effective only when the parent is entitled, and stamped on the namespace as `namespace.cozystack.io/vgpu`, where only `"true"` means entitled. It has its own section in the proposal, outside the budgets. The VMI webhook refuses a vGPU class entry or claim without it at every start, and in step 2 the claim order webhook refuses a vGPU order without it. Budgets limit how many vGPUs an entitled tenant uses, through the class keys and the key over every class. + +## Why not the alternatives + +- **A vGPU budget key with 0 by default.** A missing key means no limit for every other key, so a tenant without the key would read as unlimited to anyone who knows the others, and this one key would mean the opposite. A cap on the license pool is a budget over the vGPU classes and can be added as one, apart from the entitlement. +- **Entitlement on the class.** A class is visible to every tenant and names an offer, not who may take it; per-tenant lists on a class would move tenant administration into the hoster's catalog. + +## Consequences + +- A hoster turns vGPU on from the root tenant down before tenants use vGPU classes. +- A write that names a vGPU class is not refused for the entitlement, only the start; the option source shows the class with a remaining budget of 0. + +## Revisit if + +The platform stops copying license Secrets into tenant namespaces (License delivery in section 6), for example once a license reaches the guest without a copy per tenant, so the flag guards nothing but the use of vGPU classes and a budget over those classes could replace it. diff --git a/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md b/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md new file mode 100644 index 00000000..04261448 --- /dev/null +++ b/design-proposals/gpudevice/decisions/0018-a-vgpu-license-reaches-each-vm-as-a-secret-volume.md @@ -0,0 +1,39 @@ +# 0018. A vGPU license reaches each VM as a Secret volume, copied only into entitled namespaces + +- **Number:** `0018` +- **Date:** `2026-10-29` +- **Status:** Accepted +- **Deciders:** `@lexfrei` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99) +- **Implemented in:** not yet + +## Context + +A vGPU guest needs two things before its driver licenses the GPU: the client configuration token the hoster downloads from its NVIDIA license server, and a `gridd.conf` with the `FeatureType` that fits the profile. The proposal left both to the hoster: in VMs through the guest image or at boot, and in Kairos worker pools through a separate proposal, which copied the hoster's Secrets into the namespaces of entitled tenants and put the token into each pool's bootstrap data. Rotating a token there meant replacing every vGPU machine. + +Golden images in `cozy-public` are open to every tenant: any tenant can clone one and boot it as root. A token baked into such an image is readable by every tenant, entitled or not. The license is a property of the offer, the class, and not of holding, so its delivery belongs to step 1. + +## Decision + +`GPUClass` carries a `license`, the name of the hoster's token Secret in `cozy-system` and the `FeatureType`, `1` by default or `4`, on vGPU classes only. NVIDIA's [client licensing guide](https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/configuring-nls-licensed-client.html) documents `1` for a VM with a vGPU, since "NVIDIA vGPU software automatically selects the correct type of license based on the vGPU type", and its `gridd.conf` template lists `4` as "NVIDIA vGPU for Compute"; `4` has been observed working on a C-series profile, and the field lets a hoster set it per class. The GPU controller copies each such Secret into every namespace whose `namespace.cozystack.io/vgpu` label is `"true"`, adding a `gridd.conf` with the class's `FeatureType`, and deletes the copies when the label, the class or its `license` goes. At VMI creation the VMI webhook adds a KubeVirt `secret` volume for the copy of the VMI's vGPU class, with the volume label `GPULICENSE`, and at every boot a Linux guest copies the token from that disk into a directory on a memory file system and puts a `ClientConfigTokenPath` line naming that directory and the `FeatureType` into its `gridd.conf`, so the token stays off its root disk; `vgpu-vms` covers Windows guests, whose token folder is on the system disk. VMs and pool workers work the same way, and neither chart changes for the license. Rotation is a VMI restart. + +## Why not the alternatives + +- **A token baked into a shared guest image.** Anyone who can clone the image reads the token, and every tenant can clone a golden image. Rotation means rebuilding the image and every disk cloned from it. +- **A license server reachable only from tenant networks, with the token in the image**, the way KMS serves Windows images. It limits where a leaked token can lease licenses, but the token still reaches every tenant that clones the image, and the network rule has to follow every tenant network. +- **The token in the VM's cloud-init or the pool's bootstrap data.** The data lives in values a tenant reads, and bootstrap data cannot change on a running node, so rotation replaces every machine. + +## Consequences + +- Only the guests of entitled tenants get the token, and they get it as a disk, so a tenant's root user inside its VM can read it; that is what licensing a guest needs. +- A guest image or a worker image with the vGPU guest driver mounts the `GPULICENSE` disk at boot; an image that does not ignores the disk and is licensed the way it was, or not at all. +- The GPU controller writes Secrets in tenant namespaces, through the cluster-wide Secret write it already has. +- A token names the allocation that serves it, and NVIDIA's license system documents no way to revoke one before its expiry, which defaults to 12 years. Rotation therefore moves to a new allocation or license server instance and disables leasing on the old one, after which a remaining guest keeps its lease at most for that allocation's maximum lease duration ([NLS user guide](https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html), sections 4.4, 4.6 and 4.7). +- Clearing the entitlement takes effect at each VM's next start. A running guest renews its lease while the allocation serves its token, and a released client can check a license out again while the token is served (NLS user guide, section 4.9.1), so an immediate cut-off also stops the tenant's vGPU VMs. +- Every entitled namespace gets a copy for every licensed class, so a token does not tell tenants apart, and the platform cannot give one tenant a token of its own. +- A licensed passthrough GPU for graphics still gets its token from a private image the hoster gives the tenants it licenses, not from the shared golden images, since a passthrough class carries no `license`. + +## Revisit if + +KubeVirt or the NVIDIA stack delivers a license to a guest without a Secret in the tenant namespace, or hosters need licensed passthrough classes for graphics (`FeatureType` 2), which would turn the vGPU entitlement into an entitlement to licensed classes. Drop `featureType` if `1` proves sufficient on every C-series profile the platform ships.