docs(design-proposals): add GPUDevice proposal

design-proposals/gpudevice/decisions/0001-a-claim-outside-permittedhostdevices-is-refused-at-admission.md

@@ -0,0 +1,35 @@
+# 0001. A class whose resource name KubeVirt does not permit is refused at VMI admission
+
+- **Number:** `0001`
+- **Date:** `2026-10-20`
+- **Status:** Accepted
+- **Deciders:** `@lexfrei`
+- **Proposal:** [`../README.md`](../README.md)
+- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99)
+- **Implemented in:** [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (the class condition), [`cozystack/cozystack#5030`](https://github.com/cozystack/cozystack/pull/5030) (the refusal at VMI admission)
+
+## Context
+
+On the `default` and `vgpu` variants the platform writes a `permittedHostDevices` table into the KubeVirt CR whenever it has entries. The proposal assumed that KubeVirt refuses a GPU outside that table when the VMI is admitted. It does not. In KubeVirt v1.9.0 no admitter refuses a GPU or host-device name the table leaves out; the GraceIOVirtualization admitter reads the table only to apply its rules to names listed there, and the refusal is `validatePermittedHostDevices` in [`renderresources.go`](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/services/renderresources.go), which runs when virt-controller renders the virt-launcher pod. A render error creates no pod, and the VMI stays `Pending` with `Synchronized=False`, reason `FailedCreate`, retried with backoff. It never reaches a final phase.
+
+A VM that names such a class would start and never run, with nothing telling the tenant why at start. With ordered claims (step 2) it is worse: a claim leaves `Attached` only when its VMI is final or removed, so a claim of a class whose `resourceName` the table leaves out would stay attached to a VMI that never runs, held and in the usage records, until someone deletes the VMI by hand. A hoster gets there easily: a class is created for a new resource name before the table lists it, or the table is edited and drops an entry.
+
+## Decision
+
+The VMI webhook reads the effective table from the KubeVirt CR, the resource names of its `pciHostDevices` and `mediatedDevices` entries, and refuses a VMI with a class entry or a claim of a class whose `resourceName` is not in it, with a message naming the class and the name. An unset table means KubeVirt checks nothing, and then neither does the webhook. The controller also sets a `ResourceNameNotPermitted` condition on such a class, so the hoster sees the gap before a tenant hits it.
+
+## Why not the alternatives
+
+- **Rely on KubeVirt's own refusal**, as the proposal did. Rejected on the mechanism above: the refusal comes after admission, as a `Pending` VMI that holds its claims, not as an error the tenant sees at start.
+- **Treat a `Pending` VMI with `Synchronized=False` and `FailedCreate` as final, and collect it.** It would release the claims, but only after the tenant has started a VM that cannot run, and the reason would surface as a VMI condition rather than in the start request. `FailedCreate` is also the reason KubeVirt gives for other failures to render or create the pod, so the rule would have to parse messages to tell this one apart.
+- **Refuse such a class when it is created.** A class may legitimately come before its table entry, and the table can change after the class exists, so a create-time check catches only one of the two paths.
+
+## Consequences
+
+- A tenant learns at start, from the webhook message, that the offer is not usable yet; in step 2 the claim stays unattached and moves by the usual rules.
+- The webhook and the controller condition read the same lists of the table, so the condition predicts the refusal. KubeVirt's own check also covers `usb` entries, which no GPU class uses.
+- Every VMI admission with class or claim entries reads the KubeVirt CR from the API server, not from the controller's cache: the replicas serve admission and a lagging cache could admit what should be refused.
+
+## Revisit if
+
+KubeVirt starts refusing a claim outside `permittedHostDevices` at VMI admission, or the table is dropped for DRA-backed devices.

design-proposals/gpudevice/decisions/0002-gpuclaim-serves-vms-only-where-gpu-operator-serves-gpus-to-vms.md

@@ -0,0 +1,39 @@
+# 0002. GPU classes serve VMs only where gpu-operator serves GPUs to VMs, while claim finalizers and usage records always run
+
+- **Number:** `0002`
+- **Date:** `2026-10-21`
+- **Status:** Accepted
+- **Deciders:** `@lexfrei`
+- **Proposal:** [`../README.md`](../README.md)
+- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99)
+- **Implemented in:** [`cozystack/cozystack#5008`](https://github.com/cozystack/cozystack/pull/5008) (the gate and the policies under it), [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (finalizers and usage records on every cluster), [`cozystack/cozystack#5030`](https://github.com/cozystack/cozystack/pull/5030) (the webhooks, the `vm-instance` refusal and the refusal of `forbidden`)
+
+## Context
+
+The proposal rolled step 1 out "on the existing variants" and said nothing about a cluster where GPUs do not reach VMs: gpu-operator under the `container` variant, gpu-operator not enabled, or no iaas bundle. On such a cluster KubeVirt runs without the `HostDevices` feature gate and without a `permittedHostDevices` table, and nothing there resolves a claim reference.
+
+The `vm-instance` and `kubernetes-nodes` charts render only the entries, on the VMI template, and leave it to the VMI webhook to fill in the GPUs (section 4). Without the webhook a class or claim entry would boot the VM with no GPU and no error. The CRDs, on the other hand, ship with the controller on every cluster, so a `GPUClass` or, in step 2, a `GPUClaim` can exist on a cluster that has no placement running, and in step 2 both carry finalizers.
+
+## Decision
+
+The platform treats GPUs as serving VMs when the iaas bundle is on, `cozystack.gpu-operator` is enabled, and its variant is `default` or `vgpu`. Only there does it turn on the VMI and virt-launcher pod webhooks, the class status controller and its metrics scrape, and in step 2 the binder, the controller's watches on KubeVirt objects, the GPU admission policies and the tenant admin's write access to `GPUClaim`. Elsewhere:
+
+- `vm-instance` and `kubernetes-nodes` refuse to render class and claim entries, through a cluster value the platform writes only where GPUs serve VMs, and `name` entries work as today.
+- In step 2 the platform refuses `gpu.rawResourceNames: forbidden`, because the policy that enforces it does not render there.
+- In step 2 the controller still adds and removes the finalizers of `GPUClass` and `GPUClaim` and keeps the usage records of ordered claims, so deleting a class, a claim or a tenant namespace never hangs.
+
+## Why not the alternatives
+
+- **Run everything on every cluster.** The webhooks and the binder read KubeVirt objects, which do not exist without KubeVirt, and under the `container` variant the GPUs gpu-operator advertises are meant for pods, not VFIO devices a VM can take. A claim bound there would be held and recorded for a GPU no VM can use.
+- **Gate the finalizer paths with the rest.** A hoster who disables GPUs, or moves to the `container` variant, would then leave every existing class and claim stuck on its finalizer, and with it the tenant namespaces that hold claims.
+- **Let the charts render class and claim entries anyway and rely on the webhook.** On these clusters no webhook exists, so the failure is a VM that runs without the GPU it asked for, which the tenant may notice only from inside the guest.
+
+## Consequences
+
+- On a cluster without GPUs for VMs, a tenant reads the catalog, and in step 2 its claims, but cannot use them, and a VM with class or claim entries fails at render with a message that says GPUs for VMs are not served.
+- Turning GPUs for VMs on writes one new cluster value, which every tenant release reads, so each upgrades once, and only on such clusters.
+- In step 2 the controller keeps a resync loop while a deleted claim waits for its VMI, because with the feature off nothing watches VMIs.
+
+## Revisit if
+
+GPUs reach VMs through another path than gpu-operator's VM variants, for example a DRA driver installed on its own (section 10).

design-proposals/gpudevice/decisions/0004-gpunode-health-is-kubelet-allocatability.md

@@ -0,0 +1,42 @@
+# 0004. GPUNode health is what kubelet lists as allocatable, and the node reports its GPU resource names and per-function counts
+
+- **Number:** `0004`
+- **Date:** `2026-10-20`
+- **Status:** Superseded by [`0015`](./0015-node-allocatable-is-the-gpu-inventory.md)
+- **Deciders:** `@lexfrei`
+- **Proposal:** [`../README.md`](../README.md)
+- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99)
+- **Implemented in:** [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998) (the fields), [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013) (the node agent)
+
+## Context
+
+Section 1 has the node agent write `GPUNode` from sysfs and kubelet's pod-resources API, with a `healthy` flag per PCI function, and section 2 decides lost capacity from devices the agent reports missing or unhealthy. Section 3 adds every resource name a GPU device plugin advertises to the GPU resource names, "as the agent reports it". Building the agent showed three gaps in that picture.
+
+The pod-resources API carries no device health. It lists the allocatable devices per resource name and the devices each pod holds, and kubelet lists as allocatable only the devices their plugin reports healthy.
+
+One PCI function is not always one unit. An mdev parent serves several plugin devices, so counting one healthy unit per function would make a node with more than one held mdev device look as if it had lost capacity.
+
+Not every GPU device ID maps to a PCI function. The GPU Operator's device plugin of the `container` variant reports NVML UUIDs, `GPU-…` or `MIG-…`, which sysfs cannot resolve, so the agent has no function to put such a GPU under, and section 3 still needs its resource name, such as `nvidia.com/gpu`.
+
+## Decision
+
+- A function is healthy when every plugin device on it is allocatable, and unhealthy when a pod that has not finished holds one of its devices that is not. A function no plugin device maps to has no health.
+- Each function reports `allocatable`, the number of its plugin devices kubelet lists as allocatable: 1 for a VF or a GPU passed through whole, N for an mdev parent. A plugin device that names a whole IOMMU group counts once, on the group's lowest-addressed GPU function, and 0 on its other functions. The controller sums it over the functions that carry the resource name and caps the sum at the node's allocatable count from kubelet, which alone counts for a held name no function carries. A node has lost capacity for a resource name when it holds more units than that.
+- `GPUNode.status.resourceNames` lists every resource name on the node whose devices are GPUs, including names whose device IDs are NVML UUIDs the agent cannot place under a function. The names join the cluster's GPU resource names of section 3.
+- A function serving several pods, such as an mdev parent, names one of them in `usedBy`, the smallest by namespace and name, so the report is stable.
+
+## Why not the alternatives
+
+- **Read health from each device plugin's socket.** The agent would talk to every plugin the host layer runs, while kubelet already folds their health into what it lists as allocatable, and kubelet's view is the one the scheduler acts on. A judgement call, not a hard limit.
+- **Count one unit per function.** Wrong for mdev parents, as above, and it would fire lost capacity on nodes that have lost nothing.
+- **Leave GPUs the agent cannot place out of the report.** Section 3 then misses `nvidia.com/gpu`, so a VMI or a pod that names it would not count as carrying a GPU entry.
+
+## Consequences
+
+- An unhealthy device that no pod holds disappears from the allocatable list and looks the same as a missing one. Both lower `allocatable`, and both become lost capacity after the grace period once the node holds more units than remain.
+- `usedBy` traces only one pod per function. Section 1's "ownership is exact for running GPUs" holds per function for VFs and whole GPUs, and for an mdev parent only through the claims' own `status.attachedTo`.
+- The agent does not fill `profile` yet; writing and reading VF profiles is host-layer work (Rollout, step 4).
+
+## Revisit if
+
+kubelet's pod-resources API starts reporting device health, or a device plugin that Cozystack ships reports IDs the agent can neither resolve through sysfs nor recognise as NVML UUIDs.

design-proposals/gpudevice/decisions/0005-the-taint-rule-skips-exactly-three-kubernetes-taints.md

@@ -0,0 +1,31 @@
+# 0005. The taint rule skips exactly the three taints that mirror a cordon or readiness
+
+- **Number:** `0005`
+- **Date:** `2026-10-20`
+- **Status:** Superseded by [`0010`](./0010-gpuclass-carries-the-tolerations-binding-reads.md)
+- **Deciders:** `@lexfrei`
+- **Proposal:** [`../README.md`](../README.md)
+- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99)
+- **Implemented in:** [`cozystack/cozystack#4998`](https://github.com/cozystack/cozystack/pull/4998)
+
+## Context
+
+Section 2 kept binds, moves, rebinds and gathers off nodes with a `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate, and left out "the `node.kubernetes.io/*` taints Kubernetes sets itself for a cordon or a node that is not `Ready`", because the cordon and readiness rules already cover those nodes. Read as a prefix, that sentence would also skip the other well-known taints Kubernetes defines under `node.kubernetes.io/`: `memory-pressure`, `disk-pressure`, `pid-pressure`, `network-unavailable` and `out-of-service`. None of them mirrors a cordon or readiness, and a node carrying one of them as `NoSchedule` or `NoExecute` would still be bound to, though the scheduler keeps a virt-launcher pod that does not tolerate it off the node.
+
+## Decision
+
+The rule skips exactly `node.kubernetes.io/unschedulable`, `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`. Every other `NoSchedule` or `NoExecute` taint that virt-launcher pods do not tolerate counts, under `node.kubernetes.io/` or not. The rule and its cases live in `internal/gpu/model/node.go` and `node_test.go`.
+
+## Why not the alternatives
+
+- **Skip the whole `node.kubernetes.io/` prefix.** It binds claims onto nodes that the scheduler then refuses, and keeps their units in `free`, which is the failure the taint rule exists to prevent.
+- **Count the three as well.** Kubernetes sets `node.kubernetes.io/unschedulable` on every cordoned node, so a cordoned node would also read as tainted. A taint keeps a bound claim where it is, so under `Move` a cordon would never move a claim, and `onCordon` would stop meaning anything.
+
+## Consequences
+
+- A node under memory, disk or PID pressure, one whose network is unavailable, and one tainted `out-of-service` get no binds, moves or gathers while the taint holds and virt-launcher pods do not tolerate it, and their units leave `free` and `maxFreePerNode`. Outside a gather at VMI creation, a claim already bound there stays, as on any tainted node.
+- A pressure taint comes and goes with the node's condition, so `free` on a busy class can drop and return without any claim moving.
+
+## Revisit if
+
+Kubernetes adds a taint of its own that mirrors a cordon or readiness, or Cozystack's virt-launcher pods start tolerating taints by default.

design-proposals/gpudevice/decisions/0006-a-stay-pin-yields-only-to-a-node-that-cannot-serve-the-claim.md

@@ -0,0 +1,32 @@
+# 0006. A `Stay` pin holds through `NotReady` and taints, and breaks when the node cannot serve the claim
+
+- **Number:** `0006`
+- **Date:** `2026-10-20`
+- **Status:** Accepted
+- **Deciders:** `@lexfrei`
+- **Proposal:** [`../README.md`](../README.md)
+- **Decided in:** [`cozystack/community#99`](https://github.com/cozystack/community/pull/99)
+- **Implemented in:** [`cozystack/cozystack#5013`](https://github.com/cozystack/cozystack/pull/5013); the gather after admission of [0012](./0012-the-vmi-webhook-checks-and-records-and-the-controller-prepares.md): not yet
+
+## Context
+
+Section 3 pinned a VMI to a cordoned node when a claim of a `Stay` class was there and all of the VMI's claims were on that node, with no template claims: there was no gather, "even if the node is also not `Ready`", and the VMI waited. Section 2 had a `Stay` claim keep its node through a cordon, while lost capacity, a node that is gone and a node that no longer serves the class were hard reasons for a claim to leave. The proposal did not say which of the two wins when a pinned node also has a hard reason, so a node under maintenance that then died could hold a VM's claims, and keep them in the usage records, for as long as the dead node stayed registered.
+
+## Decision
+
+The pin holds while the node's only problems are the cordon, a `NotReady` spell shorter than the lost-capacity grace period, or a taint its classes do not tolerate. It breaks when a member of the set on that node has lost capacity there, when the node is gone, or when it no longer serves a member's class. Then the set is treated as any set with a hard reason to leave (section 9): a new VMI's set is gathered onto one node with room for all of it while its launcher pod waits at the gate, or the pod stays gated when no node fits; a VMI whose pod is already pinned there is deleted when its claims must leave, and the unattached set moves together, or goes to `Lost` together when no node has room. The gather at VMI start and the controller's background pass read the pin through one function, so a VMI is never pinned where the next pass deletes it.
+
+## Why not the alternatives
+
+- **The pin always wins.** A `Stay` class exists for a hoster who wants VMs back on the same hardware after maintenance. A dead node is not coming back to the same state on any known schedule, and the VM's claims would stay held and recorded against hardware that serves nothing.
+- **Break the pin on `NotReady` alone.** It undoes the point of `Stay` for the most common maintenance step, a reboot, and step 2 already uses the lost-capacity grace period so that a short outage moves nothing.
+- **Break the pin on any taint.** Apart from a gather at VMI creation, a taint never moves a bound claim; it only keeps new binds off the node (section 9). Breaking the pin on one would make a `Stay` claim leave in a case where a claim of a `Move` class on an uncordoned node stays.
+
+## Consequences
+
+- A `Stay` VM waits through a reboot of its cordoned node and comes back on it, and after a node failure its claims move like any other VM's once the grace period has passed.
+- A pinned VM waits on a dead node for the lost-capacity grace period, five minutes by default, before its claims can move.
+
+## Revisit if
+
+Hosters ask for a `Stay` policy that also outlasts a node failure, for example to wait for a repaired node rather than move off it.