[Aleksei Sviridkin (lexfrei)]

Design proposal for GPUs in Kairos worker pools of tenant Kubernetes clusters, in vGPU mode and in whole-GPU passthrough. Cozystack cannot ship the NVIDIA vGPU guest driver, since it is licensed per customer and may not be published, for example in a public image or repository. So the proposal adds an extension point instead of a driver. Cozystack ships a build recipe that layers a hoster-supplied driver .run onto the platform Kairos worker image, one image per mode. The hoster names those images in platform values, together with the vGPU license Secret and, when needed, registry credentials and a CA. A Kairos pool then boots the image of its mode, and vGPU pools get the license files.

This proposal is implemented after GPUDevice (#99), one of two related proposals. Kairos pools take GPUs only through GPUDevice's deviceSelector or deviceTemplate, never through raw resource names. vGPU is allowed only for tenants whose GPUDevice gpu.vgpu flag holds. The hoster sets that flag on top-level tenants and each parent tenant on its children, and it holds only when it also holds on the parent. GPUDevice also owns the host layer and usage records of allocated GPUs (model, count, owner) that an external billing system reads. GPU virtual machines (#100) covers tenant virtual machines. Existing Talos GPU pools are left as they are.

It builds on Kairos pools in kubernetes-nodes, from cozystack/cozystack#4911 (draft). Their bootstrap provider needs Cluster API 1.13.4 or newer, and cozystack/cozystack#4902 moves Cozystack's Cluster API to v1.13.7.

What is verified and what is not:

• Verified: a Hadron guest with the 595 vGPU guest driver was licensed against an NVIDIA License System server (C-series profile) and ran CUDA in a VM. The provider's KairosConfig files has no contentFrom at b19dd350e84a, so the token needs either an upstream change or a Cozystack-side substitution (Design §3).
• Verified on a test VM: Kairos applies the userdata it saved at install on every boot and does not re-read a changed datasource. Token rotation therefore replaces machines (Design §5).
• Not verified: the data center driver in a Hadron guest with a passthrough GPU; whether Cozystack may publish the passthrough image itself; which misuse restrictions DLS and CLS support; whether the gpu-operator default finds a driver installed in a glibc prefix and nvidia-ctk through a boot-time link; whether HAMi works on these pools. All of these are listed under Open questions.

I would like review mainly on the per-mode images and why the pool states its mode in kairos.gpuMode instead of taking it from the device class (§2, §3), on the token path and rotation (§3, §5), and on where the hoster setting lives (§2).

Before review

• If this revises a merged design proposal: it adds a decision record under that proposal's decisions/ directory, or says below why none is needed.

This is a new proposal and revises none.

DCO

• Commits are signed off (git commit --signoff).

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: b251328b-a81c-4f0a-9fa8-4d44508ea63a

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Andrei Kvapil (kvaps)]

I'd postpone this one until we decide on Kairos for tenant worker pools. A second worker OS is a bigger decision than GPU: its own image pipeline, a young CAPI bootstrap provider and a migration path for existing Talos pools. Today it comes as code in cozystack/cozystack#4911 without a design proposal. Let's have that proposal first, and then come back to how GPUs attach to Kairos workers.

Two things for later, no need to address them now:

• workerImages entries must match the chart's pinned patch, so every Cozystack upgrade that moves the patch breaks the render of all GPU pools until the hoster rebuilds, and it can't be prepared beforehand. Taking the Machine version from the hoster's entry would avoid it.
• A KubeVirt secret volume for the token. CAPK appends its config drive to the template volumes instead of replacing them (pkg/kubevirt/utils.go:144-155), so the token doesn't have to be inline in KairosConfigTemplate, and rotation becomes a VMI restart instead of replacing every vGPU machine.

[Andrei Kvapil (kvaps)]

Read it as if #99 had landed, and the blocker is still the one from my previous comment: Kairos pools need their own proposal first. cozystack/cozystack#4911 also carries downstream patches to the Kairos CAPI provider, and this proposal only mentions the merged upstream kairos-io/cluster-api-provider-kairos#141. That proposal should say what we carry and for how long.

I'd also like that proposal to move the OS settings off the pool into a hoster-side class, NodeGroupClass in kubernetes.cozystack.io/v1alpha1, like StorageClass: the OS, the worker image, the GPU mode, and the references to the license and registry Secrets. Tenants can't create Secrets through the Cozystack API, so a Secret reference on a pool is really a hoster setting anyway. With a class the platform reads the hoster's Secrets directly, and the per-tenant copies, the entitlement check on them and licenseRevision go away. The class should be immutable, a new version is a new name, otherwise one change rolls the pools of every tenant.

That also removes kairos.gpuMode. If mode on GPUClass is immutable and the pool entry names the class (className in deviceSelector), the chart can take the mode from the class.

For when we come back to it:

• The license token is just a file, so the KubeVirt secret volume from my previous comment should be the main path. Rotation becomes a VMI restart.
• gpu=on is set for HAMi in both modes. Is HAMi meant to share a whole passthrough GPU between pods there?
• cozystack/cozystack#4949 counts gpu-count as the number of gpus entries. With a count on devices, scale from zero will see the wrong number of GPUs per node.

[myasnikovdaniil]

Hi! We removed Status and Date from proposal headers in #111. Could you remove these fields when you next update your proposal? Decision records keep their metadata.

Sorry for the extra work, I wanted to mention this now so it doesn't come up late in review.

[Aleksei Sviridkin (lexfrei)]

The Kairos tenant pool proposal kvaps asked for is now #115; #101 waits for it.

[Timofei Larkin (lllamnyp)]

Thanks for this. The Hadron findings in Context (the tmpfiles.d entry, ordering nvidia-modprobe before nvidia-persistenced, the glibc prefix) are exactly the hands-on detail a hoster needs, and I'd like to keep all of it. I know #101 is waiting on #115. Here's the shape I'd hope it comes back in.

The direction I'd like: worker pools that don't know about GPUs or OSes beyond what they must. A worker is a VM that happens to join a control plane. Whatever a tenant VM can have (a GPU, a licensed guest driver, an image from a private registry) a worker should get through the same mechanism, not through a parallel one in KubernetesNodes. Karpenter on AWS is a good model. The node class names an OS family and an image, the instance type carries the accelerator, and nothing about GPUs is implemented per OS or per node pool.

Measured against that, most of #101 sits in the wrong layer. Roughly a quarter of the Design section is about Kairos: the image recipe (§1), the file entries in the bootstrap template (§3), and gpu-operator on Kairos (§4). The other three quarters have nothing to do with Kairos or with worker pools:

• the license Secret and FeatureType;
• the vGPU entitlement;
• replicating the token and registry credentials into namespaces;
• rotation;
• choosing an image by GPU mode.

A tenant VM with a vGPU (#100) needs every one of these, and today #100 answers them differently: the hoster may bake the token into a shared golden image. If they stay here, the next worker OS and the VM path each get their own copy.

What I'd suggest moving where:

1. GPU mode and license settings belong on GPUClass (#99), not on the worker side. The mode already lives there and can't change (#99, decision 0009). The license also follows the profile: compute profiles are licensed through NVIDIA AI Enterprise and graphics profiles through vGPU software, and FeatureType depends on the profile type. A reference to the license Secret and the FeatureType on the class would serve VMs and workers alike. I'd go a step further than Andrei Kvapil (@kvaps)'s suggestion of putting the GPU mode and Secret references on NodeGroupClass, and keep them off the worker side altogether.

2. License delivery belongs to the VM, defined once with #99 and #100. I agree with Andrei Kvapil (@kvaps) that a KubeVirt secret volume should be the main path. A KubeVirt secret volume references a Secret in the VM's own namespace, so a per-namespace copy is still needed, but #99's controller already knows which namespaces hold entitled vGPU claims. One option is for #99's VMI webhook, which already resolves claim references at VMI creation, to add the license volume for a vGPU claim. Then a plain VM and a worker get it identically, with no chart change in vm-instance or kubernetes-nodes. Rotation becomes a VMI restart instead of rolling every vGPU worker of every tenant. Tokens stop accumulating in old bootstrap templates. The dependency on upstream contentFrom disappears, and so do licenseRevision and the gate. The only part left for the OS is "mount the volume and put the file where nvidia-gridd reads it", which belongs in the image.

3. Driver images belong to image distribution (#115 for workers, #100 for VMs). A GPU worker image is a hoster NodeGroupClass whose image is the hoster's GPU build. That removes gpu.kairosGuest.workerImages, the per-mode registry copies, and the render failure when the pinned patch moves. Whether an image carries the vGPU guest driver is a property of the image. I'd mark and check it the same way for VM golden images and worker classes. A vGPU VM booted from an image without the guest driver gets a GPU that doesn't work, and #100 doesn't refuse that. A worker pool should be treated the same, or both should get the same check. Keeping the vGPU driver away from tenants without the entitlement is then one requirement on image distribution, and it feeds directly into #115's open question 7 (clone rights on goldens in cozy-public).

4. What stays in #101 is the Kairos guest: the image recipe for both modes, its test stage, the unit that places the license file, and the gpu-operator contract below. On the KubernetesNodes side the only change left would be lifting the gpus refusal on Kairos pools and passing the entries through to the VM template unchanged. That's small enough to be a section of #115 plus a hoster guide, rather than a proposal of its own.

More broadly, KubernetesNodes and VMInstance. The worker pool chart currently has its own copy of the VM shape: instance type or explicit resources, disks, firmware, networking and gpus. #99 then adds GPU references to both charts in different shapes: claim/claimTemplate on VMs, claimSelector/claimTemplate on pools. I'd like worker pools to be "N VMs from a VM template, plus a bootstrap config", where the template is shared with vm-instance, so a VM feature doesn't have to be implemented twice. CAPK takes a full VirtualMachineSpec, so this works with Cluster API as it is. It's a separate proposal, but it's the direction I'd like this one to move toward, not away from.

On the Kairos image itself:

• gpu-operator in mixed clusters. The gpu-operator addon is per tenant cluster, but the OS is per pool. A cluster with a Talos pool and a Kairos pool, which is every cluster moving pools from one OS to the other, runs one gpu-operator whose DRIVER_ROOT, NVIDIA_DRIVER_ROOT and NVIDIA_CTK_PATH must work on both. So "the addon gets an override for Kairos" (L163) isn't available. Could the image be required to meet the existing contract (driver libraries resolvable from /, nvidia-ctk at /usr/local/bin), with the test stage checking it?
• musl vs glibc. The NVIDIA userspace is glibc-linked, and that drives the prefix and most of the open questions (the validator and CDI lookup, HAMi, the data center driver). #115 lists an Ubuntu-based Kairos flavour as an alternative. Would a glibc flavour for GPU images remove most of these? A sentence in Alternatives either way would help.
• Testing. The passthrough image hasn't been tested on Hadron yet. It would be good to run it once, alongside the vGPU path, before the image contract is written down.

[Timofei Larkin (lllamnyp)]

I think #115 supersedes this. A hoster class carries kubernetesVersion, and the Machine version comes from it, so a Cozystack upgrade that moves the patch no longer breaks GPU pools. Could this check go when the proposal moves onto the class?

[Timofei Larkin (lllamnyp)]

#115 avoids copying registry credentials into tenant namespaces by importing catalog disks once into a golden. I'd rather this proposal state its requirement (the vGPU guest driver must not reach tenants without the entitlement) and leave the mechanism to #115 and #100, so VM images and worker images follow one rule.

[Timofei Larkin (lllamnyp)]

Following up on Andrei Kvapil (@kvaps)'s question: what is HAMi meant to do on these pools? On passthrough it would share a whole GPU between pods. On vGPU it would subdivide a slice that's already fractional. If it isn't intended, I'd leave the label off. If it is, it's a GPU-sharing decision that belongs with #99 rather than in the pool chart.

[Timofei Larkin (lllamnyp)]

Small one: the default is 1, but the only live test used 4 on a C-series profile. Could the text say which value has been validated? If the setting moves to GPUClass as suggested in the review comment, the class can carry the value that matches its profile.

[Timofei Larkin (lllamnyp)]

Per #111, the Status and Date header fields can go. The ## Decisions heading (L23) is empty; either drop it or add records.