Repository navigation
docs(design-proposals): add vGPU licensing for virtual machines proposal - #100
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
3176d84 to
0cbd40c
Compare
Hosters that sell GPUs need to offer tenants VM images with a working guest driver, but the vGPU guest driver may not be published and the golden image collection can only fetch from anonymous HTTP. The proposal keeps the image on the hoster's side: private sources and a GPU marking for golden images, an additive extraImages list, and the marking shown in the image picker. The VM takes its GPUs as device entries backed by GPUDevice, so the proposal is implemented after the GPU device proposal. It also states what a GPU VM does on a node drain and after a node failure. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
0cbd40c to
c5b99df
Compare
Andrei Kvapil (kvaps)
left a comment
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Let's come back to this one after GPUDevice (#99) is implemented. Until then a hoster can add GPU guest images by hand, as a DataVolume in cozy-public (hack/cdi_golden_image_create.sh), with any CDI source including secretRef and certConfigMap.
Notes for later:
- I'd leave per-tenant visibility of GPU images out, including the open question about limiting clones of
mode: vgpuimages to entitled namespaces. A shared golden image with the token baked in and a license server reachable only from tenant networks is enough, the same way KMS works for Windows images. - helm-controller takes over an existing object by default, so hand-made images should get names that a future
extraImagesentry won't reuse. - I checked the CDI v1.66.1 claims and they hold, including
tls.crtfor nbdkit.
Andrei Kvapil (kvaps)
left a comment
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Thinking about it more, I don't see what problem this solves for GPUs specifically. A GPU VM already works with #99: vm-instance with a device entry, booted from an image the hoster adds by hand as a DataVolume in cozy-public. Nothing in #99 waits for this one, so it shouldn't block the GPUDevice implementation.
Of the three changes, only the gpu marking is GPU-specific, and it only changes a label in the picker, section 3 says it enforces nothing. I'd keep the image lists flat and not add it.
The other two are not about GPUs. Pulling an image from a private source with secretRef and certConfigMap, and adding images without restating the defaults, are useful for any golden image. Same for a license key baked into a shared image: a Windows image with a KMS key has the same problem. If we need that, let's have a generic proposal for golden images, and it should stay with the standard RBAC model rather than visibility by namespace labels.
The GPU proposal this one builds on renamed GPUDevice to GPUClaim and its gpus entries to claim and claimTemplate, so the references here use the new names. "Device" stays where it means the GPU itself. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The GPU proposal this one builds on no longer refuses a VM whose claims cannot be placed yet: the launcher pod waits at a scheduling gate until they can, so the KubeVirt create backoff described here no longer applies. It also no longer names a host OS for vGPU hosts, and its first rollout step adds template claims as well as held ones. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
Hi! We removed Sorry for the extra work, I wanted to mention this now so it doesn't come up late in review. |
Proposal status and dates now come from the PR and Git history. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The gpudevice proposal now lands in two steps: a VM names a GPUClass and holds nothing in step 1, and references held GPUClaims only in step 2. GPU VMs follow it and need only its class entries. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Timofei Larkin (lllamnyp)
left a comment
Timofei Larkin (lllamnyp)
left a comment
There was a problem hiding this comment.
Thanks for this. I'm reading it alongside Andrei Kvapil (@kvaps)'s review, which I agree with, and I'd like to add a few points on the shape before line-level review.
Hosters can already add images today. There are two documented ways: override images[] on the cozystack.vm-default-images Package, or create a DataVolume named vm-default-images-<name> in cozy-public with any CDI source (hack/cdi_golden_image_create.sh, and the Windows golden image guide uses the same route). The picker and vm-disk both match on that name, so a hand-made image is a full golden image. Could the problem statement say that? As written, it reads as if hosters had no way to offer a GPU guest image.
A private source doesn't need a public URL. CDI's importer runs inside the cluster, so a url reachable only from the cluster network works with today's chart. secretRef and certConfigMap matter only when the image server asks for credentials or uses a private CA, and a hand-made DataVolume already accepts both.
On extraImages: I understand why it's there, since Helm replaces lists, so adding to images means restating the defaults. But I'd rather not grow one values file that holds every image. I'd prefer the default set to stay a package of its own, and hoster images to be separate objects the hoster manages however they like. That needs one generic change: the picker recognises golden images by a label on the DataVolume/PVC in cozy-public rather than by the vm-default-images- name prefix. Then any DataVolume a hoster applies, from any CDI source, is a first-class golden image, and nothing GPU-specific is needed for it.
On the marker: an annotation or label is the right shape for "this image carries the vGPU guest driver for branch R595". It shouldn't grow into typed fields or a GPU-specific API, because the next request will be nested virtualization or some other device, and GPUs aren't special in that respect. (On the wider question of a dedicated GPU API, see my latest comment on #99. §4 and §5 here build on that model, so I'd hold off elaborating them until #99's shape settles.)
What I'd most like this proposal to cover is licensing, because that's the genuinely GPU-specific part, and today it's one sentence: the hoster "decides whether the token is baked into the image or injected at boot". A GPU-ready image doesn't tell us how a VM gets licensed. Concretely:
- Who is the licensee: the hoster's NVIDIA entitlement and license server, the tenant's own (bring-your-own-license), or both?
- How does the client configuration token get into the guest? The tenant pastes it in over SSH? The tenant passes it through
vm-instancecloud-init? The hoster bakes it into a shared image? Then every tenant who can clone the image can read it, rotating it means rebuilding the image, and our own Windows golden image guide says never to bake licences. Or does the platform inject it on the hoster's behalf for entitled tenants? #101 does this for Kairos workers (it copies the token into entitled namespaces, rendersgridd.conf, and rotates by revision); should VMs reuse that, and if not, why not? - How does a VM on an isolated tenant network reach the license server?
- How does the hoster map license usage, which NVIDIA counts per concurrent client, back to tenants?
- What does a tenant see when the guest is unlicensed or the token is revoked, and how does rotation reach running VMs?
So I'd suggest picking one of two directions:
- A generic golden-images proposal, as Andrei Kvapil (@kvaps) suggested: label-based discovery, private sources (auth, CA, a
registrysource), and no GPU fields. GPU guest image guidance becomes a docs page. - A GPU-focused proposal that drops the image plumbing (hosters can already add images) and spends its length on licensing delivery for VMs, consistent with #101.
Doing both in one document is what makes it long without settling either. §5's restatement of #99's node-failure rules and the private-endpoint details could largely go either way.
Small things: per #111, the Status and Date header fields can go, and the ## Decisions heading is empty.
|
Andrei Kvapil (@kvaps) Timofei Larkin (@lllamnyp) I'm taking the second direction: a GPU proposal about licensing for VMs, without the image plumbing. Hosters can already add a GPU guest image by hand or through the What stays is the part that is specific to GPUs: who holds the license, how the client token reaches the guest, how a VM on a tenant network reaches the license server, how usage maps back to tenants, and what happens on rotation or revoke. It will be written to match #101, or say why VMs need something else. I'll rewrite it once the shape of #99 settles, since sections 4 and 5 build on it. The header fields and the empty Decisions heading get fixed in the same push. |
Hosters can already add GPU guest images by hand or through the images[] override, so the golden image plumbing, the gpu marking and per-tenant visibility go. What is specific to GPUs is licensing: the hoster's license delivered by gpudevice to entitled tenants as a disk, a tenant's own license on classes without one, the guest contract, the network path to the license server, usage reconciliation, rotation, revocation and what an unlicensed guest shows. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin (lexfrei)
changed the title
|
Timofei Larkin (@lllamnyp) the rewrite is up at 5473ee2: the image plumbing is gone, and the proposal now covers vGPU licensing for VMs only, built on section 6 and record 0018 of #99. Your CHANGES_REQUESTED is against the old text; could you take another look? |
Timofei Larkin (lllamnyp)
left a comment
Timofei Larkin (lllamnyp)
left a comment
There was a problem hiding this comment.
Thanks for the rewrite. This is the proposal I was hoping for: it settles who holds the license, how the token reaches the guest, how usage maps to tenants and what a tenant sees, and it leaves the image plumbing out. What's left is mostly about altitude.
1. Keep the design at the design level. Line 12 says NVIDIA's stack changes quickly. That's a good reason to keep vendor specifics on the documentation page, where they can be updated without amending the proposal. Concretely:
- §3 and the related open question and alternative (lines 30, 81, 85, 162, 169, 173): the proposal concludes that a DLS can't run in the management cluster because tenant namespaces can't reach it today. That's a technicality, not a design constraint: a hoster who wants that setup adds a network policy towards it. I'd reduce §3 to "a guest must reach the license server named in its token; where the hoster runs it and how it opens the path is the hoster's call; a guest that can't reach it stays unlicensed (§6)", and drop lines 30, 85, 162 and 173 and the network half of 169.
- Ports (28, 81, 146), lease-release limits and filters (99), degradation timings and fps figures (108–111), the driver-compatibility matrix exegesis (118), the token-expiry caveat for specific vGPU releases (120), and the NGC and version-number details (122) would all sit better on the docs page this proposal adds. In the design, one line each is enough: stuck leases can be released on the license server; an unlicensed guest runs degraded under NVIDIA's rules and recovers once licensed; host vGPU Manager upgrades are checked against the guest drivers in use.
- Lines 121 (Unified Memory and debuggers) and 124 (disk size, golden-image name prefixes) aren't about licensing and can go. Line 24 can shrink to one sentence that hosters already add images through the package or by hand.
- §5 (93–97) and the Linux half of §2 (70) restate #99 §6 and record 0018, now merged. Linking them avoids the two texts drifting. For tenant-visible states (106, 145), the outcome is enough without the event names, which #99 owns.
- The rollback section (133) could be two sentences: rolling back
gpudeviceremoves the disk, and guests that follow §2 then boot unlicensed.
2. One image for licensed and bring-your-own classes. As described (line 62), the reference scripts make an image unusable on a bring-your-own class. Could the guest contract say the scripts change nothing when the GPULICENSE disk is absent, beyond removing the hoster's token, so that one image serves both kinds of class?
3. Windows. Line 71 notes that the license service may read the previous boot's token before the script runs, and that what happens then is undocumented. Either the reference script guarantees it runs first, or the proposal states in Security that a restarted Windows guest without the disk can still hold the last token, so rotation and revocation are weaker there than on Linux.
4. For an incompatible guest driver after a host upgrade (line 148), an in-guest driver update path for entitled tenants would be a gentler documented remedy than rebuilding VMs.
Line numbers refer to 5473ee2.
Vendor release details belong in the hoster guide, where they can change without amending the design. Keep the accepted delivery contract as the source of truth and make the Windows token limitation explicit. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
Timofei Larkin (@lllamnyp) I shortened the design and kept the vendor details for the hoster guide. The shared-image suggestion changes the contract already accepted in |
This proposes licensing guidance and reference guest scripts for vGPU VMs, on top of the accepted
gpudevicedelivery contract. It covers the licensee, server reachability, usage reconciliation and failure recovery. Vendor-specific settings belong in the hoster guide.The hoster chooses where to run DLS and how guests reach it. Windows has an explicit limitation: the licensing service may read the previous boot's token before cleanup. A tenant can update an incompatible driver inside the guest instead of recreating the VM.
The suggestion to use one image for platform licensing and BYOL needs a separate decision on guest configuration ownership. This PR keeps the accepted contract unchanged. Should Cozystack read license server counts beside tenant usage, or leave reconciliation to the hoster's billing system?
Before review
decisions/directory, or says below why none is needed.This is a new proposal, revised before merge. It links the accepted delivery contract without changing it.
DCO
git commit --signoff).