[Aleksei Sviridkin (lexfrei)]

I propose opt-in Kairos workers for managed tenant Kubernetes, selected through WorkerClass in cozystack.io. The class's CRD, controller and admission ship in a managed-Kubernetes system package rather than core. A hoster can build an immutable worker image with its own driver or module from the platform Kairos image.

Existing pools keep their classless Talos path and render. New application creates get the platform's default class, initially talos. Supported tenant Talos settings and reachability controls remain available as explicit overrides, with fixed-image compatibility checked after resolution. Moving existing pools onto the class-based path is a required later stage with no-rollout proof and rollback, not a dependency of shipping Kairos.

GPU mode and licensing stay with GPUClass, not WorkerClass. Private worker images wait for the shared VM/worker catalog. A future custom bootstrap needs a reusable, rotatable per-cluster join credential and a tenant-writable place for cloud-init.

Isolated tests demonstrated generic admission and HelmRelease persistence using research fixtures. The proposal records those results separately from production package wiring, authorization and the full image contract, which remain to be validated. It does not claim that the generic main storage failures in cozystack/cozystack#5234 were fixed or that the correction was tested.

Before review

• If this revises a merged design proposal: it adds a decision record under that proposal's decisions/ directory, or says below why none is needed.

This is a new proposal. The two forward links added to kubernetes-nodes-split point to it without changing that proposal's phase decisions, so no decision record is needed there.

DCO

• Commits are signed off (git commit --signoff).

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 8229c042-0449-4461-9de3-73447f5babba

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[Timofei Larkin (lllamnyp)]

Thanks for writing this up. It answers a lot of what Andrei Kvapil (@kvaps) asked for on #101, and the provider-patch carry table with its exit rule is exactly what I'd want for a young upstream. The immutable hoster-side class is the right shape, and moving the image and its kubernetesVersion into it removes the shared-default and patch-pin problems #101 had.

My concerns are about where the line between pool, OS and VM sits. They're the same ones I raised on #101, and I'd rather settle them here, since #101 will build on this.

1. Make the class the whole OS boundary, for both OSes. As proposed, Kairos settings move into the class, but Talos settings stay on the pool (Non-goals, open question 8). The pool then carries talos.*, osImage, kernelModules and proxmox.templateTags for Talos, plus a new kairos.sshPublicKey. Every OS adds pool fields, and the chart has to refuse or ignore the other OS's fields. L125 shows the cost: talos.* can't be refused on a Kairos pool, because defaults and a tenant's setting look the same, so it's silently ignored. I'd answer open question 8 with "yes" now. The pool keeps only fields that mean the same thing on every OS: scaling, health checks, version, labels and taints (as #103 already does), the VM shape and gpus. The class carries everything OS-specific, for Talos as well as Kairos. Moving the existing Talos pool fields into classes would be a rollout step. A design that leaves Talos fields on the pool is hard to undo once Kairos pools exist.

2. A worker is a VM plus a join. What a worker actually needs from its OS is a bootstrap provider, a bootstrap template and an image. Everything else is a VM feature that vm-instance already has: firmware, SSH keys, disk import and clone, sizing, devices. This proposal adds EFI firmware handling and kairos.sshPublicKey to the pool chart, next to its existing copy of sizing, disks and gpus. I'd like worker pools to move toward "N VMs from a VM template shared with vm-instance, plus a bootstrap config". At a minimum, a new OS shouldn't add VM-level fields to the pool.

3. One image catalog for VMs and workers. Kairos disks that need credentials go into the Talos worker catalog (kubernetes-worker-image, goldens named talos-worker-* in cozy-public). Meanwhile #100 extends the VM catalog (vm-default-images, goldens named vm-default-images-* in the same namespace) with private sources. A worker disk is a VM disk. Two catalogs means two ways to add a private source, two pull-Secret conventions, and open question 7 (clone rights on goldens in cozy-public) answered twice. Could the class's catalogEntry refer to a golden in one catalog that serves both? For example, goldens found by a label in cozy-public rather than by name prefix, as suggested on #100.

4. Keep GPU fields off the class. "Extending the class" reserves room for fields #99 and #101 will define. As I wrote on #101, I'd keep GPU mode, licensing and Secret references off the worker side altogether: GPU mode and licensing on GPUClass (#99), and license delivery at the VM level (#99/#100). Relatedly, Kairos pools refuse gpus because the platform image has no driver. That's a property of the image, not of the OS. A hoster image with the driver should let gpus pass through to the VM, the same way on every OS.

5. The default OS. The pool defaults to talos, and the names talos and kairos are resolved by the chart without an object. With StorageClass, the analogy the proposal uses, the default is a cluster setting rather than a built-in name. Could the default class be a platform value (or a default-class annotation), so the design doesn't assume which OS stays the default?

6. A new API group. kubernetes.cozystack.io doesn't exist today, and the proposal doesn't say why the class needs a group of its own. cozystack.io already holds a hoster-side class, SchedulingClass. A CRD can't move groups in place, so I'd like a short decision on this: why a new group, and what else is expected to live in it. It's the same question I asked about gpu.cozystack.io on #99.

7. The base distribution versus the stated purpose. The stated reason for Kairos is hosting vendor drivers, and vendor userspace such as NVIDIA's is glibc-linked. That's why #101 needs a glibc prefix, and it's behind most of #101's open questions. Hadron being musl also means Cozystack builds kubelet from source for every minor and architecture. The Ubuntu-based flavour is rejected as "a second base distribution to track", but Hadron is already that second distribution. Could Alternatives weigh a glibc flavour against the driver use case specifically?

8. Substrate × OS. Kairos is refused on Proxmox, and each substrate and OS combination becomes a branch in one chart. If the class is the OS boundary and the substrate is the infrastructure boundary, it would help to write down what an OS has to provide per substrate (image format, bootstrap template), so adding either stays additive.

9. The next OS as a test of the design. Picture adding Ubuntu, Debian or Fedora workers. As Kairos flavours, the class already handles it, which is a real strength. As plain cloud images joined by CAPI's kubeadm bootstrap provider (the path Cozystack used before Talos), the design as written needs:

• a new os value;
• a new reconcile Job for the runtime addresses;
• new pool fields;
• refusal rules against every other OS's fields;
• another naming scheme in the image catalog.

EKS solves this with a documented, OS-neutral join contract (endpoint, CA, node identity), a short list of first-party OS families for managed node groups, and a CUSTOM escape hatch where the user owns the bootstrap. Could #115 do the same?

• Publish what a worker needs to join (endpoint, CA, token source, Service and DNS addresses) in one object per tenant cluster.
• Let the class name a bootstrap family (Talos, kubeadm, Kairos) instead of an os enum.
• Offer a custom class for hoster-supplied images and bootstrap templates.

The platform would then commit to supporting the families it ships, and to the integration for everything else.

[Timofei Larkin (lllamnyp)]

See point 1 of the review: I'd make this a goal, with migrating the existing pool fields as a rollout step. Otherwise the pool's API stays Talos-shaped, and each new OS adds fields beside it.

[Timofei Larkin (lllamnyp)]

Could the proposal say why this is a new group rather than cozystack.io, which already holds SchedulingClass? See point 6.

[Timofei Larkin (lllamnyp)]

I understand the upgrade-safety reason for resolving these without an object. Could the default still come from a platform setting, as a default StorageClass does, rather than being fixed to talos in the chart?

[Timofei Larkin (lllamnyp)]

I'd rather the class not grow GPU fields; see point 4 and my review on #101. Could this section say the class carries OS and image settings only, and leave GPUs to #99/#100?

[Timofei Larkin (lllamnyp)]

This is the same problem #100 solves for VM golden images, in a second catalog in the same namespace. Could both use one catalog? See point 3.

[Timofei Larkin (lllamnyp)]

This is the clearest sign that OS fields shouldn't live on the pool: the chart can't tell a tenant's Talos setting from a default, so it has to ignore it silently. Moving Talos fields into Talos classes (point 1) removes the problem instead of working around it.

[Timofei Larkin (lllamnyp)]

vm-instance already has sshKeys. A break-glass SSH key is a VM concern, not a Kairos one. Could it come from the same field, or go into the class if it's meant to be hoster-set?

[Timofei Larkin (lllamnyp)]

See point 7: given the driver use case, I'd like this weighed on glibc compatibility, not only on the cost of tracking another base. Hadron is already a base Cozystack doesn't track today.

[Timofei Larkin (lllamnyp)]

The ## Decisions heading is empty; either drop it or add records.

[Andrei Kvapil (kvaps)]

I agree with Timofei Larkin (@lllamnyp)'s review, including the group: NodeGroupClass should go into cozystack.io next to SchedulingClass, not into a new kubernetes.cozystack.io. That reverses what I suggested on #101, sorry for the back and forth. Same for GPU fields: with the mode coming from GPUClass and the license from GPUClass.license as a secret volume (#99, 0018), the pool class doesn't need any.

One thing to add: deleting a class that pools use breaks their next render. I'd answer open question 2 now, with a finalizer that keeps the class while pools reference it.

[Aleksei Sviridkin (lexfrei)]

Timofei Larkin (@lllamnyp) I reworked the proposal around your review. The class is now the whole OS boundary: Talos fields move into Talos classes in Rollout step 3, and the migration keeps every pool's render. The class sits in cozystack.io next to SchedulingClass, names a bootstrap family instead of an os, and carries no GPU fields (section 1). kairos.sshPublicKey is replaced by sshKeys with the vm-instance shape, and it stays the one VM-level field on the pool until workers come from a shared VM template. The API writes the default class into the pool at create, and where that default comes from is open question 4. In the numbering you reviewed, old OQ7 (credentialed goldens) now waits for the shared catalog, and old OQ8 (Talos settings in the class) is answered yes. gpus passes through on Kairos pools the same way as on Talos, and the substrate × OS table is in section 3. Alternatives now weighs Hadron against a glibc flavour for the driver case, and the empty ## Decisions heading is dropped.

[Aleksei Sviridkin (lexfrei)]

Andrei Kvapil (@kvaps) Done: NodeGroupClass is in cozystack.io and carries no GPU fields. Open question 2 is closed in section 1, under "Class deletion". A controller in cozystack-controller puts a finalizer on every class except the built-in talos and kairos, and removes it once no pool names the class. Deleting a referenced class leaves it Terminating, and its pools keep rendering. The API also refuses a new pool, or a switch of an existing one, onto a class being deleted. The built-in two get no finalizer, since pools on them resolve without the object. The CRD carries helm.sh/resource-policy: keep, so a revert doesn't delete the classes and leave them stuck behind finalizers. Rollout step 2 lists all of this as changes #4911 needs before it merges.

[Timofei Larkin (lllamnyp)]

Thanks for the rework, and for walking through each point. The class as the whole OS boundary, the bootstrap family in place of os, a class without GPU fields, the substrate × OS table and the finalizer all read well. The deferrals are explicit and written so each one can land later. A few things remain; the first is the one I'd consider blocking.

1. Ship the class with the managed Kubernetes packages, not in Cozystack core. The class serves two apps, Kubernetes and KubernetesNodes, but the proposal puts its machinery into core:

• The CRD reaches the cluster with the cozystack-controller chart, and when it's missing the migration applies it from the migrations image (L264).
• The finalizer controller lives in cozystack-controller (L121).
• The Cozystack API gains KubernetesNodes-specific behaviour:
  • it writes the default class into a new pool (L140);
  • it refuses a pool on a class being deleted (L121);
  • it refuses old Talos fields by comparing an update with the stored object (L260, L306).
• A platform migration rewrites KubernetesNodes and Kubernetes values (L264).

The rest of managed Kubernetes already ships as system packages: kubernetes-rd, kubernetes-nodes-rd, kamaji, capi-operator, the capi-providers-* packages and kubernetes-worker-image. I'd like the CRD and its controller in a system package of their own that ships with those, so a platform without managed Kubernetes carries none of it.

The API behaviour can move into that package as well. The Cozystack API runs the generic admission chain (RecommendedOptions in pkg/cmd/server/start.go), so the package can ship, for kubernetesnodes.apps.cozystack.io:

• a mutating webhook that writes the default class (it has to be a webhook, since MutatingAdmissionPolicy is disabled there);
• ValidatingAdmissionPolicy objects for the refusals.

If something can't be expressed that way, I'd rather add a generic extension point to the API than code specific to one app.

2. The name, and the group. Nothing in the class is about a group of nodes. bootstrap, the image, the Talos settings and the image source for each substrate all describe how one node boots and joins. The group-level settings (replicas, scaling, health checks, labels and taints) stay on the pool. That's the split Karpenter makes between NodePool and NodeClass, so I'd rename the kind to something like WorkerClass.

For the group, cozystack.io is fine. If the kind should have a group of its own, I'd name the group after the app it serves, as in workerclasses.kubernetes.apps.cozystack.io, rather than kubernetes.cozystack.io. Either works for me. Where the CRD ships (point 1) matters more than its group.

3. Kairos is gated on migrating every existing pool. Step 2 removes the chart's fallback to talos, steps 2 and 3 ship together, and step 2 can't merge until open question 16 is settled (L326). Open question 16 is a platform question: the migration hook runs only in the values-isp-* variants, and the default variant has migrations disabled. As written, no Kairos pool ships until Cozystack can run migrations in every variant.

The migration has also grown its own machinery:

• generated per-namespace classes, bound by a UID label;
• a transition rule for pools that still carry the old fields;
• a restore script;
• open questions 12 and 14–16.

Nearly all of it exists because every existing pool needs a class before the new chart renders it. I'd like the proposal to separate shipping Kairos from moving existing pools. One option is a frozen legacy path: a pool without a class renders from its own Talos fields as it does today, new pools get a class at creation, and existing pools move later. There may be better ones, so I'll leave the choice to you.

4. What the tenant can still choose. After the move, the Talos version and schematic, the kernel modules a pool loads and the registry mirrors all become class content. On a Kairos class with its own image, the Kubernetes patch comes from the class too (L258). That may be the right call, but it takes choices from the tenant that have little to do with how a node bootstraps, such as which modules a workload needs or which Kubernetes patch a pool runs. Today a tenant that needs a module sets it on its own pool; after the move it has to ask the hoster for a class. Could the proposal argue this explicitly, and say whether some of these choices should stay on the pool as fields that mean the same thing on every OS?

5. Smaller decisions, your call:

• sshKeys has no effect on a Talos pool (L136). Refusing a non-empty value on a pool whose class is in the Talos family, and accepting it with a documented no-op, both seem defensible to me. I have no preference, but the text should say which one and why.
• gpudevice refuses a vGPU class on a Talos pool by bootstrap family (L138). That's the one place where GPU capability follows the OS family rather than the image. Recording it as a known exception is fine, or handle it differently if you prefer.

6. Looking ahead: a custom bootstrap family. This isn't for this proposal, but since the design makes room for it, here are two notes for open question 8. With a published join contract, a custom family, where the class supplies the cloud-init, becomes possible. Two parts of it go beyond publishing an endpoint and a CA:

• The join credential. A MachineDeployment shares one bootstrap Secret across all its machines, so the credential has to be reusable, and rotatable per tenant cluster.
• Who may supply the cloud-init. Classes are hoster-only, so a template that the tenant supplies would need somewhere the tenant can write.

[Aleksei Sviridkin (lexfrei)]

Timofei Larkin (@lllamnyp) I moved the class machinery into a managed-Kubernetes system package and renamed the kind to WorkerClass (section 1). Existing pools keep their classless Talos path; moving them onto classes is still required, but it no longer blocks Kairos (section 4). Tenant Talos settings and reachability controls stay as supported per-bootstrap overrides, resolved before fixed-image compatibility is checked (section 3). Talos rejects nonempty sshKeys, and the vGPU/Talos refusal is recorded as the family-based exception. A reusable, rotatable per-cluster join credential and tenant-supplied cloud-init remain requirements for the future custom-bootstrap proposal (OQ8).

Isolated tests demonstrated package-style mutation, validation and persistence through the real aggregated API, using research fixtures. They did not validate the production package's DNS/backend TLS, RBAC/HA or complete image contract. They also reproduced the generic main storage failures tracked in cozystack/cozystack#5234; I am not claiming that fix was tested.

[Timofei Larkin (lllamnyp)]

Thanks, this addresses everything from my last review: the class lives in its own managed-Kubernetes package with nothing in core, it's WorkerClass, Kairos no longer waits on migrating existing pools, tenants keep their Talos choices, and the sshKeys and vGPU/Talos decisions are explicit. The isolated admission test is useful, and thanks for being clear about what it did and didn't cover.

Approving, with a few notes that don't need to block:

• Webhook outages and classless pools (L101–105). Since a classless update needs no class lookup, the webhooks could exclude it with a matchConditions entry that skips UPDATEs where neither the old nor the new object names a class. Classless pools would then never depend on the package being available, instead of only having that dependency documented.
• ComputePlane and the legacy path (§4). ComputePlane keeps producing classless pools, so the later migration can't retire the legacy path until ComputePlane's module contract changes. Could the proposal say which comes first, or that the legacy path stays for ComputePlane indefinitely?
• The Kairos recovery failure (§7). The text mentions an unresolved recovery failure with an unknown cause. Could it describe the symptom, or link an issue? It's one of the gates for leaving experimental, so it helps to know what it is.
• Evidence vs design (optional). The isolated-test findings (L103–107) might read better as a linked test report, with the proposal keeping only the requirements that came out of them.

One design preference, not a blocker: kernel modules and registry mirrors mean the same thing on every OS ("load these modules", "pull through this mirror"). I'd rather they become pool fields that each bootstrap family implements, so a future Kairos override doesn't add Kairos-specific pool fields next to the Talos ones. The Talos version, schematic, factory and installer settings are genuinely Talos-only and fine as they are.