From e17b6c6622e1c66ee5d4958739a01f630d74d537 Mon Sep 17 00:00:00 2001 From: Marian Koreniuk Date: Tue, 20 Oct 2026 11:22:00 +0200 Subject: [PATCH 1/2] docs(kubernetes): describe the Proxmox VE substrate for tenant clusters Add a page under Managed Kubernetes for substrate: proxmox, the opt-in option that runs the workers of a tenant cluster on an external Proxmox VE cluster through capmox: what runs where, the support matrix, enabling the packages, Proxmox user/token/pool setup with the privileges the charts document, the Talos template script, Kubernetes and KubernetesNodes values, verification, the isolation model, limitations and troubleshooting. Assisted-by: LLM Signed-off-by: Marian Koreniuk --- .../docs/next/kubernetes/proxmox-substrate.md | 333 ++++++++++++++++++ 1 file changed, 333 insertions(+) create mode 100644 content/en/docs/next/kubernetes/proxmox-substrate.md diff --git a/content/en/docs/next/kubernetes/proxmox-substrate.md b/content/en/docs/next/kubernetes/proxmox-substrate.md new file mode 100644 index 00000000..420febd0 --- /dev/null +++ b/content/en/docs/next/kubernetes/proxmox-substrate.md @@ -0,0 +1,333 @@ +--- +title: "Running Tenant Kubernetes Workers on Proxmox VE" +linkTitle: "Proxmox VE Substrate" +description: "Run the worker nodes of managed Kubernetes clusters on an external Proxmox VE cluster instead of KubeVirt, using the Cluster API Proxmox provider, the Proxmox cloud-controller-manager and the Proxmox CSI driver." +weight: 70 +--- + +{{< note >}} +The Proxmox VE substrate is **opt-in and experimental** as of Cozystack v1.7. +It is shipped in the platform, tested by its maintainer on Proxmox VE 9.2, and +not yet covered by the project's release gate. Track the decision on support +tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481). +{{< /note >}} + +## What it is + +By default a managed (tenant) Kubernetes cluster in Cozystack has its control +plane run by Kamaji in the management cluster and its worker nodes run as +KubeVirt virtual machines inside the same cluster. + +With `substrate: proxmox` the control plane stays exactly where it is, and the +**workers are cloned as VMs on an external Proxmox VE cluster** through the +Cluster API provider for Proxmox (capmox). No nested virtualization, no +KubeVirt on the worker path. + +``` +Management cluster (Cozystack) Proxmox VE cluster +┌──────────────────────────────────┐ ┌─────────────────────────┐ +│ Kamaji control plane (tenant A) │◄── join ──│ worker VM (pool: a) │ +│ capmox + in-cluster IPAM │── clone ─►│ worker VM (pool: a) │ +│ proxmox-ccm (tenant ns) │── API ───►│ worker VM (pool: b) │ +│ proxmox-csi controller (tenant ns)│── API ──►│ Talos VM template │ +└──────────────────────────────────┘ └─────────────────────────┘ +``` + +| Component | Where it runs | What it does | +|---|---|---| +| capmox (`ProxmoxCluster`, `ProxmoxMachineTemplate`) | management cluster, `cozy-cluster-api` | clones worker VMs from a Talos template, assigns static IPs from an in-cluster IPAM pool | +| Proxmox cloud-controller-manager | management cluster, tenant namespace | sets `providerID`, region/zone labels, clears the `uninitialized` taint on tenant Nodes over the tenant's admin kubeconfig | +| Proxmox CSI controller | management cluster, tenant namespace | creates and attaches Proxmox disks for tenant PersistentVolumes | +| Proxmox CSI node plugin | tenant cluster | mounts the disks; carries no hypervisor credential | + +The hypervisor token never enters the tenant cluster. + +## Support matrix + +| | Verified | Notes | +|---|---|---| +| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 | +| Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x | +| capmox | v0.7.7 (shipped) | | +| Talos | the version in `kubernetes-nodes` `talos.version` | template must match it | +| Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested | +| Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* | +| Networking | VM NIC on a Proxmox bridge, optional VLAN tag | workers must reach the tenant apiserver's LoadBalancer IP | + +## Prerequisites + +On the **Proxmox VE** side: + +- an API endpoint `https://:8006` reachable from the management cluster; +- a bridge the workers attach to, with L3 reachability to the management + cluster's LoadBalancer range (the tenant apiserver is published as a + `LoadBalancer` Service on this substrate); +- a range of free static IPv4 addresses on that L2 for the workers, not + overlapping any DHCP range; +- a storage the Talos template lives on (workers are linked clones of it by default); +- reachable DNS resolvers for the workers (they cannot use the management + cluster's CoreDNS). + +On the **Cozystack** side: + +- a working management cluster with the `iaas` bundle; +- a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address. + +## 1. Enable the packages + +Add the Proxmox infrastructure provider to the platform's enabled packages. +The in-cluster IPAM provider is pulled in as its dependency. + +```yaml +# ConfigMap cozystack in cozy-system, or the platform values +bundles: + enabledPackages: + - cozystack.capi-provider-infra-proxmox +``` + +Before the provider is deployed, create the Secret it authenticates with. +capi-operator fetches it first; while it is missing nothing is installed and +the `InfrastructureProvider` reports the fetch error. + +```bash +kubectl -n cozy-cluster-api create secret generic capmox-config \ + --from-literal=PROXMOX_URL='https://pve.example.com:8006' \ + --from-literal=PROXMOX_TOKEN='capmox@pve!cozystack' \ + --from-literal=PROXMOX_SECRET='' +``` + +Three things that are easy to get wrong: + +- `PROXMOX_URL` is the bare `https://host:8006`; capmox appends `/api2/json` itself. +- `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone. +- a rotated token is picked up only after a restart of the capmox manager pod. + +Check: + +```bash +kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders +kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' +``` + +## 2. Prepare Proxmox: user, tokens, pool + +Create a user and tokens for the platform. Use **privilege-separated tokens** +and grant the privileges to the user as well: Proxmox caps a token at its +user's permissions. + +For capmox itself, follow the provider's own +[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): + +```bash +pveum user add capmox@pve +pveum aclmod / -user capmox@pve -role PVEVMAdmin +pveum user token add capmox@pve cozystack -privsep 0 +``` + +`PVEVMAdmin` on `/` is what the provider documents; a narrower set is +described in its advanced-setup docs. This token is platform-wide: capmox +clones the workers of every tenant with it (see *Isolation*). + +For each **tenant** create a resource pool and a dedicated storage, and grant +the tenant's CCM and CSI tokens their privileges on those paths only: + +| Component | Privileges | Path | +|---|---|---| +| CSI controller | `VM.Audit`, `VM.Config.Disk` | `/pool/` | +| CSI controller | `Datastore.Allocate`, `Datastore.AllocateSpace`, `Datastore.Audit` | `/storage/` | +| CCM | `VM.Audit`, `VM.GuestAgent.Audit` | `/pool/` | +| CCM | `Sys.Audit` | `/` | + +```bash +pveum pool add tenant-a +pveum user add tenant-a@pve +pveum user token add tenant-a@pve csi --privsep 1 +pveum user token add tenant-a@pve ccm --privsep 1 +pveum role add CozyCSI -privs "VM.Audit VM.Config.Disk" +pveum role add CozyCSIStorage -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.Audit" +pveum role add CozyCCM -privs "VM.Audit VM.GuestAgent.Audit" +# the user ... +pveum acl modify /pool/tenant-a --users tenant-a@pve --roles CozyCSI,CozyCCM +pveum acl modify /storage/tenant-a-disks --users tenant-a@pve --roles CozyCSIStorage +pveum acl modify / --users tenant-a@pve --roles PVEAuditor +# ... and each token, which Proxmox caps at the user's permissions +pveum acl modify /pool/tenant-a --tokens 'tenant-a@pve!csi' --roles CozyCSI +pveum acl modify /storage/tenant-a-disks --tokens 'tenant-a@pve!csi' --roles CozyCSIStorage +pveum acl modify /pool/tenant-a --tokens 'tenant-a@pve!ccm' --roles CozyCCM +pveum acl modify / --tokens 'tenant-a@pve!ccm' --roles PVEAuditor +pveum user token permissions tenant-a@pve csi --path /pool/tenant-a +``` + +Why a pool: capmox adds every worker VM it clones to `proxmox.pool`, +replacements included, so an ACL on `/pool/` follows the tenant's VMs +without being granted again. + +## 3. Build the Talos VM template + +Workers are cloned from a Talos VM template that already exists on the +hypervisor. Build it with the script shipped in the repository; it reads +`talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so +the template matches what the chart will render into the machine config. + +```bash +# prints the program it would run, for review +hack/proxmox-talos-template.sh print 9000 local-zfs + +# creates the template over ssh on the PVE node +COZY_PVE_SSH=root@pve.example.com hack/proxmox-talos-template.sh create 9000 local-zfs +``` + +The script prints the tags to put into `proxmox.templateTags`. Do **not** +build the template from the platform's `nocloud-amd64.raw.xz`: that is the +bare-metal node image, and its `zfs` extension keeps a VM from completing its +boot, rebooting it every ~70 minutes. + +## 4. Create the tenant cluster + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Kubernetes +metadata: + name: px + namespace: tenant-a +spec: + substrate: proxmox + proxmox: + allowedNodes: ["pve1", "pve2"] # empty = every node + dnsServers: ["10.0.0.1"] + insecure: true # stock PVE serves a self-signed cert + ipv4Config: + addresses: ["10.0.0.120-10.0.0.170"] + gateway: "10.0.0.1" + prefix: 24 + ccm: + credentialsSecretName: proxmox-ccm # keys: url, token_id, token_secret, region + csi: + enabled: true + credentialsSecretName: proxmox-csi + storageClasses: + - name: proxmox + storage: tenant-a-disks + fstype: ext4 + reclaimPolicy: Delete + # ... the rest of the usual Kubernetes spec +``` + +The two credential Secrets live in the **tenant's namespace of the management +cluster** with the keys `url`, `token_id`, `token_secret`, `region` +(`region` is the Proxmox cluster's name; the controllers put it on the +Nodes as `topology.kubernetes.io/region`. Note the URL here carries +`/api2/json`, unlike capmox's `PROXMOX_URL`): + +```bash +kubectl -n tenant-a create secret generic proxmox-csi \ + --from-literal=url='https://pve.example.com:8006/api2/json' \ + --from-literal=token_id='tenant-a@pve!csi' \ + --from-literal=token_secret='' \ + --from-literal=region='pve' +``` + +`substrate` is immutable: switching an existing cluster is refused. Create a +new one. + +## 5. Add a worker pool + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: KubernetesNodes +metadata: + name: px-md0 + namespace: tenant-a +spec: + cluster: px + substrate: proxmox + replicas: 2 + resources: + cpu: 4 + memory: 8Gi + proxmox: + templateTags: ["cozystack", "talos", "talos-v1.13.6", "schematic-ce4c9805"] + full: false # linked clone (default) + network: + bridge: vmbr0 + vlan: 60 # optional + pool: tenant-a + dnsServers: ["10.0.0.1"] +``` + +`templateTags` must be **equal** to the template's tag set, not a subset: a +missing tag yields `found 0 VM templates with tags ...` even though the tags +are plainly present on the template. + +`resources.cpu` / `memory` are converted to capmox's `numCores` / `memoryMiB` +(rounded up). + +## 6. Verify + +```bash +# management cluster +kubectl -n tenant-a get cluster,proxmoxcluster,machinedeployment,machine +kubectl -n tenant-a get proxmoxmachines -o custom-columns='NAME:.metadata.name,VMID:.spec.virtualMachineID,NODE:.status.proxmoxNode,IP:.status.ipAddresses.net0.ipv4,STATE:.status.vmStatus' +kubectl -n tenant-a get inclusterippools,ipaddresses.ipam.cluster.x-k8s.io +kubectl -n tenant-a get deploy px-pccm px-pcsi-controller + +# tenant cluster +kubectl --kubeconfig get nodes -o wide +kubectl --kubeconfig get nodes -o custom-columns='NODE:.metadata.name,PROVIDER-ID:.spec.providerID,TAINTS:.spec.taints[*].key' +kubectl --kubeconfig -n csi-proxmox get pods +``` + +A healthy node has the same `providerID` as its CAPI `Machine` +(`proxmox://`; the CCM runs in its `capmox` mode for that), the +`topology.kubernetes.io/region|zone` labels, and no +`node.cloudprovider.kubernetes.io/uninitialized` taint. + +What is covered by the integration and was verified live on PVE 9.2: +PVC create → attach → online resize → delete; Cluster Autoscaler scale-up and +scale-down through the pool; rolling replacement of a worker; +MachineHealthCheck. + +## Isolation: what the design gives and what it does not + +- The hypervisor token is composed into the CCM and CSI controllers in the + management cluster; nothing copies it into the tenant cluster. +- The tenant is cluster-admin of its own cluster, and the controllers act on + what its objects say: the CSI controller attaches the disk a PV names to the + VM a Node names. **Scope comes from the Proxmox ACL**, not from the + controllers: one pool and one storage per tenant. +- A storage shared between tenants cannot be scoped: `Datastore.Allocate` on + it reaches every volume on it. +- capmox itself (VM creation) currently uses one platform-wide token for all + tenants; per-tenant `credentialsRef` on `ProxmoxCluster` is being worked on + upstream. + +## Limitations (v1.7) + +- Opt-in, experimental; no release gate yet (an e2e lane exists upstream, but its nightly run is opt-in and currently skipped). +- Workers only. Cozystack's `vm-disk` / `vm-instance` applications still run on KubeVirt; a Proxmox substrate for them is in design. +- Tenant `Kubernetes` kinds are published as part of the KubeVirt variant, so a management cluster still needs the KubeVirt packages installed even if every pool runs on Proxmox. +- No DHCP: capmox assigns static addresses from `ipv4Config` only. +- Linked clones keep the template's base snapshot referenced; rotate templates by creating a new VMID with new tags and rolling the pool. +- No live migration awareness: the CCM reports a VM it cannot see as gone and deletes the Node. +- Cozystack's Terraform provider does not model `substrate`/`proxmox.*` yet; `terraform apply` on such a cluster overwrites them. + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `InfrastructureProvider` not installing, fetch error | `capmox-config` Secret missing or in the wrong namespace | create it in `cozy-cluster-api` | +| every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` | +| `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly | +| Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS | +| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n logs deploy/-pccm` | +| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM; the CSI release depends on it | +| VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` | +| `TalosConfigTemplate.Spec is immutable` | apiserver address changed on an existing cluster | delete the template; the reconcile Job recreates it | +| capmox stuck at `Waiting for Cluster Controller to set OwnerRef` | `cluster.x-k8s.io/managed-by` annotation on the `ProxmoxCluster` | not expected from the chart; report it | + +## See also + +- [Managed Kubernetes]({{% ref "/docs/next/kubernetes" %}}) — the application this page extends +- Cluster API provider for Proxmox: https://github.com/ionos-cloud/cluster-api-provider-proxmox +- Proxmox CCM / CSI: https://github.com/sergelogvinov/proxmox-cloud-controller-manager, https://github.com/sergelogvinov/proxmox-csi-plugin +- Support-tier proposal: https://github.com/cozystack/community/pull/98 From c6c4e9cc3fcf66ef9d232c78be2c877bf2494d79 Mon Sep 17 00:00:00 2001 From: Marian Koreniuk Date: Fri, 30 Oct 2026 21:25:51 +0100 Subject: [PATCH 2/2] docs(kubernetes): make the Proxmox VE substrate enablement followable The page used the capmox token in step 1 and created it in step 2, and showed bundles.enabledPackages as a values fragment without saying how to set it. Order the steps so each uses only what exists, give the Package patch, and add the CCM Secret the example references. Link the page from the deployment guide, which had no path to it. Carry the four v1.7 corrections from #782 to next/. v1.7.0 runs proxmox-cloud-controller-manager:v0.15.0, a tag that now resolves to a build that panics at start, so the v1.7 copy gets the image override until the backport in cozystack/cozystack#5150 is released. Assisted-by: LLM Signed-off-by: Marian Koreniuk --- content/en/docs/next/install/_index.md | 7 ++ .../docs/next/kubernetes/proxmox-substrate.md | 90 +++++++++++------ content/en/docs/v1.7/install/_index.md | 7 ++ .../docs/v1.7/kubernetes/proxmox-substrate.md | 99 +++++++++++++------ 4 files changed, 142 insertions(+), 61 deletions(-) diff --git a/content/en/docs/next/install/_index.md b/content/en/docs/next/install/_index.md index e0e9740f..d7782f5c 100644 --- a/content/en/docs/next/install/_index.md +++ b/content/en/docs/next/install/_index.md @@ -45,6 +45,13 @@ If that's your case, we recommend using the guides below: - [Servers.com]({{% ref "/docs/next/install/providers/servers-com" %}}) +## Proxmox VE + +There are two ways to use Cozystack with an existing Proxmox VE cluster, and they can be combined: + +- Run the Cozystack cluster itself in Proxmox VE VMs: follow the [generic installation path](#generic-installation-path), treating the VMs like any other machines. Set the VM CPU type to `host` if tenants will run KubeVirt VMs on them. +- Run the workers of tenant Kubernetes clusters as VMs on Proxmox VE instead of KubeVirt: [enable the Proxmox VE substrate]({{% ref "/docs/next/kubernetes/proxmox-substrate" %}}). This is opt-in and experimental. + ## Upgrading and Post-deployment Configuration After you've deployed a cluster, proceed to the [Cluster Administration]({{% ref "/docs/next/operations" %}}) section for diff --git a/content/en/docs/next/kubernetes/proxmox-substrate.md b/content/en/docs/next/kubernetes/proxmox-substrate.md index 420febd0..202a9ab6 100644 --- a/content/en/docs/next/kubernetes/proxmox-substrate.md +++ b/content/en/docs/next/kubernetes/proxmox-substrate.md @@ -12,6 +12,10 @@ not yet covered by the project's release gate. Track the decision on support tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481). {{< /note >}} +This page is about running the **workers of tenant clusters** on Proxmox VE. +To run the management cluster itself on Proxmox VE VMs, follow the regular +[deployment guide]({{% ref "/docs/next/install" %}}). + ## What it is By default a managed (tenant) Kubernetes cluster in Cozystack has its control @@ -49,6 +53,7 @@ The hypervisor token never enters the tenant cluster. | Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 | | Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x | | capmox | v0.7.7 (shipped) | | +| Proxmox CCM | v0.16.1 (shipped) | pinned by digest | | Talos | the version in `kubernetes-nodes` `talos.version` | template must match it | | Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested | | Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* | @@ -70,22 +75,28 @@ On the **Proxmox VE** side: On the **Cozystack** side: -- a working management cluster with the `iaas` bundle; +- a working management cluster on a variant with the `iaas` bundle, such as + `isp-full` or `isp-full-generic`; - a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address. -## 1. Enable the packages +## 1. Enable the Proxmox provider -Add the Proxmox infrastructure provider to the platform's enabled packages. -The in-cluster IPAM provider is pulled in as its dependency. +The provider is an opt-in package, because it needs a Proxmox API token before +it can do anything. On a Proxmox VE node, create a user and a token for capmox, +following the provider's own +[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): -```yaml -# ConfigMap cozystack in cozy-system, or the platform values -bundles: - enabledPackages: - - cozystack.capi-provider-infra-proxmox +```bash +pveum user add capmox@pve +pveum aclmod / -user capmox@pve -role PVEVMAdmin +pveum user token add capmox@pve cozystack -privsep 0 ``` -Before the provider is deployed, create the Secret it authenticates with. +`PVEVMAdmin` on `/` is what the provider documents; a narrower set is +described in its advanced-setup docs. This token is platform-wide: capmox +clones the workers of every tenant with it (see *Isolation*). + +Put the token into the Secret the provider authenticates with. capi-operator fetches it first; while it is missing nothing is installed and the `InfrastructureProvider` reports the fetch error. @@ -102,34 +113,41 @@ Three things that are easy to get wrong: - `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone. - a rotated token is picked up only after a restart of the capmox manager pod. -Check: +Then add the provider to the platform's enabled packages. The in-cluster IPAM +provider is pulled in as its dependency. ```bash -kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders -kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' +kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p '{ + "spec": { + "components": { + "platform": { + "values": { + "bundles": { + "enabledPackages": ["cozystack.capi-provider-infra-proxmox"] + } + } + } + } + } +}' ``` -## 2. Prepare Proxmox: user, tokens, pool +A merge patch replaces the whole list: if `bundles.enabledPackages` already has +entries, repeat them here. -Create a user and tokens for the platform. Use **privilege-separated tokens** -and grant the privileges to the user as well: Proxmox caps a token at its -user's permissions. - -For capmox itself, follow the provider's own -[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): +Check: ```bash -pveum user add capmox@pve -pveum aclmod / -user capmox@pve -role PVEVMAdmin -pveum user token add capmox@pve cozystack -privsep 0 +kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders +kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' ``` -`PVEVMAdmin` on `/` is what the provider documents; a narrower set is -described in its advanced-setup docs. This token is platform-wide: capmox -clones the workers of every tenant with it (see *Isolation*). +## 2. Prepare a tenant: pool and tokens For each **tenant** create a resource pool and a dedicated storage, and grant -the tenant's CCM and CSI tokens their privileges on those paths only: +the tenant's CCM and CSI tokens their privileges on those paths only. Use +**privilege-separated tokens** and grant the privileges to the user as well: +Proxmox caps a token at its user's permissions. | Component | Privileges | Path | |---|---|---| @@ -168,6 +186,7 @@ Workers are cloned from a Talos VM template that already exists on the hypervisor. Build it with the script shipped in the repository; it reads `talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so the template matches what the chart will render into the machine config. +Run it from a checkout of the Cozystack version the management cluster runs. ```bash # prints the program it would run, for review @@ -223,7 +242,13 @@ Nodes as `topology.kubernetes.io/region`. Note the URL here carries kubectl -n tenant-a create secret generic proxmox-csi \ --from-literal=url='https://pve.example.com:8006/api2/json' \ --from-literal=token_id='tenant-a@pve!csi' \ - --from-literal=token_secret='' \ + --from-literal=token_secret='' \ + --from-literal=region='pve' + +kubectl -n tenant-a create secret generic proxmox-ccm \ + --from-literal=url='https://pve.example.com:8006/api2/json' \ + --from-literal=token_id='tenant-a@pve!ccm' \ + --from-literal=token_secret='' \ --from-literal=region='pve' ``` @@ -241,7 +266,8 @@ metadata: spec: cluster: px substrate: proxmox - replicas: 2 + minReplicas: 2 + maxReplicas: 2 resources: cpu: 4 memory: 8Gi @@ -269,7 +295,7 @@ are plainly present on the template. kubectl -n tenant-a get cluster,proxmoxcluster,machinedeployment,machine kubectl -n tenant-a get proxmoxmachines -o custom-columns='NAME:.metadata.name,VMID:.spec.virtualMachineID,NODE:.status.proxmoxNode,IP:.status.ipAddresses.net0.ipv4,STATE:.status.vmStatus' kubectl -n tenant-a get inclusterippools,ipaddresses.ipam.cluster.x-k8s.io -kubectl -n tenant-a get deploy px-pccm px-pcsi-controller +kubectl -n tenant-a get deploy kubernetes-px-pccm kubernetes-px-pcsi-controller # tenant cluster kubectl --kubeconfig get nodes -o wide @@ -319,8 +345,8 @@ MachineHealthCheck. | every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` | | `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly | | Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS | -| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n logs deploy/-pccm` | -| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM; the CSI release depends on it | +| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n logs deploy/kubernetes--pccm` | +| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM | | VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` | | `TalosConfigTemplate.Spec is immutable` | apiserver address changed on an existing cluster | delete the template; the reconcile Job recreates it | | capmox stuck at `Waiting for Cluster Controller to set OwnerRef` | `cluster.x-k8s.io/managed-by` annotation on the `ProxmoxCluster` | not expected from the chart; report it | diff --git a/content/en/docs/v1.7/install/_index.md b/content/en/docs/v1.7/install/_index.md index d286a44a..da0198a7 100644 --- a/content/en/docs/v1.7/install/_index.md +++ b/content/en/docs/v1.7/install/_index.md @@ -45,6 +45,13 @@ If that's your case, we recommend using the guides below: - [Servers.com]({{% ref "/docs/v1.7/install/providers/servers-com" %}}) +## Proxmox VE + +There are two ways to use Cozystack with an existing Proxmox VE cluster, and they can be combined: + +- Run the Cozystack cluster itself in Proxmox VE VMs: follow the [generic installation path](#generic-installation-path), treating the VMs like any other machines. Set the VM CPU type to `host` if tenants will run KubeVirt VMs on them. +- Run the workers of tenant Kubernetes clusters as VMs on Proxmox VE instead of KubeVirt: [enable the Proxmox VE substrate]({{% ref "/docs/v1.7/kubernetes/proxmox-substrate" %}}). This is opt-in and experimental. + ## Upgrading and Post-deployment Configuration After you've deployed a cluster, proceed to the [Cluster Administration]({{% ref "/docs/v1.7/operations" %}}) section for diff --git a/content/en/docs/v1.7/kubernetes/proxmox-substrate.md b/content/en/docs/v1.7/kubernetes/proxmox-substrate.md index 40d7a07e..5dadbe21 100644 --- a/content/en/docs/v1.7/kubernetes/proxmox-substrate.md +++ b/content/en/docs/v1.7/kubernetes/proxmox-substrate.md @@ -12,6 +12,10 @@ not yet covered by the project's release gate. Track the decision on support tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481). {{< /note >}} +This page is about running the **workers of tenant clusters** on Proxmox VE. +To run the management cluster itself on Proxmox VE VMs, follow the regular +[deployment guide]({{% ref "/docs/v1.7/install" %}}). + ## What it is By default a managed (tenant) Kubernetes cluster in Cozystack has its control @@ -46,9 +50,10 @@ The hypervisor token never enters the tenant cluster. | | Verified | Notes | |---|---|---| -| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 | +| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6; v1.7.0 needs the CCM fix in step 4 | | Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x | | capmox | v0.7.7 (shipped) | | +| Proxmox CCM | v0.16.1 | v1.7.0 ships v0.15.0, which panics; see step 4 | | Talos | the version in `kubernetes-nodes` `talos.version` | template must match it | | Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested | | Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* | @@ -70,22 +75,28 @@ On the **Proxmox VE** side: On the **Cozystack** side: -- a working management cluster with the `iaas` bundle; +- a working management cluster on a variant with the `iaas` bundle, such as + `isp-full` or `isp-full-generic`; - a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address. -## 1. Enable the packages +## 1. Enable the Proxmox provider -Add the Proxmox infrastructure provider to the platform's enabled packages. -The in-cluster IPAM provider is pulled in as its dependency. +The provider is an opt-in package, because it needs a Proxmox API token before +it can do anything. On a Proxmox VE node, create a user and a token for capmox, +following the provider's own +[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): -```yaml -# spec.components.platform.values of the cozystack.cozystack-platform Package -bundles: - enabledPackages: - - cozystack.capi-provider-infra-proxmox +```bash +pveum user add capmox@pve +pveum aclmod / -user capmox@pve -role PVEVMAdmin +pveum user token add capmox@pve cozystack -privsep 0 ``` -Before the provider is deployed, create the Secret it authenticates with. +`PVEVMAdmin` on `/` is what the provider documents; a narrower set is +described in its advanced-setup docs. This token is platform-wide: capmox +clones the workers of every tenant with it (see *Isolation*). + +Put the token into the Secret the provider authenticates with. capi-operator fetches it first; while it is missing nothing is installed and the `InfrastructureProvider` reports the fetch error. @@ -102,34 +113,41 @@ Three things that are easy to get wrong: - `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone. - a rotated token is picked up only after a restart of the capmox manager pod. -Check: +Then add the provider to the platform's enabled packages. The in-cluster IPAM +provider is pulled in as its dependency. ```bash -kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders -kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' +kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p '{ + "spec": { + "components": { + "platform": { + "values": { + "bundles": { + "enabledPackages": ["cozystack.capi-provider-infra-proxmox"] + } + } + } + } + } +}' ``` -## 2. Prepare Proxmox: user, tokens, pool +A merge patch replaces the whole list: if `bundles.enabledPackages` already has +entries, repeat them here. -Create a user and tokens for the platform. Use **privilege-separated tokens** -and grant the privileges to the user as well: Proxmox caps a token at its -user's permissions. - -For capmox itself, follow the provider's own -[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): +Check: ```bash -pveum user add capmox@pve -pveum aclmod / -user capmox@pve -role PVEVMAdmin -pveum user token add capmox@pve cozystack -privsep 0 +kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders +kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' ``` -`PVEVMAdmin` on `/` is what the provider documents; a narrower set is -described in its advanced-setup docs. This token is platform-wide: capmox -clones the workers of every tenant with it (see *Isolation*). +## 2. Prepare a tenant: pool and tokens For each **tenant** create a resource pool and a dedicated storage, and grant -the tenant's CCM and CSI tokens their privileges on those paths only: +the tenant's CCM and CSI tokens their privileges on those paths only. Use +**privilege-separated tokens** and grant the privileges to the user as well: +Proxmox caps a token at its user's permissions. | Component | Privileges | Path | |---|---|---| @@ -168,6 +186,7 @@ Workers are cloned from a Talos VM template that already exists on the hypervisor. Build it with the script shipped in the repository; it reads `talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so the template matches what the chart will render into the machine config. +Run it from a checkout of the Cozystack version the management cluster runs. ```bash # prints the program it would run, for review @@ -223,13 +242,34 @@ Nodes as `topology.kubernetes.io/region`. Note the URL here carries kubectl -n tenant-a create secret generic proxmox-csi \ --from-literal=url='https://pve.example.com:8006/api2/json' \ --from-literal=token_id='tenant-a@pve!csi' \ - --from-literal=token_secret='' \ + --from-literal=token_secret='' \ + --from-literal=region='pve' + +kubectl -n tenant-a create secret generic proxmox-ccm \ + --from-literal=url='https://pve.example.com:8006/api2/json' \ + --from-literal=token_id='tenant-a@pve!ccm' \ + --from-literal=token_secret='' \ --from-literal=region='pve' ``` `substrate` is immutable: switching an existing cluster is refused. Create a new one. +{{< warning >}} +Cozystack v1.7.0 ships a broken CCM image. The chart runs +`proxmox-cloud-controller-manager:v0.15.0`, and that tag now resolves to a build +that panics at start, so workers keep the `uninitialized` taint. The fix is in +[cozystack/cozystack#5150](https://github.com/cozystack/cozystack/pull/5150) and +is not released yet. Until then, set the image by hand once the cluster exists. +The next Helm upgrade of the cluster puts the chart's image back, so repeat this +after every change to the `Kubernetes` resource. +{{< /warning >}} + +```bash +kubectl -n tenant-a set image deploy/kubernetes-px-pccm \ + cloud-controller-manager=ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.16.1@sha256:474dbfd3be5c2e0866e520bc6ddb60b7accef554ca7401488eee35252cdf8734 +``` + ## 5. Add a worker pool ```yaml @@ -320,6 +360,7 @@ MachineHealthCheck. | every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` | | `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly | | Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS | +| CCM in `CrashLoopBackOff`, `panic: version string "0.15.0"` | the image v1.7.0 ships | set the image as in step 4 | | Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n logs deploy/kubernetes--pccm` | | CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM | | VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` |