diff --git a/content/en/docs/next/install/_index.md b/content/en/docs/next/install/_index.md index e0e9740f..d7782f5c 100644 --- a/content/en/docs/next/install/_index.md +++ b/content/en/docs/next/install/_index.md @@ -45,6 +45,13 @@ If that's your case, we recommend using the guides below: - [Servers.com]({{% ref "/docs/next/install/providers/servers-com" %}}) +## Proxmox VE + +There are two ways to use Cozystack with an existing Proxmox VE cluster, and they can be combined: + +- Run the Cozystack cluster itself in Proxmox VE VMs: follow the [generic installation path](#generic-installation-path), treating the VMs like any other machines. Set the VM CPU type to `host` if tenants will run KubeVirt VMs on them. +- Run the workers of tenant Kubernetes clusters as VMs on Proxmox VE instead of KubeVirt: [enable the Proxmox VE substrate]({{% ref "/docs/next/kubernetes/proxmox-substrate" %}}). This is opt-in and experimental. + ## Upgrading and Post-deployment Configuration After you've deployed a cluster, proceed to the [Cluster Administration]({{% ref "/docs/next/operations" %}}) section for diff --git a/content/en/docs/next/kubernetes/proxmox-substrate.md b/content/en/docs/next/kubernetes/proxmox-substrate.md new file mode 100644 index 00000000..202a9ab6 --- /dev/null +++ b/content/en/docs/next/kubernetes/proxmox-substrate.md @@ -0,0 +1,359 @@ +--- +title: "Running Tenant Kubernetes Workers on Proxmox VE" +linkTitle: "Proxmox VE Substrate" +description: "Run the worker nodes of managed Kubernetes clusters on an external Proxmox VE cluster instead of KubeVirt, using the Cluster API Proxmox provider, the Proxmox cloud-controller-manager and the Proxmox CSI driver." +weight: 70 +--- + +{{< note >}} +The Proxmox VE substrate is **opt-in and experimental** as of Cozystack v1.7. +It is shipped in the platform, tested by its maintainer on Proxmox VE 9.2, and +not yet covered by the project's release gate. Track the decision on support +tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481). +{{< /note >}} + +This page is about running the **workers of tenant clusters** on Proxmox VE. +To run the management cluster itself on Proxmox VE VMs, follow the regular +[deployment guide]({{% ref "/docs/next/install" %}}). + +## What it is + +By default a managed (tenant) Kubernetes cluster in Cozystack has its control +plane run by Kamaji in the management cluster and its worker nodes run as +KubeVirt virtual machines inside the same cluster. + +With `substrate: proxmox` the control plane stays exactly where it is, and the +**workers are cloned as VMs on an external Proxmox VE cluster** through the +Cluster API provider for Proxmox (capmox). No nested virtualization, no +KubeVirt on the worker path. + +``` +Management cluster (Cozystack) Proxmox VE cluster +┌──────────────────────────────────┐ ┌─────────────────────────┐ +│ Kamaji control plane (tenant A) │◄── join ──│ worker VM (pool: a) │ +│ capmox + in-cluster IPAM │── clone ─►│ worker VM (pool: a) │ +│ proxmox-ccm (tenant ns) │── API ───►│ worker VM (pool: b) │ +│ proxmox-csi controller (tenant ns)│── API ──►│ Talos VM template │ +└──────────────────────────────────┘ └─────────────────────────┘ +``` + +| Component | Where it runs | What it does | +|---|---|---| +| capmox (`ProxmoxCluster`, `ProxmoxMachineTemplate`) | management cluster, `cozy-cluster-api` | clones worker VMs from a Talos template, assigns static IPs from an in-cluster IPAM pool | +| Proxmox cloud-controller-manager | management cluster, tenant namespace | sets `providerID`, region/zone labels, clears the `uninitialized` taint on tenant Nodes over the tenant's admin kubeconfig | +| Proxmox CSI controller | management cluster, tenant namespace | creates and attaches Proxmox disks for tenant PersistentVolumes | +| Proxmox CSI node plugin | tenant cluster | mounts the disks; carries no hypervisor credential | + +The hypervisor token never enters the tenant cluster. + +## Support matrix + +| | Verified | Notes | +|---|---|---| +| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 | +| Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x | +| capmox | v0.7.7 (shipped) | | +| Proxmox CCM | v0.16.1 (shipped) | pinned by digest | +| Talos | the version in `kubernetes-nodes` `talos.version` | template must match it | +| Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested | +| Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* | +| Networking | VM NIC on a Proxmox bridge, optional VLAN tag | workers must reach the tenant apiserver's LoadBalancer IP | + +## Prerequisites + +On the **Proxmox VE** side: + +- an API endpoint `https://:8006` reachable from the management cluster; +- a bridge the workers attach to, with L3 reachability to the management + cluster's LoadBalancer range (the tenant apiserver is published as a + `LoadBalancer` Service on this substrate); +- a range of free static IPv4 addresses on that L2 for the workers, not + overlapping any DHCP range; +- a storage the Talos template lives on (workers are linked clones of it by default); +- reachable DNS resolvers for the workers (they cannot use the management + cluster's CoreDNS). + +On the **Cozystack** side: + +- a working management cluster on a variant with the `iaas` bundle, such as + `isp-full` or `isp-full-generic`; +- a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address. + +## 1. Enable the Proxmox provider + +The provider is an opt-in package, because it needs a Proxmox API token before +it can do anything. On a Proxmox VE node, create a user and a token for capmox, +following the provider's own +[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): + +```bash +pveum user add capmox@pve +pveum aclmod / -user capmox@pve -role PVEVMAdmin +pveum user token add capmox@pve cozystack -privsep 0 +``` + +`PVEVMAdmin` on `/` is what the provider documents; a narrower set is +described in its advanced-setup docs. This token is platform-wide: capmox +clones the workers of every tenant with it (see *Isolation*). + +Put the token into the Secret the provider authenticates with. +capi-operator fetches it first; while it is missing nothing is installed and +the `InfrastructureProvider` reports the fetch error. + +```bash +kubectl -n cozy-cluster-api create secret generic capmox-config \ + --from-literal=PROXMOX_URL='https://pve.example.com:8006' \ + --from-literal=PROXMOX_TOKEN='capmox@pve!cozystack' \ + --from-literal=PROXMOX_SECRET='' +``` + +Three things that are easy to get wrong: + +- `PROXMOX_URL` is the bare `https://host:8006`; capmox appends `/api2/json` itself. +- `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone. +- a rotated token is picked up only after a restart of the capmox manager pod. + +Then add the provider to the platform's enabled packages. The in-cluster IPAM +provider is pulled in as its dependency. + +```bash +kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p '{ + "spec": { + "components": { + "platform": { + "values": { + "bundles": { + "enabledPackages": ["cozystack.capi-provider-infra-proxmox"] + } + } + } + } + } +}' +``` + +A merge patch replaces the whole list: if `bundles.enabledPackages` already has +entries, repeat them here. + +Check: + +```bash +kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders +kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' +``` + +## 2. Prepare a tenant: pool and tokens + +For each **tenant** create a resource pool and a dedicated storage, and grant +the tenant's CCM and CSI tokens their privileges on those paths only. Use +**privilege-separated tokens** and grant the privileges to the user as well: +Proxmox caps a token at its user's permissions. + +| Component | Privileges | Path | +|---|---|---| +| CSI controller | `VM.Audit`, `VM.Config.Disk` | `/pool/` | +| CSI controller | `Datastore.Allocate`, `Datastore.AllocateSpace`, `Datastore.Audit` | `/storage/` | +| CCM | `VM.Audit`, `VM.GuestAgent.Audit` | `/pool/` | +| CCM | `Sys.Audit` | `/` | + +```bash +pveum pool add tenant-a +pveum user add tenant-a@pve +pveum user token add tenant-a@pve csi --privsep 1 +pveum user token add tenant-a@pve ccm --privsep 1 +pveum role add CozyCSI -privs "VM.Audit VM.Config.Disk" +pveum role add CozyCSIStorage -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.Audit" +pveum role add CozyCCM -privs "VM.Audit VM.GuestAgent.Audit" +# the user ... +pveum acl modify /pool/tenant-a --users tenant-a@pve --roles CozyCSI,CozyCCM +pveum acl modify /storage/tenant-a-disks --users tenant-a@pve --roles CozyCSIStorage +pveum acl modify / --users tenant-a@pve --roles PVEAuditor +# ... and each token, which Proxmox caps at the user's permissions +pveum acl modify /pool/tenant-a --tokens 'tenant-a@pve!csi' --roles CozyCSI +pveum acl modify /storage/tenant-a-disks --tokens 'tenant-a@pve!csi' --roles CozyCSIStorage +pveum acl modify /pool/tenant-a --tokens 'tenant-a@pve!ccm' --roles CozyCCM +pveum acl modify / --tokens 'tenant-a@pve!ccm' --roles PVEAuditor +pveum user token permissions tenant-a@pve csi --path /pool/tenant-a +``` + +Why a pool: capmox adds every worker VM it clones to `proxmox.pool`, +replacements included, so an ACL on `/pool/` follows the tenant's VMs +without being granted again. + +## 3. Build the Talos VM template + +Workers are cloned from a Talos VM template that already exists on the +hypervisor. Build it with the script shipped in the repository; it reads +`talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so +the template matches what the chart will render into the machine config. +Run it from a checkout of the Cozystack version the management cluster runs. + +```bash +# prints the program it would run, for review +hack/proxmox-talos-template.sh print 9000 local-zfs + +# creates the template over ssh on the PVE node +COZY_PVE_SSH=root@pve.example.com hack/proxmox-talos-template.sh create 9000 local-zfs +``` + +The script prints the tags to put into `proxmox.templateTags`. Do **not** +build the template from the platform's `nocloud-amd64.raw.xz`: that is the +bare-metal node image, and its `zfs` extension keeps a VM from completing its +boot, rebooting it every ~70 minutes. + +## 4. Create the tenant cluster + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Kubernetes +metadata: + name: px + namespace: tenant-a +spec: + substrate: proxmox + proxmox: + allowedNodes: ["pve1", "pve2"] # empty = every node + dnsServers: ["10.0.0.1"] + insecure: true # stock PVE serves a self-signed cert + ipv4Config: + addresses: ["10.0.0.120-10.0.0.170"] + gateway: "10.0.0.1" + prefix: 24 + ccm: + credentialsSecretName: proxmox-ccm # keys: url, token_id, token_secret, region + csi: + enabled: true + credentialsSecretName: proxmox-csi + storageClasses: + - name: proxmox + storage: tenant-a-disks + fstype: ext4 + reclaimPolicy: Delete + # ... the rest of the usual Kubernetes spec +``` + +The two credential Secrets live in the **tenant's namespace of the management +cluster** with the keys `url`, `token_id`, `token_secret`, `region` +(`region` is the Proxmox cluster's name; the controllers put it on the +Nodes as `topology.kubernetes.io/region`. Note the URL here carries +`/api2/json`, unlike capmox's `PROXMOX_URL`): + +```bash +kubectl -n tenant-a create secret generic proxmox-csi \ + --from-literal=url='https://pve.example.com:8006/api2/json' \ + --from-literal=token_id='tenant-a@pve!csi' \ + --from-literal=token_secret='' \ + --from-literal=region='pve' + +kubectl -n tenant-a create secret generic proxmox-ccm \ + --from-literal=url='https://pve.example.com:8006/api2/json' \ + --from-literal=token_id='tenant-a@pve!ccm' \ + --from-literal=token_secret='' \ + --from-literal=region='pve' +``` + +`substrate` is immutable: switching an existing cluster is refused. Create a +new one. + +## 5. Add a worker pool + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: KubernetesNodes +metadata: + name: px-md0 + namespace: tenant-a +spec: + cluster: px + substrate: proxmox + minReplicas: 2 + maxReplicas: 2 + resources: + cpu: 4 + memory: 8Gi + proxmox: + templateTags: ["cozystack", "talos", "talos-v1.13.6", "schematic-ce4c9805"] + full: false # linked clone (default) + network: + bridge: vmbr0 + vlan: 60 # optional + pool: tenant-a + dnsServers: ["10.0.0.1"] +``` + +`templateTags` must be **equal** to the template's tag set, not a subset: a +missing tag yields `found 0 VM templates with tags ...` even though the tags +are plainly present on the template. + +`resources.cpu` / `memory` are converted to capmox's `numCores` / `memoryMiB` +(rounded up). + +## 6. Verify + +```bash +# management cluster +kubectl -n tenant-a get cluster,proxmoxcluster,machinedeployment,machine +kubectl -n tenant-a get proxmoxmachines -o custom-columns='NAME:.metadata.name,VMID:.spec.virtualMachineID,NODE:.status.proxmoxNode,IP:.status.ipAddresses.net0.ipv4,STATE:.status.vmStatus' +kubectl -n tenant-a get inclusterippools,ipaddresses.ipam.cluster.x-k8s.io +kubectl -n tenant-a get deploy kubernetes-px-pccm kubernetes-px-pcsi-controller + +# tenant cluster +kubectl --kubeconfig get nodes -o wide +kubectl --kubeconfig get nodes -o custom-columns='NODE:.metadata.name,PROVIDER-ID:.spec.providerID,TAINTS:.spec.taints[*].key' +kubectl --kubeconfig -n csi-proxmox get pods +``` + +A healthy node has the same `providerID` as its CAPI `Machine` +(`proxmox://`; the CCM runs in its `capmox` mode for that), the +`topology.kubernetes.io/region|zone` labels, and no +`node.cloudprovider.kubernetes.io/uninitialized` taint. + +What is covered by the integration and was verified live on PVE 9.2: +PVC create → attach → online resize → delete; Cluster Autoscaler scale-up and +scale-down through the pool; rolling replacement of a worker; +MachineHealthCheck. + +## Isolation: what the design gives and what it does not + +- The hypervisor token is composed into the CCM and CSI controllers in the + management cluster; nothing copies it into the tenant cluster. +- The tenant is cluster-admin of its own cluster, and the controllers act on + what its objects say: the CSI controller attaches the disk a PV names to the + VM a Node names. **Scope comes from the Proxmox ACL**, not from the + controllers: one pool and one storage per tenant. +- A storage shared between tenants cannot be scoped: `Datastore.Allocate` on + it reaches every volume on it. +- capmox itself (VM creation) currently uses one platform-wide token for all + tenants; per-tenant `credentialsRef` on `ProxmoxCluster` is being worked on + upstream. + +## Limitations (v1.7) + +- Opt-in, experimental; no release gate yet (an e2e lane exists upstream, but its nightly run is opt-in and currently skipped). +- Workers only. Cozystack's `vm-disk` / `vm-instance` applications still run on KubeVirt; a Proxmox substrate for them is in design. +- Tenant `Kubernetes` kinds are published as part of the KubeVirt variant, so a management cluster still needs the KubeVirt packages installed even if every pool runs on Proxmox. +- No DHCP: capmox assigns static addresses from `ipv4Config` only. +- Linked clones keep the template's base snapshot referenced; rotate templates by creating a new VMID with new tags and rolling the pool. +- No live migration awareness: the CCM reports a VM it cannot see as gone and deletes the Node. +- Cozystack's Terraform provider does not model `substrate`/`proxmox.*` yet; `terraform apply` on such a cluster overwrites them. + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `InfrastructureProvider` not installing, fetch error | `capmox-config` Secret missing or in the wrong namespace | create it in `cozy-cluster-api` | +| every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` | +| `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly | +| Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS | +| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n logs deploy/kubernetes--pccm` | +| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM | +| VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` | +| `TalosConfigTemplate.Spec is immutable` | apiserver address changed on an existing cluster | delete the template; the reconcile Job recreates it | +| capmox stuck at `Waiting for Cluster Controller to set OwnerRef` | `cluster.x-k8s.io/managed-by` annotation on the `ProxmoxCluster` | not expected from the chart; report it | + +## See also + +- [Managed Kubernetes]({{% ref "/docs/next/kubernetes" %}}) — the application this page extends +- Cluster API provider for Proxmox: https://github.com/ionos-cloud/cluster-api-provider-proxmox +- Proxmox CCM / CSI: https://github.com/sergelogvinov/proxmox-cloud-controller-manager, https://github.com/sergelogvinov/proxmox-csi-plugin +- Support-tier proposal: https://github.com/cozystack/community/pull/98 diff --git a/content/en/docs/v1.7/install/_index.md b/content/en/docs/v1.7/install/_index.md index d286a44a..da0198a7 100644 --- a/content/en/docs/v1.7/install/_index.md +++ b/content/en/docs/v1.7/install/_index.md @@ -45,6 +45,13 @@ If that's your case, we recommend using the guides below: - [Servers.com]({{% ref "/docs/v1.7/install/providers/servers-com" %}}) +## Proxmox VE + +There are two ways to use Cozystack with an existing Proxmox VE cluster, and they can be combined: + +- Run the Cozystack cluster itself in Proxmox VE VMs: follow the [generic installation path](#generic-installation-path), treating the VMs like any other machines. Set the VM CPU type to `host` if tenants will run KubeVirt VMs on them. +- Run the workers of tenant Kubernetes clusters as VMs on Proxmox VE instead of KubeVirt: [enable the Proxmox VE substrate]({{% ref "/docs/v1.7/kubernetes/proxmox-substrate" %}}). This is opt-in and experimental. + ## Upgrading and Post-deployment Configuration After you've deployed a cluster, proceed to the [Cluster Administration]({{% ref "/docs/v1.7/operations" %}}) section for diff --git a/content/en/docs/v1.7/kubernetes/proxmox-substrate.md b/content/en/docs/v1.7/kubernetes/proxmox-substrate.md index 40d7a07e..5dadbe21 100644 --- a/content/en/docs/v1.7/kubernetes/proxmox-substrate.md +++ b/content/en/docs/v1.7/kubernetes/proxmox-substrate.md @@ -12,6 +12,10 @@ not yet covered by the project's release gate. Track the decision on support tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481). {{< /note >}} +This page is about running the **workers of tenant clusters** on Proxmox VE. +To run the management cluster itself on Proxmox VE VMs, follow the regular +[deployment guide]({{% ref "/docs/v1.7/install" %}}). + ## What it is By default a managed (tenant) Kubernetes cluster in Cozystack has its control @@ -46,9 +50,10 @@ The hypervisor token never enters the tenant cluster. | | Verified | Notes | |---|---|---| -| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 | +| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6; v1.7.0 needs the CCM fix in step 4 | | Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x | | capmox | v0.7.7 (shipped) | | +| Proxmox CCM | v0.16.1 | v1.7.0 ships v0.15.0, which panics; see step 4 | | Talos | the version in `kubernetes-nodes` `talos.version` | template must match it | | Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested | | Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* | @@ -70,22 +75,28 @@ On the **Proxmox VE** side: On the **Cozystack** side: -- a working management cluster with the `iaas` bundle; +- a working management cluster on a variant with the `iaas` bundle, such as + `isp-full` or `isp-full-generic`; - a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address. -## 1. Enable the packages +## 1. Enable the Proxmox provider -Add the Proxmox infrastructure provider to the platform's enabled packages. -The in-cluster IPAM provider is pulled in as its dependency. +The provider is an opt-in package, because it needs a Proxmox API token before +it can do anything. On a Proxmox VE node, create a user and a token for capmox, +following the provider's own +[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): -```yaml -# spec.components.platform.values of the cozystack.cozystack-platform Package -bundles: - enabledPackages: - - cozystack.capi-provider-infra-proxmox +```bash +pveum user add capmox@pve +pveum aclmod / -user capmox@pve -role PVEVMAdmin +pveum user token add capmox@pve cozystack -privsep 0 ``` -Before the provider is deployed, create the Secret it authenticates with. +`PVEVMAdmin` on `/` is what the provider documents; a narrower set is +described in its advanced-setup docs. This token is platform-wide: capmox +clones the workers of every tenant with it (see *Isolation*). + +Put the token into the Secret the provider authenticates with. capi-operator fetches it first; while it is missing nothing is installed and the `InfrastructureProvider` reports the fetch error. @@ -102,34 +113,41 @@ Three things that are easy to get wrong: - `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone. - a rotated token is picked up only after a restart of the capmox manager pod. -Check: +Then add the provider to the platform's enabled packages. The in-cluster IPAM +provider is pulled in as its dependency. ```bash -kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders -kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' +kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p '{ + "spec": { + "components": { + "platform": { + "values": { + "bundles": { + "enabledPackages": ["cozystack.capi-provider-infra-proxmox"] + } + } + } + } + } +}' ``` -## 2. Prepare Proxmox: user, tokens, pool +A merge patch replaces the whole list: if `bundles.enabledPackages` already has +entries, repeat them here. -Create a user and tokens for the platform. Use **privilege-separated tokens** -and grant the privileges to the user as well: Proxmox caps a token at its -user's permissions. - -For capmox itself, follow the provider's own -[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): +Check: ```bash -pveum user add capmox@pve -pveum aclmod / -user capmox@pve -role PVEVMAdmin -pveum user token add capmox@pve cozystack -privsep 0 +kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders +kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' ``` -`PVEVMAdmin` on `/` is what the provider documents; a narrower set is -described in its advanced-setup docs. This token is platform-wide: capmox -clones the workers of every tenant with it (see *Isolation*). +## 2. Prepare a tenant: pool and tokens For each **tenant** create a resource pool and a dedicated storage, and grant -the tenant's CCM and CSI tokens their privileges on those paths only: +the tenant's CCM and CSI tokens their privileges on those paths only. Use +**privilege-separated tokens** and grant the privileges to the user as well: +Proxmox caps a token at its user's permissions. | Component | Privileges | Path | |---|---|---| @@ -168,6 +186,7 @@ Workers are cloned from a Talos VM template that already exists on the hypervisor. Build it with the script shipped in the repository; it reads `talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so the template matches what the chart will render into the machine config. +Run it from a checkout of the Cozystack version the management cluster runs. ```bash # prints the program it would run, for review @@ -223,13 +242,34 @@ Nodes as `topology.kubernetes.io/region`. Note the URL here carries kubectl -n tenant-a create secret generic proxmox-csi \ --from-literal=url='https://pve.example.com:8006/api2/json' \ --from-literal=token_id='tenant-a@pve!csi' \ - --from-literal=token_secret='' \ + --from-literal=token_secret='' \ + --from-literal=region='pve' + +kubectl -n tenant-a create secret generic proxmox-ccm \ + --from-literal=url='https://pve.example.com:8006/api2/json' \ + --from-literal=token_id='tenant-a@pve!ccm' \ + --from-literal=token_secret='' \ --from-literal=region='pve' ``` `substrate` is immutable: switching an existing cluster is refused. Create a new one. +{{< warning >}} +Cozystack v1.7.0 ships a broken CCM image. The chart runs +`proxmox-cloud-controller-manager:v0.15.0`, and that tag now resolves to a build +that panics at start, so workers keep the `uninitialized` taint. The fix is in +[cozystack/cozystack#5150](https://github.com/cozystack/cozystack/pull/5150) and +is not released yet. Until then, set the image by hand once the cluster exists. +The next Helm upgrade of the cluster puts the chart's image back, so repeat this +after every change to the `Kubernetes` resource. +{{< /warning >}} + +```bash +kubectl -n tenant-a set image deploy/kubernetes-px-pccm \ + cloud-controller-manager=ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.16.1@sha256:474dbfd3be5c2e0866e520bc6ddb60b7accef554ca7401488eee35252cdf8734 +``` + ## 5. Add a worker pool ```yaml @@ -320,6 +360,7 @@ MachineHealthCheck. | every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` | | `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly | | Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS | +| CCM in `CrashLoopBackOff`, `panic: version string "0.15.0"` | the image v1.7.0 ships | set the image as in step 4 | | Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n logs deploy/kubernetes--pccm` | | CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM | | VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` |