[Andrei Kvapil (kvaps)]

Summary

Placeholder for Phase 3 of the kubernetes-application reshape: workers in environments outside the Cozystack management cluster (cloud autoscaling against Hetzner/Azure/AWS/GCP, BYO clusters, bare-metal/on-prem workers).

Held in draft pending PR #8 (Phase 1 + Phase 2: Talos migration + package split). Phase 3 depends on the architectural seam delivered by PR #8 and on operational experience from Phase 1+2.

This proposal does not commit to any specific shape for Phase 3. It documents the intended scope, the reasons for deferral, and a set of non-committal sketches collected during early discussion of PR #8 — so the design conversation does not restart from zero when work resumes.

Test plan

This is a placeholder proposal — no implementation, no tests. Implementation testing will be scoped when this proposal is filled in.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8c10b4fb-ac15-49ee-95bb-9369408b0d24

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces a design proposal for Phase 3 of the hybrid Kubernetes cluster implementation, focusing on supporting worker nodes in external environments such as public clouds and on-premises datacenters. The feedback provided focuses on improving the technical documentation's clarity and grammatical precision, including corrections for terminology like 'on-premises', ensuring proper pronoun agreement, removing redundant phrasing, and standardizing US English spelling.

[gemini-code-assist]

In technical documentation, "on-premises" is the correct adjective to describe hardware or software located on the site of an organization. "On-premise" refers to a thought or proposition. Also, "tenant" is singular, so "its own" is more appropriate than "their own".

Suggested change

	- **Bare metal / on-premise workers**: a tenant wanting nodes in their own datacenter joined to a Cozystack-hosted control-plane.
	- **Bare metal / on-premises workers**: a tenant wanting nodes in its own datacenter joined to a Cozystack-hosted control-plane.

[gemini-code-assist]

Grammatical correction: "each" is singular and should be followed by "its own" rather than "their own".

Suggested change

	The Novolos use case is the concrete driving example: workers in different tenant clouds, each with their own `cluster-autoscaler`, all joining a single managed Kamaji control-plane.
	The Novolos use case is the concrete driving example: workers in different tenant clouds, each with its own cluster-autoscaler, all joining a single managed Kamaji control-plane.

[gemini-code-assist]

The phrase "tenant-owned location ownership" is redundant. Using "tenant-level location ownership" or "ownership of tenant-owned locations" is more concise.

Suggested change

	3. Several open Cozystack-side decisions (admin- vs tenant-owned location ownership, credential model for BYO clouds, default deny vs explicit advertise, dashboard surfacing) are best made with concrete Phase 1 + 2 operational experience in hand, not in advance.
	3. Several open Cozystack-side decisions (admin- vs tenant-level location ownership, credential model for BYO clouds, default deny vs explicit advertise, dashboard surfacing) are best made with concrete Phase 1 + 2 operational experience in hand, not in advance.

[gemini-code-assist]

The project appears to follow US English spelling conventions (e.g., "reshape", "autoscaler"). "Realize" should be used instead of "realise".

Suggested change

	- **New `backend.type` field in `kubernetes-nodes`.** The single-backend "kubevirt-talos" shape from Phase 2 grows a discriminator: `kubevirt-talos`, `cloud-talos-hetzner`, `cloud-talos-azure`, etc. Per-backend sub-charts realise the actual lifecycle (CAPI for KubeVirt-VM backends; `cluster-autoscaler` directly against the cloud's native API for cloud backends).
	- **New `backend.type` field in `kubernetes-nodes`.** The single-backend "kubevirt-talos" shape from Phase 2 grows a discriminator: `kubevirt-talos`, `cloud-talos-hetzner`, `cloud-talos-azure`, etc. Per-backend sub-charts realize the actual lifecycle (CAPI for KubeVirt-VM backends; `cluster-autoscaler` directly against the cloud's native API for cloud backends).

[IvanHunters]

The sketches cover how a pool of another backend gets its machines. I wrote up the other half: what the kubernetes and kubernetes-nodes charts already run in every tenant cluster on the assumption that each worker is a KubeVirt VM, and what happens to each of those components on a node from another backend. It is one commit on top of this branch, so you can cherry-pick it or take only parts of it:

6e4e1f5 (branch proposal/hybrid-backend-bound-components)

The inventory is pinned to cozystack ff52aeb44 and to the upstream revisions it cites. As a real-world reference it uses aenix-org/kubernetes-switchcloud (OpenStack/CAPO workers joined to a Kamaji control plane), which had to ship its own providerID setter, an in-cluster apiserver proxy, a load-balancer controller and an SNI router for trustd and konnectivity.

Two points touch text that is already in the proposal, so they are your call:

• The last possible direction (CAPI infrastructure providers for cloud pools) conflicts with "CAPI is bypassed" under Out of scope. I left that line unchanged and marked the direction as unresolved.
• CAPO and CAPH both resolve their infra cluster from Cluster.spec.infrastructureRef, so a CAPI pool of another provider cannot join a cluster whose infrastructure is KubevirtCluster. This is now one of the open questions.

Like the sketches, none of this is a commitment.

[IvanHunters]

Moved this into a separate supplementary proposal, #86, so this PR stays untouched. It has the same content as the commit linked above, reworked into the proposal template and re-reviewed.

[IvanHunters]

Update: #86 now takes a position instead of only listing components. For single-backend clusters it proposes one CAPI infrastructure provider per cluster, with backends as separate package sources on a shared library chart, modelled on kubernetes-switchcloud. It uses CAPO rather than bypassing CAPI there, which differs from the sketches here; mixed clusters and the non-CAPI path stay with this proposal.

[Marian Koreniuk (themoriarti)]

Andrei Kvapil (@kvaps), this proposal is very relevant to us, and we are moving toward implementing it. PR #8 merged in June and kubernetes-nodes has shipped since v1.6.0, so the prerequisite this placeholder waits for is in place. We would like to restart Phase 3 together, and we have a concrete case and a stand to test it on.

Our case. One tenant cluster, with its control plane in Kamaji on our Cozystack management cluster and worker pools in two places:

• a Proxmox VE cluster in our lab, the same one that runs the management cluster;
• Proxmox VE on Hetzner dedicated servers. This is a separate PVE cluster with its own API endpoint and no shared corosync, reachable only over the WAN.

Both pools are Talos and both are Proxmox. This is the "bare metal / on-premise workers" case from the proposal, with one difference: the backend is the same but the location is not.

Where it meets the current shape. #86 leaves mixed clusters to this proposal, and our case turns out to be one even though both sides are Proxmox. capmox resolves the infra cluster of a ProxmoxMachine from Cluster.spec.infrastructureRef (getInfraCluster in internal/controller/proxmoxmachine_controller.go) and builds its PVE client from the credentialsRef of that single ProxmoxCluster. ProxmoxMachine has no credentials of its own. So one CAPI Cluster maps to exactly one PVE endpoint. #86 found the same constraint for CAPO and CAPH; here it applies to two locations of one provider.

What we would like to agree on before writing code:

1. Does a pool in a second location of the same backend belong here or in design-proposal: per-cluster infrastructure backends for tenant Kubernetes clusters #86? We would put it here, because it needs a location concept either way.

2. How a pool reaches the API of its location. The options we see:

   • (a) a LocationProfile-style object referenced per pool, with the infrastructure provider able to take credentials from it. For capmox that is a change in capmox itself;
   • (b) one CAPI Cluster per location, all sharing one Kamaji control plane. This keeps MHC, rolling replacement and the clusterapi autoscaler, but a Cluster without a control plane of its own is outside what CAPI expects, and we have not tested it;
   • (c) the non-CAPI path from the sketches: Talos machine config plus a bootstrap token, with an autoscaler and NLC.

   Is there a preferred direction, or a known reason to rule one of them out?

3. Networking between locations. Workers on Hetzner need the Kamaji API, konnectivity and trustd, and pod traffic between locations crosses the internet. Should this be the Kilo mesh, Cilium WireGuard encryption, or an L3 link the operator provides as a prerequisite?

4. Location labels on nodes, so that CSI, CCM and load balancers can be confined to one location. design-proposal: per-cluster infrastructure backends for tenant Kubernetes clusters #86 notes that none exist today.

What we can bring. The stand described in #98: a Cozystack management cluster on Proxmox with tenant pools on capmox. We can add a Hetzner location to it, run the hybrid case end to end and report the results here. We can also do the writing: fill in this placeholder, or open a follow-up proposal if you would rather keep this one yours, with Proxmox in two locations as the first concrete case.

Is now a good time to resume this, and would you be OK with us driving the next draft?

cc IvanHunters, since #86 draws the line between the two proposals.

[myasnikovdaniil]

Hi! We removed Status and Date from proposal headers in #111. Could you remove these fields when you next update your proposal? Decision records keep their metadata.

Sorry for the extra work, I wanted to mention this now so it doesn't come up late in review.