[Marian Koreniuk (themoriarti)]

Design proposal for Proxmox-backed VPC subnets: a tenant puts Proxmox VMs, including the workers of a Proxmox-backed Kubernetes cluster, on a subnet of its own VirtualPrivateCloud, and the platform allocates the VLAN, the bridge and the IP pool instead of taking them from tenant values.

It comes from cozystack/cozystack#3481, which asks for a deliberate decision on Proxmox and for the provisioning layer to be specified first. That layer is in main now (cozystack/cozystack#4087, cozystack/cozystack#4483, cozystack/cozystack#4088, cozystack/cozystack#4089); this proposal is the networking piece on top of it and does not change how VMs are provisioned. It meets #86 (per-cluster infrastructure backends) only at the proxmox.network values of the kubernetes and kubernetes-nodes applications, which would move with a Proxmox backend if #86 is accepted.

What it asks reviewers to weigh:

• a new API group, proxmox.cozystack.io/v1alpha1, with an admin-owned ProxmoxNetworkZone and a per-subnet ProxmoxNetwork;
• a controller that writes HA_Chassis_Group rows to the OVN northbound database, outside Kube-OVN, so that a VM reached through a localnet port can resolve its gateway, as a stopgap until Kube-OVN does it itself;
• egress through VpcEgressGateway onto a shared transit network, with the L3 boundary between VPCs held by an admin-run transit router until a platform-owned address space allows a drop inside OVN;
• the open questions and the alternatives listed in the proposal, Proxmox SDN models among them.

The implementation is a draft pull request in cozystack/cozystack: cozystack/cozystack#4813. The proposal's numbers come from a run of its shell-based end-to-end suite on a two-host Proxmox VE 9 cluster; the Chainsaw suites that replace it have not run on a cluster yet.

Before review

• If this revises a merged design proposal: it adds a decision record under that proposal's decisions/ directory, or says below why none is needed.

Not a revision: this is a new proposal.

DCO

• Commits are signed off (git commit --signoff).

Assisted-by: LLM

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Aleksei Sviridkin (lexfrei)]

Marian Koreniuk (@themoriarti) The problem is real. On main a tenant picks the Proxmox bridge and vlan for its workers in the kubernetes-nodes values, and the address range in proxmox.ipv4Config, so nothing stops one tenant from putting its workers on another tenant's VLAN. That gap needs closing. I don't think this design is the way to close it now, for two reasons.

First, the size. The isolation gap only needs the platform to own the bridge, VLAN and pool of each tenant network: the VLAN allocator, the namespaced InClusterIPPool, and the admission policy that keeps machine NICs on the namespace's own networks. Everything else here is a feature on top of that. It puts VMs on L2 next to pods of a VPC subnet, adds a controller that writes HA_Chassis_Group rows into the OVN northbound database with Kube-OVN's own certificate, runs egress gateways with port security off, and needs a transit router outside the cluster. The NB writer is the part I would not take into the tree. The proposal itself says the right fix is in Kube-OVN, the gateway cells were run on v1.15.10 while main ships v1.15.26, and every Kube-OVN bump would have to be checked against it. So my answer to open question 2 is to wait for Kube-OVN, or to send the change there first.

Second, the boundary between tenants. The Goals say isolation holds at every layer, but traffic from one VPC to another VPC's CIDR is stopped only by the operator's transit router (Security, open question 1). Cozystack does not ship that router, does not test it, and cannot see when it is misconfigured. A boundary between tenants has to be something the platform owns, so the drop inside OVN belongs in the design and not in the open questions.

I'd split it. A smaller proposal for platform-allocated VLANs and pools with the admission policy closes the gap on main and stands on its own. The VPC integration can follow once Kube-OVN handles the gateway chassis and the drop between VPCs is designed in. Whether Proxmox becomes a supported platform is still the open question in cozystack/cozystack#3481, and a smaller first piece is easier to commit to while that is undecided.

[Marian Koreniuk (themoriarti)]

Thanks, and sorry for the slow reply. I agree with both points and I'm splitting it the way you suggest.

The first piece is smaller than what you asked for. It has no controller at all:

• The allocation is a static table. A cluster-scoped ProxmoxNetworkZone holds the bridge and, per tenant network, the namespace, the VLAN, the address range, the gateway and the prefix. The admin writes it in the package values and the package chart renders it, together with one namespaced InClusterIPPool per network. No allocator, no status, no image.
• A ValidatingAdmissionPolicy with paramKind: ProxmoxNetworkZone covers ProxmoxMachine, ProxmoxMachineTemplate and ProxmoxCluster. One binding per zone, each judging only its own bridge: a NIC on the zone bridge must be one of the (VLAN, pool) pairs the zone gives the request's namespace, and the ProxmoxCluster pool must be the sentinel that holds only the gateway. The policy also covers an update that changes a machine's network, since the capmox webhook does not make that field immutable.
• kubernetes and kubernetes-nodes take proxmox.network.name and read bridge, VLAN, pool, gateway and prefix from the zone with lookup, in the release namespace only.

No Kube-OVN objects, no OVN northbound writes, no transit network, no egress gateway. Tenants have no RBAC on infrastructure.cluster.x-k8s.io or ipam.cluster.x-k8s.io, so the charts rendered from tenant values are today's enforcement point and admission is defense in depth. L3 between tenant VLANs stays on the operator's router, exactly as on main today; the proposal says so rather than claiming more.

On open question 2: the NB writer stays out of the tree. The VPC integration waits until Kube-OVN manages the gateway chassis of these subnets itself.

On the boundary between VPCs: agreed, it belongs to the platform and inside OVN. It is designed in the follow-up, as a router policy on each VPC rendered from a platform-owned list and placed ahead of the egress reroute, with test cells of its own. I've turned this pull request into a draft and retitled it as that follow-up, so your review stays with it.

The first piece is proposed in #104. Its implementation will be a new, small PR from main; cozystack/cozystack#4813 stays a draft for reference only. Two things from #4813 are independent of both pieces: the Proxmox CCM v0.16.1 fix went separately as cozystack/cozystack#4917, and the /30 netmask for the VPC peering address goes as its own small PR.

[myasnikovdaniil]

Hi! We removed Status and Date from proposal headers in #111. Could you remove these fields when you next update your proposal? Decision records keep their metadata.

Sorry for the extra work, I wanted to mention this now so it doesn't come up late in review.

[Marian Koreniuk (themoriarti)]

myasnikovdaniil thanks for the heads-up. Status and Date are gone from the header (7134604). The proposal stays a draft on hold, since its first part moved to #104.