From 91182de61c4fb502e4376d450d0573aab073d1db Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Tue, 14 Jul 2026 15:23:10 +0300 Subject: [PATCH 01/16] design-proposal: public IPs as a first-class resource (PublicIP/PublicIPClaim/PublicIPClass) A tenant cannot own a public address today: it exists only as a side effect of a Service type: LoadBalancer and evaporates with it. Nothing can be held across a workload rebuild, moved between workloads, quota'd, or enumerated. Applies the PersistentVolume pattern to addresses -- PublicIPClass (the class), PublicIP (the cluster-scoped inventory object), PublicIPClaim (the namespaced request) -- plus a provisioner contract in the shape of CSI, with an explicit Pin capability that the whole model depends on. Records that this is a regression rather than a wishlist: Cozystack has reserve-then-associate today through kube-ovn's OvnEip, and migrating off kube-ovn drops it. Stub: object model and primitives survey are worked out; controller mechanics, provisioner contract details and rollout are sketched, and opened early to settle the shape before code. Signed-off-by: Timofei Larkin --- design-proposals/public-ip-claims/README.md | 433 ++++++++++++++++++++ 1 file changed, 433 insertions(+) create mode 100644 design-proposals/public-ip-claims/README.md diff --git a/design-proposals/public-ip-claims/README.md b/design-proposals/public-ip-claims/README.md new file mode 100644 index 00000000..a5f25739 --- /dev/null +++ b/design-proposals/public-ip-claims/README.md @@ -0,0 +1,433 @@ +# Public IPs as a first-class resource: `PublicIP`, `PublicIPClaim`, `PublicIPClass` + +- **Title:** `Public IPs as a first-class resource: PublicIP, PublicIPClaim, PublicIPClass, and a provisioner contract` +- **Author(s):** `@lllamnyp` +- **Date:** `2026-07-14` +- **Status:** Draft + +> **This is a stub.** The problem statement, the primitives survey, and the object +> model are worked out; the controller mechanics, the provisioner contract details, +> and the rollout are deliberately sketched. It is opened early to settle the +> **shape** — is an address a resource? — before anyone writes code. + +## Overview + +A Cozystack tenant cannot *own* a public address. They can only cause one to appear +as a side effect of creating a `Service type: LoadBalancer`, and it evaporates when +that Service does. There is no object to hold, keep, quota, bill, hand to another +workload, or point DNS at with confidence. + +This proposal makes the address a resource. Applying the `PersistentVolume` pattern +to addresses: + +| storage | addresses | role | +|---|---|---| +| `StorageClass` | **`PublicIPClass`** | which pool, which provisioner, which announcer | +| `PersistentVolume` | **`PublicIP`** (cluster-scoped) | the address itself, with a `claimRef` and a reclaim policy | +| `PersistentVolumeClaim` | **`PublicIPClaim`** (namespaced) | "give me one" — what a tenant creates | +| CSI driver | **a provisioner contract** | how a class's backend allocates, adopts, and pins | + +The experience we are after is the AWS one: **allocate an address, keep it, attach it +to something, detach it, attach it to something else, release it when done.** Not: +"create a Service and hope you get the same IP back." + +## Scope and related proposals + +- **Adjacent, not a dependency:** `design-proposals/structured-external-exposure` + (community #29) restructures how a managed application *requests* exposure. This + proposal is about what an address **is**. The two meet at one point — an exposure + entry should be able to name a `PublicIPClaim` instead of implicitly minting an + address — but neither needs the other to land first, and this proposal does not + assume #29's shape. +- **Related:** an `ExposureClass` kind exists today in `network.cozystack.io`. This + proposal takes a position on it (see *Alternatives*): the **class** idea is right + and should survive in some form; binding an address's lifetime to a `Service` is + the part that cannot serve this use case. +- **Deferred to a sibling:** the *datapath* for whole-IP 1:1 NAT (all ports in, and + the workload egressing **as** that address) — today provided by + [cozy-proxy](https://github.com/cozystack/cozy-proxy)'s nftables rules. That is a + CNI concern, tracked separately. **This proposal is the allocation half; that is + the forwarding half.** They are independent: allocation is useful with a stock + LoadBalancer Service and no 1:1 NAT at all. + +## Context + +### What exists today + +Cozystack installs MetalLB but renders no `IPAddressPool`/`L2Advertisement` itself; +the admin configures an allocator to suit the environment (MetalLB L2/BGP, Cilium +LB-IPAM, a cloud LB, or `Service.spec.externalIPs` pinning). + +Across every one of those, the model is the same: **an address is allocated to a +Service, at the moment the Service is created, and freed when it is deleted.** + +### The problem + +> *"I gave my customer this IP address. It is in their firewall allow-list. I need to +> rebuild the VM behind it, and I need the address to still be mine afterwards."* + +> *"I want to move this address from the old VM to the new one during a cutover."* + +> *"I want to reserve four addresses now, because the range is nearly full, and attach +> them over the next month."* + +None of these are expressible. The address has no independent existence, so: + +- **It cannot be held.** Delete the Service, lose the address. Someone else may get it. +- **It cannot be moved.** There is no detach/attach; there is only "hope the allocator + hands you the same one." +- **It cannot be quota'd.** There is no object to count, so nothing bounds how many + public addresses a tenant consumes. The only lever is pre-carving a pool per tenant. +- **It cannot be gated.** Anyone who can create a `Service` in a namespace can cause a + public address to be allocated. RBAC authorizes *verbs on resources*, not *fields*. +- **It cannot be shown.** No inventory. "Which addresses do we own, and who has them?" + has no answer short of listing every Service in the cluster. + +### This is a regression, not a wishlist + +Cozystack has reserve-then-associate **today**, through kube-ovn: `OvnEip` is a +cluster-scoped object created independently, then bound by an `OvnFip` (1:1 NAT), +`OvnSnatRule`, or `OvnDnatRule` that references it by name. The legacy iptables +gateway mode has the same shape (`IptablesEIP` + `IptablesFIPRule`). + +**Migrating off kube-ovn drops this capability on the floor** unless something +replaces it. That reframes the proposal: the question is not "should Cozystack gain an +AWS-style EIP", it is "on what terms does Cozystack keep the EIP it already has." + +## Goals + +- A tenant can request an address and receive one, without an admin pre-creating a + pool object for them. +- An address **survives** the deletion of the workload it was attached to (reclaim + policy `Retain`), and can be attached to a different workload afterwards. +- An operator can enumerate every public address the cluster owns, and see who holds it. +- The number of addresses a tenant may hold is bounded by a **plain `ResourceQuota`** + (`count/publicipclaims.`), with no new quota machinery. +- An admin configures the address source **once per class**, not once per tenant and + not once per address. No pool-per-tenant, no `/32` pools. +- The tenant-facing API does not name the backend. A tenant asks for an address; it + does not learn whether MetalLB, Cilium, or a cloud API produced it. +- The design admits at least: MetalLB, Cilium LB-IPAM, and a cloud provider whose + addresses are allocated outside the cluster. + +### Non-goals + +- **Announcing addresses.** Attraction (ARP/NDP, BGP, a cloud VNIC assignment) stays + with the LB implementation. This proposal allocates and binds; it never puts a + packet on a wire. +- **The 1:1 NAT datapath.** See *Scope*. +- **Replacing the LB implementation.** MetalLB/Cilium/cloud stay exactly where they + are; this sits above them. +- **IPv4-only thinking.** The model is family-agnostic; a claim may request v4, v6, or + both. (Whether one `PublicIP` carries both families or a claim binds two is an open + question below.) +- **Solving field-level authorization in general.** It is *used* here (see *Security*) + but the general problem is bigger than this proposal. + +## Design + +### 1. The primitives that already exist (survey) + +Load-bearing findings, because the design is shaped by them: + +1. **Nothing has a standing reservation.** Not MetalLB, not Cilium LB-IPAM, not + kube-vip, not PureLB, not OpenELB — and not Kubernetes core. KEP-1880 shipped + cluster-scoped `ServiceCIDR`/`IPAddress` kinds (the name *is* the IP, with a + generic `spec.parentRef`), and then explicitly named *"any generalization onto + something like an IPAM API"* a **non-goal**. Nothing upstream is coming to provide + this. It is ours to build. + +2. **Every backend has a "pin this exact address" hook, and it is always an + annotation** — because `Service.spec.loadBalancerIP` was deprecated in Kubernetes + 1.24 with *no core replacement*, so every implementation invented its own: + + | backend | pin mechanism | + |---|---| + | MetalLB | `metallb.io/loadBalancerIPs` | + | Cilium LB-IPAM | `lbipam.cilium.io/ips` | + | kube-vip | `kube-vip.io/loadbalancerIPs` | + | OpenELB | `eip.openelb.kubesphere.io/v1alpha1` | + | AWS LB Controller | `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` | + | GCP / GKE | `networking.gke.io/load-balancer-ip-addresses` | + | PureLB | **none found** — see *Open questions* | + + Different spelling, identical semantics. **This is the seam the whole design rests + on:** if we can pin, we can bind a reserved address to a workload. + +3. **A reserved-but-unattached address is inert, everywhere.** MetalLB never announces + an address that is not the live `status.loadBalancer.ingress` of a Service it + allocated. Cilium LB-IPAM only *assigns*; something else announces. A cloud EIP + routes nowhere until associated. This is not an obstacle — **it is correct + semantics**, and it is exactly why AWS bills for an idle EIP. + +4. **One pool can serve many tenants.** MetalLB's + `IPAddressPool.spec.serviceAllocation.{namespaces,namespaceSelectors,serviceSelectors}` + and Cilium's `CiliumLoadBalancerIPPool.spec.serviceSelector` both scope a *single* + pool by namespace or label. And `L2Advertisement`/`BGPAdvertisement` select + **pools**, not addresses — so one advertisement covers arbitrarily many addresses. + **The pool-per-tenant and `/32`-pool patterns were never necessary.** + +5. **`Service.spec.loadBalancerClass`** (GA in 1.24) is the standard way for multiple + LB implementations to coexist; MetalLB honours it via `--lb-class`. It is how a + cluster can run more than one backend without them fighting. + +### 2. The object model + +```yaml +# Admin, once per address source. +apiVersion: network.cozystack.io/v1alpha1 +kind: PublicIPClass +metadata: + name: public + annotations: + publicipclass.network.cozystack.io/is-default-class: "true" +spec: + provisioner: metallb.cozystack.io # who fulfils claims of this class + reclaimPolicy: Retain # Retain | Delete (default Retain) + parameters: # opaque to the core controller + addresses: ["203.0.113.0/24"] +--- +# Tenant. This is the whole tenant-facing API. +apiVersion: network.cozystack.io/v1alpha1 +kind: PublicIPClaim +metadata: {name: web, namespace: tenant-a} +spec: + className: public # empty => the default class + family: IPv4 # IPv4 | IPv6 | Dual +status: + phase: Bound + publicIPName: pip-203-0-113-7 + address: 203.0.113.7 # what the tenant reads, and puts in DNS +--- +# Cluster-scoped. The inventory. Created by the provisioner, not the tenant. +apiVersion: network.cozystack.io/v1alpha1 +kind: PublicIP +metadata: {name: pip-203-0-113-7} +spec: + className: public + address: 203.0.113.7 + reclaimPolicy: Retain + claimRef: {namespace: tenant-a, name: web} + source: # a union — see below + fromClass: {} # this controller allocated it + # providerRef: {id: eipalloc-0a1b} # ...or it wraps a provider-side reservation +status: + phase: Bound + associatedTo: # nil => reserved but inert + kind: Service + namespace: tenant-a + name: web-lb +``` + +### 3. The address-source union — the reason this is not a MetalLB adapter + +There are two structurally different worlds, and the model must admit both: + +- **Self-allocating backends** (MetalLB, Cilium, kube-vip, OpenELB): *nothing anywhere* + holds a reservation. **Our provisioner is the IPAM of record.** `PublicIP` is + authoritative, and its address is carved from the class's range (`source.fromClass`). +- **Provider-reservation backends** (AWS, GCP, Hetzner): the reservation is already a + real object with a stable handle (`eipalloc-…`, a named static address), guarded by + the provider's own IAM. **Our provisioner must adopt, not allocate** + (`source.providerRef`). + +This is precisely the `PersistentVolume` volume-source union (`spec.csi` / +`spec.nfs` / …), and it is the strongest evidence the shape is not bent around any one +backend: it survives a backend where allocation is not ours at all. The tenant's +`PublicIPClaim` never learns the difference. + +### 4. The provisioner contract (the "CSI analogue") + +`PublicIPClass.spec.provisioner` names a controller, exactly as `StorageClass` names a +CSI driver. Each provisioner declares **capabilities**, and the core controller refuses +what a backend cannot do rather than half-supporting it: + +| capability | meaning | consequence if absent | +|---|---|---| +| `Allocate` | can carve an address from a class range | class must use `providerRef` adoption | +| `Adopt` | can wrap a provider-side reservation | `providerRef` claims rejected | +| **`Pin`** | **can bind a specific address to a Service** | **the class cannot serve claims at all** | + +`Pin` is not optional decoration — **it is the capability the entire model depends on.** +A backend that cannot be told *which* address to use can never attach a reserved one, +and a class over such a backend must reject `PublicIPClaim`s outright instead of +allocating an address it can never bind. (PureLB may be exactly this case; see *Open +questions*.) + +Whether provisioners are in-tree (a switch in one controller) or out-of-tree (separate +deployments, a real CRD contract, as CSI did) is an **open question**. In-tree is +right for three backends; the CSI lesson is that it stops being right at about five. + +### 5. Association + +Association is a **separate, reversible act** — that is the entire point, and the thing +the current model cannot express. + +A workload references the **claim**, never the raw address: + +```yaml +kind: Service +metadata: + annotations: + network.cozystack.io/public-ip-claim: web # a claim in THIS namespace +spec: {type: LoadBalancer, ...} +``` + +The controller resolves the claim → address, verifies the claim is in the Service's own +namespace, and writes the backend's pin annotation (§1.2). MetalLB/Cilium/the cloud then +allocates and announces exactly that address. + +Remove the annotation and the announcement is withdrawn, but the `PublicIP` stays +`Bound` to its claim and simply becomes inert — **a reserved address, held, attached to +nothing.** Which is what an unassociated EIP is. + +### 6. What an admin actually configures + +For MetalLB, forever, for the whole cluster: + +- **one** `IPAddressPool` covering the range, and +- **one** `L2Advertisement` (or `BGPAdvertisement`) selecting it. + +That is all. No pool per tenant, no `/32`s. The `PublicIPClass` carries the range; the +provisioner renders the backend objects. + +## User-facing changes + +- **Tenants** gain one kind: `PublicIPClaim`. Create it, read `status.address`, put that + in DNS, reference it by name from a Service. They never see `PublicIP`, + `PublicIPClass`, MetalLB, or a pin annotation. +- **Admins** gain `PublicIPClass` (one per address source) and `PublicIP` (a read-mostly + inventory: `kubectl get publicips` finally answers "what do we own and who has it"). +- **Quota** works with no new machinery: `count/publicipclaims.network.cozystack.io` in a + stock `ResourceQuota`. + +## Upgrade and rollback compatibility + +- **Additive.** Nothing about today's `Service type: LoadBalancer` path changes; a + Service with no claim annotation behaves exactly as now. +- **Adoption path.** A cluster with existing LoadBalancer Services can import their + addresses as `PublicIP` objects with `source.providerRef`/`fromClass` and a `claimRef` + to a generated claim — an offline, reversible migration. +- **kube-ovn.** `OvnEip` holders need a migration story. Sketch only: enumerate `OvnEip`s, + mint an equivalent `PublicIP` + `PublicIPClaim` per address, re-point the association. + **Not designed here.** +- **Rollback.** Deleting the CRDs with `reclaimPolicy: Retain` leaves the backend objects + and the live Services untouched; addresses stay where they are. With `Delete`, it does + not — flagged as the irreversible case. + +## Security + +**The pin annotation is a privilege-escalation surface, and this is the sharp edge of +the proposal.** + +RBAC authorizes *verbs on resources*, not *fields*. Today, anyone who can create a +`Service` in a namespace can write `metallb.io/loadBalancerIPs: ` — and +because a self-allocating backend has **no concept of a reservation**, it will happily +hand over an address that another tenant has reserved but not yet attached. That is a +theft window, and it exists on MetalLB and Cilium alike. It is *not* a cloud problem: +there, IAM already guards `eipalloc-…`. + +The fix is the same move Cozystack already makes elsewhere — **turn the ungatable field +into a reference to an RBAC-gated object**: + +1. A tenant may only write `network.cozystack.io/public-ip-claim`, naming an object in + their **own namespace**, which RBAC *can* gate. +2. The **controller** writes the backend's raw pin annotation. +3. A `ValidatingAdmissionPolicy` **rejects any principal but the controller** writing + *any* backend pin annotation (§1.2's list) on a Service. + +Without (3) the whole model is advisory. It should land in the same release as the CRDs, +not after. + +Note this does not, by itself, close field-level authorization in general — a tenant who +can create a `PublicIPClaim` can still consume an address. That is what the class and the +`ResourceQuota` are for, and it is a bound, not a gate. + +## Failure and edge cases + +- **Class range exhausted** → claim stays `Pending` with a reason; no partial binding. +- **Claim deleted, `reclaimPolicy: Retain`** → `PublicIP` goes `Released`, keeps the + address, is not reusable until an admin clears the `claimRef`. (PV semantics, deliberately.) +- **Claim deleted, `Delete`** → address returned to the range; the backend object is torn down. +- **Service references a claim in another namespace** → rejected. Cross-namespace address + sharing is not a thing. +- **Two Services reference one claim** → the second is rejected. A 1:1 binding is 1:1; + silently letting the second win is how an address goes quietly dead. +- **Backend lacks `Pin`** → the class rejects claims at admission, loudly, rather than + allocating an address that can never be attached. +- **Address adopted from a provider, then released provider-side** → `PublicIP` goes + `Lost`. It must not silently re-allocate. + +## Testing + +- **Unit:** the allocator (range carving, exhaustion, reclaim transitions, the source union). +- **Integration:** claim → bind → associate → *delete the Service* → assert the address is + still held → associate it to a **different** workload → assert the same address comes back. + **That single test is the proposal.** It is precisely what cannot pass today. +- **e2e (per backend):** MetalLB and Cilium, on a real cluster: an external client reaches + the workload on the reserved address, and still reaches it after the workload is + rebuilt behind the same claim. +- **Admission:** a tenant writing a raw pin annotation is rejected; a tenant reserving an + address and a second tenant attempting to pin it directly is rejected. + +## Rollout + +Sketch — sequencing is an open question: + +1. CRDs + core controller + the **MetalLB** provisioner + the admission policy. (The + admission policy is not phase 2. See *Security*.) +2. The Cilium provisioner. Proves the abstraction is not a MetalLB adapter — **this is + the phase that either validates or falsifies the design**, and it should come early. +3. A cloud provisioner (adoption path, `source.providerRef`). Proves the source union. +4. kube-ovn `OvnEip` migration. + +## Open questions + +1. **Is this Cozystack's job at all?** It is not a CNI concern (a CNI consumes an address + that lands on a node; it has no business owning the inventory). It is not upstream's + (KEP-1880 says so explicitly). Platform seems right — but it is worth asking whether + this belongs in a standalone project rather than in Cozystack core. +2. **Provisioners: in-tree or out-of-tree?** In-tree for three backends; CSI's history + says that stops scaling. Where is the line, and do we want the CRD contract on day one? +3. **Reuse Cluster API's IPAM?** `ipam.cluster.x-k8s.io` `IPAddressClaim`/`IPAddress` is + *literally* this pattern — namespaced claim, concrete address object, a documented + third-party provider contract. It has only ever been wired to **Machine** addressing, + never Services. Do we adopt those kinds, or mint our own and merely copy the contract? + (Leaning: our own — we need reclaim policy and association, which it has no concept of. + But the shape is not novel and review should know that.) +4. **`ExposureClass`.** A class kind already exists in `network.cozystack.io`. Is + `PublicIPClass` a second class kind, or the same one grown a provisioner? Answering + this depends on the fate of `ExposureClass`/`ServiceExposure`, which is being + re-examined independently. +5. **Dual-stack.** Does one `PublicIP` carry a v4 and a v6 address, or does a `Dual` claim + bind two `PublicIP`s? (PV has no precedent. Leaning: two objects, one claim.) +6. **PureLB.** Does it have *any* "request this exact IP" mechanism? If not, it cannot be a + backend, and that should be stated rather than discovered. +7. **Sharing one address across Services** (different ports — GCP and Cilium both permit + it). Does a claim bind to one Service, or may several reference it on disjoint ports? + +## Alternatives considered + +**Bind the address to a Service (`ServiceExposure`-shaped).** A namespaced object naming +a `serviceRef`, which allocates an address and reports it in status. **Rejected as the +model for *this* problem:** it fuses *allocation* with *association*, so the address's +lifetime is the Service's lifetime — which is the exact thing being fixed. It cannot +express "keep this address, the workload is gone", which is the entire user request. +The **class** half of that idea is right and is kept here; the exposure half is not a +reservation and cannot be made into one. + +**Per-tenant pools.** Give each tenant an `IPAddressPool` (or a `/32` per address) and let +MetalLB do the rest. **Rejected:** it is the status quo's failure mode, it makes the admin +the allocator, it scales as O(tenants) or O(addresses) in operator-managed objects, and it +*still* provides no reservation — the address is released the moment the Service goes away. + +**Let the tenant write the backend's pin annotation directly.** **Rejected:** unauthorizable +(RBAC does not gate fields), and it hands every tenant the ability to steal any reserved +address that is not currently attached. See *Security*. + +**Wait for upstream.** **Rejected:** KEP-1880 lists generalizing to an IPAM API as an +explicit non-goal, and `loadBalancerIP` was deprecated with no replacement. There is +nothing to wait for. + +**Do nothing.** **Rejected:** it is a regression. Cozystack has reserve-then-associate today +via kube-ovn's `OvnEip`, and the migration off kube-ovn removes it. From 01be56d97cbde29433f73c70dc9dd0d63ef85fe8 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Wed, 15 Jul 2026 21:28:38 +0300 Subject: [PATCH 02/16] design-proposal: revise public-IP resource proposal per review Rework the stub in response to review and maintainer steers: - Rename the kinds PublicIP/PublicIPClaim/PublicIPClass -> IPAddress/ IPAddressClaim/IPAddressClass, in the ipam.cozystack.io group. A class names a pool, and pools are not necessarily public; the resource is the general address. - Drop the "regression, not a wishlist" framing entirely. Cozystack uses kube-ovn only for subnets and VPCs, never its EIP objects, so there is no prior reserve-then-associate capability to preserve. Argue on merits: hold, move, quota, enumerate, bill. - Positioning: standalone-valuable AND part of the Cozystack API surface, both -- precedent etcd-operator.cozystack.io. Designed for clean Cozystack integration without hard-depending on Cozystack machinery. - Rework Security and the worked examples into two honest layers rather than a static invariant. autoAssign:false only stops automatic selection; a Service can still name the pool (metallb.io/address-pool) or pin an IP, and a cluster-admin can impersonate the controller SA past any identity-keyed admission policy. Layer 1 (admission) reduces the blast radius for tenants; layer 2 (reconciliation) turns any collision -- including impersonation -- into a detected IPAddress Conflict rather than silent theft. The cluster-admin/impersonation case is stated as a trust boundary, not handwaved. The VAP also covers pool-select annotations, identifies the controller by ServiceAccount, and allowlists GitOps/system principals so reconciliation is not broken. - Add a section on allocator-first allocation: MetalLB assigns an address the moment it sees a Service, before any claim exists, so the controller must also reconcile Service -> inventory after the fact. Commit only to eventual consistency; note the volumeClaimTemplate analogy is weak because an IP is non-fungible and the allocator does not wait for our controller. - Status schema is a list (dual-stack), addressing the scalar-field gap. - Defer multiple platform-owned addresses per workload as out of scope, with a note that the case may not be real (a router/VPN VM's extra addresses live on a tunnel the platform neither owns nor sees). - Add a Naming note on the IPAddress overlap with core (KEP-1880) and Cluster API IPAM. Assisted-by: LLM Signed-off-by: Timofei Larkin --- design-proposals/public-ip-claims/README.md | 410 +++++++++++++++----- 1 file changed, 305 insertions(+), 105 deletions(-) diff --git a/design-proposals/public-ip-claims/README.md b/design-proposals/public-ip-claims/README.md index a5f25739..c348eb3f 100644 --- a/design-proposals/public-ip-claims/README.md +++ b/design-proposals/public-ip-claims/README.md @@ -1,8 +1,8 @@ -# Public IPs as a first-class resource: `PublicIP`, `PublicIPClaim`, `PublicIPClass` +# IP addresses as a first-class resource: `IPAddress`, `IPAddressClaim`, `IPAddressClass` -- **Title:** `Public IPs as a first-class resource: PublicIP, PublicIPClaim, PublicIPClass, and a provisioner contract` +- **Title:** `IP addresses as a first-class resource: IPAddress, IPAddressClaim, IPAddressClass, and a provisioner contract` - **Author(s):** `@lllamnyp` -- **Date:** `2026-07-14` +- **Date:** `2026-07-15` - **Status:** Draft > **This is a stub.** The problem statement, the primitives survey, and the object @@ -12,31 +12,55 @@ ## Overview -A Cozystack tenant cannot *own* a public address. They can only cause one to appear -as a side effect of creating a `Service type: LoadBalancer`, and it evaporates when -that Service does. There is no object to hold, keep, quota, bill, hand to another -workload, or point DNS at with confidence. +A tenant cannot *own* an address. They can only cause one to appear as a side effect +of creating a `Service type: LoadBalancer`, and it evaporates when that Service does. +There is no object to hold, keep, quota, bill, hand to another workload, or point DNS +at with confidence. This proposal makes the address a resource. Applying the `PersistentVolume` pattern to addresses: | storage | addresses | role | |---|---|---| -| `StorageClass` | **`PublicIPClass`** | which pool, which provisioner, which announcer | -| `PersistentVolume` | **`PublicIP`** (cluster-scoped) | the address itself, with a `claimRef` and a reclaim policy | -| `PersistentVolumeClaim` | **`PublicIPClaim`** (namespaced) | "give me one" — what a tenant creates | +| `StorageClass` | **`IPAddressClass`** | which pool, which provisioner, which announcer | +| `PersistentVolume` | **`IPAddress`** (cluster-scoped) | the address itself, with a `claimRef` and a reclaim policy | +| `PersistentVolumeClaim` | **`IPAddressClaim`** (namespaced) | "give me one" — what a tenant creates | | CSI driver | **a provisioner contract** | how a class's backend allocates, adopts, and pins | -The experience we are after is the AWS one: **allocate an address, keep it, attach it -to something, detach it, attach it to something else, release it when done.** Not: -"create a Service and hope you get the same IP back." +The driving use case is the **public** address — the AWS Elastic IP experience: +**allocate an address, keep it, attach it to something, detach it, attach it to +something else, release it when done.** Not: "create a Service and hope you get the +same IP back." But the resource is not public-specific. A class names a pool, and a +platform can hand out more than one kind of pool — public ranges, a private internal +LB range, a provider's static addresses. So the kinds are named for the general thing +(`IPAddress`), and "public" is a property of the class's pool, not of the API. + +### Naming + +The kinds `IPAddress` and `IPAddressClaim` are deliberate echoes of an existing +pattern, and the overlap is worth naming up front so reviewers are not surprised: + +- **Kubernetes core** ships an `IPAddress` kind in `networking.k8s.io` (KEP-1880). It + is *not* this: it is a low-level record where the object name **is** the IP, with a + generic `parentRef`, and its authors named "any generalization onto something like + an IPAM API" an explicit **non-goal** (see Context). Ours lives in a different API + group, carries a reclaim policy, a claim binding, and a class, and is the IPAM + object core declined to build. +- **Cluster API** ships `IPAddressClaim`/`IPAddress` in `ipam.cluster.x-k8s.io` with a + documented third-party provider contract — the same claim-plus-concrete-address + shape, but only ever wired to **Machine** addressing, never Services, and with no + reclaim policy or association. Whether we adopt that group and contract or mint our + own is an open question (below); the name choice keeps that door open. + +Kinds are qualified by API group, so the short-name overlap is not a collision. The +kinds live in `ipam.cozystack.io` (see *Positioning*). ## Scope and related proposals - **Adjacent, not a dependency:** `design-proposals/structured-external-exposure` (community #29) restructures how a managed application *requests* exposure. This proposal is about what an address **is**. The two meet at one point — an exposure - entry should be able to name a `PublicIPClaim` instead of implicitly minting an + entry should be able to name an `IPAddressClaim` instead of implicitly minting an address — but neither needs the other to land first, and this proposal does not assume #29's shape. - **Related:** an `ExposureClass` kind exists today in `network.cozystack.io`. This @@ -50,6 +74,28 @@ to something, detach it, attach it to something else, release it when done.** No the forwarding half.** They are independent: allocation is useful with a stock LoadBalancer Service and no 1:1 NAT at all. +## Positioning + +**This should stand on its own, and also ship as part of the Cozystack API surface — +both, not either.** A controller that reserves an address, binds it to a workload +through whatever pin mechanism the backend exposes, and gates that binding behind an +admission policy is useful to any Kubernetes platform running MetalLB, Cilium LB-IPAM, +or a cloud LB — not only Cozystack. So it is designed to be **valuable standalone**: the +tenant contract is a plain namespaced CRD, the quota is a stock `ResourceQuota`, the +enforcement is a `ValidatingAdmissionPolicy` — all vanilla Kubernetes, with no +Cozystack-only machinery required to run it. + +Standalone value does **not** mean it lives outside the Cozystack API group. Cozystack +already ships components that are fully standalone products yet live under +`*.cozystack.io` — `etcd-operator.cozystack.io` is the precedent. This belongs there the +same way: the kinds live in **`ipam.cozystack.io`**, and the component is deliberately +shaped so Cozystack integrates it cleanly (a default class, and the exposure-request path +of #29 able to name a claim). Standalone-valuable and in-the-Cozystack-group are not in +tension, and designing for easy Cozystack integration is a goal, not a compromise. + +Whether the kinds are greenfield or an adoption of Cluster API's IPAM kinds is an open +question (below). + ## Context ### What exists today @@ -77,22 +123,16 @@ None of these are expressible. The address has no independent existence, so: - **It cannot be moved.** There is no detach/attach; there is only "hope the allocator hands you the same one." - **It cannot be quota'd.** There is no object to count, so nothing bounds how many - public addresses a tenant consumes. The only lever is pre-carving a pool per tenant. -- **It cannot be gated.** Anyone who can create a `Service` in a namespace can cause a - public address to be allocated. RBAC authorizes *verbs on resources*, not *fields*. + addresses a tenant consumes. The only lever is pre-carving a pool per tenant. +- **It cannot be gated.** Anyone who can create a `Service` in a namespace can cause an + address to be allocated. RBAC authorizes *verbs on resources*, not *fields*. - **It cannot be shown.** No inventory. "Which addresses do we own, and who has them?" has no answer short of listing every Service in the cluster. -### This is a regression, not a wishlist - -Cozystack has reserve-then-associate **today**, through kube-ovn: `OvnEip` is a -cluster-scoped object created independently, then bound by an `OvnFip` (1:1 NAT), -`OvnSnatRule`, or `OvnDnatRule` that references it by name. The legacy iptables -gateway mode has the same shape (`IptablesEIP` + `IptablesFIPRule`). - -**Migrating off kube-ovn drops this capability on the floor** unless something -replaces it. That reframes the proposal: the question is not "should Cozystack gain an -AWS-style EIP", it is "on what terms does Cozystack keep the EIP it already has." +This is a **new capability**, argued on its own merits: hold, move, quota, enumerate, +bill. Cozystack has no prior reserve-then-associate mechanism to preserve — the +platform uses kube-ovn only for subnets and VPCs, not for its EIP objects — so there +is nothing here being restored, only something being added. ## Goals @@ -100,15 +140,18 @@ AWS-style EIP", it is "on what terms does Cozystack keep the EIP it already has. pool object for them. - An address **survives** the deletion of the workload it was attached to (reclaim policy `Retain`), and can be attached to a different workload afterwards. -- An operator can enumerate every public address the cluster owns, and see who holds it. +- An operator can enumerate every address the cluster owns, and see who holds it. - The number of addresses a tenant may hold is bounded by a **plain `ResourceQuota`** - (`count/publicipclaims.`), with no new quota machinery. + (`count/ipaddressclaims.`), with no new quota machinery. - An admin configures the address source **once per class**, not once per tenant and not once per address. No pool-per-tenant, no `/32` pools. - The tenant-facing API does not name the backend. A tenant asks for an address; it does not learn whether MetalLB, Cilium, or a cloud API produced it. - The design admits at least: MetalLB, Cilium LB-IPAM, and a cloud provider whose addresses are allocated outside the cluster. +- A reserved address is never *silently* lost to a plain `Service type: LoadBalancer`: + reservations are reconciled against live Service assignments, so a collision is a + detected, surfaced state rather than a quiet theft (see *Design §8* and *Security*). ### Non-goals @@ -116,11 +159,19 @@ AWS-style EIP", it is "on what terms does Cozystack keep the EIP it already has. with the LB implementation. This proposal allocates and binds; it never puts a packet on a wire. - **The 1:1 NAT datapath.** See *Scope*. +- **Multiple platform-owned addresses per workload.** A claim binds to one Service; + a workload holds one claim. Requesting *N* owned addresses for one workload is a + step beyond "an address is a resource", and it is not clear the case is even real: + the obvious example (a VM acting as a router or VPN concentrator) attaches its extra + addresses on a **tunnel interface the platform neither owns nor sees**, which makes + it orthogonal to address *reservation* rather than an extension of it. Deferred until + a concrete platform-owned use case exists. Dual-stack (one v4 + one v6) is a separate, + narrower question and is treated below, not folded into this. - **Replacing the LB implementation.** MetalLB/Cilium/cloud stay exactly where they are; this sits above them. - **IPv4-only thinking.** The model is family-agnostic; a claim may request v4, v6, or - both. (Whether one `PublicIP` carries both families or a claim binds two is an open - question below.) + both. (Whether one `IPAddress` carries both families or a `Dual` claim binds two is + an open question below.) - **Solving field-level authorization in general.** It is *used* here (see *Security*) but the general problem is bigger than this proposal. @@ -175,34 +226,35 @@ Load-bearing findings, because the design is shaped by them: ```yaml # Admin, once per address source. -apiVersion: network.cozystack.io/v1alpha1 -kind: PublicIPClass +apiVersion: ipam.cozystack.io/v1alpha1 +kind: IPAddressClass metadata: name: public annotations: - publicipclass.network.cozystack.io/is-default-class: "true" + ipaddressclass.ipam.cozystack.io/is-default-class: "true" spec: - provisioner: metallb.cozystack.io # who fulfils claims of this class + provisioner: metallb.ipam.cozystack.io # who fulfils claims of this class reclaimPolicy: Retain # Retain | Delete (default Retain) parameters: # opaque to the core controller addresses: ["203.0.113.0/24"] --- # Tenant. This is the whole tenant-facing API. -apiVersion: network.cozystack.io/v1alpha1 -kind: PublicIPClaim +apiVersion: ipam.cozystack.io/v1alpha1 +kind: IPAddressClaim metadata: {name: web, namespace: tenant-a} spec: className: public # empty => the default class family: IPv4 # IPv4 | IPv6 | Dual status: phase: Bound - publicIPName: pip-203-0-113-7 - address: 203.0.113.7 # what the tenant reads, and puts in DNS + addresses: # a list, so a Dual claim can report v4 + v6 + - name: ip-203-0-113-7 # the IPAddress object + address: 203.0.113.7 # what the tenant reads, and puts in DNS --- # Cluster-scoped. The inventory. Created by the provisioner, not the tenant. -apiVersion: network.cozystack.io/v1alpha1 -kind: PublicIP -metadata: {name: pip-203-0-113-7} +apiVersion: ipam.cozystack.io/v1alpha1 +kind: IPAddress +metadata: {name: ip-203-0-113-7} spec: className: public address: 203.0.113.7 @@ -219,12 +271,18 @@ status: name: web-lb ``` +`IPAddressClaim.status.addresses` is a **list** deliberately: a `Dual` claim binds a +v4 and a v6 `IPAddress` and must report both, and a scalar `address` field could not. +For a single-family claim the list has one entry. (Whether `Dual` is even one claim +binding two objects, or two claims, is an open question — but the status shape must not +foreclose the list case, so it is a list from the start.) + ### 3. The address-source union — the reason this is not a MetalLB adapter There are two structurally different worlds, and the model must admit both: - **Self-allocating backends** (MetalLB, Cilium, kube-vip, OpenELB): *nothing anywhere* - holds a reservation. **Our provisioner is the IPAM of record.** `PublicIP` is + holds a reservation. **Our provisioner is the IPAM of record.** `IPAddress` is authoritative, and its address is carved from the class's range (`source.fromClass`). - **Provider-reservation backends** (AWS, GCP, Hetzner): the reservation is already a real object with a stable handle (`eipalloc-…`, a named static address), guarded by @@ -234,11 +292,11 @@ There are two structurally different worlds, and the model must admit both: This is precisely the `PersistentVolume` volume-source union (`spec.csi` / `spec.nfs` / …), and it is the strongest evidence the shape is not bent around any one backend: it survives a backend where allocation is not ours at all. The tenant's -`PublicIPClaim` never learns the difference. +`IPAddressClaim` never learns the difference. ### 4. The provisioner contract (the "CSI analogue") -`PublicIPClass.spec.provisioner` names a controller, exactly as `StorageClass` names a +`IPAddressClass.spec.provisioner` names a controller, exactly as `StorageClass` names a CSI driver. Each provisioner declares **capabilities**, and the core controller refuses what a backend cannot do rather than half-supporting it: @@ -250,7 +308,7 @@ what a backend cannot do rather than half-supporting it: `Pin` is not optional decoration — **it is the capability the entire model depends on.** A backend that cannot be told *which* address to use can never attach a reserved one, -and a class over such a backend must reject `PublicIPClaim`s outright instead of +and a class over such a backend must reject `IPAddressClaim`s outright instead of allocating an address it can never bind. (PureLB may be exactly this case; see *Open questions*.) @@ -269,7 +327,7 @@ A workload references the **claim**, never the raw address: kind: Service metadata: annotations: - network.cozystack.io/public-ip-claim: web # a claim in THIS namespace + ipam.cozystack.io/ip-address-claim: web # a claim in THIS namespace spec: {type: LoadBalancer, ...} ``` @@ -277,85 +335,222 @@ The controller resolves the claim → address, verifies the claim is in the Serv namespace, and writes the backend's pin annotation (§1.2). MetalLB/Cilium/the cloud then allocates and announces exactly that address. -Remove the annotation and the announcement is withdrawn, but the `PublicIP` stays +Remove the annotation and the announcement is withdrawn, but the `IPAddress` stays `Bound` to its claim and simply becomes inert — **a reserved address, held, attached to nothing.** Which is what an unassociated EIP is. -### 6. What an admin actually configures +This is the claim-first path: the address exists before the Service names it. §6 covers +the opposite order — when an eager allocator hands a Service an address before any claim +exists. + +### 6. When the allocator gets there first + +An eager allocator does not wait to be asked. MetalLB assigns an address the moment it sees +a `Service type: LoadBalancer`, before any `IPAddressClaim` exists and whether or not one +ever will. So the inventory cannot assume every address originates from a claim: the +controller must also work **backwards** — observing `status.loadBalancer` on Services and +reconciling the addresses it finds into `IPAddress` objects after the fact — so the +cluster-scoped inventory converges on what is actually in use rather than only on what was +asked for. + +This is loosely the shape of a `StatefulSet` `volumeClaimTemplate` — a workload causing a +first-class object to exist without anyone hand-writing it — but the analogy is weak and +not worth leaning on: + +- **An address is non-fungible.** Any 20 GiB PV substitutes for any other; a specific IP + does not. "Give me one" and "give me *this* one" are different requests, and that exact, + scarce identity is the whole reason an inventory is needed. +- **The allocator does not wait for us.** A `volumeClaimTemplate` PVC materialises a volume + only through the provisioner; here MetalLB has already acted by the time the controller + looks. Claim-first and allocator-first allocations can therefore race, and the inventory + can transiently disagree with the live `status.loadBalancer` set. + +The design commits only to **eventual consistency** here, and to Service → inventory +reconciliation as a first-class direction alongside claim → address. How the race is +resolved is implementation, not settled in this document. + +And eventual consistency really is the ceiling for now. An eager allocator that gets there +first can hand a plain Service an address that is already `Bound` to a claim but not yet +associated — a reserved-but-inert address it has no way of knowing is spoken for. Layer 2 +(§8) detects the collision, but there is no clean remedy today: the address is already +announced for the wrong Service, and the allocator will not surrender it on request. A real +fix likely waits for the per-provider provisioners (§4), where allocation is ours and can +decline a reserved address in the first place. Until then this is an acknowledged gap, +deferred to the implementation phase. + +### 7. What an admin actually configures For MetalLB, forever, for the whole cluster: -- **one** `IPAddressPool` covering the range, and +- **one** `IPAddressPool` covering the reserved range, **with `autoAssign: false`**, and - **one** `L2Advertisement` (or `BGPAdvertisement`) selecting it. -That is all. No pool per tenant, no `/32`s. The `PublicIPClass` carries the range; the -provisioner renders the backend objects. +That is all. No pool per tenant, no `/32`s. The `IPAddressClass` carries the range; the +provisioner renders the backend objects. `autoAssign: false` keeps MetalLB from handing a +reserved address out *automatically* — but it does **not** make the pool unreachable, +because a Service can still name the pool explicitly (`metallb.io/address-pool`) or pin an +address (`metallb.io/loadBalancerIPs`). Closing that path is a security concern, not a +pool-config one; see §8 and *Security*. + +### 8. Worked examples: how this rides real allocators, and where it does not hold + +No static pool layout can *guarantee* a reservation. On MetalLB, `autoAssign: false` +stops only automatic selection — a Service can still name the reserved pool +(`metallb.io/address-pool`) or pin a specific address (`metallb.io/loadBalancerIPs`) and +be handed a reserved address. And any principal who can impersonate the controller's +ServiceAccount — a cluster-admin can — sails past an admission policy keyed on identity. +So the design does **not** rest on an invariant it cannot enforce. It uses **two layers**, +and is explicit about which one covers which threat: + +1. **Admission reduces the blast radius for the parties multi-tenancy is about** — + tenants and namespaced principals. A `ValidatingAdmissionPolicy` forbids them from + writing *any* backend annotation that selects a pool or pins an address (MetalLB's + `metallb.io/address-pool` **and** `metallb.io/loadBalancerIPs`, plus the §1.2 + equivalents), leaving the claim reference as their only route to the reserved range. +2. **Reconciliation makes a collision a detected, correctable state — however it arose.** + The `IPAddress` ledger is authoritative. The controller watches Services and their + assigned `status.loadBalancer.ingress`; if a Service holds an address belonging to an + `IPAddress` whose `claimRef`/`associatedTo` does not name that Service, it sets + `IPAddress.status.phase: Conflict`, raises a condition and a Service event, and does + not let the rightful claim silently lose its address. This is what "dealt with" means + for the cases admission cannot prevent (see the trust boundary in *Security*). + +A Service arriving *without* a claim reference is **not** by itself the thing being +defended against — an eager allocator handing a Service an address is the normal case §6 +reconciles after the fact. The conflict is narrower and specific: a Service ending up with +an address already `Bound` to a *different* claim — by an explicit pin, a race, or an +impersonated write. Layer 2 keys on exactly that — a `claimRef`/`associatedTo` mismatch — +not on the mere absence of an annotation. What separates legitimate allocation from theft +is whose reservation the address belongs to, not whether a claim annotation was present. + +**MetalLB.** The class's `IPAddressPool` is `autoAssign: false`, so plain Services never +draw from it *automatically*; they draw from the admin's other pools. Explicit selection +(`address-pool`/`loadBalancerIPs`) is blocked for tenants by layer 1 and detected by layer +2 for anyone else. Scoping the pool further with `IPAddressPool.spec.serviceAllocation` +(limit to a controller-set label or namespace) is available defense-in-depth — not a +guarantee. + +> *"A cluster-admin runs `kubectl create -f bad-service.yaml --as +> system:serviceaccount:cozy-ipam:ipam-controller`, referencing the reserved pool and +> maybe pinning a specific address."* — Admission cannot stop this: the request carries +> the controller's identity, and a cluster-admin is the cluster's trust root anyway (they +> can rewrite the policy or edit MetalLB objects directly). Layer 2 is what applies. The +> Service is assigned a reserved address; the controller sees an assignment no claim +> authorizes; the `IPAddress` goes `Conflict` with the offending Service named. The +> reservation is not silently overwritten — the conflict is surfaced for an operator (or +> an automated policy) to resolve. That is the honest boundary: the model defends tenants +> against each other, and turns admin/impersonator collisions into **visible faults** +> rather than pretending to prevent them. + +**Cilium LB-IPAM.** Cilium has no `autoAssign` flag; the equivalent is `serviceSelector`. +The reserved `CiliumLoadBalancerIPPool` carries a `serviceSelector` matching a label the +controller stamps on associated Services only, so a plain Service lacks it and draws from +an unselected default pool. The same two layers apply: admission stops a tenant from +setting that label directly, and reconciliation catches an assignment no claim authorizes. + +**Cloud (adopt).** The provider holds the reservation (`eipalloc-…`) under its own IAM. A +plain LB Service with no annotation is handed an ephemeral address from the provider's +general pool; the provider's API refuses to double-assign a reserved EIP. Here the +separation is enforced by the provider, not by us — which is exactly why +`source.providerRef` (adopt, don't allocate) is the right model for this world, and why +this is the one backend where the reservation genuinely cannot be raided from inside the +cluster. ## User-facing changes -- **Tenants** gain one kind: `PublicIPClaim`. Create it, read `status.address`, put that - in DNS, reference it by name from a Service. They never see `PublicIP`, - `PublicIPClass`, MetalLB, or a pin annotation. -- **Admins** gain `PublicIPClass` (one per address source) and `PublicIP` (a read-mostly - inventory: `kubectl get publicips` finally answers "what do we own and who has it"). -- **Quota** works with no new machinery: `count/publicipclaims.network.cozystack.io` in a - stock `ResourceQuota`. +- **Tenants** gain one kind: `IPAddressClaim`. Create it, read `status.addresses[].address`, + put that in DNS, reference the claim by name from a Service. They never see `IPAddress`, + `IPAddressClass`, MetalLB, or a pin annotation. +- **Admins** gain `IPAddressClass` (one per address source) and `IPAddress` (a read-mostly + inventory: `kubectl get ipaddresses` finally answers "what do we own and who has it"). +- **Quota** works with no new machinery: `count/ipaddressclaims.` in a stock + `ResourceQuota`. ## Upgrade and rollback compatibility - **Additive.** Nothing about today's `Service type: LoadBalancer` path changes; a Service with no claim annotation behaves exactly as now. - **Adoption path.** A cluster with existing LoadBalancer Services can import their - addresses as `PublicIP` objects with `source.providerRef`/`fromClass` and a `claimRef` + addresses as `IPAddress` objects with `source.providerRef`/`fromClass` and a `claimRef` to a generated claim — an offline, reversible migration. -- **kube-ovn.** `OvnEip` holders need a migration story. Sketch only: enumerate `OvnEip`s, - mint an equivalent `PublicIP` + `PublicIPClaim` per address, re-point the association. - **Not designed here.** - **Rollback.** Deleting the CRDs with `reclaimPolicy: Retain` leaves the backend objects and the live Services untouched; addresses stay where they are. With `Delete`, it does not — flagged as the irreversible case. ## Security -**The pin annotation is a privilege-escalation surface, and this is the sharp edge of -the proposal.** +**Selecting a pool or pinning an address is a privilege-escalation surface, and this is +the sharp edge of the proposal.** -RBAC authorizes *verbs on resources*, not *fields*. Today, anyone who can create a -`Service` in a namespace can write `metallb.io/loadBalancerIPs: ` — and -because a self-allocating backend has **no concept of a reservation**, it will happily -hand over an address that another tenant has reserved but not yet attached. That is a -theft window, and it exists on MetalLB and Cilium alike. It is *not* a cloud problem: -there, IAM already guards `eipalloc-…`. +RBAC authorizes *verbs on resources*, not *fields* or *annotations*. Today, anyone who +can create a `Service` in a namespace can write `metallb.io/loadBalancerIPs: ` — or name the reserved pool with `metallb.io/address-pool` — and because a +self-allocating backend has **no concept of a reservation**, it will happily hand over an +address another tenant reserved but has not yet attached. That is a theft window, on +MetalLB and Cilium alike. It is *not* a cloud problem: there, IAM already guards +`eipalloc-…`. -The fix is the same move Cozystack already makes elsewhere — **turn the ungatable field -into a reference to an RBAC-gated object**: +No single mechanism closes this, so the response has **two layers** (see *Design §8*), +and the design is explicit about which threat each one covers. -1. A tenant may only write `network.cozystack.io/public-ip-claim`, naming an object in - their **own namespace**, which RBAC *can* gate. -2. The **controller** writes the backend's raw pin annotation. -3. A `ValidatingAdmissionPolicy` **rejects any principal but the controller** writing - *any* backend pin annotation (§1.2's list) on a Service. +**Layer 1 — admission, against tenants.** Turn the ungatable annotation into a reference +to an RBAC-gated object: -Without (3) the whole model is advisory. It should land in the same release as the CRDs, -not after. +1. A tenant may only write `ipam.cozystack.io/ip-address-claim`, naming an object in + their **own namespace**, which RBAC *can* gate. +2. The **controller** writes the backend's raw pool/pin annotations. +3. A `ValidatingAdmissionPolicy` **rejects a non-allowlisted principal** writing *any* + backend annotation that selects a pool or pins an address — MetalLB's + `metallb.io/address-pool` and `metallb.io/loadBalancerIPs`, plus the §1.2 equivalents + for other backends — on a Service. + +*Identifying the controller, and not breaking the platform.* The policy must not rely on +a hardcoded username. It matches the controller's **ServiceAccount** by name and +namespace (`system:serviceaccount::`), configured at install time. And it must +carry an **allowlist**: real clusters have non-tenant principals that legitimately write +these annotations — GitOps reconcilers (Flux/Argo SAs) and system controllers (system +Istio Services ship with pool annotations already set). A naive "reject everyone but the +controller" breaks reconciliation on day one, so the policy denies the write only for +principals **outside** an explicit, admin-configurable allowlist. + +**Layer 2 — reconciliation, for everyone admission cannot bind.** Layer 1 is keyed on +identity, so it is defeated by anyone who can impersonate an allowlisted ServiceAccount — +which a cluster-admin can (`--as system:serviceaccount:cozy-ipam:ipam-controller`), along +with rewriting the policy or editing MetalLB objects directly. **This is a real trust +boundary, not a bug to be closed:** a cluster-admin is the cluster's trust root, and +handing out a reserved address is within their authority, not an escalation. What the +design owes here is not prevention but **detection**. Because the `IPAddress` ledger is +authoritative and continuously reconciled, any collision — impersonation, a direct +backend edit, a misconfigured pool, MetalLB assigning a reserved address to a Service with +no matching claim — surfaces as `IPAddress.status.phase: Conflict` with the offending +Service named, and the rightful claim is never *silently* overwritten. Admission defends +tenants against each other; reconciliation turns the cases admission cannot reach into +visible faults instead of quiet theft. + +Landing layer 1 is worth doing on its own, independent of the rest of the model: the +theft window it closes is live in any cluster running a shared auto-assign pool today. Note this does not, by itself, close field-level authorization in general — a tenant who -can create a `PublicIPClaim` can still consume an address. That is what the class and the -`ResourceQuota` are for, and it is a bound, not a gate. +can create an `IPAddressClaim` can still consume an address. That is what the class and +the `ResourceQuota` are for, and it is a bound, not a gate. ## Failure and edge cases - **Class range exhausted** → claim stays `Pending` with a reason; no partial binding. -- **Claim deleted, `reclaimPolicy: Retain`** → `PublicIP` goes `Released`, keeps the +- **Claim deleted, `reclaimPolicy: Retain`** → `IPAddress` goes `Released`, keeps the address, is not reusable until an admin clears the `claimRef`. (PV semantics, deliberately.) - **Claim deleted, `Delete`** → address returned to the range; the backend object is torn down. - **Service references a claim in another namespace** → rejected. Cross-namespace address sharing is not a thing. - **Two Services reference one claim** → the second is rejected. A 1:1 binding is 1:1; silently letting the second win is how an address goes quietly dead. +- **Plain Service pulls a reserved address** (explicit `address-pool`/pin, or a request + made under an impersonated controller identity) → not prevented in general; **detected**. + The `IPAddress` goes `Conflict`, the offending Service is named, and the reservation is + not silently overwritten. See §8 and *Security*. - **Backend lacks `Pin`** → the class rejects claims at admission, loudly, rather than allocating an address that can never be attached. -- **Address adopted from a provider, then released provider-side** → `PublicIP` goes +- **Address adopted from a provider, then released provider-side** → `IPAddress` goes `Lost`. It must not silently re-allocate. ## Testing @@ -367,8 +562,13 @@ can create a `PublicIPClaim` can still consume an address. That is what the clas - **e2e (per backend):** MetalLB and Cilium, on a real cluster: an external client reaches the workload on the reserved address, and still reaches it after the workload is rebuilt behind the same claim. -- **Admission:** a tenant writing a raw pin annotation is rejected; a tenant reserving an - address and a second tenant attempting to pin it directly is rejected. +- **Admission:** a tenant writing a raw pool-select or pin annotation is rejected; a tenant + reserving an address and a second tenant attempting to pin it directly is rejected; an + allowlisted GitOps/system principal writing a pool annotation is **admitted** (the + allowlist does not over-block). +- **Reconciliation:** a Service assigned a reserved address with no authorizing claim + (e.g. created under an impersonated controller identity) drives the `IPAddress` to + `Conflict` and emits an event; the rightful claim's binding is not lost. ## Rollout @@ -379,31 +579,29 @@ Sketch — sequencing is an open question: 2. The Cilium provisioner. Proves the abstraction is not a MetalLB adapter — **this is the phase that either validates or falsifies the design**, and it should come early. 3. A cloud provisioner (adoption path, `source.providerRef`). Proves the source union. -4. kube-ovn `OvnEip` migration. ## Open questions -1. **Is this Cozystack's job at all?** It is not a CNI concern (a CNI consumes an address - that lands on a node; it has no business owning the inventory). It is not upstream's - (KEP-1880 says so explicitly). Platform seems right — but it is worth asking whether - this belongs in a standalone project rather than in Cozystack core. -2. **Provisioners: in-tree or out-of-tree?** In-tree for three backends; CSI's history +1. **Provisioners: in-tree or out-of-tree?** In-tree for three backends; CSI's history says that stops scaling. Where is the line, and do we want the CRD contract on day one? -3. **Reuse Cluster API's IPAM?** `ipam.cluster.x-k8s.io` `IPAddressClaim`/`IPAddress` is +2. **Reuse Cluster API's IPAM?** `ipam.cluster.x-k8s.io` `IPAddressClaim`/`IPAddress` is *literally* this pattern — namespaced claim, concrete address object, a documented - third-party provider contract. It has only ever been wired to **Machine** addressing, - never Services. Do we adopt those kinds, or mint our own and merely copy the contract? - (Leaning: our own — we need reclaim policy and association, which it has no concept of. - But the shape is not novel and review should know that.) -4. **`ExposureClass`.** A class kind already exists in `network.cozystack.io`. Is - `PublicIPClass` a second class kind, or the same one grown a provisioner? Answering + third-party provider contract — and shares even the kind names we chose. It has only + ever been wired to **Machine** addressing, never Services, and has no reclaim policy or + association. Do we adopt those kinds (and that group), or mint our own and merely copy + the contract? (Leaning: our own — we need reclaim policy and association, which it has + no concept of. But the shape is not novel and review should know that.) This is the + greenfield-vs-adopt question *Positioning* leaves open. +3. **`ExposureClass`.** A class kind already exists in `network.cozystack.io`. Is + `IPAddressClass` a second class kind, or the same one grown a provisioner? Answering this depends on the fate of `ExposureClass`/`ServiceExposure`, which is being re-examined independently. -5. **Dual-stack.** Does one `PublicIP` carry a v4 and a v6 address, or does a `Dual` claim - bind two `PublicIP`s? (PV has no precedent. Leaning: two objects, one claim.) -6. **PureLB.** Does it have *any* "request this exact IP" mechanism? If not, it cannot be a +4. **Dual-stack.** Does one `IPAddress` carry a v4 and a v6 address, or does a `Dual` claim + bind two `IPAddress`es? (PV has no precedent. Leaning: two objects, one claim — the + status list already admits it.) +5. **PureLB.** Does it have *any* "request this exact IP" mechanism? If not, it cannot be a backend, and that should be stated rather than discovered. -7. **Sharing one address across Services** (different ports — GCP and Cilium both permit +6. **Sharing one address across Services** (different ports — GCP and Cilium both permit it). Does a claim bind to one Service, or may several reference it on disjoint ports? ## Alternatives considered @@ -429,5 +627,7 @@ address that is not currently attached. See *Security*. explicit non-goal, and `loadBalancerIP` was deprecated with no replacement. There is nothing to wait for. -**Do nothing.** **Rejected:** it is a regression. Cozystack has reserve-then-associate today -via kube-ovn's `OvnEip`, and the migration off kube-ovn removes it. +**Do nothing.** **Rejected:** the capability simply does not exist today — an address +cannot be held, moved, quota'd, or enumerated — and the pin-annotation theft window in +*Security* is live in any cluster running a shared auto-assign pool. Doing nothing leaves +both the missing capability and the open security hole in place. From f463a10de1602dda244dc868f32202f2355b8441 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Fri, 28 Aug 2026 15:13:04 +0300 Subject: [PATCH 03/16] design-proposal: settle the API group, record the implementation Align the document with what is built and argue the group choice on the record rather than leaving it in a commit message of another repository. - API group: ipam.cozystack.io -> local.sdn.cozystack.io throughout, including the association annotation, the default-class annotation and the example provisioner name, which now match what the controllers actually ship. Positioning carries the argument: Cozystack networking is already sdn.cozystack.io (SecurityGroup, served by the Cozystack API), cozyplane serves its own SecurityGroup there plus local.sdn.cozystack.io for its CRD-served kinds, and addresses are the third networking feature to want a home. One sdn family, with the local. prefix marking CRD-served kinds and the bare group the aggregated ones; ipam.cozystack.io would add a fourth name to a subject area that should have one. Stated plainly as a proposal, not a precedent: nothing serves local.sdn.cozystack.io today. - New Implementation status section: the core controller and the MetalLB reference driver exist, and cozyplane consumes reservations through the contract with no module import, no CRD dependency and no informer on the claim kinds -- the association seam is exercised by a second consumer, while Rollout phase 2, a second provisioner, remains outstanding. The ValidatingAdmissionPolicy is recorded as implemented nowhere and still part of phase 1. - The stub banner no longer says the proposal predates any code, and Rollout phases 1 and 2 say what exists. - Scope records that community #45 depends on this proposal, which does not wait on it. - Directory renamed public-ip-claims -> ip-address-claims; the kinds lost the "public" prefix two revisions ago. Assisted-by: LLM Signed-off-by: Timofei Larkin --- .../README.md | 115 +++++++++++++----- 1 file changed, 86 insertions(+), 29 deletions(-) rename design-proposals/{public-ip-claims => ip-address-claims}/README.md (85%) diff --git a/design-proposals/public-ip-claims/README.md b/design-proposals/ip-address-claims/README.md similarity index 85% rename from design-proposals/public-ip-claims/README.md rename to design-proposals/ip-address-claims/README.md index c348eb3f..ccba0048 100644 --- a/design-proposals/public-ip-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -5,10 +5,11 @@ - **Date:** `2026-07-15` - **Status:** Draft -> **This is a stub.** The problem statement, the primitives survey, and the object -> model are worked out; the controller mechanics, the provisioner contract details, -> and the rollout are deliberately sketched. It is opened early to settle the -> **shape** — is an address a resource? — before anyone writes code. +> **The shape is settled and implemented; the contract details are not.** The problem +> statement, the primitives survey, and the object model are worked out, and phase 1 of +> *Rollout* now exists in code (see *Implementation status*). Deliberately still open: +> the provisioner contract's own API (capabilities are not modelled yet), the admission +> policy of *Security*, the sequencing beyond phase 1, and the open questions below. ## Overview @@ -53,7 +54,7 @@ pattern, and the overlap is worth naming up front so reviewers are not surprised own is an open question (below); the name choice keeps that door open. Kinds are qualified by API group, so the short-name overlap is not a collision. The -kinds live in `ipam.cozystack.io` (see *Positioning*). +kinds live in `local.sdn.cozystack.io` (see *Positioning*). ## Scope and related proposals @@ -67,6 +68,10 @@ kinds live in `ipam.cozystack.io` (see *Positioning*). proposal takes a position on it (see *Alternatives*): the **class** idea is right and should survive in some form; binding an address's lifetime to a `Service` is the part that cannot serve this use case. +- **Depended on by:** `design-proposals/endpoint-attachments` (community #45), which + attaches one endpoint of an application to an external address. It consumes an + `IPAddressClaim` and the association annotation of §5; this proposal does not wait on + it and lands independently. - **Deferred to a sibling:** the *datapath* for whole-IP 1:1 NAT (all ports in, and the workload egressing **as** that address) — today provided by [cozy-proxy](https://github.com/cozystack/cozy-proxy)'s nftables rules. That is a @@ -88,14 +93,62 @@ Cozystack-only machinery required to run it. Standalone value does **not** mean it lives outside the Cozystack API group. Cozystack already ships components that are fully standalone products yet live under `*.cozystack.io` — `etcd-operator.cozystack.io` is the precedent. This belongs there the -same way: the kinds live in **`ipam.cozystack.io`**, and the component is deliberately -shaped so Cozystack integrates it cleanly (a default class, and the exposure-request path -of #29 able to name a claim). Standalone-valuable and in-the-Cozystack-group are not in -tension, and designing for easy Cozystack integration is a goal, not a compromise. +same way, and the component is deliberately shaped so Cozystack integrates it cleanly (a +default class, and an exposure path able to name a claim). Standalone-valuable and +in-the-Cozystack-group are not in tension, and designing for easy Cozystack integration +is a goal, not a compromise. + +It also belongs in the **networking** group rather than a new one of its own. Cozystack +networking is `sdn.cozystack.io` today: `SecurityGroup` lives there, served by the +Cozystack API as a tenant-facing interface over Cilium network policies. Cozyplane serves +its own `SecurityGroup` in that same group, plus `local.sdn.cozystack.io` for the kinds it +serves as CRDs. Addresses are the third networking feature to want a home, and the +argument here is only that networking APIs should not accumulate several different group +names: **one `sdn` family**, with the `local.` prefix marking CRD-served kinds and the +bare group the aggregated ones. So the kinds live in **`local.sdn.cozystack.io`**. +`ipam.cozystack.io` is not wrong on its own terms — it adds a fourth name to a subject +area that should have one. + +To be clear that this is a proposal and not an appeal to precedent: +`local.sdn.cozystack.io` is new, nothing serves it today, and this would be among the +first components to introduce it. Nor does the group +string create a Cozystack dependency for a standalone user — the same three CRDs install +under any name. Whether the kinds are greenfield or an adoption of Cluster API's IPAM kinds is an open question (below). +## Implementation status + +Phase 1 of *Rollout* exists: + +- **[address-controller](https://github.com/lllamnyp/address-controller)** — the + class-agnostic core: the three kinds, claim–address binding, status, reclaim, and the + contract per-class drivers plug into. It never touches a Service. +- **[metallb-iad](https://github.com/lllamnyp/metallb-iad)** — the reference per-class + driver, for addresses announced by MetalLB. A reservation is held as a placeholder + Service in a driver-owned namespace, so MetalLB's own accounting refuses to + double-assign a held address. + +One backend proves little on its own, so the load-bearing evidence is a **second +consumer**. [Cozyplane](https://github.com/lllamnyp/cozyplane) wears reserved addresses +through this contract with **no module import, no CRD dependency, and no informer on the +claim kinds**: it copies a claim name into the association annotation (§5) on a Service it +already owns, and the driver does the rest. With the mechanism absent entirely it behaves +identically, auto-assigning instead — reserved and dynamic are one code path. Validated +end to end on a three-node cluster running address-controller + metallb-iad + MetalLB + +cozyplane: an address survived delete-and-rebind and stayed externally reachable +throughout. + +Note precisely which seam that tests. *Rollout* §2 — a second **provisioner** — is the +*allocation* seam, and it remains outstanding. What is demonstrated is the *association* +seam: that a backend with an entirely different datapath can wear a reserved address while +knowing nothing about the ledger beyond one annotation. The coupling turned out looser +than this document assumed it would need to be. + +The `ValidatingAdmissionPolicy` of *Security* is implemented in neither repository. It is +the outstanding half of phase 1, not a phase 2 item. + ## Context ### What exists today @@ -226,20 +279,20 @@ Load-bearing findings, because the design is shaped by them: ```yaml # Admin, once per address source. -apiVersion: ipam.cozystack.io/v1alpha1 +apiVersion: local.sdn.cozystack.io/v1alpha1 kind: IPAddressClass metadata: name: public annotations: - ipaddressclass.ipam.cozystack.io/is-default-class: "true" + ipaddressclass.local.sdn.cozystack.io/is-default-class: "true" spec: - provisioner: metallb.ipam.cozystack.io # who fulfils claims of this class + provisioner: metallb.drivers.local.sdn.cozystack.io # who fulfils claims of this class reclaimPolicy: Retain # Retain | Delete (default Retain) parameters: # opaque to the core controller addresses: ["203.0.113.0/24"] --- # Tenant. This is the whole tenant-facing API. -apiVersion: ipam.cozystack.io/v1alpha1 +apiVersion: local.sdn.cozystack.io/v1alpha1 kind: IPAddressClaim metadata: {name: web, namespace: tenant-a} spec: @@ -252,7 +305,7 @@ status: address: 203.0.113.7 # what the tenant reads, and puts in DNS --- # Cluster-scoped. The inventory. Created by the provisioner, not the tenant. -apiVersion: ipam.cozystack.io/v1alpha1 +apiVersion: local.sdn.cozystack.io/v1alpha1 kind: IPAddress metadata: {name: ip-203-0-113-7} spec: @@ -327,7 +380,7 @@ A workload references the **claim**, never the raw address: kind: Service metadata: annotations: - ipam.cozystack.io/ip-address-claim: web # a claim in THIS namespace + local.sdn.cozystack.io/ip-address-claim: web # a claim in THIS namespace spec: {type: LoadBalancer, ...} ``` @@ -431,16 +484,16 @@ draw from it *automatically*; they draw from the admin's other pools. Explicit s guarantee. > *"A cluster-admin runs `kubectl create -f bad-service.yaml --as -> system:serviceaccount:cozy-ipam:ipam-controller`, referencing the reserved pool and -> maybe pinning a specific address."* — Admission cannot stop this: the request carries -> the controller's identity, and a cluster-admin is the cluster's trust root anyway (they -> can rewrite the policy or edit MetalLB objects directly). Layer 2 is what applies. The -> Service is assigned a reserved address; the controller sees an assignment no claim -> authorizes; the `IPAddress` goes `Conflict` with the offending Service named. The -> reservation is not silently overwritten — the conflict is surfaced for an operator (or -> an automated policy) to resolve. That is the honest boundary: the model defends tenants -> against each other, and turns admin/impersonator collisions into **visible faults** -> rather than pretending to prevent them. +> system:serviceaccount:cozy-address-controller:address-controller`, referencing the +> reserved pool and maybe pinning a specific address."* — Admission cannot stop this: the +> request carries the controller's identity, and a cluster-admin is the cluster's trust +> root anyway (they can rewrite the policy or edit MetalLB objects directly). Layer 2 is +> what applies. The Service is assigned a reserved address; the controller sees an +> assignment no claim authorizes; the `IPAddress` goes `Conflict` with the offending +> Service named. The reservation is not silently overwritten — the conflict is surfaced +> for an operator (or an automated policy) to resolve. That is the honest boundary: the +> model defends tenants against each other, and turns admin/impersonator collisions into +> **visible faults** rather than pretending to prevent them. **Cilium LB-IPAM.** Cilium has no `autoAssign` flag; the equivalent is `serviceSelector`. The reserved `CiliumLoadBalancerIPPool` carries a `serviceSelector` matching a label the @@ -496,7 +549,7 @@ and the design is explicit about which threat each one covers. **Layer 1 — admission, against tenants.** Turn the ungatable annotation into a reference to an RBAC-gated object: -1. A tenant may only write `ipam.cozystack.io/ip-address-claim`, naming an object in +1. A tenant may only write `local.sdn.cozystack.io/ip-address-claim`, naming an object in their **own namespace**, which RBAC *can* gate. 2. The **controller** writes the backend's raw pool/pin annotations. 3. A `ValidatingAdmissionPolicy` **rejects a non-allowlisted principal** writing *any* @@ -515,8 +568,9 @@ principals **outside** an explicit, admin-configurable allowlist. **Layer 2 — reconciliation, for everyone admission cannot bind.** Layer 1 is keyed on identity, so it is defeated by anyone who can impersonate an allowlisted ServiceAccount — -which a cluster-admin can (`--as system:serviceaccount:cozy-ipam:ipam-controller`), along -with rewriting the policy or editing MetalLB objects directly. **This is a real trust +which a cluster-admin can (`--as +system:serviceaccount:cozy-address-controller:address-controller`), along with rewriting +the policy or editing MetalLB objects directly. **This is a real trust boundary, not a bug to be closed:** a cluster-admin is the cluster's trust root, and handing out a reserved address is within their authority, not an escalation. What the design owes here is not prevention but **detection**. Because the `IPAddress` ledger is @@ -575,9 +629,12 @@ the `ResourceQuota` are for, and it is a bound, not a gate. Sketch — sequencing is an open question: 1. CRDs + core controller + the **MetalLB** provisioner + the admission policy. (The - admission policy is not phase 2. See *Security*.) + admission policy is not phase 2. See *Security*.) The controllers and the driver + exist; the admission policy does not — see *Implementation status*. 2. The Cilium provisioner. Proves the abstraction is not a MetalLB adapter — **this is the phase that either validates or falsifies the design**, and it should come early. + A second *consumer* has since exercised the association half (*Implementation + status*); a second *provisioner* is what remains. 3. A cloud provisioner (adoption path, `source.providerRef`). Proves the source union. ## Open questions From 39a257cfb4db4e548201d27563e6a9ed114c3faa Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Fri, 28 Aug 2026 15:48:26 +0300 Subject: [PATCH 04/16] design-proposal: correct the claims review found overstated Review found several places where the document asserts something that is not true or not achievable. Each is corrected rather than annotated. - Section 6 described the reserved-but-unattached theft window as an acknowledged gap with no clean remedy. Holders close it: for every reserved address not currently attached the provisioner keeps a placeholder consumer, so the backend's own accounting refuses to double-assign, and neither MetalLB nor Cilium will give one address to a second Service without an explicit sharing key. The section now leads with that as the primary guarantee and keeps detection as the backstop, naming what genuinely remains: the non-atomic handoff, and explicit annotation writers. The backwards inventory is scoped to ranges a class manages, so foreign auto-assign pools are observed rather than adjudicated. - Security claimed more for layer 1 than an identity allowlist can deliver. A privileged controller applying user-authored content -- a GitOps reconciler that does not impersonate, a cloud-controller-manager copying annotations verbatim out of a user-controlled cluster -- hands every user behind it the gated write. Annotation filtering at the deputy is now stated as a precondition of layer 1, which is partial until it holds, with layer 2 covering those users meanwhile. - Two failure cases named admission as the mechanism where admission cannot be one. A ValidatingAdmissionPolicy sees the request object and its params, not arbitrary cluster state, so it can read neither the claim nor a backend's capabilities: the second Service on a claim is resolved by reconciliation with a deterministic winner and a Conflicted loser, and a class over a Pin-less backend is refused by its provisioner. - Section 5 assumed the annotated object is always a Service. Under Gateway API the implementation renders the data-plane Service from the Gateway, so the contract is an annotated object that resolves to an address consumer. Where a provisioner lands the pin is its own business; the part that is not is that Gateway.spec.addresses names an address by value and therefore falls under the same admission policy as a backend pin annotation. - The claim carries spec.addressName, the PVC spec.volumeName analogue, which the document never mentioned. - A Dual claim against a single-stack Service was missing from the failure cases. - Dropped "bill" from the motivation: the object model delivers enumeration and quota, and billing is something a platform builds on the inventory. Assisted-by: LLM Signed-off-by: Timofei Larkin --- design-proposals/ip-address-claims/README.md | 126 +++++++++++++------ 1 file changed, 87 insertions(+), 39 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index ccba0048..6452b24f 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -15,7 +15,7 @@ A tenant cannot *own* an address. They can only cause one to appear as a side effect of creating a `Service type: LoadBalancer`, and it evaporates when that Service does. -There is no object to hold, keep, quota, bill, hand to another workload, or point DNS +There is no object to hold, keep, quota, hand to another workload, or point DNS at with confidence. This proposal makes the address a resource. Applying the `PersistentVolume` pattern @@ -182,10 +182,10 @@ None of these are expressible. The address has no independent existence, so: - **It cannot be shown.** No inventory. "Which addresses do we own, and who has them?" has no answer short of listing every Service in the cluster. -This is a **new capability**, argued on its own merits: hold, move, quota, enumerate, -bill. Cozystack has no prior reserve-then-associate mechanism to preserve — the -platform uses kube-ovn only for subnets and VPCs, not for its EIP objects — so there -is nothing here being restored, only something being added. +This is a **new capability**, argued on its own merits: hold, move, quota, enumerate. +Cozystack has no prior reserve-then-associate mechanism to preserve — the platform uses +kube-ovn only for subnets and VPCs, not for its EIP objects — so there is nothing here +being restored, only something being added. ## Goals @@ -298,6 +298,8 @@ metadata: {name: web, namespace: tenant-a} spec: className: public # empty => the default class family: IPv4 # IPv4 | IPv6 | Dual + # addressName: ip-203-0-113-7 # optional: bind this specific Available address -- + # the PVC spec.volumeName analogue, single-family only status: phase: Bound addresses: # a list, so a Dual claim can report v4 + v6 @@ -392,44 +394,66 @@ Remove the annotation and the announcement is withdrawn, but the `IPAddress` sta `Bound` to its claim and simply becomes inert — **a reserved address, held, attached to nothing.** Which is what an unassociated EIP is. +**The annotated object is not always a Service.** Some consumers *own* the Service they +are announced on rather than being one: under Gateway API the implementation renders the +data-plane Service from the `Gateway`, so there is no user-authored Service to annotate, +and the object the user does write has no way to name an address. The contract is therefore +an **annotated object that resolves to an address consumer** — `Service` is the first +member and `Gateway` the obvious second. Which object a given backend annotates, and where +it lands the pin (`Gateway.spec.addresses`, infrastructure annotations propagated onto the +generated Service, or the generated Service itself), is the provisioner's business (§4). +The consequence that is *not* the provisioner's business is in *Security*: +`Gateway.spec.addresses` names an address **by value**, so it is the same escalation +surface as a backend pin annotation and the admission policy has to cover it. + This is the claim-first path: the address exists before the Service names it. §6 covers the opposite order — when an eager allocator hands a Service an address before any claim exists. ### 6. When the allocator gets there first -An eager allocator does not wait to be asked. MetalLB assigns an address the moment it sees -a `Service type: LoadBalancer`, before any `IPAddressClaim` exists and whether or not one -ever will. So the inventory cannot assume every address originates from a claim: the -controller must also work **backwards** — observing `status.loadBalancer` on Services and -reconciling the addresses it finds into `IPAddress` objects after the fact — so the -cluster-scoped inventory converges on what is actually in use rather than only on what was -asked for. +An eager allocator does not wait to be asked. MetalLB assigns an address the moment it +sees a `Service type: LoadBalancer`, before any `IPAddressClaim` exists and whether or not +one ever will. Two things follow. + +**A reservation is held in the allocator's own books, not merely recorded in ours.** For +every address that is reserved but not currently attached, the provisioner keeps a +**holder**: for MetalLB, a selectorless `type: LoadBalancer` Service in a provisioner-owned +namespace, which MetalLB assigns the address to and, having no endpoints, never announces — +held in the backend's accounting, silent on the wire. Neither MetalLB nor Cilium will give +one address to a second Service without an explicit sharing key, so for as long as the +holder exists the reservation is enforced by the allocator itself rather than by our +detection. This is the primary guarantee, and it is what makes a reserved-but-unattached +address safe to leave lying around. + +**The inventory still works backwards.** The ledger cannot assume every address originates +from a claim: the controller also observes `status.loadBalancer` on Services and reconciles +what it finds into `IPAddress` objects after the fact, so the inventory converges on what is +actually in use rather than only on what was asked for. Its authority is **scoped to ranges +a class manages**; addresses from pools no class owns are observed, not adjudicated — +otherwise every foreign auto-assign pool in the cluster becomes a source of `Conflict` +noise. This is loosely the shape of a `StatefulSet` `volumeClaimTemplate` — a workload causing a -first-class object to exist without anyone hand-writing it — but the analogy is weak and -not worth leaning on: - -- **An address is non-fungible.** Any 20 GiB PV substitutes for any other; a specific IP - does not. "Give me one" and "give me *this* one" are different requests, and that exact, - scarce identity is the whole reason an inventory is needed. -- **The allocator does not wait for us.** A `volumeClaimTemplate` PVC materialises a volume - only through the provisioner; here MetalLB has already acted by the time the controller - looks. Claim-first and allocator-first allocations can therefore race, and the inventory - can transiently disagree with the live `status.loadBalancer` set. - -The design commits only to **eventual consistency** here, and to Service → inventory -reconciliation as a first-class direction alongside claim → address. How the race is -resolved is implementation, not settled in this document. - -And eventual consistency really is the ceiling for now. An eager allocator that gets there -first can hand a plain Service an address that is already `Bound` to a claim but not yet -associated — a reserved-but-inert address it has no way of knowing is spoken for. Layer 2 -(§8) detects the collision, but there is no clean remedy today: the address is already -announced for the wrong Service, and the allocator will not surrender it on request. A real -fix likely waits for the per-provider provisioners (§4), where allocation is ours and can -decline a reserved address in the first place. Until then this is an acknowledged gap, -deferred to the implementation phase. +first-class object to exist without anyone hand-writing it — but the analogy is weak and not +worth leaning on. An address is **non-fungible**: any 20 GiB PV substitutes for any other, a +specific IP does not, and that scarce identity is the whole reason an inventory is needed. +And **the allocator does not wait for us**, so claim-first and allocator-first allocations +can race and the inventory can transiently disagree with the live `status.loadBalancer` set. +The design commits to **eventual consistency** there, and to Service → inventory +reconciliation as a first-class direction alongside claim → address. + +What remains once holders are in place is narrower than a general theft window, and worth +naming precisely rather than waving at: + +- **The handoff.** Releasing the holder and pinning the workload's own Service is not + atomic. For that window the address is unheld and an eager allocator could hand it to + whichever Service happens to be asking. Narrow, but real — and one of the intervals where + layer 1 of *Security* earns its place. +- **Annotation writers.** With a reserved pool set `autoAssign: false` a plain Service + cannot *automatically* draw a reserved address; it can only get one by naming the pool or + pinning the address explicitly. That is precisely the write layer 1 gates and layer 2 + detects. ### 7. What an admin actually configures @@ -566,6 +590,21 @@ Istio Services ship with pool annotations already set). A naive "reject everyone controller" breaks reconciliation on day one, so the policy denies the write only for principals **outside** an explicit, admin-configurable allowlist. +*The allowlist's hard case: trusted principals applying untrusted content.* An identity +allowlist assumes a trusted principal writes content it authored, and that assumption fails +wherever a privileged controller applies material some user supplied. Two shapes of this are +ordinary rather than exotic: a GitOps reconciler that applies user-authored releases under +its **own** identity instead of impersonating the user, and a cloud-controller-manager that +copies Service annotations from a user-controlled cluster into an infrastructure Service +verbatim. Allowlist such a deputy and every user behind it inherits the ability to write a +pin annotation; deny it and legitimate reconciliation breaks. So **annotation filtering at +the deputy is a precondition of layer 1, not a hardening extra**: a strip-list for the gated +keys in any controller that forwards user-supplied annotations, and impersonation rather +than ambient authority in any reconciler applying user-authored content. Until those hold +for a given deputy, layer 1 is **partial** for the users behind it and layer 2 is what +covers them. The design says so rather than assuming a trusted identity implies trusted +content. + **Layer 2 — reconciliation, for everyone admission cannot bind.** Layer 1 is keyed on identity, so it is defeated by anyone who can impersonate an allowlisted ServiceAccount — which a cluster-admin can (`--as @@ -596,14 +635,23 @@ the `ResourceQuota` are for, and it is a bound, not a gate. - **Claim deleted, `Delete`** → address returned to the range; the backend object is torn down. - **Service references a claim in another namespace** → rejected. Cross-namespace address sharing is not a thing. -- **Two Services reference one claim** → the second is rejected. A 1:1 binding is 1:1; - silently letting the second win is how an address goes quietly dead. +- **Two Services reference one claim** → resolved by reconciliation, not at admission. A + `ValidatingAdmissionPolicy` cannot see the claim — CEL is given the request object and the + policy's params, not arbitrary cluster state — so admission-time rejection would need a + webhook. Instead one association wins by a deterministic rule, the loser reports + `Conflicted`, and the address is never silently moved. A 1:1 binding is 1:1; letting the + second win quietly is how an address goes dead. - **Plain Service pulls a reserved address** (explicit `address-pool`/pin, or a request made under an impersonated controller identity) → not prevented in general; **detected**. The `IPAddress` goes `Conflict`, the offending Service is named, and the reservation is not silently overwritten. See §8 and *Security*. -- **Backend lacks `Pin`** → the class rejects claims at admission, loudly, rather than - allocating an address that can never be attached. +- **A `Dual` claim associated with a single-stack Service** → the Service carries the one + family it has; the other address stays `Bound` to the claim and inert. The claim is not + `Lost`, and the unattached family is never silently released. +- **Backend lacks `Pin`** → claims against that class are refused loudly, rather than + allocating an address that can never be attached. With no capability registry there is + nothing for admission to consult, so this is the provisioner refusing and reporting it on + the claim, not an admission-time rejection. - **Address adopted from a provider, then released provider-side** → `IPAddress` goes `Lost`. It must not silently re-allocate. From 84211a8bcfcf1c9e7ed0bad2fe56d4c8a95b1304 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Mon, 31 Aug 2026 12:50:28 +0300 Subject: [PATCH 05/16] design-proposal: specify addressName, and correct two stale claims Review approved the shape and left four specifics. Three change the document. - spec.addressName appeared once, commented out in the claim example, with no prose anywhere. Two reviewers read it opposite ways -- one as absent entirely, one as an ungoverned way for a tenant to name a cluster-scoped object. It is now specified: it narrows the candidate set and never widens it, honoured only for an address that is Available, unclaimed, not being deleted, and of the claim's own resolved class and family, with a name that matches nothing leaving the claim Pending rather than failing. Failure cases cover the absent, Bound, Released and wrong-class names. Security now states where this component stops. It secures the surface a reservation can be stolen through; it does not decide which class a namespace may draw from, and neither addressName nor className can be gated by namespace RBAC. That is deliberate: the capability is reserve/hold/attach/release, and encoding a tenancy model this controller cannot see belongs above it -- an admission policy over those two fields, or machinery that mints claims on a tenant's behalf. Both fields are named so a deployment hardening this does not have to find them. - ExposureClass was described as a kind that exists in network.cozystack.io. It was proposed alongside community #29 and never shipped, and that group does not exist; the direction went to loadBalancerClass. Open question 3 was anchored to its fate and so could not be answered. It now asks the live question -- whether a class should declare the loadBalancerClass it corresponds to, so an association can check the pair -- and records that loadBalancerClass is immutable on an existing Service, so a mismatch has to be a refused association rather than a rewrite. - The non-goal said a workload holds one claim, which forbids a case that is not ours to forbid and that community #45 depends on: a managed service reachable on a public address and an internal routable one is two Services with two claims. The 1:1 rule that is real -- one claim, one consumer at a time -- is kept, and what stays deferred is N owned addresses on a single consumer. Also records, in Positioning, why the group is not ipam.cozystack.io: this is not all of IPAM and cannot become it, since allocation stays with the allocator. Assisted-by: LLM Signed-off-by: Timofei Larkin --- design-proposals/ip-address-claims/README.md | 100 ++++++++++++++----- 1 file changed, 77 insertions(+), 23 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index 6452b24f..16a89f47 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -64,10 +64,13 @@ kinds live in `local.sdn.cozystack.io` (see *Positioning*). entry should be able to name an `IPAddressClaim` instead of implicitly minting an address — but neither needs the other to land first, and this proposal does not assume #29's shape. -- **Related:** an `ExposureClass` kind exists today in `network.cozystack.io`. This - proposal takes a position on it (see *Alternatives*): the **class** idea is right - and should survive in some form; binding an address's lifetime to a `Service` is - the part that cannot serve this use case. +- **Related:** `Service.spec.loadBalancerClass`, the upstream field selecting which LB + implementation serves a Service. An `IPAddressClass` names a provisioner and a pool, + which in practice implies the implementation that will announce the address, so the two + classes are adjacent and have to agree — see the open question below. (An + `ExposureClass` kind was proposed alongside community #29 and never shipped; there is no + `network.cozystack.io` group. That direction went to `loadBalancerClass` instead — + cozystack/cozystack#3164, cozystack/cozystack#3218.) - **Depended on by:** `design-proposals/endpoint-attachments` (community #45), which attaches one endpoint of an application to an external address. It consumes an `IPAddressClaim` and the association annotation of §5; this proposal does not wait on @@ -115,6 +118,15 @@ first components to introduce it. Nor does the group string create a Cozystack dependency for a standalone user — the same three CRDs install under any name. +The obvious objection is that `ipam.` describes the object better than `sdn.` does: an +address ledger is not a datapath concern, and its consumers include Gateways and cloud +providers that are not SDN either. That is a fair reading, and the reason it does not win +here is that this is **not all of IPAM and cannot become it** — allocation stays with the +allocator, since AWS and MetalLB both keep it — so an `ipam.cozystack.io` group would +promise a generality the component deliberately does not have. If upstream moves and that +generality becomes real, it is a v2 question. Until then the kinds sit with the rest of +networking. + Whether the kinds are greenfield or an adoption of Cluster API's IPAM kinds is an open question (below). @@ -212,14 +224,18 @@ being restored, only something being added. with the LB implementation. This proposal allocates and binds; it never puts a packet on a wire. - **The 1:1 NAT datapath.** See *Scope*. -- **Multiple platform-owned addresses per workload.** A claim binds to one Service; - a workload holds one claim. Requesting *N* owned addresses for one workload is a - step beyond "an address is a resource", and it is not clear the case is even real: - the obvious example (a VM acting as a router or VPN concentrator) attaches its extra - addresses on a **tunnel interface the platform neither owns nor sees**, which makes - it orthogonal to address *reservation* rather than an extension of it. Deferred until - a concrete platform-owned use case exists. Dual-stack (one v4 + one v6) is a separate, - narrower question and is treated below, not folded into this. +- **Multiple owned addresses on one consumer.** A claim binds to **one consumer at a + time**, and that 1:1 rule is real and enforced. No limit on a *workload* is claimed: + nothing stops an application from having several Services, each with its own claim and + its own address, and a managed service reachable on a public address and an internal + routable one at the same time is exactly that — two Services, two claims, no new + concept. What stays out of scope is *N* owned addresses on a single consumer. The + obvious example (a VM acting as a router or VPN concentrator) attaches its extra + addresses on a **tunnel interface the platform neither owns nor sees**, which makes it + orthogonal to address *reservation*; the platform-owned variant would need an internal + address and in-guest configuration per address. Deferred until a concrete platform-owned + use case exists. Dual-stack (one v4 + one v6) is a separate, narrower question, treated + below rather than folded into this. - **Replacing the LB implementation.** MetalLB/Cilium/cloud stay exactly where they are; this sits above them. - **IPv4-only thinking.** The model is family-agnostic; a claim may request v4, v6, or @@ -298,8 +314,7 @@ metadata: {name: web, namespace: tenant-a} spec: className: public # empty => the default class family: IPv4 # IPv4 | IPv6 | Dual - # addressName: ip-203-0-113-7 # optional: bind this specific Available address -- - # the PVC spec.volumeName analogue, single-family only + # addressName: ip-203-0-113-7 # optional: bind this specific address (see below) status: phase: Bound addresses: # a list, so a Dual claim can report v4 + v6 @@ -326,6 +341,23 @@ status: name: web-lb ``` +**`spec.addressName` — asking for a *specific* address.** The +`PersistentVolumeClaim.spec.volumeName` analogue: a claim may name one `IPAddress` rather +than take whatever its class yields. It is meaningful only for a single-family claim, and +the controller honours it only if every ordinary condition already holds — the address is +`Available`, carries no `claimRef`, is not being deleted, and its class and family match +the ones the claim resolved. A name that matches nothing eligible is not an error: the +claim stays `Pending` and keeps waiting, exactly as it would with no `addressName` at all. +The field therefore **narrows a candidate set and never widens one**. It cannot pull an +address out of another class, take one away from another claim, or reclaim a `Released` +address before an admin has cleared its `claimRef`. + +What it deliberately does *not* do is authorize. `IPAddress` is cluster-scoped, so a claim +naming one names an object outside its own namespace. Neither this field nor +`spec.className` — equally free-text, and equally unable to be gated by namespace RBAC — +decides which classes a namespace may draw from. That gap is identical with or without +`addressName`, and closing it belongs above this controller; see *Security*. + `IPAddressClaim.status.addresses` is a **list** deliberately: a `Dual` claim binds a v4 and a v6 `IPAddress` and must report both, and a scalar `address` field could not. For a single-family claim the list has one entry. (Whether `Dual` is even one claim @@ -623,9 +655,22 @@ visible faults instead of quiet theft. Landing layer 1 is worth doing on its own, independent of the rest of the model: the theft window it closes is live in any cluster running a shared auto-assign pool today. -Note this does not, by itself, close field-level authorization in general — a tenant who -can create an `IPAddressClaim` can still consume an address. That is what the class and -the `ResourceQuota` are for, and it is a bound, not a gate. +**Where this component stops.** The two layers above cover the surface a reservation can +be *stolen* through. They do not close field-level authorization in general: a tenant who +can create an `IPAddressClaim` can consume an address, may name any `spec.className`, and +may name a specific `IPAddress` through `spec.addressName` (§2). `ResourceQuota` bounds how +many addresses they hold; nothing here decides *which class* a namespace is entitled to +draw from. + +That is deliberate rather than unfinished. This component's job is the **capability** — +reserve an address, hold it, attach it, detach it, release it — and a capability that also +tried to be its own policy engine would have to encode a tenancy model it does not have and +cannot see. The gating belongs where the tenancy model lives: a `ValidatingAdmissionPolicy` +restricting which classes a namespace may name (the same mechanism as layer 1, with a +different subject), or platform machinery that mints claims on a tenant's behalf instead of +letting tenants write claims directly. **`spec.className` and `spec.addressName` are the two +fields such a policy must cover** — naming them here is the point, so that a deployment +hardening this does not have to discover them. ## Failure and edge cases @@ -645,6 +690,9 @@ the `ResourceQuota` are for, and it is a bound, not a gate. made under an impersonated controller identity) → not prevented in general; **detected**. The `IPAddress` goes `Conflict`, the offending Service is named, and the reservation is not silently overwritten. See §8 and *Security*. +- **`spec.addressName` names an address that is absent, `Bound`, `Released`, or of another + class or family** → nothing binds; the claim stays `Pending` with the ordinary waiting + reason. Naming an address filters, it never seizes. - **A `Dual` claim associated with a single-stack Service** → the Service carries the one family it has; the other address stays `Bound` to the claim and inert. The claim is not `Lost`, and the unattached family is never silently released. @@ -697,10 +745,15 @@ Sketch — sequencing is an open question: the contract? (Leaning: our own — we need reclaim policy and association, which it has no concept of. But the shape is not novel and review should know that.) This is the greenfield-vs-adopt question *Positioning* leaves open. -3. **`ExposureClass`.** A class kind already exists in `network.cozystack.io`. Is - `IPAddressClass` a second class kind, or the same one grown a provisioner? Answering - this depends on the fate of `ExposureClass`/`ServiceExposure`, which is being - re-examined independently. +3. **`IPAddressClass` and `Service.spec.loadBalancerClass`.** A class names a provisioner + and a pool, which implies the LB implementation that will announce the address; a + Service names its implementation directly in `spec.loadBalancerClass` (§1.5). A Service + whose `loadBalancerClass` selects an implementation other than the one behind its + claim's class is asking two backends for one address, and nothing notices today. Should + a class declare the `loadBalancerClass` it corresponds to, so association can resolve + the pair and check it? Note what the answer cannot be: `loadBalancerClass` is immutable + on an existing Service, so a mismatch must surface as a **refused association with a + reason**, never a silent rewrite. 4. **Dual-stack.** Does one `IPAddress` carry a v4 and a v6 address, or does a `Dual` claim bind two `IPAddress`es? (PV has no precedent. Leaning: two objects, one claim — the status list already admits it.) @@ -711,8 +764,9 @@ Sketch — sequencing is an open question: ## Alternatives considered -**Bind the address to a Service (`ServiceExposure`-shaped).** A namespaced object naming -a `serviceRef`, which allocates an address and reports it in status. **Rejected as the +**Bind the address to a Service** — the shape community #29 proposed as +`ServiceExposure`. A namespaced object naming a `serviceRef`, which allocates an address +and reports it in status. **Rejected as the model for *this* problem:** it fuses *allocation* with *association*, so the address's lifetime is the Service's lifetime — which is the exact thing being fixed. It cannot express "keep this address, the workload is gone", which is the entire user request. From 8dc2e30507c4676894425a373bcd6d21f546ed6e Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Mon, 31 Aug 2026 13:05:03 +0300 Subject: [PATCH 06/16] design-proposal: the class fixes the loadBalancerClass; open the cardinality question Two corrections from review. - IPAddressClass and Service.spec.loadBalancerClass were left as an open question about whether they should be related. They should not be merely related: a class names the pool and the provisioner that carves it, which determines the implementation that announces anything drawn from it, so the class states that implementation in spec.loadBalancerClass and a consuming Service must match it, empty matching empty. A Service selecting one implementation while its claim's address is held by another asks two backends for one address and both are wrong about who owns it; the provisioner's holder Services carry the class's value for the same reason. The check is a refusal and never a repair, since loadBalancerClass is immutable on an existing Service, so whoever creates the Service sets it at creation -- and a controller that both creates the Service and names the claim can read the class first. The open question is dropped and the rest renumbered. - The non-goal claimed a claim binds one consumer and left it there, which reads as though the cardinality is settled in both directions. It is not. Gateway.spec.addresses is a list, so one Gateway legitimately wears several addresses, and nothing here says how several claims attach to one consumer or how a provisioner renders that. What stays settled is the half that is enforced: a claim is consumed by one consumer at a time. How many claims one consumer may hold is acknowledged as open and deferred to future work, with the tunnel-interface and platform-owned variants deferred alongside it. The open question on sharing one address across Services is reworded to match: it is the mirror of that case, and the two want answering together. Assisted-by: LLM Signed-off-by: Timofei Larkin --- design-proposals/ip-address-claims/README.md | 72 ++++++++++++-------- 1 file changed, 45 insertions(+), 27 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index 16a89f47..3fc376b8 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -66,8 +66,8 @@ kinds live in `local.sdn.cozystack.io` (see *Positioning*). assume #29's shape. - **Related:** `Service.spec.loadBalancerClass`, the upstream field selecting which LB implementation serves a Service. An `IPAddressClass` names a provisioner and a pool, - which in practice implies the implementation that will announce the address, so the two - classes are adjacent and have to agree — see the open question below. (An + which determines the implementation that will announce the address. An `IPAddressClass` + therefore **fixes** the `loadBalancerClass` a consuming Service must carry (§5). (An `ExposureClass` kind was proposed alongside community #29 and never shipped; there is no `network.cozystack.io` group. That direction went to `loadBalancerClass` instead — cozystack/cozystack#3164, cozystack/cozystack#3218.) @@ -224,18 +224,23 @@ being restored, only something being added. with the LB implementation. This proposal allocates and binds; it never puts a packet on a wire. - **The 1:1 NAT datapath.** See *Scope*. -- **Multiple owned addresses on one consumer.** A claim binds to **one consumer at a - time**, and that 1:1 rule is real and enforced. No limit on a *workload* is claimed: - nothing stops an application from having several Services, each with its own claim and - its own address, and a managed service reachable on a public address and an internal - routable one at the same time is exactly that — two Services, two claims, no new - concept. What stays out of scope is *N* owned addresses on a single consumer. The - obvious example (a VM acting as a router or VPN concentrator) attaches its extra - addresses on a **tunnel interface the platform neither owns nor sees**, which makes it - orthogonal to address *reservation*; the platform-owned variant would need an internal - address and in-guest configuration per address. Deferred until a concrete platform-owned - use case exists. Dual-stack (one v4 + one v6) is a separate, narrower question, treated - below rather than folded into this. +- **Several owned addresses on one consumer.** One rule is settled and enforced: a claim + is consumed by **one consumer at a time**, so two consumers cannot quietly share a + reservation. The converse is *not* settled, and this proposal does not pretend it is — + how many claims one consumer may hold is open. No limit on a *workload* is claimed: + nothing stops an application owning several Services, each with its own claim, and a + managed service reachable on a public address and an internal routable one at once is + exactly that — two Services, two claims, no new concept. But a single consumer wearing + several addresses is a real case rather than a hypothetical one, and Gateway API states + it outright: `Gateway.spec.addresses` is a **list**, so one Gateway legitimately wears + several addresses, and nothing in the model below says how several claims attach to one + consumer or how a provisioner renders that. **Acknowledged and deferred to future + work.** Two neighbouring cases stay out with it: *N* owned addresses on a workload whose + extra addresses live on a **tunnel interface the platform neither owns nor sees** (a VM + acting as a router or VPN concentrator), which is orthogonal to reservation; and the + platform-owned variant of it, which would need an internal address and in-guest + configuration per address. Dual-stack (one v4 + one v6) is a separate, narrower + question, treated below rather than folded into this. - **Replacing the LB implementation.** MetalLB/Cilium/cloud stay exactly where they are; this sits above them. - **IPv4-only thinking.** The model is family-agnostic; a claim may request v4, v6, or @@ -303,6 +308,8 @@ metadata: ipaddressclass.local.sdn.cozystack.io/is-default-class: "true" spec: provisioner: metallb.drivers.local.sdn.cozystack.io # who fulfils claims of this class + loadBalancerClass: metallb.io/public # the implementation that announces it; + # empty => the cluster's default implementation reclaimPolicy: Retain # Retain | Delete (default Retain) parameters: # opaque to the core controller addresses: ["203.0.113.0/24"] @@ -422,6 +429,22 @@ The controller resolves the claim → address, verifies the claim is in the Serv namespace, and writes the backend's pin annotation (§1.2). MetalLB/Cilium/the cloud then allocates and announces exactly that address. +**The class fixes the `loadBalancerClass`.** An `IPAddressClass` names the pool and the +provisioner that carves it, which determines the LB implementation that will announce +anything drawn from it; the class states that implementation in +`spec.loadBalancerClass`, and a consuming Service must match it — empty matching empty, +meaning the cluster's default implementation (§1.5). This is not a preference: a Service +whose `loadBalancerClass` selects one implementation while its claim's address is held by +another is asking two backends for one address, and both will be wrong about who owns it. +The provisioner's own holder Services (§6) carry the class's value for the same reason. + +The check can only ever be a **refusal**, never a repair: `loadBalancerClass` is immutable +on an existing Service, so a mismatch surfaces as a refused association naming both +values, and whoever creates the Service — a chart, a controller minting one on a tenant's +behalf, or a user — is responsible for setting it at creation. A controller that creates +the Service *and* names the claim has no excuse for getting it wrong, since it can read +the class first. + Remove the annotation and the announcement is withdrawn, but the `IPAddress` stays `Bound` to its claim and simply becomes inert — **a reserved address, held, attached to nothing.** Which is what an unassociated EIP is. @@ -745,22 +768,17 @@ Sketch — sequencing is an open question: the contract? (Leaning: our own — we need reclaim policy and association, which it has no concept of. But the shape is not novel and review should know that.) This is the greenfield-vs-adopt question *Positioning* leaves open. -3. **`IPAddressClass` and `Service.spec.loadBalancerClass`.** A class names a provisioner - and a pool, which implies the LB implementation that will announce the address; a - Service names its implementation directly in `spec.loadBalancerClass` (§1.5). A Service - whose `loadBalancerClass` selects an implementation other than the one behind its - claim's class is asking two backends for one address, and nothing notices today. Should - a class declare the `loadBalancerClass` it corresponds to, so association can resolve - the pair and check it? Note what the answer cannot be: `loadBalancerClass` is immutable - on an existing Service, so a mismatch must surface as a **refused association with a - reason**, never a silent rewrite. -4. **Dual-stack.** Does one `IPAddress` carry a v4 and a v6 address, or does a `Dual` claim +3. **Dual-stack.** Does one `IPAddress` carry a v4 and a v6 address, or does a `Dual` claim bind two `IPAddress`es? (PV has no precedent. Leaning: two objects, one claim — the status list already admits it.) -5. **PureLB.** Does it have *any* "request this exact IP" mechanism? If not, it cannot be a +4. **PureLB.** Does it have *any* "request this exact IP" mechanism? If not, it cannot be a backend, and that should be stated rather than discovered. -6. **Sharing one address across Services** (different ports — GCP and Cilium both permit - it). Does a claim bind to one Service, or may several reference it on disjoint ports? +5. **Sharing one address across Services** (different ports — GCP and Cilium both permit + it). Today one consumer at a time holds a claim and a second reference is refused (§5). + Should that relax to several consumers on disjoint ports? It is the mirror of the + deferred several-addresses-on-one-consumer case in *Non-goals*, and the two probably + want answering together, since both are really "what is the cardinality between a claim + and the things that wear it". ## Alternatives considered From bc43ac6cdaf9626cd65fb0f7cfd34515a84d408b Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Fri, 18 Sep 2026 14:50:25 +0300 Subject: [PATCH 07/16] design-proposal(ip-address-claims): note EndpointAttachment joins the local.sdn family Community #45 now homes its EndpointAttachment kind in local.sdn.cozystack.io rather than consuming the substrate from another group. Record it as another CRD-served resident of the family, which exercises the local.-prefix quota guarantee the Positioning section argues for. Signed-off-by: Timofei Larkin --- design-proposals/ip-address-claims/README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index 3fc376b8..0df5c409 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -73,8 +73,11 @@ kinds live in `local.sdn.cozystack.io` (see *Positioning*). cozystack/cozystack#3164, cozystack/cozystack#3218.) - **Depended on by:** `design-proposals/endpoint-attachments` (community #45), which attaches one endpoint of an application to an external address. It consumes an - `IPAddressClaim` and the association annotation of §5; this proposal does not wait on - it and lands independently. + `IPAddressClaim` and the association annotation of §5, and its `EndpointAttachment` + kind lives in **this same `local.sdn.cozystack.io` family** — the exposure half beside + the allocation half, another CRD-served resident relying on the `local.` prefix's + quota guarantee (*Positioning*). This proposal does not wait on it and lands + independently. - **Deferred to a sibling:** the *datapath* for whole-IP 1:1 NAT (all ports in, and the workload egressing **as** that address) — today provided by [cozy-proxy](https://github.com/cozystack/cozy-proxy)'s nftables rules. That is a From e804115df7849a23ec3fd0b1ab7b6994feafe9b6 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Mon, 2 Nov 2026 15:30:13 +0100 Subject: [PATCH 08/16] design-proposal(ip-address-claims): follow main's proposal conventions Drop the Status and Date header fields, which community #111 removed from proposals in favour of deriving both from PR and Git history, and add the empty Decisions section the template now carries. Join the three pull-quotes in The problem into one blockquote so the lint does not flag bare blank lines inside it. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index 0df5c409..fee4bb04 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -2,8 +2,6 @@ - **Title:** `IP addresses as a first-class resource: IPAddress, IPAddressClaim, IPAddressClass, and a provisioner contract` - **Author(s):** `@lllamnyp` -- **Date:** `2026-07-15` -- **Status:** Draft > **The shape is settled and implemented; the contract details are not.** The problem > statement, the primitives survey, and the object model are worked out, and phase 1 of @@ -85,6 +83,11 @@ kinds live in `local.sdn.cozystack.io` (see *Positioning*). the forwarding half.** They are independent: allocation is useful with a stock LoadBalancer Service and no 1:1 NAT at all. +## Decisions + + + ## Positioning **This should stand on its own, and also ship as part of the Cozystack API surface — @@ -179,9 +182,9 @@ Service, at the moment the Service is created, and freed when it is deleted.** > *"I gave my customer this IP address. It is in their firewall allow-list. I need to > rebuild the VM behind it, and I need the address to still be mine afterwards."* - +> > *"I want to move this address from the old VM to the new one during a cutover."* - +> > *"I want to reserve four addresses now, because the range is nearly full, and attach > them over the next month."* From 828be75b73868839a74b783fdc7be4186ced21ac Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Mon, 2 Nov 2026 15:31:56 +0100 Subject: [PATCH 09/16] design-proposal(ip-address-claims): reconcile the text with the implementation The code moved while the document did not, in both directions. Record where the document runs ahead of the binaries, and correct what it got wrong about them. - Implementation status points at the cozystack organisation, the packaging PR and the driver's admission-policy and auto-claim PRs, and lists every rule specified here and not yet implemented, plus the detail the code carries that the model does not need. - Section 6 no longer reconciles foreign Service assignments into the ledger. Nothing justified catch-all adoption: with a class pool at autoAssign false a plain Service can only reach a class address by a pin, which is a conflict to surface. The dropped design is recorded under Alternatives. In its place, the opt-in auto-claim the core now has, and why: a cloud-controller-manager renders host-side LoadBalancer Services that have never heard of claims. Two rules follow: a claim with a controller owner follows its owner rather than the class reclaim policy, and a namespace's held claims are attached before a new one is minted. - Section 5: the class determines the loadBalancerClass, which an explicit field is the simplest but not the only way to express; the check is marked specified, not implemented. Dropping the annotation does not make MetalLB withdraw an assignment from a live Service, so say so and say what detaching from a live Service means. - Section 3: the reference driver lets MetalLB pick the address through the holder and records the result, after a driver-side allocator was tried and replaced. The ledger is the record of reservation, not the allocator. - The principal that writes pins is the driver's ServiceAccount, never the core's; the examples said otherwise. Conflict adjudication is per class, so two drivers do not judge each other's handoff windows. - Two Services on one claim: the live holder keeps it, the loser gets an event and the claim an Associated condition. IPAddress gains conditions with transition times, so the ledger answers when as well as who. Dual is settled as one claim binding two objects, because a dual-stack Service names one claim; the driver pins only the families the Service carries. - Rollout gains Gateway as its second phase, ahead of the Cilium provisioner, since the accepted database-exposure proposal puts tenant traffic on the Gateway's one address. Open questions on Cluster API reuse and dual-stack close; Gateway and the entitlement object open. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 247 +++++++++++++------ 1 file changed, 172 insertions(+), 75 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index fee4bb04..20b03f98 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -5,9 +5,10 @@ > **The shape is settled and implemented; the contract details are not.** The problem > statement, the primitives survey, and the object model are worked out, and phase 1 of -> *Rollout* now exists in code (see *Implementation status*). Deliberately still open: -> the provisioner contract's own API (capabilities are not modelled yet), the admission -> policy of *Security*, the sequencing beyond phase 1, and the open questions below. +> *Rollout* exists in code (see *Implementation status*, which also lists where the +> document runs ahead of the code). Deliberately still open: the provisioner contract's +> own API (capabilities are not modelled yet), the entitlement object of *Security*, the +> sequencing beyond phase 1, and the open questions below. ## Overview @@ -48,8 +49,8 @@ pattern, and the overlap is worth naming up front so reviewers are not surprised - **Cluster API** ships `IPAddressClaim`/`IPAddress` in `ipam.cluster.x-k8s.io` with a documented third-party provider contract — the same claim-plus-concrete-address shape, but only ever wired to **Machine** addressing, never Services, and with no - reclaim policy or association. Whether we adopt that group and contract or mint our - own is an open question (below); the name choice keeps that door open. + reclaim policy or association. We mint our own (*Alternatives considered*); the shared + names are an acknowledgement that the shape is not novel. Kinds are qualified by API group, so the short-name overlap is not a collision. The kinds live in `local.sdn.cozystack.io` (see *Positioning*). @@ -65,7 +66,7 @@ kinds live in `local.sdn.cozystack.io` (see *Positioning*). - **Related:** `Service.spec.loadBalancerClass`, the upstream field selecting which LB implementation serves a Service. An `IPAddressClass` names a provisioner and a pool, which determines the implementation that will announce the address. An `IPAddressClass` - therefore **fixes** the `loadBalancerClass` a consuming Service must carry (§5). (An + therefore **determines** the `loadBalancerClass` a consuming Service must carry (§5). (An `ExposureClass` kind was proposed alongside community #29 and never shipped; there is no `network.cozystack.io` group. That direction went to `loadBalancerClass` instead — cozystack/cozystack#3164, cozystack/cozystack#3218.) @@ -133,20 +134,27 @@ promise a generality the component deliberately does not have. If upstream moves generality becomes real, it is a v2 question. Until then the kinds sit with the rest of networking. -Whether the kinds are greenfield or an adoption of Cluster API's IPAM kinds is an open -question (below). +The kinds are minted rather than adopted from Cluster API's IPAM; *Alternatives +considered* says why. ## Implementation status -Phase 1 of *Rollout* exists: +Phase 1 of *Rollout* exists, under the Cozystack organisation: -- **[address-controller](https://github.com/lllamnyp/address-controller)** — the +- **[address-controller](https://github.com/cozystack/address-controller)** — the class-agnostic core: the three kinds, claim–address binding, status, reclaim, and the - contract per-class drivers plug into. It never touches a Service. -- **[metallb-iad](https://github.com/lllamnyp/metallb-iad)** — the reference per-class + contract per-class drivers plug into. It touches a Service only on the opt-in + auto-claim path of §6 (cozystack/address-controller#7). +- **[metallb-iad](https://github.com/cozystack/metallb-iad)** — the reference per-class driver, for addresses announced by MetalLB. A reservation is held as a placeholder Service in a driver-owned namespace, so MetalLB's own accounting refuses to - double-assign a held address. + double-assign a held address. cozystack/metallb-iad#6 ships the admission policy of + *Security* as an opt-in chart template; cozystack/metallb-iad#5 opts the placeholders + out of auto-claim. +- **cozystack/cozystack#3382** packages both as optional system packages, with tenant + RBAC on claims and a distribution-specific admission policy. Which policy a deployment + runs is its own business: the driver's generic one is the reference, and a distribution + that needs a different shape disables it and ships its own. One backend proves little on its own, so the load-bearing evidence is a **second consumer**. [Cozyplane](https://github.com/lllamnyp/cozyplane) wears reserved addresses @@ -158,14 +166,23 @@ end to end on a three-node cluster running address-controller + metallb-iad + Me cozyplane: an address survived delete-and-rebind and stayed externally reachable throughout. -Note precisely which seam that tests. *Rollout* §2 — a second **provisioner** — is the +Note precisely which seam that tests. *Rollout* names a second **provisioner** as the *allocation* seam, and it remains outstanding. What is demonstrated is the *association* seam: that a backend with an entirely different datapath can wear a reserved address while knowing nothing about the ledger beyond one annotation. The coupling turned out looser than this document assumed it would need to be. -The `ValidatingAdmissionPolicy` of *Security* is implemented in neither repository. It is -the outstanding half of phase 1, not a phase 2 item. +**Where the document runs ahead of the code.** This is a design proposal; the following +are specified here and not yet implemented, so that nobody mistakes the text for a +description of the binaries: the class-determined `loadBalancerClass` check of §5; a +driver pinning only the address families its Service carries (§2); conditions on +`IPAddress` (§2) and the `Associated` condition on a claim (§5); per-class scoping of +conflict adjudication (§8); the held-before-minted rule of §6; the release delay and the +class-deletion guard of *Failure and edge cases*; Gateway as an association target (§5); +the entitlement object of *Security*. Where the code carries detail the text does not — +`claimRef.uid`, a sticky `status.className` on the claim, claim conditions and a `Lost` +claim phase, a `Pending` address phase, the provisioner stamp annotation, the finalizers — +the implementation's own design document is the reference; none of it changes the model. ## Context @@ -250,8 +267,7 @@ being restored, only something being added. - **Replacing the LB implementation.** MetalLB/Cilium/cloud stay exactly where they are; this sits above them. - **IPv4-only thinking.** The model is family-agnostic; a claim may request v4, v6, or - both. (Whether one `IPAddress` carries both families or a `Dual` claim binds two is - an open question below.) + both; a `Dual` claim binds one `IPAddress` per family (§2). - **Solving field-level authorization in general.** It is *used* here (see *Security*) but the general problem is bigger than this proposal. @@ -348,6 +364,13 @@ spec: # providerRef: {id: eipalloc-0a1b} # ...or it wraps a provider-side reservation status: phase: Bound + conditions: # metav1.Condition; lastTransitionTime is the + - type: Bound # audit trail of who held the address, and when + status: "True" + lastTransitionTime: "2026-07-15T09:12:00Z" + - type: Associated + status: "True" + lastTransitionTime: "2026-07-15T09:15:31Z" associatedTo: # nil => reserved but inert kind: Service namespace: tenant-a @@ -373,17 +396,30 @@ decides which classes a namespace may draw from. That gap is identical with or w `IPAddressClaim.status.addresses` is a **list** deliberately: a `Dual` claim binds a v4 and a v6 `IPAddress` and must report both, and a scalar `address` field could not. -For a single-family claim the list has one entry. (Whether `Dual` is even one claim -binding two objects, or two claims, is an open question — but the status shape must not -foreclose the list case, so it is a list from the start.) +For a single-family claim the list has one entry. `Dual` is one claim binding two objects +— the implementation settled it that way, and the reason is the association contract: a +dual-stack Service names *one* claim in §5, so one claim must be able to yield both +families. The driver pins only the families the Service carries (*Failure and edge cases*). + +`IPAddress.status.conditions` exist so the ledger answers *when* as well as *who*. `Bound` +and `Associated` follow `metav1.Condition` semantics, and their `lastTransitionTime` is the +record of which claim held an address, and which consumer wore it, over time. An operator +reconstructing that after the fact — an abuse report naming an address and an hour — reads +it from the ledger, not from Service events, which expire. `Conflict` is reported the same +way, as a condition beside the phase, so that the offending consumer is named somewhere +durable. ### 3. The address-source union — the reason this is not a MetalLB adapter There are two structurally different worlds, and the model must admit both: - **Self-allocating backends** (MetalLB, Cilium, kube-vip, OpenELB): *nothing anywhere* - holds a reservation. **Our provisioner is the IPAM of record.** `IPAddress` is - authoritative, and its address is carved from the class's range (`source.fromClass`). + holds a reservation. **The ledger is the record of reservation**, and the address is + carved from the class's range (`source.fromClass`). Which component picks the address + is the driver's business, and the reference driver deliberately lets the backend do it: + it asks MetalLB for an address through the holder of §6 and records what MetalLB + assigned, so the backend's allocator and the ledger can never disagree about who holds + what. A driver-side allocator was tried first and replaced for exactly that reason. - **Provider-reservation backends** (AWS, GCP, Hetzner): the reservation is already a real object with a stable handle (`eipalloc-…`, a named static address), guarded by the provider's own IAM. **Our provisioner must adopt, not allocate** @@ -435,14 +471,17 @@ The controller resolves the claim → address, verifies the claim is in the Serv namespace, and writes the backend's pin annotation (§1.2). MetalLB/Cilium/the cloud then allocates and announces exactly that address. -**The class fixes the `loadBalancerClass`.** An `IPAddressClass` names the pool and the -provisioner that carves it, which determines the LB implementation that will announce -anything drawn from it; the class states that implementation in -`spec.loadBalancerClass`, and a consuming Service must match it — empty matching empty, -meaning the cluster's default implementation (§1.5). This is not a preference: a Service -whose `loadBalancerClass` selects one implementation while its claim's address is held by -another is asking two backends for one address, and both will be wrong about who owns it. -The provisioner's own holder Services (§6) carry the class's value for the same reason. +**The class determines the `loadBalancerClass`.** An `IPAddressClass` names the pool and +the provisioner that carves it, which determines the LB implementation that will announce +anything drawn from it, so a class unambiguously implies one `loadBalancerClass` value — +empty meaning the cluster's default implementation (§1.5) — and a consuming Service must +carry that value. This is not a preference: a Service whose `loadBalancerClass` selects one +implementation while its claim's address is held by another is asking two backends for one +address, and both will be wrong about who owns it. The provisioner's own holder Services +(§6) carry the same value for the same reason. The simplest way to express the mapping is +an explicit `spec.loadBalancerClass` on the class, as in §2; a driver that can derive it +from its own parameters may do that instead. What is required is the check, and it is +**specified here and not yet implemented** (*Implementation status*). The check can only ever be a **refusal**, never a repair: `loadBalancerClass` is immutable on an existing Service, so a mismatch surfaces as a refused association naming both @@ -451,9 +490,15 @@ behalf, or a user — is responsible for setting it at creation. A controller th the Service *and* names the claim has no excuse for getting it wrong, since it can read the class first. -Remove the annotation and the announcement is withdrawn, but the `IPAddress` stays -`Bound` to its claim and simply becomes inert — **a reserved address, held, attached to -nothing.** Which is what an unassociated EIP is. +Remove the annotation and the driver withdraws the pin it wrote, clears `associatedTo`, +and holds the address again; the `IPAddress` stays `Bound` to its claim and becomes inert — +**a reserved address, held, attached to nothing.** Which is what an unassociated EIP is. +One honest caveat: the backend does not necessarily follow. MetalLB keeps an assignment +until the Service changes or goes away, so a Service that drops the annotation but lives +on keeps announcing the address, and the ledger reports it as a `Conflict` naming that +Service until it is deleted or re-addressed. Detaching an address from a *live* Service +therefore means recreating that Service, which is the natural shape of a cutover anyway; +the common path — the workload is deleted, the address is kept — has no such wrinkle. **The annotated object is not always a Service.** Some consumers *own* the Service they are announced on rather than being one: under Gateway API the implementation renders the @@ -487,22 +532,49 @@ holder exists the reservation is enforced by the allocator itself rather than by detection. This is the primary guarantee, and it is what makes a reserved-but-unattached address safe to leave lying around. -**The inventory still works backwards.** The ledger cannot assume every address originates -from a claim: the controller also observes `status.loadBalancer` on Services and reconciles -what it finds into `IPAddress` objects after the fact, so the inventory converges on what is -actually in use rather than only on what was asked for. Its authority is **scoped to ranges -a class manages**; addresses from pools no class owns are observed, not adjudicated — -otherwise every foreign auto-assign pool in the cluster becomes a source of `Conflict` -noise. - -This is loosely the shape of a `StatefulSet` `volumeClaimTemplate` — a workload causing a -first-class object to exist without anyone hand-writing it — but the analogy is weak and not -worth leaning on. An address is **non-fungible**: any 20 GiB PV substitutes for any other, a -specific IP does not, and that scarce identity is the whole reason an inventory is needed. -And **the allocator does not wait for us**, so claim-first and allocator-first allocations -can race and the inventory can transiently disagree with the live `status.loadBalancer` set. -The design commits to **eventual consistency** there, and to Service → inventory -reconciliation as a first-class direction alongside claim → address. +**Nothing adopts what it did not reserve.** An earlier draft had the controller observe +`status.loadBalancer` on every Service and reconcile foreign assignments into the ledger +after the fact. That is dropped (*Alternatives considered*): with a class's pool at +`autoAssign: false` (§7), a plain Service cannot draw a class address except by naming the +pool or pinning the address, and that is a conflict to detect (§8), not an allocation to +adopt. Addresses from pools no class manages are not in the ledger and are of no interest +to it. The inventory therefore answers "what has been reserved, and who holds it", not +"every address in the cluster" — which is the question an operator actually asks. + +**Auto-claim: the ledger reaching a Service that has never heard of it.** The opposite +need is real: a Service that *should* draw from a class but names no claim, because +whoever rendered it knows nothing about claims. The concrete case is a +cloud-controller-manager for tenant Kubernetes clusters, which creates a `type: +LoadBalancer` Service on the host for every LoadBalancer Service in the guest, copying the +guest's annotations once at creation and nothing afterwards. Under `autoAssign: false` +such a Service sits `Pending` forever, and asking every guest to name a claim is not an +answer. So the core offers an **opt-in** path, enabled per class by an annotation on the +`IPAddressClass` and off everywhere otherwise: for a LoadBalancer Service with no claim +annotation, no `loadBalancerClass`, no pin and no assigned address, the core mints an +`IPAddressClaim` of the same name in the same namespace, **owned by the Service**, and +writes the §5 annotation onto the Service; the driver then associates exactly as if a +tenant had written it. A Service may name a class explicitly, or opt out, through +annotations the core defines — which is what a guest cluster's annotations, copied by the +deputy, reach for. A class marked for auto-claim is independent of the default class: the +two decisions have different owners. + +Two rules follow from "owned by the Service". **A claim with a controller owner follows +its owner, not the class's reclaim policy**: when the Service goes, its claim goes, and +the claim's addresses are deleted even under `Retain`, because nobody asked to keep them +and a `Released` address per deleted Service is a leak, not a reservation. To keep such an +address, remove the owner reference and the claim becomes an ordinary held one. The rule +is stated for any controller owner, not only Services, so machinery that mints claims on a +tenant's behalf (community #45) inherits it. And **held comes before minted**: when the +namespace already holds an unattached claim of that class, auto-claim attaches the oldest +one rather than minting, and mints only when none is free. In a brokered deployment where +tenants cannot create claims (*User-facing changes*), that is what makes a plain +LoadBalancer Service wear the address the tenant owns, and a second Service with no spare +claim stay `Pending` rather than silently mint one. The first rule is implemented +(cozystack/address-controller#7); the second is specified here. + +Allocator-first and claim-first still race in the handoff below, and the inventory can +transiently disagree with the live `status.loadBalancer` set. The design commits to +**eventual consistency** there. What remains once holders are in place is narrower than a general theft window, and worth naming precisely rather than waving at: @@ -552,6 +624,8 @@ and is explicit about which one covers which threat: `IPAddress.status.phase: Conflict`, raises a condition and a Service event, and does not let the rightful claim silently lose its address. This is what "dealt with" means for the cases admission cannot prevent (see the trust boundary in *Security*). + Adjudication is **per class**: the driver serving the address's class is the one that + marks it, so with two drivers neither judges the other's handoff window as theft. A Service arriving *without* a claim reference is **not** by itself the thing being defended against — an eager allocator handing a Service an address is the normal case §6 @@ -569,7 +643,7 @@ draw from it *automatically*; they draw from the admin's other pools. Explicit s guarantee. > *"A cluster-admin runs `kubectl create -f bad-service.yaml --as -> system:serviceaccount:cozy-address-controller:address-controller`, referencing the +> system:serviceaccount:metallb-iad-system:metallb-iad`, impersonating the driver, referencing the > reserved pool and maybe pinning a specific address."* — Admission cannot stop this: the > request carries the controller's identity, and a cluster-admin is the cluster's trust > root anyway (they can rewrite the policy or edit MetalLB objects directly). Layer 2 is @@ -636,7 +710,8 @@ to an RBAC-gated object: 1. A tenant may only write `local.sdn.cozystack.io/ip-address-claim`, naming an object in their **own namespace**, which RBAC *can* gate. -2. The **controller** writes the backend's raw pool/pin annotations. +2. The **driver** of the claim's class writes the backend's raw pool/pin annotations; the + core never does. 3. A `ValidatingAdmissionPolicy` **rejects a non-allowlisted principal** writing *any* backend annotation that selects a pool or pins an address — MetalLB's `metallb.io/address-pool` and `metallb.io/loadBalancerIPs`, plus the §1.2 equivalents @@ -669,7 +744,7 @@ content. **Layer 2 — reconciliation, for everyone admission cannot bind.** Layer 1 is keyed on identity, so it is defeated by anyone who can impersonate an allowlisted ServiceAccount — which a cluster-admin can (`--as -system:serviceaccount:cozy-address-controller:address-controller`), along with rewriting +system:serviceaccount:metallb-iad-system:metallb-iad`), along with rewriting the policy or editing MetalLB objects directly. **This is a real trust boundary, not a bug to be closed:** a cluster-admin is the cluster's trust root, and handing out a reserved address is within their authority, not an escalation. What the @@ -712,9 +787,12 @@ hardening this does not have to discover them. - **Two Services reference one claim** → resolved by reconciliation, not at admission. A `ValidatingAdmissionPolicy` cannot see the claim — CEL is given the request object and the policy's params, not arbitrary cluster state — so admission-time rejection would need a - webhook. Instead one association wins by a deterministic rule, the loser reports - `Conflicted`, and the address is never silently moved. A 1:1 binding is 1:1; letting the - second win quietly is how an address goes dead. + webhook. The rule is **the live holder keeps it**: the Service already in `associatedTo` + stays associated for as long as it exists, a later Service naming the claim is refused + with an `AssociationRejected` event, and the claim reports the refusal in an `Associated` + condition naming both; only when the holder is gone does the next one take over. The + address is never silently moved. A 1:1 binding is 1:1; letting the second win quietly is + how an address goes dead. - **Plain Service pulls a reserved address** (explicit `address-pool`/pin, or a request made under an impersonated controller identity) → not prevented in general; **detected**. The `IPAddress` goes `Conflict`, the offending Service is named, and the reservation is @@ -722,8 +800,8 @@ hardening this does not have to discover them. - **`spec.addressName` names an address that is absent, `Bound`, `Released`, or of another class or family** → nothing binds; the claim stays `Pending` with the ordinary waiting reason. Naming an address filters, it never seizes. -- **A `Dual` claim associated with a single-stack Service** → the Service carries the one - family it has; the other address stays `Bound` to the claim and inert. The claim is not +- **A `Dual` claim associated with a single-stack Service** → the driver pins only the + family the Service carries; the other address stays `Bound` to the claim and inert. The claim is not `Lost`, and the unattached family is never silently released. - **Backend lacks `Pin`** → claims against that class are refused loudly, rather than allocating an address that can never be attached. With no capability registry there is @@ -754,37 +832,40 @@ hardening this does not have to discover them. Sketch — sequencing is an open question: 1. CRDs + core controller + the **MetalLB** provisioner + the admission policy. (The - admission policy is not phase 2. See *Security*.) The controllers and the driver - exist; the admission policy does not — see *Implementation status*. -2. The Cilium provisioner. Proves the abstraction is not a MetalLB adapter — **this is + admission policy is not phase 2. See *Security*.) All three exist, the policy as the + driver's opt-in template — see *Implementation status*. The §5 `loadBalancerClass` + check and the held-before-minted rule of §6 complete this phase. +2. **Gateway as an association target** (§5), with `Gateway.spec.addresses` fenced. This + is the consumer tenants will meet first: the accepted external-database-exposure + proposal puts a tenant's database traffic on the single address of the tenant Gateway, + so the address a tenant most wants to keep is one a Service annotation cannot reach. + Which Cilium release honours `spec.addresses` is open question 4. +3. The Cilium provisioner. Proves the abstraction is not a MetalLB adapter — **this is the phase that either validates or falsifies the design**, and it should come early. A second *consumer* has since exercised the association half (*Implementation status*); a second *provisioner* is what remains. -3. A cloud provisioner (adoption path, `source.providerRef`). Proves the source union. +4. A cloud provisioner (adoption path, `source.providerRef`). Proves the source union. ## Open questions 1. **Provisioners: in-tree or out-of-tree?** In-tree for three backends; CSI's history says that stops scaling. Where is the line, and do we want the CRD contract on day one? -2. **Reuse Cluster API's IPAM?** `ipam.cluster.x-k8s.io` `IPAddressClaim`/`IPAddress` is - *literally* this pattern — namespaced claim, concrete address object, a documented - third-party provider contract — and shares even the kind names we chose. It has only - ever been wired to **Machine** addressing, never Services, and has no reclaim policy or - association. Do we adopt those kinds (and that group), or mint our own and merely copy - the contract? (Leaning: our own — we need reclaim policy and association, which it has - no concept of. But the shape is not novel and review should know that.) This is the - greenfield-vs-adopt question *Positioning* leaves open. -3. **Dual-stack.** Does one `IPAddress` carry a v4 and a v6 address, or does a `Dual` claim - bind two `IPAddress`es? (PV has no precedent. Leaning: two objects, one claim — the - status list already admits it.) -4. **PureLB.** Does it have *any* "request this exact IP" mechanism? If not, it cannot be a +2. **PureLB.** Does it have *any* "request this exact IP" mechanism? If not, it cannot be a backend, and that should be stated rather than discovered. -5. **Sharing one address across Services** (different ports — GCP and Cilium both permit +3. **Sharing one address across Services** (different ports — GCP and Cilium both permit it). Today one consumer at a time holds a claim and a second reference is refused (§5). Should that relax to several consumers on disjoint ports? It is the mirror of the deferred several-addresses-on-one-consumer case in *Non-goals*, and the two probably want answering together, since both are really "what is the cardinality between a claim and the things that wear it". +4. **Gateway.** `Gateway.spec.addresses` is the natural pin for a Gateway consumer (§5), + and Cilium supports it with LB-IPAM; whether the Cilium release a platform ships honours + it, and whether the infrastructure-annotation path is needed as a fallback, is + unverified. The provisioner decides where the pin lands; which object a tenant + annotates is the part that needs settling here. +5. **The entitlement object's shape** (*Security*). Namespace-keyed, per-class, a count; + whether it is one object per namespace or one per class, who may read it, and whether + the reference admission policy consumes it directly as its `paramRef` are open. ## Alternatives considered @@ -806,6 +887,22 @@ the allocator, it scales as O(tenants) or O(addresses) in operator-managed objec (RBAC does not gate fields), and it hands every tenant the ability to steal any reserved address that is not currently attached. See *Security*. +**Reuse Cluster API's IPAM kinds.** `ipam.cluster.x-k8s.io` `IPAddressClaim`/`IPAddress` is +*literally* this pattern — namespaced claim, concrete address object, a documented +third-party provider contract — and shares even the kind names chosen here. **Rejected:** +it has only ever been wired to **Machine** addressing, never Services, and has no reclaim +policy and no association, which are the two things this proposal exists to add. The +contract is copied; the group is not. + +**Adopt every live LoadBalancer assignment into the ledger.** An earlier draft reconciled +`status.loadBalancer` on every Service into `IPAddress` objects, so the inventory would +converge on everything in use. **Rejected:** catch-all adoption of objects nobody asked for +is uncharacteristic of Kubernetes, and there is nothing to adopt — a class's pool is +`autoAssign: false`, so a Service holding a class address without a claim got it through a +pin, which is a conflict to surface, not a reservation to record; and addresses outside +every class are not the ledger's concern. The real need underneath — Services that should +draw from a class but cannot name a claim — is met by the opt-in auto-claim of §6. + **Wait for upstream.** **Rejected:** KEP-1880 lists generalizing to an IPAM API as an explicit non-goal, and `loadBalancerIP` was deprecated with no replacement. There is nothing to wait for. From 71e0383fcae579b74094e5b6f5d24d3e859a2090 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Mon, 2 Nov 2026 15:33:27 +0100 Subject: [PATCH 10/16] design-proposal(ip-address-claims): answer the review on fencing, entitlement and lifecycle Review on this PR, on the packaging PR and on the sibling proposal asked for the same handful of things, which the text now carries. - The admission fence covers every input the backend honours: MetalLB's legacy annotation prefix and Service.spec.loadBalancerIP, which the controller still parses, are gated beside the current keys, and the comparison is old against new. Not every platform controller is allowlisted: a deputy forwarding user content is denied, and its Services reach the ledger through the claim annotation or auto-claim. The policy ships with Warn and Audit modes, and already-pinned Services are adopted by pre-binding, not broken. - Who may draw from which class is stated as a contract rather than handed to a sibling: the check keys on the namespace so that machinery-minted claims are covered, the empty className is an explicit decision, and addressName is refused to tenants or checked in a webhook, since CEL cannot resolve it. The natural home is an optional per-namespace, per-class count in the core, the per-class quota scoping Kubernetes applies to storage; its shape is an open question and the distribution writes it. addressName itself is exactly volumeName: unauthorized by design, with pre-binding as the admin's lever. - Two operating modes fall out of RBAC alone: self-service, where tenants create claims, and brokered, where platform machinery does and tenants attach and detach but never release. Metering follows the claim, since a held address is held whether or not a Service wears it. Both are stated under user-facing changes. - Lifecycle cases the implementation surfaced: a namespace deleted while the core is down waits on the finalizer, a class with bound addresses refuses deletion rather than withdrawing them all, and an optional releaseDelay on the class returns Released addresses to Available after a cooling-off period, with a hold marker for addresses an operator does not want reissued. - One sentence on Service.spec.externalIPs, the field behind CVE-2020-8554, as a further reason the kinds are not ExternalIP*. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 159 +++++++++++++++---- 1 file changed, 125 insertions(+), 34 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index 20b03f98..d354261b 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -51,6 +51,10 @@ pattern, and the overlap is worth naming up front so reviewers are not surprised shape, but only ever wired to **Machine** addressing, never Services, and with no reclaim policy or association. We mint our own (*Alternatives considered*); the shared names are an acknowledgement that the shape is not novel. +- **`Service.spec.externalIPs`** is the older, unrelated mechanism — a Service claiming an + address by value with no allocator behind it, the field behind CVE-2020-8554 — and + `ExternalIP*` kinds, proposed in review, would invite confusion with it. One more reason + the kinds are not named that way. Kinds are qualified by API group, so the short-name overlap is not a collision. The kinds live in `local.sdn.cozystack.io` (see *Positioning*). @@ -224,8 +228,9 @@ being restored, only something being added. ## Goals -- A tenant can request an address and receive one, without an admin pre-creating a - pool object for them. +- A tenant can request an address and receive one — or, in a brokered deployment, have + one requested on their behalf and attach it — without an admin pre-creating a pool + object for them. - An address **survives** the deletion of the workload it was attached to (reclaim policy `Retain`), and can be attached to a different workload afterwards. - An operator can enumerate every address the cluster owns, and see who holds it. @@ -333,6 +338,8 @@ spec: loadBalancerClass: metallb.io/public # the implementation that announces it; # empty => the cluster's default implementation reclaimPolicy: Retain # Retain | Delete (default Retain) + # releaseDelay: 720h # optional, Retain only: Released -> Available + # # after this long (Failure and edge cases) parameters: # opaque to the core controller addresses: ["203.0.113.0/24"] --- @@ -394,6 +401,13 @@ naming one names an object outside its own namespace. Neither this field nor decides which classes a namespace may draw from. That gap is identical with or without `addressName`, and closing it belongs above this controller; see *Security*. +This is precisely how Kubernetes treats `volumeName`: any claim may name any `Available` +volume of its class, nothing authorizes the name, and the admin's lever for a volume meant +for one tenant is to **pre-bind** it by setting its `claimRef` first. Pre-provisioned +addresses work the same way — an admin who wants an address to go to one namespace binds +it before anyone can name it. Where a deployment wants the field gated rather than merely +narrowed, *Security* says how. + `IPAddressClaim.status.addresses` is a **list** deliberately: a `Dual` claim binds a v4 and a v6 `IPAddress` and must report both, and a scalar `address` field could not. For a single-family claim the list has one entry. `Dual` is one claim binding two objects @@ -673,10 +687,25 @@ cluster. - **Tenants** gain one kind: `IPAddressClaim`. Create it, read `status.addresses[].address`, put that in DNS, reference the claim by name from a Service. They never see `IPAddress`, `IPAddressClass`, MetalLB, or a pin annotation. +- **Two operating modes, decided by RBAC alone.** In **self-service** mode tenants create + and delete their own claims, bounded by quota. In **brokered** mode tenants hold *read* + access to claims, and platform machinery — a billing integration, a portal, a tenant + controller — creates them on the tenant's behalf: the tenant may attach and detach the + address, through the §5 annotation or the held-before-minted rule of §6, but cannot + release it, and an address moved between namespaces is moved by the machinery through + `spec.addressName`. Nothing in the model differs between the two. A hosting provider + that sells addresses on a minimum term runs the second; a platform team with internal + tenants runs the first. - **Admins** gain `IPAddressClass` (one per address source) and `IPAddress` (a read-mostly inventory: `kubectl get ipaddresses` finally answers "what do we own and who has it"). - **Quota** works with no new machinery: `count/ipaddressclaims.` in a stock - `ResourceQuota`. + `ResourceQuota`. (A distribution that renders quotas from its own tenant object has to + pass `count/` keys through; Cozystack's tenant chart does not yet.) +- **Metering follows the claim.** A platform that meters `type: LoadBalancer` Services + today meters the wrong thing once addresses are claims: a held address is held whether + or not a Service wears it, and a placeholder in the driver's namespace is nobody's + workload. The `IPAddress` ledger is the unit of accounting — its `claimRef` says whose, + its conditions say since when — and a platform that bills addresses bills claims. ## Upgrade and rollback compatibility @@ -688,6 +717,16 @@ cluster. - **Rollback.** Deleting the CRDs with `reclaimPolicy: Retain` leaves the backend objects and the live Services untouched; addresses stay where they are. With `Delete`, it does not — flagged as the irreversible case. +- **Adopting Services that are already pinned.** A cluster that enables the fence with + hand-pinned Services in place does not break them: an unchanged pin always passes, since + the policy compares old against new. To bring such an address under the ledger, an admin + creates its `IPAddress` pre-bound to a claim (`claimRef` set, exactly as a pre-provisioned + PV), then adds the claim annotation to the Service; the driver's pin equals the existing + one and nothing moves. Removing a pin is a change and is gated, so unpinning a legacy + Service is an allowlisted operation. +- **Turning the fence on.** The policy ships with `validationActions` configurable, so a + deployment runs `Warn` or `Audit` first, reads what would have been denied, extends the + allowlist, and only then switches to `Deny`. ## Security @@ -717,14 +756,25 @@ to an RBAC-gated object: `metallb.io/address-pool` and `metallb.io/loadBalancerIPs`, plus the §1.2 equivalents for other backends — on a Service. -*Identifying the controller, and not breaking the platform.* The policy must not rely on -a hardcoded username. It matches the controller's **ServiceAccount** by name and -namespace (`system:serviceaccount::`), configured at install time. And it must -carry an **allowlist**: real clusters have non-tenant principals that legitimately write -these annotations — GitOps reconcilers (Flux/Argo SAs) and system controllers (system -Istio Services ship with pool annotations already set). A naive "reject everyone but the -controller" breaks reconciliation on day one, so the policy denies the write only for -principals **outside** an explicit, admin-configurable allowlist. +*Identifying the driver, and not breaking the platform.* The policy must not rely on a +hardcoded username. It matches the driver's **ServiceAccount** by name and namespace +(`system:serviceaccount::`), configured at install time, and it must carry an +**allowlist**: real clusters have non-tenant principals that legitimately write these +annotations — GitOps reconcilers applying content their operators authored, and system +components whose own Services ship with pool annotations pre-set (an Istio installation, +say). A naive "reject everyone but the driver" breaks reconciliation on day one, so the +policy denies the write only for principals **outside** an explicit, admin-configurable +allowlist of usernames and groups, and it ships with `Warn`/`Audit` modes for rollout +(*Upgrade and rollback compatibility*). + +*The fence covers every input the backend honours.* Not only the current annotation: a +backend's deprecated spellings are still parsed, and so may be a core field. For MetalLB +the gated set is `metallb.io/loadBalancerIPs`, `metallb.io/address-pool` and +`metallb.io/allow-shared-ip`, the same three under the legacy `metallb.universe.tf/` +prefix, **and `Service.spec.loadBalancerIP`**, which MetalLB still reads despite the +field's deprecation in Kubernetes 1.24. Omit any one and the fence has a door. The +comparison is old-versus-new, not presence: the driver pins the tenant's own Service, and +the tenant must stay able to edit the rest of it. *The allowlist's hard case: trusted principals applying untrusted content.* An identity allowlist assumes a trusted principal writes content it authored, and that assumption fails @@ -732,14 +782,18 @@ wherever a privileged controller applies material some user supplied. Two shapes ordinary rather than exotic: a GitOps reconciler that applies user-authored releases under its **own** identity instead of impersonating the user, and a cloud-controller-manager that copies Service annotations from a user-controlled cluster into an infrastructure Service -verbatim. Allowlist such a deputy and every user behind it inherits the ability to write a -pin annotation; deny it and legitimate reconciliation breaks. So **annotation filtering at -the deputy is a precondition of layer 1, not a hardening extra**: a strip-list for the gated -keys in any controller that forwards user-supplied annotations, and impersonation rather -than ambient authority in any reconciler applying user-authored content. Until those hold -for a given deputy, layer 1 is **partial** for the users behind it and layer 2 is what -covers them. The design says so rather than assuming a trusted identity implies trusted -content. +verbatim. So **not every platform controller belongs on the allowlist**. A deputy that +forwards user content is *denied*, like the users behind it, and the Services it creates +get their addresses through the one path the fence leaves open: the claim annotation, or +the auto-claim of §6, whose class-selection and opt-out annotations are exactly the ones a +deputy may safely copy. Allowlist such a deputy instead and every user behind it inherits +the ability to write a pin; deny it without that path and legitimate Services stay +`Pending`. Where a deputy must forward gated keys for some other reason, **annotation +filtering at the deputy is a precondition of layer 1, not a hardening extra**: a strip-list +for the gated keys, and impersonation rather than ambient authority in any reconciler +applying user-authored content. Until those hold for a given deputy, layer 1 is +**partial** for the users behind it and layer 2 is what covers them. The design says so +rather than assuming a trusted identity implies trusted content. **Layer 2 — reconciliation, for everyone admission cannot bind.** Layer 1 is keyed on identity, so it is defeated by anyone who can impersonate an allowlisted ServiceAccount — @@ -759,22 +813,45 @@ visible faults instead of quiet theft. Landing layer 1 is worth doing on its own, independent of the rest of the model: the theft window it closes is live in any cluster running a shared auto-assign pool today. -**Where this component stops.** The two layers above cover the surface a reservation can -be *stolen* through. They do not close field-level authorization in general: a tenant who -can create an `IPAddressClaim` can consume an address, may name any `spec.className`, and -may name a specific `IPAddress` through `spec.addressName` (§2). `ResourceQuota` bounds how -many addresses they hold; nothing here decides *which class* a namespace is entitled to -draw from. +**Where this component stops, and the entitlement it leaves room for.** The two layers +above cover the surface a reservation can be *stolen* through. They do not decide *which +class* a namespace may draw from: a tenant who can create an `IPAddressClaim` can consume +an address and may name any `spec.className`. `ResourceQuota` bounds how many addresses +they hold, not from where. Kubernetes has the same gap for `StorageClass` — any namespace +may name any class — and closes it, where it bothers to, with per-class quota scoping. That is deliberate rather than unfinished. This component's job is the **capability** — -reserve an address, hold it, attach it, detach it, release it — and a capability that also -tried to be its own policy engine would have to encode a tenancy model it does not have and -cannot see. The gating belongs where the tenancy model lives: a `ValidatingAdmissionPolicy` -restricting which classes a namespace may name (the same mechanism as layer 1, with a -different subject), or platform machinery that mints claims on a tenant's behalf instead of -letting tenants write claims directly. **`spec.className` and `spec.addressName` are the two -fields such a policy must cover** — naming them here is the point, so that a deployment -hardening this does not have to discover them. +reserve an address, hold it, attach it, detach it, release it — and encoding a tenancy +model it cannot see belongs above it. Three things are stated here so that whoever closes +the gap does not have to discover them: + +1. **The check keys on the namespace, never on the requesting principal.** Platform + machinery that mints claims on a tenant's behalf — community #45's attachment + controller, a billing integration — creates them under its *own* identity, so a policy + keyed on the requester never sees the tenant. A namespace-keyed check covers + tenant-authored and machinery-minted claims alike, and a `ValidatingAdmissionPolicy` + can express it synchronously, since `paramRef` resolves a parameter object in the + request's namespace. +2. **The empty `className` is a decision, not a convenience.** "Empty means the default + class" is the documented path, so a rule of the form *empty or entitled* admits every + namespace to the default class. Whether the default class is entitled is a separate + boolean. +3. **`spec.addressName` cannot be checked in CEL**, because resolving an address to its + class needs a second object read. A deployment that gates it either refuses the field + to tenant principals outright — the reference policy's choice, and the right one where + platform machinery is what moves specific addresses — or checks it in a webhook. + +The natural home for the data is an **optional entitlement object in the core**: per +namespace, per class, the number of claims that namespace may hold of that class, with +zero meaning not entitled and absence meaning unrestricted — the per-class quota scoping +Kubernetes applies to storage, which gives entitlement and a per-tenant ceiling in one +object. When present, the core refuses a claim that exceeds it with a condition, and the +reference admission policy enforces the same bound synchronously on the tenant-facing +path. The object's shape is **deferred** (*Open questions*); what is fixed is that it is +namespace-keyed and that the distribution, not the tenant, writes it. In Cozystack the +`Tenant` object is the obvious source and the tenant chart the obvious renderer, which is +also how a brokered deployment (*User-facing changes*) expresses "this tenant owns three +addresses of the public class". ## Failure and edge cases @@ -809,6 +886,19 @@ hardening this does not have to discover them. the claim, not an admission-time rejection. - **Address adopted from a provider, then released provider-side** → `IPAddress` goes `Lost`. It must not silently re-allocate. +- **Controllers down while a namespace holding claims is deleted** → the claim finalizer + holds the namespace until the core returns, exactly as PVC protection does; a controller + outage never leaks an address, at the cost of a deletion that waits. +- **Class deleted while addresses of it are bound** → refused, not cascaded. The rendered + pool and advertisement are owned by the class, so letting the deletion through would + withdraw every address of the class at once; a finalizer holds the class until its + addresses are gone. +- **`Released` addresses accumulate under `Retain`** → by default an admin clears + `claimRef`, PV semantics. A class may set an optional **`releaseDelay`**: once it + elapses the controller clears `claimRef` itself and the address returns to `Available`, + so a release followed by a cooling-off period before reuse needs no operator. An + address marked held — an annotation the controller honours — is exempt, which is how an + operator quarantines an address they do not want reissued. ## Testing @@ -819,7 +909,8 @@ hardening this does not have to discover them. - **e2e (per backend):** MetalLB and Cilium, on a real cluster: an external client reaches the workload on the reserved address, and still reaches it after the workload is rebuilt behind the same claim. -- **Admission:** a tenant writing a raw pool-select or pin annotation is rejected; a tenant +- **Admission:** a tenant writing a raw pool-select or pin annotation, a legacy-prefixed + one, or `spec.loadBalancerIP` is rejected; a tenant reserving an address and a second tenant attempting to pin it directly is rejected; an allowlisted GitOps/system principal writing a pool annotation is **admitted** (the allowlist does not over-block). From d6e35ac2c258e35d5f05c43d200b4330d68eee69 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Mon, 2 Nov 2026 15:44:18 +0100 Subject: [PATCH 11/16] design-proposal(ip-address-claims): record the two decisions implementation forced Two changes in the preceding revision were not review feedback but implementation findings that changed the design's course, and the repository's process asks for a decision record in that case so the reasoning outlives the pull-request thread. 0001 records that the reference driver reserves through placeholder Services and lets MetalLB allocate, after a driver-side allocator proved invisible to the backend and able to split-brain against its live state, and what that costs: the handoff gap, and sticky assignments that keep a live Service announcing after its pin is withdrawn. 0002 records that the ledger adopts nothing and that the core instead claims, opt-in per class, on behalf of Services that cannot name a claim, after tenant-cluster Services rendered by a cloud-controller- manager sat pending under autoAssign false; and why such claims follow their owner rather than the class reclaim policy. Both are linked from the proposal's Decisions section. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 7 +- ...ackend-allocates-and-the-ledger-records.md | 89 +++++++++++++++++ ...t-the-core-claims-on-behalf-of-services.md | 95 +++++++++++++++++++ 3 files changed, 189 insertions(+), 2 deletions(-) create mode 100644 design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md create mode 100644 design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index d354261b..46222570 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -90,8 +90,11 @@ kinds live in `local.sdn.cozystack.io` (see *Positioning*). ## Decisions - +Implementation ran ahead of this document, and two of its findings changed the design +rather than merely filling it in. Each is recorded where the reasoning can be found again: + +- [0002. The ledger does not adopt assignments; the core claims on behalf of Services that cannot](./decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md) — why §6's backwards inventory was dropped and the opt-in auto-claim took its place. +- [0001. The backend allocates; the ledger records what it assigned](./decisions/0001-the-backend-allocates-and-the-ledger-records.md) — why the reference driver reserves through placeholder Services instead of carving addresses itself, and what that costs. ## Positioning diff --git a/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md b/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md new file mode 100644 index 00000000..3b486610 --- /dev/null +++ b/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md @@ -0,0 +1,89 @@ +# 0001. The backend allocates; the ledger records what it assigned + +- **Number:** `0001` +- **Date:** `2026-07-20` +- **Status:** Accepted +- **Deciders:** `@lllamnyp` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/metallb-iad#1`](https://github.com/cozystack/metallb-iad/pull/1) +- **Implemented in:** [`cozystack/metallb-iad#1`](https://github.com/cozystack/metallb-iad/pull/1) (merged 2026-07-23) + +## Context + +The proposal's §3 said that for self-allocating backends — MetalLB, Cilium LB-IPAM, +kube-vip — *"our provisioner is the IPAM of record"*: nothing in those backends holds a +reservation, so the ledger would carve addresses from the class range itself and the +backend would merely be told which address to pin. + +The reference MetalLB driver was first built that way. It walked the class CIDRs, skipped +the network and broadcast addresses and every address already in the ledger, took the +first free one, created the `IPAddress`, and relied on create-collision on the object name +as its lock +([design §10](https://github.com/cozystack/metallb-iad/blob/main/docs/design.md#10-alternative-considered-driver-side-first-free-allocation)). + +Two things were wrong with it, and both were structural rather than bugs. A ledger-only +reservation is **invisible to MetalLB**: a held-but-unattached address was free in +MetalLB's books, so the entire anti-theft posture rested on the admission policy, which the +proposal had already conceded is defeated by impersonation. And the driver was a **second +IPAM** that could disagree with MetalLB's live state: it could pick an address some plain +Service already held from an overlapping pool, and discover the collision only at pin +time. Reviewer @lexfrei had argued the same point from the other direction on the proposal +([review, 2026-07-23](https://github.com/cozystack/community/pull/35#pullrequestreview-4763639339)): +a reservation should be enforced in the allocator's own accounting, with detection as the +backstop. + +## Decision + +The driver does not compute addresses. To reserve one it creates a selectorless +`type: LoadBalancer` **placeholder Service** in its own namespace, drawing from the class's +pool; MetalLB assigns an address to it and, the Service having no endpoints, never +announces it. The driver observes the assignment, pins the placeholder to it, and only then +records the `IPAddress`. A held address therefore exists in MetalLB's accounting for as +long as it is held, and MetalLB itself refuses to hand it to any other Service. The ledger +is the record of reservation; the backend is the allocator of record. Association is a +**handoff**: record `associatedTo`, delete the placeholder, pin the workload's Service. + +## Why not the alternatives + +- **Driver-side first-free allocation** (the first implementation, replaced in + [`56b3d95`](https://github.com/cozystack/metallb-iad/commit/56b3d95)). Lost on the two + structural facts above: a reservation MetalLB cannot see is not enforced by anything + but identity-keyed admission, and a parallel allocator can split-brain against the + backend's live state. The design remains valid for a backend with no allocation API at + all, but MetalLB is not such a backend — allocation by Service *is* its API. +- **Gapless handoff by sharing the address between placeholder and workload** + (`metallb.io/allow-shared-ip`, analysed in + [`e2e608f`](https://github.com/cozystack/metallb-iad/commit/e2e608f)). Deferred, not + rejected: MetalLB's sharing rules would impose `externalTrafficPolicy: Cluster` and + port-disjointness on tenant Services. Kept as an opportunistic optimisation for Services + that happen to satisfy both. + +## Consequences + +- A reserved-but-unattached address is safe to leave lying around, which the proposal + could not previously claim; admission and conflict detection become defence in depth + rather than the primary guarantee. Proposal §6 was rewritten around this. +- One placeholder Service, with a ClusterIP, per held address. Cheap, but not free, and + visible in the driver's namespace. +- The handoff is not atomic. Between deleting the placeholder and pinning the workload's + Service the address is briefly unheld; the window is bounded by reconcile latency and is + the one interval where the admission policy is load-bearing. +- MetalLB assignments are sticky and `autoAssign: false` gates only *new* allocation + (observed on a live cluster, + [design §5](https://github.com/cozystack/metallb-iad/blob/main/docs/design.md#5-association-the-gap-handoff)). + So withdrawing the pin from a live Service does not take the address away from it: the + ex-holder keeps announcing, the re-created placeholder sits `Pending`, and the ledger + reports `Conflict` naming the ex-holder until that Service is deleted or re-addressed. + Detaching an address from a live Service means recreating the Service; the proposal's + §5 now says so instead of claiming the announcement is withdrawn. +- The `IPAddress` is created after allocation is a fact, which is the allocate–observe–record + flow a cloud driver would use with a provider-side reservation; the placeholder plays + that object's role. This is why the proposal's §3 no longer says the provisioner is the + IPAM of record for self-allocating backends. + +## Revisit if + +MetalLB or Cilium LB-IPAM gains a first-class reservation object that can be held without +a Service, or the shared-handoff optimisation proves safe enough to make the default. A +backend with no allocation-by-Service API at all would reopen first-free allocation for +*that* driver only. diff --git a/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md b/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md new file mode 100644 index 00000000..0cf638f7 --- /dev/null +++ b/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md @@ -0,0 +1,95 @@ +# 0002. The ledger does not adopt assignments; the core claims on behalf of Services that cannot + +- **Number:** `0002` +- **Date:** `2026-10-30` +- **Status:** Accepted +- **Deciders:** `@lllamnyp` +- **Proposal:** [`../README.md`](../README.md) +- **Decided in:** [`cozystack/address-controller#7`](https://github.com/cozystack/address-controller/pull/7) +- **Implemented in:** [`cozystack/address-controller#7`](https://github.com/cozystack/address-controller/pull/7) (open); the held-before-minted rule is not yet implemented + +## Context + +The proposal's §6 originally had two halves. Holders (record 0001) were one. The other said +*"the inventory still works backwards"*: because an eager allocator assigns before any claim +exists, the controller would also observe `status.loadBalancer` on every Service and +reconcile what it found into `IPAddress` objects after the fact, so that the ledger would +converge on everything in use. It was introduced in the first revision in answer to a +production report that a shared auto-assign pool was the normal deployment +([comment, 2026-07-15](https://github.com/cozystack/community/pull/35#issuecomment-4977850733) +and the [reply](https://github.com/cozystack/community/pull/35#issuecomment-4984507704)). + +Neither controller ever implemented it, and when the reason for it was examined it was not +there. With a class's pool at `autoAssign: false`, a plain Service cannot draw a class +address *except* by naming the pool or pinning the address — which is a conflict to +surface, not an allocation to record — and addresses from pools no class manages were never +the ledger's concern. Catch-all adoption of objects nobody asked for is also uncharacteristic +of Kubernetes: the PersistentVolume controller does not adopt volumes it did not provision. + +What the implementation *did* find was the opposite problem. The packaging work for tenant +Kubernetes clusters showed that cloud-provider-kubevirt creates a host-side +`type: LoadBalancer` Service for every LoadBalancer Service in a guest, copying the guest's +annotations once at creation and nothing afterwards +([cozystack/cozystack#3382 review](https://github.com/cozystack/cozystack/pull/3382#pullrequestreview-5503155946), +[address-controller#7](https://github.com/cozystack/address-controller/pull/7)). Under +`autoAssign: false` such Services sat `Pending` with *no available IPs*. These are Services +that *should* draw from a class but cannot name a claim, because nothing in their render +path has heard of claims. + +## Decision + +The ledger records only what was reserved through it; it adopts nothing. In its place the +core offers an **opt-in auto-claim** path, enabled per class by an annotation on the +`IPAddressClass` and inert everywhere otherwise: for a live LoadBalancer Service with no +claim annotation, no `loadBalancerClass`, no pin and no assigned address, the core mints an +`IPAddressClaim` of the same name in the same namespace, **owned by the Service**, and +writes the association annotation onto the Service; the driver then associates as if a +tenant had written it. A Service may name a class, or opt out, through annotations the core +defines — the ones a deputy such as a cloud-controller-manager may safely copy. A claim with +a controller owner follows its owner rather than the class reclaim policy: when the Service +goes, its claim goes and its addresses are deleted even under `Retain`. Removing the owner +reference turns it into an ordinary held claim. + +## Why not the alternatives + +- **Reconcile every live assignment into the ledger** (the proposal's own §6 text until + this revision). Lost because there is nothing to adopt once class pools are + `autoAssign: false`, and because an inventory that adopts foreign assignments would + record addresses nobody reserved, with no claim to attribute them to. Recorded in the + proposal's *Alternatives considered*. +- **Require every guest cluster to name a claim.** Lost on the deputy's behaviour: the + cloud-controller-manager copies annotations at creation only, so a guest would have to + know about claims before creating its Service, and every Kubernetes-app tenant would + need to be taught the contract. Auto-claim keeps the guest ignorant and still gives the + Service a ledgered address. +- **Let auto-claimed addresses follow the class reclaim policy.** Lost because nobody asked + to keep them: under `Retain` every deleted Service would leave a `Released` address + behind, which is a leak, not a reservation. The same concern was raised independently + against machinery-minted claims on community #45 + ([review, 2026-10-12](https://github.com/cozystack/community/pull/45#pullrequestreview-5491027438)), + which is why the rule is stated for any controller owner, not only Services. + +## Consequences + +- The core now touches Services on this one path; the proposal's claim that it never does + is withdrawn. +- Auto-claim is independent of the default class. The two decisions have different + owners, and a class marked for auto-claim must be deployed after drivers have opted + their own placeholder Services out + ([cozystack/metallb-iad#5](https://github.com/cozystack/metallb-iad/pull/5)), or the core + will claim the driver's holders. +- A deputy that forwards user content no longer needs to be allowlisted by the admission + fence: its Services reach the ledger through the claim-reference and class-selection + annotations, which the fence leaves writable. +- The brokered operating mode gains a natural attach surface: with the held-before-minted + rule (specified, not yet implemented), a plain LoadBalancer Service in a namespace that + already holds an unattached claim wears that address rather than minting a new one, and + a second Service with no spare claim stays `Pending`. +- Deputies copy annotations at creation only, so class selection and opt-out must be set on + the guest Service before it exists; adding them later has no effect. + +## Revisit if + +cloud-provider-kubevirt gains an annotation filter or a configurable claim contract, which +would let guest clusters name claims directly; or a second deputy appears whose behaviour +the annotation-copy assumption does not fit. From abd2f6e958a2cddba556d11423f017300c2b3afa Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Tue, 3 Nov 2026 08:54:46 +0100 Subject: [PATCH 12/16] design-proposal(ip-address-claims): rewrite the prose as plain technical statements A reading pass over the proposal and both decision records, with no change of substance. Every sentence now names a component, says what it does, and gives the consequence in a sentence of its own. Evaluative asides, closing restatements, and reassurances that the rule already implied are deleted. A fact that was derived in several sections is now stated once in the section that owns it and referenced from the others. The document is a third shorter with no fact removed. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 1428 +++++++---------- ...ackend-allocates-and-the-ledger-records.md | 101 +- ...t-the-core-claims-on-behalf-of-services.md | 120 +- 3 files changed, 715 insertions(+), 934 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index 46222570..8291f6b2 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -3,22 +3,19 @@ - **Title:** `IP addresses as a first-class resource: IPAddress, IPAddressClaim, IPAddressClass, and a provisioner contract` - **Author(s):** `@lllamnyp` -> **The shape is settled and implemented; the contract details are not.** The problem -> statement, the primitives survey, and the object model are worked out, and phase 1 of -> *Rollout* exists in code (see *Implementation status*, which also lists where the -> document runs ahead of the code). Deliberately still open: the provisioner contract's -> own API (capabilities are not modelled yet), the entitlement object of *Security*, the -> sequencing beyond phase 1, and the open questions below. +> The object model, the problem statement and the survey of existing primitives are +> settled, and phase 1 of *Rollout* is implemented. *Implementation status* lists where +> the text is ahead of the code. Open: the provisioner contract's own API (capabilities +> are not modelled), the entitlement object of *Security*, sequencing after phase 1, and +> *Open questions*. ## Overview -A tenant cannot *own* an address. They can only cause one to appear as a side effect -of creating a `Service type: LoadBalancer`, and it evaporates when that Service does. -There is no object to hold, keep, quota, hand to another workload, or point DNS -at with confidence. +A tenant cannot own an address. An address appears when a tenant creates a `Service` of +`type: LoadBalancer` and disappears when the Service is deleted. No object holds it, +counts it, moves it to another workload, or is stable enough to put in DNS. -This proposal makes the address a resource. Applying the `PersistentVolume` pattern -to addresses: +This proposal makes the address a resource, on the `PersistentVolume` pattern: | storage | addresses | role | |---|---|---| @@ -27,180 +24,142 @@ to addresses: | `PersistentVolumeClaim` | **`IPAddressClaim`** (namespaced) | "give me one" — what a tenant creates | | CSI driver | **a provisioner contract** | how a class's backend allocates, adopts, and pins | -The driving use case is the **public** address — the AWS Elastic IP experience: -**allocate an address, keep it, attach it to something, detach it, attach it to -something else, release it when done.** Not: "create a Service and hope you get the -same IP back." But the resource is not public-specific. A class names a pool, and a -platform can hand out more than one kind of pool — public ranges, a private internal -LB range, a provider's static addresses. So the kinds are named for the general thing -(`IPAddress`), and "public" is a property of the class's pool, not of the API. +The driving use case is the public address, with the semantics of an AWS Elastic IP: +allocate, keep, attach, detach, attach elsewhere, release. The resource is not +public-specific. A class names a pool, and a platform can offer several pools: public +ranges, a private LB range, a provider's static addresses. The kinds are therefore named +`IPAddress`, and "public" is a property of a class's pool. ### Naming -The kinds `IPAddress` and `IPAddressClaim` are deliberate echoes of an existing -pattern, and the overlap is worth naming up front so reviewers are not surprised: - -- **Kubernetes core** ships an `IPAddress` kind in `networking.k8s.io` (KEP-1880). It - is *not* this: it is a low-level record where the object name **is** the IP, with a - generic `parentRef`, and its authors named "any generalization onto something like - an IPAM API" an explicit **non-goal** (see Context). Ours lives in a different API - group, carries a reclaim policy, a claim binding, and a class, and is the IPAM - object core declined to build. -- **Cluster API** ships `IPAddressClaim`/`IPAddress` in `ipam.cluster.x-k8s.io` with a - documented third-party provider contract — the same claim-plus-concrete-address - shape, but only ever wired to **Machine** addressing, never Services, and with no - reclaim policy or association. We mint our own (*Alternatives considered*); the shared - names are an acknowledgement that the shape is not novel. -- **`Service.spec.externalIPs`** is the older, unrelated mechanism — a Service claiming an - address by value with no allocator behind it, the field behind CVE-2020-8554 — and - `ExternalIP*` kinds, proposed in review, would invite confusion with it. One more reason - the kinds are not named that way. - -Kinds are qualified by API group, so the short-name overlap is not a collision. The -kinds live in `local.sdn.cozystack.io` (see *Positioning*). +`IPAddress` and `IPAddressClaim` share their short names with two existing kinds: + +- **Kubernetes core** has `IPAddress` in `networking.k8s.io` (KEP-1880). It is a record + whose name is the IP, with a generic `parentRef`. KEP-1880 names "any generalization + onto something like an IPAM API" a non-goal. Our kind is in a different group and + carries a reclaim policy, a claim binding and a class. +- **Cluster API** has `IPAddressClaim` and `IPAddress` in `ipam.cluster.x-k8s.io`, with a + provider contract. They serve Machine addressing only and have no reclaim policy and no + association. We define our own kinds (*Alternatives considered*). + +`ExternalIP*`, proposed in review, was not chosen. `Service.spec.externalIPs` is an older, +unrelated mechanism in which a Service claims an address by value with no allocator +(CVE-2020-8554), and the kinds would be confused with it. + +Kinds are qualified by API group, so the shared short names do not collide. The group is +`local.sdn.cozystack.io` (*Positioning*). ## Scope and related proposals -- **Adjacent, not a dependency:** `design-proposals/structured-external-exposure` - (community #29) restructures how a managed application *requests* exposure. This - proposal is about what an address **is**. The two meet at one point — an exposure - entry should be able to name an `IPAddressClaim` instead of implicitly minting an - address — but neither needs the other to land first, and this proposal does not - assume #29's shape. -- **Related:** `Service.spec.loadBalancerClass`, the upstream field selecting which LB - implementation serves a Service. An `IPAddressClass` names a provisioner and a pool, - which determines the implementation that will announce the address. An `IPAddressClass` - therefore **determines** the `loadBalancerClass` a consuming Service must carry (§5). (An - `ExposureClass` kind was proposed alongside community #29 and never shipped; there is no - `network.cozystack.io` group. That direction went to `loadBalancerClass` instead — - cozystack/cozystack#3164, cozystack/cozystack#3218.) -- **Depended on by:** `design-proposals/endpoint-attachments` (community #45), which - attaches one endpoint of an application to an external address. It consumes an - `IPAddressClaim` and the association annotation of §5, and its `EndpointAttachment` - kind lives in **this same `local.sdn.cozystack.io` family** — the exposure half beside - the allocation half, another CRD-served resident relying on the `local.` prefix's - quota guarantee (*Positioning*). This proposal does not wait on it and lands - independently. -- **Deferred to a sibling:** the *datapath* for whole-IP 1:1 NAT (all ports in, and - the workload egressing **as** that address) — today provided by - [cozy-proxy](https://github.com/cozystack/cozy-proxy)'s nftables rules. That is a - CNI concern, tracked separately. **This proposal is the allocation half; that is - the forwarding half.** They are independent: allocation is useful with a stock - LoadBalancer Service and no 1:1 NAT at all. +- **Adjacent:** community #29 (`structured-external-exposure`) restructures how a managed + application requests exposure. This proposal defines what an address is. They meet at + one point: an exposure entry can name an `IPAddressClaim` instead of minting an address. + Neither depends on the other. +- **Related:** `Service.spec.loadBalancerClass` selects the LB implementation for a + Service. An `IPAddressClass` names a provisioner and a pool, which determines the + implementation that announces the address, so the class determines the + `loadBalancerClass` a consuming Service must carry (§5). The `ExposureClass` kind + proposed with #29 never shipped, and `network.cozystack.io` does not exist; + cozystack/cozystack#3164 and cozystack/cozystack#3218 chose `loadBalancerClass`. +- **Depended on by:** community #45 (`endpoint-attachments`), which attaches one endpoint + of an application to an address. It consumes an `IPAddressClaim` and the annotation of + §5, and its `EndpointAttachment` kind is in the same `local.sdn.cozystack.io` group + (*Positioning*). This proposal does not wait for it. +- **Deferred to a sibling:** the datapath for whole-IP 1:1 NAT (all ports in, egress as + that address), provided today by [cozy-proxy](https://github.com/cozystack/cozy-proxy). + That is a CNI concern. This proposal is the allocation half; allocation works with a + stock LoadBalancer Service and no NAT. ## Decisions -Implementation ran ahead of this document, and two of its findings changed the design -rather than merely filling it in. Each is recorded where the reasoning can be found again: +Two implementation findings changed the design: -- [0002. The ledger does not adopt assignments; the core claims on behalf of Services that cannot](./decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md) — why §6's backwards inventory was dropped and the opt-in auto-claim took its place. -- [0001. The backend allocates; the ledger records what it assigned](./decisions/0001-the-backend-allocates-and-the-ledger-records.md) — why the reference driver reserves through placeholder Services instead of carving addresses itself, and what that costs. +- [0002. The ledger does not adopt assignments; the core claims on behalf of Services that cannot](./decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md) — the backwards inventory of §6 is dropped and auto-claim replaces it. +- [0001. The backend allocates; the ledger records what it assigned](./decisions/0001-the-backend-allocates-and-the-ledger-records.md) — the reference driver reserves through placeholder Services instead of allocating itself. ## Positioning -**This should stand on its own, and also ship as part of the Cozystack API surface — -both, not either.** A controller that reserves an address, binds it to a workload -through whatever pin mechanism the backend exposes, and gates that binding behind an -admission policy is useful to any Kubernetes platform running MetalLB, Cilium LB-IPAM, -or a cloud LB — not only Cozystack. So it is designed to be **valuable standalone**: the -tenant contract is a plain namespaced CRD, the quota is a stock `ResourceQuota`, the -enforcement is a `ValidatingAdmissionPolicy` — all vanilla Kubernetes, with no -Cozystack-only machinery required to run it. - -Standalone value does **not** mean it lives outside the Cozystack API group. Cozystack -already ships components that are fully standalone products yet live under -`*.cozystack.io` — `etcd-operator.cozystack.io` is the precedent. This belongs there the -same way, and the component is deliberately shaped so Cozystack integrates it cleanly (a -default class, and an exposure path able to name a claim). Standalone-valuable and -in-the-Cozystack-group are not in tension, and designing for easy Cozystack integration -is a goal, not a compromise. - -It also belongs in the **networking** group rather than a new one of its own. Cozystack -networking is `sdn.cozystack.io` today: `SecurityGroup` lives there, served by the -Cozystack API as a tenant-facing interface over Cilium network policies. Cozyplane serves -its own `SecurityGroup` in that same group, plus `local.sdn.cozystack.io` for the kinds it -serves as CRDs. Addresses are the third networking feature to want a home, and the -argument here is only that networking APIs should not accumulate several different group -names: **one `sdn` family**, with the `local.` prefix marking CRD-served kinds and the -bare group the aggregated ones. So the kinds live in **`local.sdn.cozystack.io`**. -`ipam.cozystack.io` is not wrong on its own terms — it adds a fourth name to a subject -area that should have one. - -To be clear that this is a proposal and not an appeal to precedent: -`local.sdn.cozystack.io` is new, nothing serves it today, and this would be among the -first components to introduce it. Nor does the group -string create a Cozystack dependency for a standalone user — the same three CRDs install -under any name. - -The obvious objection is that `ipam.` describes the object better than `sdn.` does: an -address ledger is not a datapath concern, and its consumers include Gateways and cloud -providers that are not SDN either. That is a fair reading, and the reason it does not win -here is that this is **not all of IPAM and cannot become it** — allocation stays with the -allocator, since AWS and MetalLB both keep it — so an `ipam.cozystack.io` group would -promise a generality the component deliberately does not have. If upstream moves and that -generality becomes real, it is a v2 question. Until then the kinds sit with the rest of -networking. - -The kinds are minted rather than adopted from Cluster API's IPAM; *Alternatives -considered* says why. +The component is useful to any Kubernetes platform that runs MetalLB, Cilium LB-IPAM or a +cloud LB, and it also ships as part of the Cozystack API. The tenant contract is a +namespaced CRD, the quota is a `ResourceQuota`, the enforcement is a +`ValidatingAdmissionPolicy`. Nothing requires Cozystack. + +Standalone does not mean outside the Cozystack API group. `etcd-operator.cozystack.io` is +a standalone product under `*.cozystack.io`, and this follows it. The component is shaped +for Cozystack integration: a default class, and an exposure path that can name a claim. + +The group is `local.sdn.cozystack.io`, in the networking family, because networking APIs +should not accumulate group names. `SecurityGroup` is in `sdn.cozystack.io`, served by the +aggregated Cozystack API. Cozyplane serves its own `SecurityGroup` in that group and its +CRD-served kinds in `local.sdn.cozystack.io`. The `local.` prefix marks CRD-served kinds; +the bare group marks aggregated ones. `ipam.cozystack.io` would be a fourth name for one +subject. + +The prefix also carries a guarantee. CRD-served kinds pass through the kube-apiserver's +ResourceQuota admission, so `count/` quotas work on them. Kinds served by the aggregated +API do not. + +`local.sdn.cozystack.io` is new; nothing serves it today. The group string creates no +Cozystack dependency: the three CRDs install under any name. + +The objection is that `ipam.` describes an address ledger better than `sdn.`: the ledger +is not a datapath, and its consumers include Gateways and cloud providers. The answer is +that this component is not all of IPAM and cannot become it. Allocation stays with the +allocator (AWS, MetalLB), so an `ipam.` group would promise more generality than the +component has. If that changes, the group is a v2 question. + +The kinds are defined here, not adopted from Cluster API's IPAM (*Alternatives +considered*). ## Implementation status -Phase 1 of *Rollout* exists, under the Cozystack organisation: - -- **[address-controller](https://github.com/cozystack/address-controller)** — the - class-agnostic core: the three kinds, claim–address binding, status, reclaim, and the - contract per-class drivers plug into. It touches a Service only on the opt-in - auto-claim path of §6 (cozystack/address-controller#7). -- **[metallb-iad](https://github.com/cozystack/metallb-iad)** — the reference per-class - driver, for addresses announced by MetalLB. A reservation is held as a placeholder - Service in a driver-owned namespace, so MetalLB's own accounting refuses to - double-assign a held address. cozystack/metallb-iad#6 ships the admission policy of - *Security* as an opt-in chart template; cozystack/metallb-iad#5 opts the placeholders - out of auto-claim. -- **cozystack/cozystack#3382** packages both as optional system packages, with tenant - RBAC on claims and a distribution-specific admission policy. Which policy a deployment - runs is its own business: the driver's generic one is the reference, and a distribution - that needs a different shape disables it and ships its own. - -One backend proves little on its own, so the load-bearing evidence is a **second -consumer**. [Cozyplane](https://github.com/lllamnyp/cozyplane) wears reserved addresses -through this contract with **no module import, no CRD dependency, and no informer on the -claim kinds**: it copies a claim name into the association annotation (§5) on a Service it -already owns, and the driver does the rest. With the mechanism absent entirely it behaves -identically, auto-assigning instead — reserved and dynamic are one code path. Validated -end to end on a three-node cluster running address-controller + metallb-iad + MetalLB + -cozyplane: an address survived delete-and-rebind and stayed externally reachable -throughout. - -Note precisely which seam that tests. *Rollout* names a second **provisioner** as the -*allocation* seam, and it remains outstanding. What is demonstrated is the *association* -seam: that a backend with an entirely different datapath can wear a reserved address while -knowing nothing about the ledger beyond one annotation. The coupling turned out looser -than this document assumed it would need to be. - -**Where the document runs ahead of the code.** This is a design proposal; the following -are specified here and not yet implemented, so that nobody mistakes the text for a -description of the binaries: the class-determined `loadBalancerClass` check of §5; a -driver pinning only the address families its Service carries (§2); conditions on -`IPAddress` (§2) and the `Associated` condition on a claim (§5); per-class scoping of -conflict adjudication (§8); the held-before-minted rule of §6; the release delay and the -class-deletion guard of *Failure and edge cases*; Gateway as an association target (§5); -the entitlement object of *Security*. Where the code carries detail the text does not — -`claimRef.uid`, a sticky `status.className` on the claim, claim conditions and a `Lost` -claim phase, a `Pending` address phase, the provisioner stamp annotation, the finalizers — -the implementation's own design document is the reference; none of it changes the model. +Phase 1 of *Rollout* exists under the Cozystack organisation: + +- **[address-controller](https://github.com/cozystack/address-controller)**: the core. + The three kinds, claim–address binding, status, reclaim, and the driver contract. It + writes to a Service only on the auto-claim path of §6 + (cozystack/address-controller#7). +- **[metallb-iad](https://github.com/cozystack/metallb-iad)**: the reference driver for + MetalLB. It holds a reservation as a placeholder Service in its own namespace, so + MetalLB's accounting refuses to assign a held address elsewhere. + cozystack/metallb-iad#6 ships the admission policy of *Security* as an opt-in chart + template; cozystack/metallb-iad#5 opts the placeholders out of auto-claim. +- **cozystack/cozystack#3382** packages both as optional system packages, with tenant RBAC + on claims and a distribution-specific admission policy. The driver's policy is the + reference; a distribution may disable it and ship its own. + +A second consumer tests the association contract. +[Cozyplane](https://github.com/lllamnyp/cozyplane) copies a claim name into the §5 +annotation on a Service it owns, with no module import, no CRD dependency and no informer +on the claim kinds. Without the mechanism it auto-assigns; reserved and dynamic are one +code path. On a three-node cluster with address-controller, metallb-iad, MetalLB and +cozyplane, an address survived delete-and-rebind and stayed reachable. + +This tests the association seam only. The allocation seam, a second provisioner, is +untested (*Rollout*). + +**Specified here and not yet implemented:** the `loadBalancerClass` check of §5; pinning +only the families a Service carries (§2); conditions on `IPAddress` (§2) and the +`Associated` condition on a claim (§5); per-class conflict adjudication (§8); the +held-before-minted rule of §6; the release delay and the class-deletion guard (*Failure +and edge cases*); Gateway as an association target (§5); the entitlement object +(*Security*). + +**In the code and not in this text:** `claimRef.uid`, `status.className` on the claim, +claim conditions, a `Lost` claim phase, a `Pending` address phase, the provisioner stamp +annotation, the finalizers. The implementation's design document is the reference for +these. None changes the model. ## Context ### What exists today -Cozystack installs MetalLB but renders no `IPAddressPool`/`L2Advertisement` itself; -the admin configures an allocator to suit the environment (MetalLB L2/BGP, Cilium -LB-IPAM, a cloud LB, or `Service.spec.externalIPs` pinning). +Cozystack installs MetalLB and renders no `IPAddressPool` or `L2Advertisement`. The admin +configures an allocator: MetalLB L2 or BGP, Cilium LB-IPAM, a cloud LB, or +`Service.spec.externalIPs`. -Across every one of those, the model is the same: **an address is allocated to a -Service, at the moment the Service is created, and freed when it is deleted.** +In every one of them an address is allocated to a Service when the Service is created and +freed when it is deleted. ### The problem @@ -212,89 +171,76 @@ Service, at the moment the Service is created, and freed when it is deleted.** > *"I want to reserve four addresses now, because the range is nearly full, and attach > them over the next month."* -None of these are expressible. The address has no independent existence, so: +None of these is expressible, because the address has no existence apart from the +Service: -- **It cannot be held.** Delete the Service, lose the address. Someone else may get it. -- **It cannot be moved.** There is no detach/attach; there is only "hope the allocator - hands you the same one." -- **It cannot be quota'd.** There is no object to count, so nothing bounds how many - addresses a tenant consumes. The only lever is pre-carving a pool per tenant. -- **It cannot be gated.** Anyone who can create a `Service` in a namespace can cause an - address to be allocated. RBAC authorizes *verbs on resources*, not *fields*. -- **It cannot be shown.** No inventory. "Which addresses do we own, and who has them?" - has no answer short of listing every Service in the cluster. +- **It cannot be held.** Deleting the Service frees the address, and another tenant may + receive it. +- **It cannot be moved.** There is no detach and attach. +- **It cannot be counted.** There is no object to quota. The only lever is a pool per + tenant. +- **It cannot be gated.** Anyone who can create a Service can allocate an address. RBAC + authorizes verbs on resources, not fields. +- **It cannot be listed.** "Which addresses do we own, and who holds them?" requires + listing every Service in the cluster. -This is a **new capability**, argued on its own merits: hold, move, quota, enumerate. -Cozystack has no prior reserve-then-associate mechanism to preserve — the platform uses -kube-ovn only for subnets and VPCs, not for its EIP objects — so there is nothing here -being restored, only something being added. +This is a new capability: hold, move, quota, enumerate. Cozystack has no +reserve-then-associate mechanism to preserve; it uses kube-ovn for subnets and VPCs, not +for its EIP objects. ## Goals -- A tenant can request an address and receive one — or, in a brokered deployment, have - one requested on their behalf and attach it — without an admin pre-creating a pool - object for them. -- An address **survives** the deletion of the workload it was attached to (reclaim - policy `Retain`), and can be attached to a different workload afterwards. -- An operator can enumerate every address the cluster owns, and see who holds it. -- The number of addresses a tenant may hold is bounded by a **plain `ResourceQuota`** - (`count/ipaddressclaims.`), with no new quota machinery. -- An admin configures the address source **once per class**, not once per tenant and - not once per address. No pool-per-tenant, no `/32` pools. -- The tenant-facing API does not name the backend. A tenant asks for an address; it - does not learn whether MetalLB, Cilium, or a cloud API produced it. -- The design admits at least: MetalLB, Cilium LB-IPAM, and a cloud provider whose - addresses are allocated outside the cluster. -- A reserved address is never *silently* lost to a plain `Service type: LoadBalancer`: - reservations are reconciled against live Service assignments, so a collision is a - detected, surfaced state rather than a quiet theft (see *Design §8* and *Security*). +- A tenant can request an address and receive one, or in a brokered deployment have one + requested on their behalf and attach it, without an admin pre-creating a pool. +- An address survives the deletion of the workload it was attached to (reclaim policy + `Retain`) and can be attached to another workload. +- An operator can list every address the cluster owns and who holds it. +- A `ResourceQuota` on `count/ipaddressclaims.` bounds how many addresses a tenant + holds. No new quota machinery. +- An admin configures an address source once per class. No pool per tenant, no `/32` + pools. +- The tenant API does not name the backend. +- The design admits MetalLB, Cilium LB-IPAM, and a cloud provider that allocates outside + the cluster. +- A reserved address is never silently lost to a plain LoadBalancer Service. A collision + is detected and reported (§8, *Security*). ### Non-goals -- **Announcing addresses.** Attraction (ARP/NDP, BGP, a cloud VNIC assignment) stays - with the LB implementation. This proposal allocates and binds; it never puts a - packet on a wire. +- **Announcing addresses.** ARP/NDP, BGP and cloud VNIC assignment stay with the LB + implementation. - **The 1:1 NAT datapath.** See *Scope*. -- **Several owned addresses on one consumer.** One rule is settled and enforced: a claim - is consumed by **one consumer at a time**, so two consumers cannot quietly share a - reservation. The converse is *not* settled, and this proposal does not pretend it is — - how many claims one consumer may hold is open. No limit on a *workload* is claimed: - nothing stops an application owning several Services, each with its own claim, and a - managed service reachable on a public address and an internal routable one at once is - exactly that — two Services, two claims, no new concept. But a single consumer wearing - several addresses is a real case rather than a hypothetical one, and Gateway API states - it outright: `Gateway.spec.addresses` is a **list**, so one Gateway legitimately wears - several addresses, and nothing in the model below says how several claims attach to one - consumer or how a provisioner renders that. **Acknowledged and deferred to future - work.** Two neighbouring cases stay out with it: *N* owned addresses on a workload whose - extra addresses live on a **tunnel interface the platform neither owns nor sees** (a VM - acting as a router or VPN concentrator), which is orthogonal to reservation; and the - platform-owned variant of it, which would need an internal address and in-guest - configuration per address. Dual-stack (one v4 + one v6) is a separate, narrower - question, treated below rather than folded into this. -- **Replacing the LB implementation.** MetalLB/Cilium/cloud stay exactly where they - are; this sits above them. -- **IPv4-only thinking.** The model is family-agnostic; a claim may request v4, v6, or - both; a `Dual` claim binds one `IPAddress` per family (§2). -- **Solving field-level authorization in general.** It is *used* here (see *Security*) - but the general problem is bigger than this proposal. +- **Several addresses on one consumer.** One rule is settled and enforced: a claim is + consumed by one consumer at a time. The converse is open: how many claims one consumer + may hold. A workload may own several Services, each with its own claim; a managed + service with a public address and an internal one is two Services with two claims. A + single consumer with several addresses is a real case: `Gateway.spec.addresses` is a + list. How several claims attach to one consumer, and how a provisioner renders that, is + deferred. Two related cases are deferred with it: extra addresses on a tunnel interface + the platform does not see (a VM acting as a router), which is unrelated to reservation; + and the platform-owned variant, which needs an internal address and in-guest + configuration per address. +- **Replacing the LB implementation.** MetalLB, Cilium and cloud LBs stay; this sits above + them. +- **IPv4-only thinking.** A claim may request IPv4, IPv6 or both; a `Dual` claim binds one + `IPAddress` per family (§2). +- **Field-level authorization in general.** It is used in *Security*; the general problem + is out of scope. ## Design ### 1. The primitives that already exist (survey) -Load-bearing findings, because the design is shaped by them: +Five findings shape the design: -1. **Nothing has a standing reservation.** Not MetalLB, not Cilium LB-IPAM, not - kube-vip, not PureLB, not OpenELB — and not Kubernetes core. KEP-1880 shipped - cluster-scoped `ServiceCIDR`/`IPAddress` kinds (the name *is* the IP, with a - generic `spec.parentRef`), and then explicitly named *"any generalization onto - something like an IPAM API"* a **non-goal**. Nothing upstream is coming to provide - this. It is ours to build. +1. **Nothing has a standing reservation.** Not MetalLB, Cilium LB-IPAM, kube-vip, PureLB, + OpenELB, or Kubernetes core. KEP-1880 shipped `ServiceCIDR` and `IPAddress` (the name + is the IP, with a generic `spec.parentRef`) and named "any generalization onto + something like an IPAM API" a non-goal. Nothing upstream will provide this. -2. **Every backend has a "pin this exact address" hook, and it is always an - annotation** — because `Service.spec.loadBalancerIP` was deprecated in Kubernetes - 1.24 with *no core replacement*, so every implementation invented its own: +2. **Every backend pins a specific address through an annotation.** + `Service.spec.loadBalancerIP` was deprecated in Kubernetes 1.24 with no replacement, so + each implementation defined its own: | backend | pin mechanism | |---|---| @@ -306,25 +252,21 @@ Load-bearing findings, because the design is shaped by them: | GCP / GKE | `networking.gke.io/load-balancer-ip-addresses` | | PureLB | **none found** — see *Open questions* | - Different spelling, identical semantics. **This is the seam the whole design rests - on:** if we can pin, we can bind a reserved address to a workload. + The semantics are identical. This is the seam the design rests on: a backend that can + pin can bind a reserved address to a workload. -3. **A reserved-but-unattached address is inert, everywhere.** MetalLB never announces - an address that is not the live `status.loadBalancer.ingress` of a Service it - allocated. Cilium LB-IPAM only *assigns*; something else announces. A cloud EIP - routes nowhere until associated. This is not an obstacle — **it is correct - semantics**, and it is exactly why AWS bills for an idle EIP. +3. **A reserved, unattached address is inert everywhere.** MetalLB announces only the live + `status.loadBalancer.ingress` of a Service it allocated. Cilium LB-IPAM assigns; + something else announces. A cloud EIP routes nowhere until associated. These are the + correct semantics; AWS bills for an idle EIP for this reason. -4. **One pool can serve many tenants.** MetalLB's - `IPAddressPool.spec.serviceAllocation.{namespaces,namespaceSelectors,serviceSelectors}` - and Cilium's `CiliumLoadBalancerIPPool.spec.serviceSelector` both scope a *single* - pool by namespace or label. And `L2Advertisement`/`BGPAdvertisement` select - **pools**, not addresses — so one advertisement covers arbitrarily many addresses. - **The pool-per-tenant and `/32`-pool patterns were never necessary.** +4. **One pool can serve many tenants.** MetalLB's `IPAddressPool.spec.serviceAllocation` + and Cilium's `CiliumLoadBalancerIPPool.spec.serviceSelector` scope a single pool by + namespace or label. `L2Advertisement` and `BGPAdvertisement` select pools, not + addresses. Pool-per-tenant and `/32` pools were never necessary. -5. **`Service.spec.loadBalancerClass`** (GA in 1.24) is the standard way for multiple - LB implementations to coexist; MetalLB honours it via `--lb-class`. It is how a - cluster can run more than one backend without them fighting. +5. **`Service.spec.loadBalancerClass`** (GA in 1.24) lets several LB implementations + coexist; MetalLB honours it via `--lb-class`. ### 2. The object model @@ -387,71 +329,53 @@ status: name: web-lb ``` -**`spec.addressName` — asking for a *specific* address.** The -`PersistentVolumeClaim.spec.volumeName` analogue: a claim may name one `IPAddress` rather -than take whatever its class yields. It is meaningful only for a single-family claim, and -the controller honours it only if every ordinary condition already holds — the address is -`Available`, carries no `claimRef`, is not being deleted, and its class and family match -the ones the claim resolved. A name that matches nothing eligible is not an error: the -claim stays `Pending` and keeps waiting, exactly as it would with no `addressName` at all. -The field therefore **narrows a candidate set and never widens one**. It cannot pull an -address out of another class, take one away from another claim, or reclaim a `Released` -address before an admin has cleared its `claimRef`. - -What it deliberately does *not* do is authorize. `IPAddress` is cluster-scoped, so a claim -naming one names an object outside its own namespace. Neither this field nor -`spec.className` — equally free-text, and equally unable to be gated by namespace RBAC — -decides which classes a namespace may draw from. That gap is identical with or without -`addressName`, and closing it belongs above this controller; see *Security*. - -This is precisely how Kubernetes treats `volumeName`: any claim may name any `Available` -volume of its class, nothing authorizes the name, and the admin's lever for a volume meant -for one tenant is to **pre-bind** it by setting its `claimRef` first. Pre-provisioned -addresses work the same way — an admin who wants an address to go to one namespace binds -it before anyone can name it. Where a deployment wants the field gated rather than merely -narrowed, *Security* says how. - -`IPAddressClaim.status.addresses` is a **list** deliberately: a `Dual` claim binds a -v4 and a v6 `IPAddress` and must report both, and a scalar `address` field could not. -For a single-family claim the list has one entry. `Dual` is one claim binding two objects -— the implementation settled it that way, and the reason is the association contract: a -dual-stack Service names *one* claim in §5, so one claim must be able to yield both -families. The driver pins only the families the Service carries (*Failure and edge cases*). - -`IPAddress.status.conditions` exist so the ledger answers *when* as well as *who*. `Bound` -and `Associated` follow `metav1.Condition` semantics, and their `lastTransitionTime` is the -record of which claim held an address, and which consumer wore it, over time. An operator -reconstructing that after the fact — an abuse report naming an address and an hour — reads -it from the ledger, not from Service events, which expire. `Conflict` is reported the same -way, as a condition beside the phase, so that the offending consumer is named somewhere -durable. - -### 3. The address-source union — the reason this is not a MetalLB adapter - -There are two structurally different worlds, and the model must admit both: - -- **Self-allocating backends** (MetalLB, Cilium, kube-vip, OpenELB): *nothing anywhere* - holds a reservation. **The ledger is the record of reservation**, and the address is - carved from the class's range (`source.fromClass`). Which component picks the address - is the driver's business, and the reference driver deliberately lets the backend do it: - it asks MetalLB for an address through the holder of §6 and records what MetalLB - assigned, so the backend's allocator and the ledger can never disagree about who holds - what. A driver-side allocator was tried first and replaced for exactly that reason. -- **Provider-reservation backends** (AWS, GCP, Hetzner): the reservation is already a - real object with a stable handle (`eipalloc-…`, a named static address), guarded by - the provider's own IAM. **Our provisioner must adopt, not allocate** - (`source.providerRef`). - -This is precisely the `PersistentVolume` volume-source union (`spec.csi` / -`spec.nfs` / …), and it is the strongest evidence the shape is not bent around any one -backend: it survives a backend where allocation is not ours at all. The tenant's -`IPAddressClaim` never learns the difference. - -### 4. The provisioner contract (the "CSI analogue") - -`IPAddressClass.spec.provisioner` names a controller, exactly as `StorageClass` names a -CSI driver. Each provisioner declares **capabilities**, and the core controller refuses -what a backend cannot do rather than half-supporting it: +**`spec.addressName`** is the analogue of `PersistentVolumeClaim.spec.volumeName`: a +claim names one `IPAddress` instead of taking any address of its class. It applies to +single-family claims. The controller binds the named address only if it is `Available`, +has no `claimRef`, is not being deleted, and matches the claim's class and family. +Otherwise the claim stays `Pending`, as it would without the field. The field narrows the +candidate set. It cannot take an address from another class, from another claim, or from +`Released` state before an admin clears the `claimRef`. + +The field does not authorize. `IPAddress` is cluster-scoped, so the claim names an object +outside its namespace, and RBAC cannot gate the name. `spec.className` has the same +property. Kubernetes treats `volumeName` the same way: any claim may name any `Available` +volume of its class, and an admin who wants a volume to go to one tenant pre-binds it by +setting its `claimRef`. Pre-provisioned addresses are pre-bound the same way. Gating the +field is covered in *Security*. + +`IPAddressClaim.status.addresses` is a list because a `Dual` claim binds one `IPAddress` +per family and reports both. A single-family claim has one entry. `Dual` is one claim +binding two objects because a dual-stack Service names one claim in §5. The driver pins +only the families the Service carries (*Failure and edge cases*). + +`IPAddress.status.conditions` record when, not only who. `Bound` and `Associated` are +`metav1.Condition`s; their `lastTransitionTime` records which claim held the address and +which consumer wore it. An operator answering "which tenant had this address at this hour" +reads the ledger, not Service events, which expire. `Conflict` is reported as a condition +beside the phase, so the offending consumer is recorded durably. + +### 3. The address-source union + +Two kinds of backend exist, and the model admits both: + +- **Self-allocating backends** (MetalLB, Cilium, kube-vip, OpenELB) hold no reservation. + The ledger is the record of reservation, and the address is carved from the class range + (`source.fromClass`). Which component picks the address is the driver's choice. The + reference driver asks MetalLB for an address through the holder of §6 and records what + MetalLB assigned, so the backend and the ledger cannot disagree (decision 0001). +- **Provider-reservation backends** (AWS, GCP, Hetzner) already hold the reservation as an + object with a stable handle (`eipalloc-…`, a named static address), under the provider's + IAM. The provisioner adopts it (`source.providerRef`) and does not allocate. + +This is the `PersistentVolume` volume-source union (`spec.csi`, `spec.nfs`, …). It is the +evidence that the shape is not specific to MetalLB: it survives a backend where allocation +is not ours. The claim does not expose the difference. + +### 4. The provisioner contract + +`IPAddressClass.spec.provisioner` names a controller, as `StorageClass` names a CSI driver. +Each provisioner declares capabilities, and the core refuses what a backend cannot do: | capability | meaning | consequence if absent | |---|---|---| @@ -459,22 +383,18 @@ what a backend cannot do rather than half-supporting it: | `Adopt` | can wrap a provider-side reservation | `providerRef` claims rejected | | **`Pin`** | **can bind a specific address to a Service** | **the class cannot serve claims at all** | -`Pin` is not optional decoration — **it is the capability the entire model depends on.** -A backend that cannot be told *which* address to use can never attach a reserved one, -and a class over such a backend must reject `IPAddressClaim`s outright instead of -allocating an address it can never bind. (PureLB may be exactly this case; see *Open -questions*.) +`Pin` is the capability the model depends on. A backend that cannot be told which address +to use cannot attach a reserved one, so a class over such a backend rejects claims instead +of allocating an address it cannot bind. PureLB may be this case (*Open questions*). Whether provisioners are in-tree (a switch in one controller) or out-of-tree (separate -deployments, a real CRD contract, as CSI did) is an **open question**. In-tree is -right for three backends; the CSI lesson is that it stops being right at about five. +deployments with a CRD contract, as CSI did) is open. In-tree suffices for three +backends; CSI's history says it stops at about five. ### 5. Association -Association is a **separate, reversible act** — that is the entire point, and the thing -the current model cannot express. - -A workload references the **claim**, never the raw address: +Association is a separate, reversible act. A workload references the claim, never the +address: ```yaml kind: Service @@ -484,524 +404,406 @@ metadata: spec: {type: LoadBalancer, ...} ``` -The controller resolves the claim → address, verifies the claim is in the Service's own -namespace, and writes the backend's pin annotation (§1.2). MetalLB/Cilium/the cloud then -allocates and announces exactly that address. - -**The class determines the `loadBalancerClass`.** An `IPAddressClass` names the pool and -the provisioner that carves it, which determines the LB implementation that will announce -anything drawn from it, so a class unambiguously implies one `loadBalancerClass` value — -empty meaning the cluster's default implementation (§1.5) — and a consuming Service must -carry that value. This is not a preference: a Service whose `loadBalancerClass` selects one -implementation while its claim's address is held by another is asking two backends for one -address, and both will be wrong about who owns it. The provisioner's own holder Services -(§6) carry the same value for the same reason. The simplest way to express the mapping is -an explicit `spec.loadBalancerClass` on the class, as in §2; a driver that can derive it -from its own parameters may do that instead. What is required is the check, and it is -**specified here and not yet implemented** (*Implementation status*). - -The check can only ever be a **refusal**, never a repair: `loadBalancerClass` is immutable -on an existing Service, so a mismatch surfaces as a refused association naming both -values, and whoever creates the Service — a chart, a controller minting one on a tenant's -behalf, or a user — is responsible for setting it at creation. A controller that creates -the Service *and* names the claim has no excuse for getting it wrong, since it can read -the class first. - -Remove the annotation and the driver withdraws the pin it wrote, clears `associatedTo`, -and holds the address again; the `IPAddress` stays `Bound` to its claim and becomes inert — -**a reserved address, held, attached to nothing.** Which is what an unassociated EIP is. -One honest caveat: the backend does not necessarily follow. MetalLB keeps an assignment -until the Service changes or goes away, so a Service that drops the annotation but lives -on keeps announcing the address, and the ledger reports it as a `Conflict` naming that -Service until it is deleted or re-addressed. Detaching an address from a *live* Service -therefore means recreating that Service, which is the natural shape of a cutover anyway; -the common path — the workload is deleted, the address is kept — has no such wrinkle. - -**The annotated object is not always a Service.** Some consumers *own* the Service they -are announced on rather than being one: under Gateway API the implementation renders the -data-plane Service from the `Gateway`, so there is no user-authored Service to annotate, -and the object the user does write has no way to name an address. The contract is therefore -an **annotated object that resolves to an address consumer** — `Service` is the first -member and `Gateway` the obvious second. Which object a given backend annotates, and where -it lands the pin (`Gateway.spec.addresses`, infrastructure annotations propagated onto the -generated Service, or the generated Service itself), is the provisioner's business (§4). -The consequence that is *not* the provisioner's business is in *Security*: -`Gateway.spec.addresses` names an address **by value**, so it is the same escalation -surface as a backend pin annotation and the admission policy has to cover it. - -This is the claim-first path: the address exists before the Service names it. §6 covers -the opposite order — when an eager allocator hands a Service an address before any claim -exists. +The driver resolves the claim to its address, checks that the claim is in the Service's +namespace, and writes the backend's pin annotation (§1.2). The backend then allocates and +announces that address. + +**The class determines the `loadBalancerClass`.** A class names a pool and a provisioner, +which determines the LB implementation that announces its addresses, so a class implies +one `loadBalancerClass` value (empty means the cluster default, §1.5). A consuming Service +must carry that value. A Service whose `loadBalancerClass` selects one implementation +while its claim's address is held by another asks two backends for one address. The +provisioner's holder Services (§6) carry the same value. The simplest expression is +`spec.loadBalancerClass` on the class, as in §2; a driver may derive it from its +parameters instead. The check is specified and not implemented (*Implementation status*). + +The check refuses and never repairs: `loadBalancerClass` is immutable on an existing +Service. A mismatch is a refused association naming both values. Whoever creates the +Service sets the field at creation; a controller that creates the Service and names the +claim reads the class first. + +**Detaching.** When the annotation is removed, the driver withdraws its pin, clears +`associatedTo`, and holds the address again. The `IPAddress` stays `Bound` to its claim: +reserved and attached to nothing, like an unassociated EIP. MetalLB keeps an assignment +until the Service is deleted or re-addressed, so a running Service that drops the +annotation keeps announcing the address, and the ledger marks the address `Conflict` +naming that Service until it is gone (decision 0001). To detach an address from a running +Service, delete or recreate the Service. + +**The annotated object is not always a Service.** Under Gateway API the implementation +renders the data-plane Service from the `Gateway`, so there is no user-authored Service to +annotate. The contract is therefore an annotated object that resolves to an address +consumer. `Service` is the first member; `Gateway` is the second. Which object a backend +annotates, and where it lands the pin (`Gateway.spec.addresses`, infrastructure +annotations propagated to the generated Service, or the generated Service), is the +provisioner's choice (§4). `Gateway.spec.addresses` names an address by value, so the +admission policy must cover it (*Security*). + +This is the claim-first path. §6 covers the allocator-first path. ### 6. When the allocator gets there first -An eager allocator does not wait to be asked. MetalLB assigns an address the moment it -sees a `Service type: LoadBalancer`, before any `IPAddressClaim` exists and whether or not -one ever will. Two things follow. - -**A reservation is held in the allocator's own books, not merely recorded in ours.** For -every address that is reserved but not currently attached, the provisioner keeps a -**holder**: for MetalLB, a selectorless `type: LoadBalancer` Service in a provisioner-owned -namespace, which MetalLB assigns the address to and, having no endpoints, never announces — -held in the backend's accounting, silent on the wire. Neither MetalLB nor Cilium will give -one address to a second Service without an explicit sharing key, so for as long as the -holder exists the reservation is enforced by the allocator itself rather than by our -detection. This is the primary guarantee, and it is what makes a reserved-but-unattached -address safe to leave lying around. - -**Nothing adopts what it did not reserve.** An earlier draft had the controller observe -`status.loadBalancer` on every Service and reconcile foreign assignments into the ledger -after the fact. That is dropped (*Alternatives considered*): with a class's pool at -`autoAssign: false` (§7), a plain Service cannot draw a class address except by naming the -pool or pinning the address, and that is a conflict to detect (§8), not an allocation to -adopt. Addresses from pools no class manages are not in the ledger and are of no interest -to it. The inventory therefore answers "what has been reserved, and who holds it", not -"every address in the cluster" — which is the question an operator actually asks. - -**Auto-claim: the ledger reaching a Service that has never heard of it.** The opposite -need is real: a Service that *should* draw from a class but names no claim, because -whoever rendered it knows nothing about claims. The concrete case is a -cloud-controller-manager for tenant Kubernetes clusters, which creates a `type: -LoadBalancer` Service on the host for every LoadBalancer Service in the guest, copying the -guest's annotations once at creation and nothing afterwards. Under `autoAssign: false` -such a Service sits `Pending` forever, and asking every guest to name a claim is not an -answer. So the core offers an **opt-in** path, enabled per class by an annotation on the -`IPAddressClass` and off everywhere otherwise: for a LoadBalancer Service with no claim -annotation, no `loadBalancerClass`, no pin and no assigned address, the core mints an -`IPAddressClaim` of the same name in the same namespace, **owned by the Service**, and -writes the §5 annotation onto the Service; the driver then associates exactly as if a -tenant had written it. A Service may name a class explicitly, or opt out, through -annotations the core defines — which is what a guest cluster's annotations, copied by the -deputy, reach for. A class marked for auto-claim is independent of the default class: the -two decisions have different owners. - -Two rules follow from "owned by the Service". **A claim with a controller owner follows -its owner, not the class's reclaim policy**: when the Service goes, its claim goes, and -the claim's addresses are deleted even under `Retain`, because nobody asked to keep them -and a `Released` address per deleted Service is a leak, not a reservation. To keep such an -address, remove the owner reference and the claim becomes an ordinary held one. The rule -is stated for any controller owner, not only Services, so machinery that mints claims on a -tenant's behalf (community #45) inherits it. And **held comes before minted**: when the -namespace already holds an unattached claim of that class, auto-claim attaches the oldest -one rather than minting, and mints only when none is free. In a brokered deployment where -tenants cannot create claims (*User-facing changes*), that is what makes a plain -LoadBalancer Service wear the address the tenant owns, and a second Service with no spare -claim stay `Pending` rather than silently mint one. The first rule is implemented -(cozystack/address-controller#7); the second is specified here. - -Allocator-first and claim-first still race in the handoff below, and the inventory can -transiently disagree with the live `status.loadBalancer` set. The design commits to -**eventual consistency** there. - -What remains once holders are in place is narrower than a general theft window, and worth -naming precisely rather than waving at: - -- **The handoff.** Releasing the holder and pinning the workload's own Service is not - atomic. For that window the address is unheld and an eager allocator could hand it to - whichever Service happens to be asking. Narrow, but real — and one of the intervals where - layer 1 of *Security* earns its place. -- **Annotation writers.** With a reserved pool set `autoAssign: false` a plain Service - cannot *automatically* draw a reserved address; it can only get one by naming the pool or - pinning the address explicitly. That is precisely the write layer 1 gates and layer 2 - detects. - -### 7. What an admin actually configures - -For MetalLB, forever, for the whole cluster: - -- **one** `IPAddressPool` covering the reserved range, **with `autoAssign: false`**, and -- **one** `L2Advertisement` (or `BGPAdvertisement`) selecting it. - -That is all. No pool per tenant, no `/32`s. The `IPAddressClass` carries the range; the -provisioner renders the backend objects. `autoAssign: false` keeps MetalLB from handing a -reserved address out *automatically* — but it does **not** make the pool unreachable, -because a Service can still name the pool explicitly (`metallb.io/address-pool`) or pin an -address (`metallb.io/loadBalancerIPs`). Closing that path is a security concern, not a -pool-config one; see §8 and *Security*. - -### 8. Worked examples: how this rides real allocators, and where it does not hold - -No static pool layout can *guarantee* a reservation. On MetalLB, `autoAssign: false` -stops only automatic selection — a Service can still name the reserved pool -(`metallb.io/address-pool`) or pin a specific address (`metallb.io/loadBalancerIPs`) and -be handed a reserved address. And any principal who can impersonate the controller's -ServiceAccount — a cluster-admin can — sails past an admission policy keyed on identity. -So the design does **not** rest on an invariant it cannot enforce. It uses **two layers**, -and is explicit about which one covers which threat: - -1. **Admission reduces the blast radius for the parties multi-tenancy is about** — - tenants and namespaced principals. A `ValidatingAdmissionPolicy` forbids them from - writing *any* backend annotation that selects a pool or pins an address (MetalLB's - `metallb.io/address-pool` **and** `metallb.io/loadBalancerIPs`, plus the §1.2 - equivalents), leaving the claim reference as their only route to the reserved range. -2. **Reconciliation makes a collision a detected, correctable state — however it arose.** - The `IPAddress` ledger is authoritative. The controller watches Services and their - assigned `status.loadBalancer.ingress`; if a Service holds an address belonging to an - `IPAddress` whose `claimRef`/`associatedTo` does not name that Service, it sets - `IPAddress.status.phase: Conflict`, raises a condition and a Service event, and does - not let the rightful claim silently lose its address. This is what "dealt with" means - for the cases admission cannot prevent (see the trust boundary in *Security*). - Adjudication is **per class**: the driver serving the address's class is the one that - marks it, so with two drivers neither judges the other's handoff window as theft. - -A Service arriving *without* a claim reference is **not** by itself the thing being -defended against — an eager allocator handing a Service an address is the normal case §6 -reconciles after the fact. The conflict is narrower and specific: a Service ending up with -an address already `Bound` to a *different* claim — by an explicit pin, a race, or an -impersonated write. Layer 2 keys on exactly that — a `claimRef`/`associatedTo` mismatch — -not on the mere absence of an annotation. What separates legitimate allocation from theft -is whose reservation the address belongs to, not whether a claim annotation was present. - -**MetalLB.** The class's `IPAddressPool` is `autoAssign: false`, so plain Services never -draw from it *automatically*; they draw from the admin's other pools. Explicit selection -(`address-pool`/`loadBalancerIPs`) is blocked for tenants by layer 1 and detected by layer -2 for anyone else. Scoping the pool further with `IPAddressPool.spec.serviceAllocation` -(limit to a controller-set label or namespace) is available defense-in-depth — not a -guarantee. - -> *"A cluster-admin runs `kubectl create -f bad-service.yaml --as -> system:serviceaccount:metallb-iad-system:metallb-iad`, impersonating the driver, referencing the -> reserved pool and maybe pinning a specific address."* — Admission cannot stop this: the -> request carries the controller's identity, and a cluster-admin is the cluster's trust -> root anyway (they can rewrite the policy or edit MetalLB objects directly). Layer 2 is -> what applies. The Service is assigned a reserved address; the controller sees an -> assignment no claim authorizes; the `IPAddress` goes `Conflict` with the offending -> Service named. The reservation is not silently overwritten — the conflict is surfaced -> for an operator (or an automated policy) to resolve. That is the honest boundary: the -> model defends tenants against each other, and turns admin/impersonator collisions into -> **visible faults** rather than pretending to prevent them. - -**Cilium LB-IPAM.** Cilium has no `autoAssign` flag; the equivalent is `serviceSelector`. -The reserved `CiliumLoadBalancerIPPool` carries a `serviceSelector` matching a label the -controller stamps on associated Services only, so a plain Service lacks it and draws from -an unselected default pool. The same two layers apply: admission stops a tenant from -setting that label directly, and reconciliation catches an assignment no claim authorizes. - -**Cloud (adopt).** The provider holds the reservation (`eipalloc-…`) under its own IAM. A -plain LB Service with no annotation is handed an ephemeral address from the provider's -general pool; the provider's API refuses to double-assign a reserved EIP. Here the -separation is enforced by the provider, not by us — which is exactly why -`source.providerRef` (adopt, don't allocate) is the right model for this world, and why -this is the one backend where the reservation genuinely cannot be raided from inside the -cluster. +MetalLB assigns an address to a `Service` of `type: LoadBalancer` as soon as it sees it, +before any claim exists. + +**A reservation is held in the allocator's own accounting.** For every reserved, +unattached address the provisioner keeps a holder. For MetalLB the holder is a +selectorless `type: LoadBalancer` Service in a provisioner-owned namespace. MetalLB assigns +the address to it and, since it has no endpoints, never announces it. MetalLB and Cilium do +not give one address to a second Service without a sharing key, so the allocator itself +enforces the reservation while the holder exists. This is the primary guarantee (decision +0001). + +**The ledger records only addresses reserved through it.** The controller does not create +`IPAddress` objects for addresses it finds on Services. An address a plain Service holds +from a class pool is a conflict (§8), not an inventory entry. Decision 0002 covers the +earlier design. + +**Auto-claim.** Some Services should draw from a class but cannot name a claim, because +the component that renders them does not know about claims. The case that forced this is +a cloud-controller-manager for tenant Kubernetes clusters: it creates a host-side +`type: LoadBalancer` Service for each LoadBalancer Service in the guest and copies the +guest's annotations once, at creation. Under `autoAssign: false` these Services stay +`Pending`. The core therefore has an opt-in path, enabled per class by an annotation on +the `IPAddressClass`: for a LoadBalancer Service with no claim annotation, no +`loadBalancerClass`, no pin and no assigned address, the core creates an `IPAddressClaim` +with the Service's name in the Service's namespace, owned by the Service, and writes the +§5 annotation on the Service. The driver associates as usual. A Service can name a class +or opt out through annotations the core defines; a deputy may copy these from a guest +Service. The auto-claim class is independent of the default class. + +Two rules follow: + +- **A claim with a controller owner follows its owner, not the class reclaim policy.** + When the Service is deleted its claim is deleted, and the claim's addresses are deleted + even under `Retain`. A `Released` address per deleted Service would be a leak. To keep + the address, remove the owner reference; the claim is then an ordinary held claim. The + rule applies to any controller owner, so machinery that mints claims for a tenant + (community #45) inherits it. Implemented in cozystack/address-controller#7. +- **Held before minted.** If the namespace holds an unattached claim of the class, + auto-claim attaches the oldest one and mints only when none is free. In a brokered + deployment (*User-facing changes*) a plain LoadBalancer Service wears the address the + tenant owns, and a second Service with no spare claim stays `Pending`. Specified, not + implemented. + +Claim-first and allocator-first allocations race during the handoff, so the inventory can +disagree with `status.loadBalancer` briefly. The design accepts eventual consistency there. + +Two gaps remain with holders in place: + +- **The handoff.** Releasing the holder and pinning the workload's Service is not atomic. + During that window the address is unheld. Layer 1 of *Security* covers it. +- **Annotation writers.** With the pool at `autoAssign: false`, a plain Service receives a + reserved address only by naming the pool or pinning the address. Layer 1 gates that + write; layer 2 detects it. + +### 7. What an admin configures + +For MetalLB, once per cluster: + +- one `IPAddressPool` for the reserved range, with `autoAssign: false`; +- one `L2Advertisement` or `BGPAdvertisement` selecting it. + +The `IPAddressClass` carries the range and the provisioner renders both objects. +`autoAssign: false` stops MetalLB from assigning a reserved address automatically. It does +not stop a Service from naming the pool (`metallb.io/address-pool`) or pinning an address +(`metallb.io/loadBalancerIPs`); §8 and *Security* cover that. + +### 8. How this rides real allocators + +No pool layout guarantees a reservation. A Service can name the reserved pool or pin an +address (§7), and a principal that can impersonate the driver's ServiceAccount passes an +identity-keyed admission policy. The design therefore uses two layers: + +1. **Admission, for tenants and namespaced principals.** A `ValidatingAdmissionPolicy` + forbids them from writing any backend input that selects a pool or pins an address + (*Security* lists them). The claim reference is their only route to the reserved range. +2. **Reconciliation, for everything else.** The driver watches + `status.loadBalancer.ingress` on Services. If a Service holds an address whose + `IPAddress` has a `claimRef` or `associatedTo` that does not name that Service, the + driver sets `IPAddress.status.phase: Conflict`, adds a condition, and emits a Service + event. The rightful claim keeps its binding. Adjudication is per class: the driver + serving the address's class marks the conflict, so two drivers do not mark each other's + handoff windows. + +A Service without a claim annotation is not the threat. The conflict is a Service holding +an address `Bound` to a different claim, through a pin, a race, or an impersonated write. +Layer 2 keys on the `claimRef`/`associatedTo` mismatch, not on the absence of an +annotation. + +**MetalLB.** The class pool is `autoAssign: false`, so plain Services draw from other +pools. Explicit selection is blocked for tenants by layer 1 and detected by layer 2 for +everyone else. `IPAddressPool.spec.serviceAllocation` can restrict the pool to a +controller-set label or namespace as defence in depth. + +A cluster-admin who runs `kubectl create --as +system:serviceaccount:metallb-iad-system:metallb-iad` with a pinned Service passes +admission, because the request carries the driver's identity. A cluster-admin is the trust +root and can rewrite the policy or edit MetalLB objects directly, so admission does not +apply to them. Layer 2 applies: the `IPAddress` goes `Conflict` naming the Service, and an +operator resolves it. The model defends tenants against each other and reports admin +collisions as faults. + +**Cilium LB-IPAM.** Cilium has no `autoAssign`; the equivalent is `serviceSelector`. The +reserved `CiliumLoadBalancerIPPool` selects a label the driver sets on associated +Services, so a plain Service draws from a default pool. Admission stops a tenant from +setting the label; reconciliation detects an assignment no claim authorizes. + +**Cloud (adopt).** The provider holds the reservation under its IAM. A plain Service +receives an ephemeral address from the provider's pool, and the provider refuses to +double-assign a reserved EIP. The provider enforces the separation, which is why +`source.providerRef` is the model for this backend. ## User-facing changes -- **Tenants** gain one kind: `IPAddressClaim`. Create it, read `status.addresses[].address`, - put that in DNS, reference the claim by name from a Service. They never see `IPAddress`, - `IPAddressClass`, MetalLB, or a pin annotation. -- **Two operating modes, decided by RBAC alone.** In **self-service** mode tenants create - and delete their own claims, bounded by quota. In **brokered** mode tenants hold *read* - access to claims, and platform machinery — a billing integration, a portal, a tenant - controller — creates them on the tenant's behalf: the tenant may attach and detach the - address, through the §5 annotation or the held-before-minted rule of §6, but cannot - release it, and an address moved between namespaces is moved by the machinery through - `spec.addressName`. Nothing in the model differs between the two. A hosting provider - that sells addresses on a minimum term runs the second; a platform team with internal - tenants runs the first. -- **Admins** gain `IPAddressClass` (one per address source) and `IPAddress` (a read-mostly - inventory: `kubectl get ipaddresses` finally answers "what do we own and who has it"). -- **Quota** works with no new machinery: `count/ipaddressclaims.` in a stock - `ResourceQuota`. (A distribution that renders quotas from its own tenant object has to - pass `count/` keys through; Cozystack's tenant chart does not yet.) -- **Metering follows the claim.** A platform that meters `type: LoadBalancer` Services - today meters the wrong thing once addresses are claims: a held address is held whether - or not a Service wears it, and a placeholder in the driver's namespace is nobody's - workload. The `IPAddress` ledger is the unit of accounting — its `claimRef` says whose, - its conditions say since when — and a platform that bills addresses bills claims. +- **Tenants** gain one kind, `IPAddressClaim`. They create it, read + `status.addresses[].address`, put it in DNS, and name the claim from a Service. They + never see `IPAddress`, `IPAddressClass`, MetalLB or a pin annotation. +- **Two operating modes, set by RBAC.** In self-service mode tenants create and delete + claims, bounded by quota. In brokered mode tenants have read access to claims, and + platform machinery (a billing integration, a portal, a tenant controller) creates them. + The tenant attaches and detaches through the §5 annotation or the held-before-minted + rule of §6 and cannot release. The machinery moves an address between namespaces + through `spec.addressName`. The model is the same in both modes. A hosting provider that + sells addresses on a minimum term runs brokered; a platform team with internal tenants + runs self-service. +- **Admins** gain `IPAddressClass`, one per address source, and `IPAddress`, the + inventory. `kubectl get ipaddresses` answers who holds what. +- **Quota** is a `ResourceQuota` on `count/ipaddressclaims.`. A distribution that + renders quotas from its own tenant object must pass `count/` keys through; Cozystack's + tenant chart does not yet. +- **Metering follows the claim.** A held address is held whether or not a Service wears + it, and a placeholder in the driver's namespace belongs to no tenant workload. A platform + that meters LoadBalancer Services meters the wrong object once addresses are claims. The + `IPAddress` ledger is the unit of accounting: `claimRef` says whose, the conditions say + since when. ## Upgrade and rollback compatibility -- **Additive.** Nothing about today's `Service type: LoadBalancer` path changes; a - Service with no claim annotation behaves exactly as now. -- **Adoption path.** A cluster with existing LoadBalancer Services can import their - addresses as `IPAddress` objects with `source.providerRef`/`fromClass` and a `claimRef` - to a generated claim — an offline, reversible migration. -- **Rollback.** Deleting the CRDs with `reclaimPolicy: Retain` leaves the backend objects - and the live Services untouched; addresses stay where they are. With `Delete`, it does - not — flagged as the irreversible case. -- **Adopting Services that are already pinned.** A cluster that enables the fence with - hand-pinned Services in place does not break them: an unchanged pin always passes, since - the policy compares old against new. To bring such an address under the ledger, an admin - creates its `IPAddress` pre-bound to a claim (`claimRef` set, exactly as a pre-provisioned - PV), then adds the claim annotation to the Service; the driver's pin equals the existing - one and nothing moves. Removing a pin is a change and is gated, so unpinning a legacy - Service is an allowlisted operation. -- **Turning the fence on.** The policy ships with `validationActions` configurable, so a - deployment runs `Warn` or `Audit` first, reads what would have been denied, extends the - allowlist, and only then switches to `Deny`. +- **Additive.** A Service with no claim annotation behaves as today. +- **Adoption.** Existing LoadBalancer Services can be imported: create an `IPAddress` with + `source.providerRef` or `fromClass` and a `claimRef` to a generated claim. The migration + is offline and reversible. +- **Rollback.** Deleting the CRDs under `reclaimPolicy: Retain` leaves backend objects and + Services untouched. Under `Delete` it does not; this is the irreversible case. +- **Already-pinned Services.** The policy compares old and new values, so an unchanged pin + passes. To bring a pinned address under the ledger, create its `IPAddress` pre-bound to + a claim (`claimRef` set, as for a pre-provisioned PV), then add the claim annotation to + the Service. The driver's pin equals the existing one. Removing a pin is a change and is + gated, so unpinning a legacy Service requires an allowlisted principal. +- **Enabling the policy.** `validationActions` is configurable. Run `Warn` or `Audit` + first, read what would have been denied, extend the allowlist, then switch to `Deny`. ## Security -**Selecting a pool or pinning an address is a privilege-escalation surface, and this is -the sharp edge of the proposal.** - -RBAC authorizes *verbs on resources*, not *fields* or *annotations*. Today, anyone who -can create a `Service` in a namespace can write `metallb.io/loadBalancerIPs: ` — or name the reserved pool with `metallb.io/address-pool` — and because a -self-allocating backend has **no concept of a reservation**, it will happily hand over an -address another tenant reserved but has not yet attached. That is a theft window, on -MetalLB and Cilium alike. It is *not* a cloud problem: there, IAM already guards -`eipalloc-…`. - -No single mechanism closes this, so the response has **two layers** (see *Design §8*), -and the design is explicit about which threat each one covers. - -**Layer 1 — admission, against tenants.** Turn the ungatable annotation into a reference -to an RBAC-gated object: - -1. A tenant may only write `local.sdn.cozystack.io/ip-address-claim`, naming an object in - their **own namespace**, which RBAC *can* gate. -2. The **driver** of the claim's class writes the backend's raw pool/pin annotations; the - core never does. -3. A `ValidatingAdmissionPolicy` **rejects a non-allowlisted principal** writing *any* - backend annotation that selects a pool or pins an address — MetalLB's - `metallb.io/address-pool` and `metallb.io/loadBalancerIPs`, plus the §1.2 equivalents - for other backends — on a Service. - -*Identifying the driver, and not breaking the platform.* The policy must not rely on a -hardcoded username. It matches the driver's **ServiceAccount** by name and namespace -(`system:serviceaccount::`), configured at install time, and it must carry an -**allowlist**: real clusters have non-tenant principals that legitimately write these -annotations — GitOps reconcilers applying content their operators authored, and system -components whose own Services ship with pool annotations pre-set (an Istio installation, -say). A naive "reject everyone but the driver" breaks reconciliation on day one, so the -policy denies the write only for principals **outside** an explicit, admin-configurable -allowlist of usernames and groups, and it ships with `Warn`/`Audit` modes for rollout -(*Upgrade and rollback compatibility*). - -*The fence covers every input the backend honours.* Not only the current annotation: a -backend's deprecated spellings are still parsed, and so may be a core field. For MetalLB -the gated set is `metallb.io/loadBalancerIPs`, `metallb.io/address-pool` and -`metallb.io/allow-shared-ip`, the same three under the legacy `metallb.universe.tf/` -prefix, **and `Service.spec.loadBalancerIP`**, which MetalLB still reads despite the -field's deprecation in Kubernetes 1.24. Omit any one and the fence has a door. The -comparison is old-versus-new, not presence: the driver pins the tenant's own Service, and -the tenant must stay able to edit the rest of it. - -*The allowlist's hard case: trusted principals applying untrusted content.* An identity -allowlist assumes a trusted principal writes content it authored, and that assumption fails -wherever a privileged controller applies material some user supplied. Two shapes of this are -ordinary rather than exotic: a GitOps reconciler that applies user-authored releases under -its **own** identity instead of impersonating the user, and a cloud-controller-manager that -copies Service annotations from a user-controlled cluster into an infrastructure Service -verbatim. So **not every platform controller belongs on the allowlist**. A deputy that -forwards user content is *denied*, like the users behind it, and the Services it creates -get their addresses through the one path the fence leaves open: the claim annotation, or -the auto-claim of §6, whose class-selection and opt-out annotations are exactly the ones a -deputy may safely copy. Allowlist such a deputy instead and every user behind it inherits -the ability to write a pin; deny it without that path and legitimate Services stay -`Pending`. Where a deputy must forward gated keys for some other reason, **annotation -filtering at the deputy is a precondition of layer 1, not a hardening extra**: a strip-list -for the gated keys, and impersonation rather than ambient authority in any reconciler -applying user-authored content. Until those hold for a given deputy, layer 1 is -**partial** for the users behind it and layer 2 is what covers them. The design says so -rather than assuming a trusted identity implies trusted content. - -**Layer 2 — reconciliation, for everyone admission cannot bind.** Layer 1 is keyed on -identity, so it is defeated by anyone who can impersonate an allowlisted ServiceAccount — -which a cluster-admin can (`--as -system:serviceaccount:metallb-iad-system:metallb-iad`), along with rewriting -the policy or editing MetalLB objects directly. **This is a real trust -boundary, not a bug to be closed:** a cluster-admin is the cluster's trust root, and -handing out a reserved address is within their authority, not an escalation. What the -design owes here is not prevention but **detection**. Because the `IPAddress` ledger is -authoritative and continuously reconciled, any collision — impersonation, a direct -backend edit, a misconfigured pool, MetalLB assigning a reserved address to a Service with -no matching claim — surfaces as `IPAddress.status.phase: Conflict` with the offending -Service named, and the rightful claim is never *silently* overwritten. Admission defends -tenants against each other; reconciliation turns the cases admission cannot reach into -visible faults instead of quiet theft. - -Landing layer 1 is worth doing on its own, independent of the rest of the model: the -theft window it closes is live in any cluster running a shared auto-assign pool today. - -**Where this component stops, and the entitlement it leaves room for.** The two layers -above cover the surface a reservation can be *stolen* through. They do not decide *which -class* a namespace may draw from: a tenant who can create an `IPAddressClaim` can consume -an address and may name any `spec.className`. `ResourceQuota` bounds how many addresses -they hold, not from where. Kubernetes has the same gap for `StorageClass` — any namespace -may name any class — and closes it, where it bothers to, with per-class quota scoping. - -That is deliberate rather than unfinished. This component's job is the **capability** — -reserve an address, hold it, attach it, detach it, release it — and encoding a tenancy -model it cannot see belongs above it. Three things are stated here so that whoever closes -the gap does not have to discover them: - -1. **The check keys on the namespace, never on the requesting principal.** Platform - machinery that mints claims on a tenant's behalf — community #45's attachment - controller, a billing integration — creates them under its *own* identity, so a policy - keyed on the requester never sees the tenant. A namespace-keyed check covers - tenant-authored and machinery-minted claims alike, and a `ValidatingAdmissionPolicy` - can express it synchronously, since `paramRef` resolves a parameter object in the - request's namespace. -2. **The empty `className` is a decision, not a convenience.** "Empty means the default - class" is the documented path, so a rule of the form *empty or entitled* admits every - namespace to the default class. Whether the default class is entitled is a separate - boolean. +**Selecting a pool or pinning an address is a privilege-escalation surface.** + +RBAC authorizes verbs on resources, not fields or annotations. Anyone who can create a +Service in a namespace can write `metallb.io/loadBalancerIPs:
` or name the +reserved pool with `metallb.io/address-pool`. A self-allocating backend has no concept of +a reservation and assigns the address, including one another tenant reserved but has not +attached. This applies to MetalLB and Cilium. It does not apply to cloud backends, where +IAM guards the reservation. + +The response has two layers (§8). + +**Layer 1: admission, against tenants.** The ungatable annotation becomes a reference to +an RBAC-gated object: + +1. A tenant writes only `local.sdn.cozystack.io/ip-address-claim`, naming an object in + their own namespace. RBAC gates that object. +2. The driver of the claim's class writes the backend's pool and pin inputs. The core + never does. +3. A `ValidatingAdmissionPolicy` rejects a non-allowlisted principal that writes any + backend input selecting a pool or pinning an address on a Service. + +*Identifying the driver.* The policy matches the driver's ServiceAccount by name and +namespace (`system:serviceaccount::`), configured at install. It carries an +allowlist of usernames and groups, because non-tenant principals legitimately write these +inputs: GitOps reconcilers applying content their operators authored, and system +components whose Services ship with pool annotations (an Istio installation, for example). +Without the allowlist, reconciliation breaks on the first such write. The policy ships with +`Warn` and `Audit` modes (*Upgrade and rollback compatibility*). + +*The gated set is every input the backend honours.* Deprecated spellings are still parsed, +and so may be a core field. For MetalLB the set is `metallb.io/loadBalancerIPs`, +`metallb.io/address-pool`, `metallb.io/allow-shared-ip`, the same three under the legacy +`metallb.universe.tf/` prefix, and `Service.spec.loadBalancerIP`, which MetalLB still +reads. The comparison is old value against new value, not presence, because the driver +pins the tenant's own Service and the tenant must be able to edit the rest of it. + +*Deputies.* An identity allowlist assumes the principal authored what it writes. A +privileged controller that applies user-supplied content breaks that assumption. Two +common cases: a GitOps reconciler that applies user-authored releases under its own +identity instead of impersonating the user, and a cloud-controller-manager that copies +Service annotations from a user-controlled cluster into an infrastructure Service. Such a +deputy is denied, like the users behind it, and is not allowlisted. Its Services obtain +addresses through the inputs the policy leaves writable: the claim annotation and the +auto-claim annotations of §6, which a deputy may copy from a guest Service. Where a deputy +must forward gated inputs, filtering at the deputy is a precondition of layer 1: a +strip-list for the gated keys, or impersonation instead of ambient authority in the +reconciler. Until a deputy filters, layer 1 is partial for the users behind it and layer 2 +covers them. + +**Layer 2: reconciliation, for everyone admission cannot bind.** Layer 1 is keyed on +identity. A cluster-admin can impersonate the driver's ServiceAccount, rewrite the policy, +or edit MetalLB objects. This is a trust boundary: a cluster-admin is the trust root, and +handing out a reserved address is within their authority. The design detects rather than +prevents. The ledger is reconciled continuously, so an impersonated write, a direct +backend edit, a misconfigured pool, or an assignment with no matching claim sets +`IPAddress.status.phase: Conflict` naming the Service, and the rightful claim keeps its +binding. + +Layer 1 is worth landing alone: the theft window exists in every cluster with a shared +auto-assign pool. + +**Where this component stops.** The two layers cover theft of a reservation. They do not +decide which class a namespace may draw from: a tenant who can create an `IPAddressClaim` +may name any `spec.className`. `ResourceQuota` bounds the count, not the source. +Kubernetes has the same gap for `StorageClass` and closes it with per-class quota scoping. + +Encoding a tenancy model belongs above this component. Three constraints apply to whatever +closes the gap: + +1. **The check keys on the namespace, not on the requesting principal.** Machinery that + mints claims for a tenant (community #45's attachment controller, a billing + integration) creates them under its own identity, so a requester-keyed policy never + sees the tenant. A `ValidatingAdmissionPolicy` can key on the namespace, since + `paramRef` resolves a parameter object in the request's namespace. +2. **The empty `className` is an explicit decision.** Empty means the default class, so a + rule of the form "empty or entitled" admits every namespace to the default class. + Whether the default class is entitled is a separate boolean. 3. **`spec.addressName` cannot be checked in CEL**, because resolving an address to its - class needs a second object read. A deployment that gates it either refuses the field - to tenant principals outright — the reference policy's choice, and the right one where - platform machinery is what moves specific addresses — or checks it in a webhook. - -The natural home for the data is an **optional entitlement object in the core**: per -namespace, per class, the number of claims that namespace may hold of that class, with -zero meaning not entitled and absence meaning unrestricted — the per-class quota scoping -Kubernetes applies to storage, which gives entitlement and a per-tenant ceiling in one -object. When present, the core refuses a claim that exceeds it with a condition, and the -reference admission policy enforces the same bound synchronously on the tenant-facing -path. The object's shape is **deferred** (*Open questions*); what is fixed is that it is -namespace-keyed and that the distribution, not the tenant, writes it. In Cozystack the -`Tenant` object is the obvious source and the tenant chart the obvious renderer, which is -also how a brokered deployment (*User-facing changes*) expresses "this tenant owns three -addresses of the public class". + class needs a second object read. A deployment either refuses the field to tenant + principals, which is the reference policy's choice, or checks it in a webhook. + +The data belongs in an optional entitlement object in the core: per namespace, per class, +the number of claims the namespace may hold. Zero means not entitled; absence means +unrestricted. This is the per-class quota scoping Kubernetes applies to storage, and it +gives entitlement and a per-tenant ceiling in one object. When present, the core refuses +a claim over the limit with a condition, and the reference admission policy enforces the +same bound synchronously. The object's shape is open (*Open questions*). The distribution +writes it, not the tenant. In Cozystack the `Tenant` object is the source and the tenant +chart the renderer; a brokered deployment (*User-facing changes*) uses it to express +"this tenant owns three addresses of the public class". ## Failure and edge cases - **Class range exhausted** → claim stays `Pending` with a reason; no partial binding. -- **Claim deleted, `reclaimPolicy: Retain`** → `IPAddress` goes `Released`, keeps the - address, is not reusable until an admin clears the `claimRef`. (PV semantics, deliberately.) -- **Claim deleted, `Delete`** → address returned to the range; the backend object is torn down. -- **Service references a claim in another namespace** → rejected. Cross-namespace address - sharing is not a thing. -- **Two Services reference one claim** → resolved by reconciliation, not at admission. A - `ValidatingAdmissionPolicy` cannot see the claim — CEL is given the request object and the - policy's params, not arbitrary cluster state — so admission-time rejection would need a - webhook. The rule is **the live holder keeps it**: the Service already in `associatedTo` - stays associated for as long as it exists, a later Service naming the claim is refused - with an `AssociationRejected` event, and the claim reports the refusal in an `Associated` - condition naming both; only when the holder is gone does the next one take over. The - address is never silently moved. A 1:1 binding is 1:1; letting the second win quietly is - how an address goes dead. -- **Plain Service pulls a reserved address** (explicit `address-pool`/pin, or a request - made under an impersonated controller identity) → not prevented in general; **detected**. - The `IPAddress` goes `Conflict`, the offending Service is named, and the reservation is - not silently overwritten. See §8 and *Security*. +- **Claim deleted, `Retain`** → `IPAddress` goes `Released` and keeps the address. It is + not reusable until an admin clears `claimRef` (PV semantics). +- **Claim deleted, `Delete`** → address returned to the range; the backend object is torn + down. +- **Service names a claim in another namespace** → rejected. +- **Two Services name one claim** → resolved by reconciliation. A + `ValidatingAdmissionPolicy` cannot read the claim (CEL sees the request object and the + policy's params), so admission-time rejection would need a webhook. The live holder + keeps the address: the Service in `associatedTo` stays associated while it exists, a + later Service is refused with an `AssociationRejected` event, and the claim reports the + refusal in an `Associated` condition naming both. When the holder is gone, the next + Service takes over. +- **Plain Service receives a reserved address** (pool annotation, pin, or impersonated + write) → detected, not prevented. The `IPAddress` goes `Conflict` naming the Service; + the reservation is not overwritten (§8, *Security*). - **`spec.addressName` names an address that is absent, `Bound`, `Released`, or of another - class or family** → nothing binds; the claim stays `Pending` with the ordinary waiting - reason. Naming an address filters, it never seizes. -- **A `Dual` claim associated with a single-stack Service** → the driver pins only the - family the Service carries; the other address stays `Bound` to the claim and inert. The claim is not - `Lost`, and the unattached family is never silently released. -- **Backend lacks `Pin`** → claims against that class are refused loudly, rather than - allocating an address that can never be attached. With no capability registry there is - nothing for admission to consult, so this is the provisioner refusing and reporting it on - the claim, not an admission-time rejection. -- **Address adopted from a provider, then released provider-side** → `IPAddress` goes - `Lost`. It must not silently re-allocate. -- **Controllers down while a namespace holding claims is deleted** → the claim finalizer - holds the namespace until the core returns, exactly as PVC protection does; a controller - outage never leaks an address, at the cost of a deletion that waits. -- **Class deleted while addresses of it are bound** → refused, not cascaded. The rendered - pool and advertisement are owned by the class, so letting the deletion through would - withdraw every address of the class at once; a finalizer holds the class until its - addresses are gone. -- **`Released` addresses accumulate under `Retain`** → by default an admin clears - `claimRef`, PV semantics. A class may set an optional **`releaseDelay`**: once it - elapses the controller clears `claimRef` itself and the address returns to `Available`, - so a release followed by a cooling-off period before reuse needs no operator. An - address marked held — an annotation the controller honours — is exempt, which is how an - operator quarantines an address they do not want reissued. + class or family** → nothing binds; the claim stays `Pending` with the ordinary reason. +- **`Dual` claim on a single-stack Service** → the driver pins only the family the Service + carries. The other address stays `Bound` and unattached. The claim is not `Lost`. +- **Backend lacks `Pin`** → the provisioner refuses claims of that class and reports it on + the claim. There is no capability registry for admission to consult. +- **Adopted address released provider-side** → `IPAddress` goes `Lost`. It is not + reallocated. +- **Namespace with claims deleted while the core is down** → the claim finalizer holds the + namespace until the core returns, as PVC protection does. No address leaks; the deletion + waits. +- **Class deleted with bound addresses** → refused. The class owns the rendered pool and + advertisement, so cascading the deletion would withdraw every address of the class. A + finalizer holds the class until its addresses are gone. +- **`Released` addresses accumulate under `Retain`** → an admin clears `claimRef`. A class + may set `releaseDelay`: after it elapses the controller clears `claimRef` and the address + returns to `Available`. An address marked held by an annotation is exempt, which + quarantines an address an operator does not want reissued. ## Testing -- **Unit:** the allocator (range carving, exhaustion, reclaim transitions, the source union). -- **Integration:** claim → bind → associate → *delete the Service* → assert the address is - still held → associate it to a **different** workload → assert the same address comes back. - **That single test is the proposal.** It is precisely what cannot pass today. -- **e2e (per backend):** MetalLB and Cilium, on a real cluster: an external client reaches - the workload on the reserved address, and still reaches it after the workload is - rebuilt behind the same claim. -- **Admission:** a tenant writing a raw pool-select or pin annotation, a legacy-prefixed - one, or `spec.loadBalancerIP` is rejected; a tenant - reserving an address and a second tenant attempting to pin it directly is rejected; an - allowlisted GitOps/system principal writing a pool annotation is **admitted** (the - allowlist does not over-block). +- **Unit:** the allocator: range carving, exhaustion, reclaim transitions, the source + union. +- **Integration:** claim → bind → associate → delete the Service → assert the address is + still held → associate it to a different workload → assert the same address returns. + This test is the proposal; it cannot pass today. +- **e2e, per backend:** MetalLB and Cilium on a real cluster. An external client reaches + the workload on the reserved address, and still reaches it after the workload is rebuilt + behind the same claim. +- **Admission:** a tenant writing a pool-select or pin annotation, a legacy-prefixed one, + or `spec.loadBalancerIP` is rejected; a second tenant pinning a reserved address is + rejected; an allowlisted principal writing a pool annotation is admitted. - **Reconciliation:** a Service assigned a reserved address with no authorizing claim - (e.g. created under an impersonated controller identity) drives the `IPAddress` to - `Conflict` and emits an event; the rightful claim's binding is not lost. + drives the `IPAddress` to `Conflict` with an event; the rightful claim keeps its + binding. ## Rollout -Sketch — sequencing is an open question: - -1. CRDs + core controller + the **MetalLB** provisioner + the admission policy. (The - admission policy is not phase 2. See *Security*.) All three exist, the policy as the - driver's opt-in template — see *Implementation status*. The §5 `loadBalancerClass` - check and the held-before-minted rule of §6 complete this phase. -2. **Gateway as an association target** (§5), with `Gateway.spec.addresses` fenced. This - is the consumer tenants will meet first: the accepted external-database-exposure - proposal puts a tenant's database traffic on the single address of the tenant Gateway, - so the address a tenant most wants to keep is one a Service annotation cannot reach. - Which Cilium release honours `spec.addresses` is open question 4. -3. The Cilium provisioner. Proves the abstraction is not a MetalLB adapter — **this is - the phase that either validates or falsifies the design**, and it should come early. - A second *consumer* has since exercised the association half (*Implementation - status*); a second *provisioner* is what remains. -4. A cloud provisioner (adoption path, `source.providerRef`). Proves the source union. +Sequencing is open: + +1. CRDs, core controller, the MetalLB provisioner, and the admission policy. All exist + (*Implementation status*); the policy is the driver's opt-in template. The §5 + `loadBalancerClass` check and the held-before-minted rule of §6 complete this phase. +2. Gateway as an association target (§5), with `Gateway.spec.addresses` fenced. The + accepted external-database-exposure proposal puts a tenant's database traffic on the + tenant Gateway's single address, so this is the consumer tenants meet first. Which + Cilium release honours `spec.addresses` is open question 4. +3. The Cilium provisioner. A second provisioner tests the allocation seam; a second + consumer has tested the association seam (*Implementation status*). +4. A cloud provisioner (`source.providerRef`). Tests the source union. ## Open questions -1. **Provisioners: in-tree or out-of-tree?** In-tree for three backends; CSI's history - says that stops scaling. Where is the line, and do we want the CRD contract on day one? -2. **PureLB.** Does it have *any* "request this exact IP" mechanism? If not, it cannot be a - backend, and that should be stated rather than discovered. -3. **Sharing one address across Services** (different ports — GCP and Cilium both permit - it). Today one consumer at a time holds a claim and a second reference is refused (§5). - Should that relax to several consumers on disjoint ports? It is the mirror of the - deferred several-addresses-on-one-consumer case in *Non-goals*, and the two probably - want answering together, since both are really "what is the cardinality between a claim - and the things that wear it". -4. **Gateway.** `Gateway.spec.addresses` is the natural pin for a Gateway consumer (§5), - and Cilium supports it with LB-IPAM; whether the Cilium release a platform ships honours - it, and whether the infrastructure-annotation path is needed as a fallback, is - unverified. The provisioner decides where the pin lands; which object a tenant - annotates is the part that needs settling here. -5. **The entitlement object's shape** (*Security*). Namespace-keyed, per-class, a count; - whether it is one object per namespace or one per class, who may read it, and whether - the reference admission policy consumes it directly as its `paramRef` are open. +1. **Provisioners: in-tree or out-of-tree?** In-tree suffices for three backends; CSI's + history says it stops scaling. Where is the line, and is the CRD contract needed on day + one? +2. **PureLB.** Does it have any "request this exact IP" mechanism? If not, it cannot be a + backend. +3. **Sharing one address across Services on disjoint ports.** GCP and Cilium permit it. + Today one consumer holds a claim at a time (§5). This is the mirror of the + several-addresses-on-one-consumer case in *Non-goals*; both are the cardinality + between a claim and its consumers. +4. **Gateway.** `Gateway.spec.addresses` is the natural pin for a Gateway consumer, and + Cilium supports it with LB-IPAM. Whether the shipped Cilium release honours it, and + whether the infrastructure-annotation path is needed as a fallback, is unverified. + Which object a tenant annotates needs settling here. +5. **The entitlement object's shape** (*Security*). One object per namespace or per class, + who may read it, and whether the reference admission policy consumes it as its + `paramRef`. ## Alternatives considered -**Bind the address to a Service** — the shape community #29 proposed as -`ServiceExposure`. A namespaced object naming a `serviceRef`, which allocates an address -and reports it in status. **Rejected as the -model for *this* problem:** it fuses *allocation* with *association*, so the address's -lifetime is the Service's lifetime — which is the exact thing being fixed. It cannot -express "keep this address, the workload is gone", which is the entire user request. -The **class** half of that idea is right and is kept here; the exposure half is not a -reservation and cannot be made into one. - -**Per-tenant pools.** Give each tenant an `IPAddressPool` (or a `/32` per address) and let -MetalLB do the rest. **Rejected:** it is the status quo's failure mode, it makes the admin -the allocator, it scales as O(tenants) or O(addresses) in operator-managed objects, and it -*still* provides no reservation — the address is released the moment the Service goes away. - -**Let the tenant write the backend's pin annotation directly.** **Rejected:** unauthorizable -(RBAC does not gate fields), and it hands every tenant the ability to steal any reserved -address that is not currently attached. See *Security*. - -**Reuse Cluster API's IPAM kinds.** `ipam.cluster.x-k8s.io` `IPAddressClaim`/`IPAddress` is -*literally* this pattern — namespaced claim, concrete address object, a documented -third-party provider contract — and shares even the kind names chosen here. **Rejected:** -it has only ever been wired to **Machine** addressing, never Services, and has no reclaim -policy and no association, which are the two things this proposal exists to add. The -contract is copied; the group is not. - -**Adopt every live LoadBalancer assignment into the ledger.** An earlier draft reconciled -`status.loadBalancer` on every Service into `IPAddress` objects, so the inventory would -converge on everything in use. **Rejected:** catch-all adoption of objects nobody asked for -is uncharacteristic of Kubernetes, and there is nothing to adopt — a class's pool is -`autoAssign: false`, so a Service holding a class address without a claim got it through a -pin, which is a conflict to surface, not a reservation to record; and addresses outside -every class are not the ledger's concern. The real need underneath — Services that should -draw from a class but cannot name a claim — is met by the opt-in auto-claim of §6. - -**Wait for upstream.** **Rejected:** KEP-1880 lists generalizing to an IPAM API as an -explicit non-goal, and `loadBalancerIP` was deprecated with no replacement. There is -nothing to wait for. - -**Do nothing.** **Rejected:** the capability simply does not exist today — an address -cannot be held, moved, quota'd, or enumerated — and the pin-annotation theft window in -*Security* is live in any cluster running a shared auto-assign pool. Doing nothing leaves -both the missing capability and the open security hole in place. +**Bind the address to a Service** (community #29's `ServiceExposure`): a namespaced object +naming a `serviceRef`, which allocates an address and reports it. Rejected: it fuses +allocation with association, so the address's lifetime is the Service's lifetime, which is +the problem being fixed. The class half of the idea is kept. + +**Per-tenant pools**: an `IPAddressPool` per tenant, or a `/32` per address. Rejected: the +admin becomes the allocator, the object count scales with tenants or addresses, and there +is still no reservation. + +**Let the tenant write the pin annotation.** Rejected: RBAC does not gate fields, so every +tenant could take any reserved, unattached address (*Security*). + +**Reuse Cluster API's IPAM kinds.** `ipam.cluster.x-k8s.io` has `IPAddressClaim` and +`IPAddress` with a provider contract and the same kind names. Rejected: they serve Machine +addressing only and have no reclaim policy and no association, which are the two things +this proposal adds. The contract is copied; the group is not. + +**Adopt every live LoadBalancer assignment into the ledger.** An earlier draft had the +controller create an `IPAddress` for every assigned address found on a Service. Rejected: +a class pool is `autoAssign: false`, so a Service holding a class address without a claim +got it through a pin, which is a conflict (§8), and addresses outside every class are out +of scope. Adoption of objects nobody asked for is also uncharacteristic of Kubernetes; the +PersistentVolume controller does not adopt volumes it did not provision. The need +underneath, Services that cannot name a claim, is met by auto-claim (§6, decision 0002). + +**Wait for upstream.** Rejected: KEP-1880 names an IPAM API a non-goal, and +`loadBalancerIP` was deprecated with no replacement. + +**Do nothing.** Rejected: an address cannot be held, moved, counted or listed, and the +theft window in *Security* exists in every cluster with a shared auto-assign pool. diff --git a/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md b/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md index 3b486610..a885026c 100644 --- a/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md +++ b/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md @@ -10,80 +10,69 @@ ## Context -The proposal's §3 said that for self-allocating backends — MetalLB, Cilium LB-IPAM, -kube-vip — *"our provisioner is the IPAM of record"*: nothing in those backends holds a -reservation, so the ledger would carve addresses from the class range itself and the -backend would merely be told which address to pin. +Proposal §3 said that for self-allocating backends (MetalLB, Cilium LB-IPAM, kube-vip) +"our provisioner is the IPAM of record": the ledger would carve addresses from the class +range and tell the backend which address to pin. -The reference MetalLB driver was first built that way. It walked the class CIDRs, skipped -the network and broadcast addresses and every address already in the ledger, took the -first free one, created the `IPAddress`, and relied on create-collision on the object name -as its lock +The first MetalLB driver did this. It walked the class CIDRs, skipped network and +broadcast addresses and every address in the ledger, took the first free one, created the +`IPAddress`, and used create-collision on the object name as its lock ([design §10](https://github.com/cozystack/metallb-iad/blob/main/docs/design.md#10-alternative-considered-driver-side-first-free-allocation)). -Two things were wrong with it, and both were structural rather than bugs. A ledger-only -reservation is **invisible to MetalLB**: a held-but-unattached address was free in -MetalLB's books, so the entire anti-theft posture rested on the admission policy, which the -proposal had already conceded is defeated by impersonation. And the driver was a **second -IPAM** that could disagree with MetalLB's live state: it could pick an address some plain -Service already held from an overlapping pool, and discover the collision only at pin -time. Reviewer @lexfrei had argued the same point from the other direction on the proposal -([review, 2026-07-23](https://github.com/cozystack/community/pull/35#pullrequestreview-4763639339)): -a reservation should be enforced in the allocator's own accounting, with detection as the -backstop. +This had two defects. MetalLB did not know about the reservation: a held, unattached +address was free in its accounting, and only the admission policy, which impersonation +defeats, stopped a pin from taking it. And the driver allocated independently of MetalLB, +so it could pick an address a plain Service already held from an overlapping pool and +detect the collision only at pin time. @lexfrei's review of the proposal +([2026-07-23](https://github.com/cozystack/community/pull/35#pullrequestreview-4763639339)) +asked for the same change from the other side: enforce the reservation in the allocator's +own accounting, with detection as the backstop. ## Decision The driver does not compute addresses. To reserve one it creates a selectorless -`type: LoadBalancer` **placeholder Service** in its own namespace, drawing from the class's -pool; MetalLB assigns an address to it and, the Service having no endpoints, never -announces it. The driver observes the assignment, pins the placeholder to it, and only then -records the `IPAddress`. A held address therefore exists in MetalLB's accounting for as -long as it is held, and MetalLB itself refuses to hand it to any other Service. The ledger -is the record of reservation; the backend is the allocator of record. Association is a -**handoff**: record `associatedTo`, delete the placeholder, pin the workload's Service. +`type: LoadBalancer` placeholder Service in its own namespace, drawing from the class +pool. MetalLB assigns an address to the placeholder and, since it has no endpoints, never +announces it. The driver pins the placeholder to the assigned address and then records the +`IPAddress`. A held address stays assigned in MetalLB's accounting, and MetalLB refuses to +assign it to any other Service. Association is a handoff: record `associatedTo`, delete +the placeholder, pin the workload's Service. ## Why not the alternatives - **Driver-side first-free allocation** (the first implementation, replaced in - [`56b3d95`](https://github.com/cozystack/metallb-iad/commit/56b3d95)). Lost on the two - structural facts above: a reservation MetalLB cannot see is not enforced by anything - but identity-keyed admission, and a parallel allocator can split-brain against the - backend's live state. The design remains valid for a backend with no allocation API at - all, but MetalLB is not such a backend — allocation by Service *is* its API. + [`56b3d95`](https://github.com/cozystack/metallb-iad/commit/56b3d95)). A reservation + MetalLB cannot see is enforced only by identity-keyed admission, and a parallel + allocator can diverge from the backend's live state. First-free remains valid for a + backend with no allocation API; MetalLB allocates by Service, so it is not such a + backend. - **Gapless handoff by sharing the address between placeholder and workload** (`metallb.io/allow-shared-ip`, analysed in - [`e2e608f`](https://github.com/cozystack/metallb-iad/commit/e2e608f)). Deferred, not - rejected: MetalLB's sharing rules would impose `externalTrafficPolicy: Cluster` and - port-disjointness on tenant Services. Kept as an opportunistic optimisation for Services - that happen to satisfy both. + [`e2e608f`](https://github.com/cozystack/metallb-iad/commit/e2e608f)). Deferred: + MetalLB's sharing rules require `externalTrafficPolicy: Cluster` and disjoint ports on + the tenant Service. It remains an optimisation for Services that satisfy both. ## Consequences -- A reserved-but-unattached address is safe to leave lying around, which the proposal - could not previously claim; admission and conflict detection become defence in depth - rather than the primary guarantee. Proposal §6 was rewritten around this. -- One placeholder Service, with a ClusterIP, per held address. Cheap, but not free, and - visible in the driver's namespace. +- A reserved, unattached address cannot be taken by another Service. Admission and + conflict detection become defence in depth. Proposal §6 is written around this. +- One placeholder Service, with a ClusterIP, per held address. - The handoff is not atomic. Between deleting the placeholder and pinning the workload's - Service the address is briefly unheld; the window is bounded by reconcile latency and is - the one interval where the admission policy is load-bearing. -- MetalLB assignments are sticky and `autoAssign: false` gates only *new* allocation - (observed on a live cluster, + Service the address is unheld for one reconcile interval. The admission policy covers + that window. +- MetalLB keeps an assignment until the Service is deleted or re-addressed; + `autoAssign: false` gates only new allocation (observed on a live cluster, [design §5](https://github.com/cozystack/metallb-iad/blob/main/docs/design.md#5-association-the-gap-handoff)). - So withdrawing the pin from a live Service does not take the address away from it: the - ex-holder keeps announcing, the re-created placeholder sits `Pending`, and the ledger - reports `Conflict` naming the ex-holder until that Service is deleted or re-addressed. - Detaching an address from a live Service means recreating the Service; the proposal's - §5 now says so instead of claiming the announcement is withdrawn. -- The `IPAddress` is created after allocation is a fact, which is the allocate–observe–record - flow a cloud driver would use with a provider-side reservation; the placeholder plays - that object's role. This is why the proposal's §3 no longer says the provisioner is the - IPAM of record for self-allocating backends. + Withdrawing the pin from a running Service does not take the address away: the Service + keeps announcing, the recreated placeholder stays `Pending`, and the ledger reports + `Conflict` naming the Service until it is deleted. Proposal §5 states this. +- The `IPAddress` is created after MetalLB has assigned the address. This is the + allocate–observe–record flow a cloud driver uses with a provider-side reservation; the + placeholder plays that object's role. Proposal §3 no longer calls the provisioner the + IPAM of record. ## Revisit if -MetalLB or Cilium LB-IPAM gains a first-class reservation object that can be held without -a Service, or the shared-handoff optimisation proves safe enough to make the default. A -backend with no allocation-by-Service API at all would reopen first-free allocation for -*that* driver only. +MetalLB or Cilium LB-IPAM gains a reservation object that can be held without a Service, +or shared handoff proves safe as the default. A backend with no allocation-by-Service API +would reopen first-free allocation for that driver only. diff --git a/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md b/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md index 0cf638f7..d575151f 100644 --- a/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md +++ b/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md @@ -10,86 +10,76 @@ ## Context -The proposal's §6 originally had two halves. Holders (record 0001) were one. The other said -*"the inventory still works backwards"*: because an eager allocator assigns before any claim -exists, the controller would also observe `status.loadBalancer` on every Service and -reconcile what it found into `IPAddress` objects after the fact, so that the ledger would -converge on everything in use. It was introduced in the first revision in answer to a -production report that a shared auto-assign pool was the normal deployment -([comment, 2026-07-15](https://github.com/cozystack/community/pull/35#issuecomment-4977850733) -and the [reply](https://github.com/cozystack/community/pull/35#issuecomment-4984507704)). +The first revision of §6 had the controller create an `IPAddress` for every assigned +address it found on a LoadBalancer Service, to inventory addresses MetalLB assigned before +a claim existed. It was added in response to a production report that a shared +auto-assign pool was the normal deployment +([comment, 2026-07-15](https://github.com/cozystack/community/pull/35#issuecomment-4977850733), +[reply](https://github.com/cozystack/community/pull/35#issuecomment-4984507704)). +Placeholder holding, the other half of §6, is record 0001. -Neither controller ever implemented it, and when the reason for it was examined it was not -there. With a class's pool at `autoAssign: false`, a plain Service cannot draw a class -address *except* by naming the pool or pinning the address — which is a conflict to -surface, not an allocation to record — and addresses from pools no class manages were never -the ledger's concern. Catch-all adoption of objects nobody asked for is also uncharacteristic -of Kubernetes: the PersistentVolume controller does not adopt volumes it did not provision. +Neither controller implemented it, and nothing needs it. A class pool is +`autoAssign: false`, so a plain Service gets a class address only by naming the pool or +pinning the address, which §8 already reports as a conflict. Other pools are outside the +ledger's scope. Adoption of objects nobody asked for is also uncharacteristic of +Kubernetes; the PersistentVolume controller does not adopt volumes it did not provision. -What the implementation *did* find was the opposite problem. The packaging work for tenant -Kubernetes clusters showed that cloud-provider-kubevirt creates a host-side -`type: LoadBalancer` Service for every LoadBalancer Service in a guest, copying the guest's -annotations once at creation and nothing afterwards +The implementation found the opposite problem. cloud-provider-kubevirt creates a host-side +`type: LoadBalancer` Service for every LoadBalancer Service in a tenant Kubernetes cluster +and copies the guest's annotations once, at creation ([cozystack/cozystack#3382 review](https://github.com/cozystack/cozystack/pull/3382#pullrequestreview-5503155946), [address-controller#7](https://github.com/cozystack/address-controller/pull/7)). Under -`autoAssign: false` such Services sat `Pending` with *no available IPs*. These are Services -that *should* draw from a class but cannot name a claim, because nothing in their render -path has heard of claims. +`autoAssign: false` these Services stayed `Pending` with "no available IPs". They should +draw from a class and cannot name a claim, because nothing in their render path knows +about claims. ## Decision -The ledger records only what was reserved through it; it adopts nothing. In its place the -core offers an **opt-in auto-claim** path, enabled per class by an annotation on the -`IPAddressClass` and inert everywhere otherwise: for a live LoadBalancer Service with no -claim annotation, no `loadBalancerClass`, no pin and no assigned address, the core mints an -`IPAddressClaim` of the same name in the same namespace, **owned by the Service**, and -writes the association annotation onto the Service; the driver then associates as if a -tenant had written it. A Service may name a class, or opt out, through annotations the core -defines — the ones a deputy such as a cloud-controller-manager may safely copy. A claim with -a controller owner follows its owner rather than the class reclaim policy: when the Service -goes, its claim goes and its addresses are deleted even under `Retain`. Removing the owner -reference turns it into an ordinary held claim. +The ledger records only addresses reserved through it. The core has an opt-in auto-claim +path, enabled per class by an annotation on the `IPAddressClass`: for a live LoadBalancer +Service with no claim annotation, no `loadBalancerClass`, no pin and no assigned address, +the core creates an `IPAddressClaim` with the Service's name in the Service's namespace, +owned by the Service, and writes the association annotation on the Service. The driver +then associates as usual. A Service can name a class or opt out through annotations the +core defines, which a deputy such as a cloud-controller-manager may copy. A claim with a +controller owner follows its owner, not the class reclaim policy: when the Service is +deleted, its claim and addresses are deleted even under `Retain`. Removing the owner +reference makes it an ordinary held claim. ## Why not the alternatives -- **Reconcile every live assignment into the ledger** (the proposal's own §6 text until - this revision). Lost because there is nothing to adopt once class pools are - `autoAssign: false`, and because an inventory that adopts foreign assignments would - record addresses nobody reserved, with no claim to attribute them to. Recorded in the - proposal's *Alternatives considered*. -- **Require every guest cluster to name a claim.** Lost on the deputy's behaviour: the - cloud-controller-manager copies annotations at creation only, so a guest would have to - know about claims before creating its Service, and every Kubernetes-app tenant would - need to be taught the contract. Auto-claim keeps the guest ignorant and still gives the - Service a ledgered address. -- **Let auto-claimed addresses follow the class reclaim policy.** Lost because nobody asked - to keep them: under `Retain` every deleted Service would leave a `Released` address - behind, which is a leak, not a reservation. The same concern was raised independently - against machinery-minted claims on community #45 +- **Create an `IPAddress` for every live assignment** (the proposal's own §6 until this + revision). There is nothing to adopt under `autoAssign: false`, and an adopted address + has no claim to attribute it to. Recorded in *Alternatives considered*. +- **Require every guest cluster to name a claim.** The cloud-controller-manager copies + annotations at creation only, so every guest would need to know the contract before + creating a Service. Auto-claim gives the Service a ledgered address without that. +- **Let auto-claimed addresses follow the class reclaim policy.** Under `Retain` every + deleted Service would leave a `Released` address. The same objection was raised against + machinery-minted claims on community #45 ([review, 2026-10-12](https://github.com/cozystack/community/pull/45#pullrequestreview-5491027438)), - which is why the rule is stated for any controller owner, not only Services. + so the rule applies to any controller owner. ## Consequences -- The core now touches Services on this one path; the proposal's claim that it never does +- The core writes to Services on this path. The proposal's statement that it never does is withdrawn. -- Auto-claim is independent of the default class. The two decisions have different - owners, and a class marked for auto-claim must be deployed after drivers have opted - their own placeholder Services out - ([cozystack/metallb-iad#5](https://github.com/cozystack/metallb-iad/pull/5)), or the core - will claim the driver's holders. -- A deputy that forwards user content no longer needs to be allowlisted by the admission - fence: its Services reach the ledger through the claim-reference and class-selection - annotations, which the fence leaves writable. -- The brokered operating mode gains a natural attach surface: with the held-before-minted - rule (specified, not yet implemented), a plain LoadBalancer Service in a namespace that - already holds an unattached claim wears that address rather than minting a new one, and - a second Service with no spare claim stays `Pending`. -- Deputies copy annotations at creation only, so class selection and opt-out must be set on - the guest Service before it exists; adding them later has no effect. +- The auto-claim class is independent of the default class. Mark a class only after + drivers have opted their placeholder Services out + ([cozystack/metallb-iad#5](https://github.com/cozystack/metallb-iad/pull/5)); otherwise + the core claims the driver's holders. +- A deputy that forwards user content does not need to be allowlisted by the admission + policy. Its Services reach the ledger through the claim-reference and class-selection + annotations, which the policy leaves writable. +- With the held-before-minted rule (specified, not implemented), a plain LoadBalancer + Service in a namespace that holds an unattached claim wears that address, and a second + Service with no spare claim stays `Pending`. This is the attach surface of the brokered + mode. +- Deputies copy annotations at creation only, so class selection and opt-out must be on + the guest Service before it is created. ## Revisit if -cloud-provider-kubevirt gains an annotation filter or a configurable claim contract, which -would let guest clusters name claims directly; or a second deputy appears whose behaviour -the annotation-copy assumption does not fit. +cloud-provider-kubevirt gains an annotation filter or a configurable claim contract, so +guest clusters can name claims directly; or a second deputy appears whose behaviour the +annotation-copy assumption does not fit. From ee24fb54240a72402904a50ea4641d280f773de6 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Tue, 3 Nov 2026 10:39:28 +0100 Subject: [PATCH 13/16] design-proposal(ip-address-claims): three corrections from review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Name cloud-provider-kubevirt where the text described its behaviour as that of deputies in general: it copies guest Service annotations once, at creation, and that constraint is its own, not a property of every controller that forwards user content. State in §7 what the MetalLB driver does with the class's address list: it renders one IPAddressPool from the CIDRs and does not adopt an existing pool, and MetalLB rejects overlapping pools. Fix the order of operations in §5: the driver reads the annotation, looks up the claim in the Service's namespace, then resolves it. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 27 ++++++++++--------- ...t-the-core-claims-on-behalf-of-services.md | 7 ++--- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index 8291f6b2..c41bc92d 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -404,9 +404,9 @@ metadata: spec: {type: LoadBalancer, ...} ``` -The driver resolves the claim to its address, checks that the claim is in the Service's -namespace, and writes the backend's pin annotation (§1.2). The backend then allocates and -announces that address. +The driver reads the annotation, looks up a claim of that name in the Service's own +namespace, resolves it to its address, and writes the backend's pin annotation (§1.2). The +backend then allocates and announces that address. **The class determines the `loadBalancerClass`.** A class names a pool and a provisioner, which determines the LB implementation that announces its addresses, so a class implies @@ -461,16 +461,16 @@ earlier design. **Auto-claim.** Some Services should draw from a class but cannot name a claim, because the component that renders them does not know about claims. The case that forced this is -a cloud-controller-manager for tenant Kubernetes clusters: it creates a host-side -`type: LoadBalancer` Service for each LoadBalancer Service in the guest and copies the -guest's annotations once, at creation. Under `autoAssign: false` these Services stay +cloud-provider-kubevirt, the cloud-controller-manager of tenant Kubernetes clusters: it +creates a host-side `type: LoadBalancer` Service for each LoadBalancer Service in the +guest and copies the guest's annotations once, at creation. Under `autoAssign: false` these Services stay `Pending`. The core therefore has an opt-in path, enabled per class by an annotation on the `IPAddressClass`: for a LoadBalancer Service with no claim annotation, no `loadBalancerClass`, no pin and no assigned address, the core creates an `IPAddressClaim` with the Service's name in the Service's namespace, owned by the Service, and writes the §5 annotation on the Service. The driver associates as usual. A Service can name a class -or opt out through annotations the core defines; a deputy may copy these from a guest -Service. The auto-claim class is independent of the default class. +or opt out through annotations the core defines; cloud-provider-kubevirt copies these +from the guest Service. The auto-claim class is independent of the default class. Two rules follow: @@ -504,7 +504,10 @@ For MetalLB, once per cluster: - one `IPAddressPool` for the reserved range, with `autoAssign: false`; - one `L2Advertisement` or `BGPAdvertisement` selecting it. -The `IPAddressClass` carries the range and the provisioner renders both objects. +The `IPAddressClass` carries the range and the provisioner renders both objects. For the +MetalLB driver, `parameters.addresses` lists the CIDRs of the pool it renders. The driver +does not adopt an existing `IPAddressPool`, and MetalLB rejects overlapping pools, so a +range moved into a class must leave its old pool first. `autoAssign: false` stops MetalLB from assigning a reserved address automatically. It does not stop a Service from naming the pool (`metallb.io/address-pool`) or pinning an address (`metallb.io/loadBalancerIPs`); §8 and *Security* cover that. @@ -635,11 +638,11 @@ pins the tenant's own Service and the tenant must be able to edit the rest of it *Deputies.* An identity allowlist assumes the principal authored what it writes. A privileged controller that applies user-supplied content breaks that assumption. Two common cases: a GitOps reconciler that applies user-authored releases under its own -identity instead of impersonating the user, and a cloud-controller-manager that copies -Service annotations from a user-controlled cluster into an infrastructure Service. Such a +identity instead of impersonating the user, and cloud-provider-kubevirt, which copies +Service annotations from the tenant cluster into the host Service it creates. Such a deputy is denied, like the users behind it, and is not allowlisted. Its Services obtain addresses through the inputs the policy leaves writable: the claim annotation and the -auto-claim annotations of §6, which a deputy may copy from a guest Service. Where a deputy +auto-claim annotations of §6, which cloud-provider-kubevirt copies from the guest Service. Where a deputy must forward gated inputs, filtering at the deputy is a precondition of layer 1: a strip-list for the gated keys, or impersonation instead of ambient authority in the reconciler. Until a deputy filters, layer 1 is partial for the users behind it and layer 2 diff --git a/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md b/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md index d575151f..6c205a8c 100644 --- a/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md +++ b/design-proposals/ip-address-claims/decisions/0002-the-ledger-does-not-adopt-the-core-claims-on-behalf-of-services.md @@ -41,7 +41,7 @@ Service with no claim annotation, no `loadBalancerClass`, no pin and no assigned the core creates an `IPAddressClaim` with the Service's name in the Service's namespace, owned by the Service, and writes the association annotation on the Service. The driver then associates as usual. A Service can name a class or opt out through annotations the -core defines, which a deputy such as a cloud-controller-manager may copy. A claim with a +core defines, which cloud-provider-kubevirt copies from the guest Service. A claim with a controller owner follows its owner, not the class reclaim policy: when the Service is deleted, its claim and addresses are deleted even under `Retain`. Removing the owner reference makes it an ordinary held claim. @@ -75,8 +75,9 @@ reference makes it an ordinary held claim. Service in a namespace that holds an unattached claim wears that address, and a second Service with no spare claim stays `Pending`. This is the attach surface of the brokered mode. -- Deputies copy annotations at creation only, so class selection and opt-out must be on - the guest Service before it is created. +- cloud-provider-kubevirt copies annotations at creation only, so class selection and + opt-out must be on the guest Service before it is created. Syncing annotations on update + in cloud-provider-kubevirt would remove this constraint. ## Revisit if From a07dd3ee510d7183e4e66996c5b5062f8fb96ecf Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Tue, 3 Nov 2026 10:46:45 +0100 Subject: [PATCH 14/16] design-proposal(ip-address-claims): require serviceability before the handoff; correct the policy's status MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the architectural review of 2026-11-02. The driver releases the hold on the annotation alone, so a Service that MetalLB will not serve ends up with the address unheld, announced nowhere, and reported as attached. §5 now requires the driver to check type and loadBalancerClass and refuse with an event before it sets associatedTo and deletes the holder. A failure case states the outcome. The admission policy is in an open PR, not in the driver's chart, so Implementation status and Rollout say layer 1 is absent on main. Auto-claim skips any Service carrying a backend pool or pin input, which excludes the driver's placeholders structurally. Duplicate default or auto-claim classes are a stated runtime condition. The handoff window lasts until the first successful pin, not one reconcile interval, in §6 and in record 0001. Record 0001 names the ClusterIP-per-placeholder ceiling. A Lost address keeps its claimRef until an admin acts. One sentence on BGP: reservations do not change route-table growth. Community #99 is cited for the per-class quota shape. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 55 +++++++++++++++---- ...ackend-allocates-and-the-ledger-records.md | 9 +-- 2 files changed, 48 insertions(+), 16 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index c41bc92d..20c1301f 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -121,9 +121,10 @@ Phase 1 of *Rollout* exists under the Cozystack organisation: (cozystack/address-controller#7). - **[metallb-iad](https://github.com/cozystack/metallb-iad)**: the reference driver for MetalLB. It holds a reservation as a placeholder Service in its own namespace, so - MetalLB's accounting refuses to assign a held address elsewhere. - cozystack/metallb-iad#6 ships the admission policy of *Security* as an opt-in chart - template; cozystack/metallb-iad#5 opts the placeholders out of auto-claim. + MetalLB's accounting refuses to assign a held address elsewhere. On `main` the chart + has no admission policy, so layer 1 of *Security* is absent as shipped. + cozystack/metallb-iad#6, open, adds the policy as an opt-in chart template; + cozystack/metallb-iad#5, open, opts the placeholders out of auto-claim. - **cozystack/cozystack#3382** packages both as optional system packages, with tenant RBAC on claims and a distribution-specific admission policy. The driver's policy is the reference; a distribution may disable it and ship its own. @@ -138,7 +139,8 @@ cozyplane, an address survived delete-and-rebind and stayed reachable. This tests the association seam only. The allocation seam, a second provisioner, is untested (*Rollout*). -**Specified here and not yet implemented:** the `loadBalancerClass` check of §5; pinning +**Specified here and not yet implemented:** the `loadBalancerClass` check and the +serviceability check of §5; the pool-or-pin exclusion of §6; pinning only the families a Service carries (§2); conditions on `IPAddress` (§2) and the `Associated` condition on a claim (§5); per-class conflict adjudication (§8); the held-before-minted rule of §6; the release delay and the class-deletion guard (*Failure @@ -422,6 +424,14 @@ Service. A mismatch is a refused association naming both values. Whoever creates Service sets the field at creation; a controller that creates the Service and names the claim reads the class first. +**Serviceability before handoff.** The driver checks that it can serve the consumer +before it releases the hold: the Service is `type: LoadBalancer`, its `loadBalancerClass` +matches the class, and the backend will act on the pin. A consumer that fails the check +is refused with an event, `associatedTo` stays unset, and the holder keeps the address. +The order matters. Setting `associatedTo` disarms the hold (§6), so a consumer recorded +before it is checked leaves the address unheld, announced nowhere, and reported as +attached. Specified, not implemented. + **Detaching.** When the annotation is removed, the driver withdraws its pin, clears `associatedTo`, and holds the address again. The `IPAddress` stays `Bound` to its claim: reserved and attached to nothing, like an unassociated EIP. MetalLB keeps an assignment @@ -470,7 +480,11 @@ the `IPAddressClass`: for a LoadBalancer Service with no claim annotation, no with the Service's name in the Service's namespace, owned by the Service, and writes the §5 annotation on the Service. The driver associates as usual. A Service can name a class or opt out through annotations the core defines; cloud-provider-kubevirt copies these -from the guest Service. The auto-claim class is independent of the default class. +from the guest Service. The core also skips any Service that carries a backend pool or pin +input (*Security* lists them): such a Service has chosen its address source. The driver's +placeholders carry the pool annotation, so this rule excludes them without the opt-out +annotation and without knowledge of the driver's namespace. The auto-claim class is +independent of the default class. Two rules follow: @@ -492,7 +506,9 @@ disagree with `status.loadBalancer` briefly. The design accepts eventual consist Two gaps remain with holders in place: - **The handoff.** Releasing the holder and pinning the workload's Service is not atomic. - During that window the address is unheld. Layer 1 of *Security* covers it. + The address is unheld until the first successful pin; `associatedTo` is already set, so + a failed pin or a driver restart does not re-arm the hold. Layer 1 of *Security* covers + the window. - **Annotation writers.** With the pool at `autoAssign: false`, a plain Service receives a reserved address only by naming the pool or pinning the address. Layer 1 gates that write; layer 2 detects it. @@ -512,6 +528,10 @@ range moved into a class must leave its old pool first. not stop a Service from naming the pool (`metallb.io/address-pool`) or pinning an address (`metallb.io/loadBalancerIPs`); §8 and *Security* cover that. +Under BGP, MetalLB announces each assigned address as a host route whether or not it is +reserved, so scattered reservations within a range do not change route-table growth. +`BGPAdvertisement.aggregationLength` summarises as it does without this proposal. + ### 8. How this rides real allocators No pool layout guarantees a reservation. A Service can name the reserved pool or pin an @@ -682,8 +702,9 @@ closes the gap: The data belongs in an optional entitlement object in the core: per namespace, per class, the number of claims the namespace may hold. Zero means not entitled; absence means -unrestricted. This is the per-class quota scoping Kubernetes applies to storage, and it -gives entitlement and a per-tenant ceiling in one object. When present, the core refuses +unrestricted. This is the per-class quota scoping Kubernetes applies to storage, and the +shape community #99 uses for GPU budgets; it gives entitlement and a per-tenant ceiling +in one object. When present, the core refuses a claim over the limit with a condition, and the reference admission policy enforces the same bound synchronously. The object's shape is open (*Open questions*). The distribution writes it, not the tenant. In Cozystack the `Tenant` object is the source and the tenant @@ -715,7 +736,16 @@ chart the renderer; a brokered deployment (*User-facing changes*) uses it to exp - **Backend lacks `Pin`** → the provisioner refuses claims of that class and reports it on the claim. There is no capability registry for admission to consult. - **Adopted address released provider-side** → `IPAddress` goes `Lost`. It is not - reallocated. + reallocated. The claim reports `Lost`, and the address keeps its `claimRef` and its + entry in `status.addresses` until an admin deletes the `IPAddress` or restores the + provider-side reservation; the claim then rebinds. +- **Service names a claim but the driver cannot serve it** (not `type: LoadBalancer`, or a + `loadBalancerClass` the backend ignores) → refused with an event before the hold is + released (§5). The address stays held. +- **Two default classes, or two auto-claim classes** → claims with an empty `className` + stay `Pending` with `MultipleDefaultClasses`; auto-claim does nothing and emits a + Warning event on the Service. A CRD schema cannot validate uniqueness across objects; + Kubernetes treats duplicate default `StorageClass`es the same way. - **Namespace with claims deleted while the core is down** → the claim finalizer holds the namespace until the core returns, as PVC protection does. No address leaks; the deletion waits. @@ -748,9 +778,10 @@ chart the renderer; a brokered deployment (*User-facing changes*) uses it to exp Sequencing is open: -1. CRDs, core controller, the MetalLB provisioner, and the admission policy. All exist - (*Implementation status*); the policy is the driver's opt-in template. The §5 - `loadBalancerClass` check and the held-before-minted rule of §6 complete this phase. +1. CRDs, core controller, the MetalLB provisioner, and the admission policy. The + controllers exist; the policy is in an open PR (*Implementation status*). The §5 + `loadBalancerClass` and serviceability checks and the held-before-minted rule of §6 + complete this phase. 2. Gateway as an association target (§5), with `Gateway.spec.addresses` fenced. The accepted external-database-exposure proposal puts a tenant's database traffic on the tenant Gateway's single address, so this is the consumer tenants meet first. Which diff --git a/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md b/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md index a885026c..a9c2fda6 100644 --- a/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md +++ b/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md @@ -56,10 +56,11 @@ the placeholder, pin the workload's Service. - A reserved, unattached address cannot be taken by another Service. Admission and conflict detection become defence in depth. Proposal §6 is written around this. -- One placeholder Service, with a ClusterIP, per held address. -- The handoff is not atomic. Between deleting the placeholder and pinning the workload's - Service the address is unheld for one reconcile interval. The admission policy covers - that window. +- One placeholder Service per held address, each with a ClusterIP. The number of + reserved addresses is therefore bounded by the Service CIDR, not by the external pool. +- The handoff is not atomic. Between deleting the placeholder and the first successful + pin the address is unheld, and because `associatedTo` is already set, a failed pin or a + driver restart does not re-arm the hold. The admission policy covers that window. - MetalLB keeps an assignment until the Service is deleted or re-addressed; `autoAssign: false` gates only new allocation (observed on a live cluster, [design §5](https://github.com/cozystack/metallb-iad/blob/main/docs/design.md#5-association-the-gap-handoff)). From 0129b2df88d094bfa30b382a9e0a78a4d5b4d303 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Tue, 3 Nov 2026 14:16:23 +0100 Subject: [PATCH 15/16] design-proposal(ip-address-claims): state what phase 1 and the provisioner contract are today MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the re-review of 2026-11-03. The banner said phase 1 is implemented; by Rollout's own definition it is not, since the policy is in an open PR and two §5 and §6 rules are unimplemented. It now says the controllers exist and are pending review. §4 described capabilities as declared and enforced by the core. Nothing declares them: the contract as implemented is the provisioner name stamped onto claims and the source union on the address, and a driver that cannot serve a claim leaves it Pending. The table is now the intended shape, with the refusal the driver's. An auto-claimed Service has no loadBalancerClass and the core cannot set one after creation, so an auto-claim class must have an empty loadBalancerClass; the core ignores the annotation otherwise. PureLB pins through the deprecated spec.loadBalancerIP field, so it is a backend after all; the §1.2 table says so and the open question is dropped. The status lists no longer contradict the text on the Lost claim phase and the claim's conditions. Record 0001 says lexfrei's review came three days after the change and made the same argument independently. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 53 ++++++++++++------- ...ackend-allocates-and-the-ledger-records.md | 7 +-- 2 files changed, 38 insertions(+), 22 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index 20c1301f..b5a008fc 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -4,8 +4,9 @@ - **Author(s):** `@lllamnyp` > The object model, the problem statement and the survey of existing primitives are -> settled, and phase 1 of *Rollout* is implemented. *Implementation status* lists where -> the text is ahead of the code. Open: the provisioner contract's own API (capabilities +> settled. The controllers of phase 1 of *Rollout* exist and are pending review; +> *Implementation status* lists what phase 1 still lacks and where the text is ahead of +> the code. Open: the provisioner contract's own API (capabilities > are not modelled), the entitlement object of *Security*, sequencing after phase 1, and > *Open questions*. @@ -113,7 +114,8 @@ considered*). ## Implementation status -Phase 1 of *Rollout* exists under the Cozystack organisation: +The controllers of phase 1 of *Rollout* exist under the Cozystack organisation, pending +review: - **[address-controller](https://github.com/cozystack/address-controller)**: the core. The three kinds, claim–address binding, status, reclaim, and the driver contract. It @@ -140,7 +142,8 @@ This tests the association seam only. The allocation seam, a second provisioner, untested (*Rollout*). **Specified here and not yet implemented:** the `loadBalancerClass` check and the -serviceability check of §5; the pool-or-pin exclusion of §6; pinning +serviceability check of §5; the pool-or-pin exclusion and the auto-claim class +constraint of §6; pinning only the families a Service carries (§2); conditions on `IPAddress` (§2) and the `Associated` condition on a claim (§5); per-class conflict adjudication (§8); the held-before-minted rule of §6; the release delay and the class-deletion guard (*Failure @@ -148,8 +151,8 @@ and edge cases*); Gateway as an association target (§5); the entitlement object (*Security*). **In the code and not in this text:** `claimRef.uid`, `status.className` on the claim, -claim conditions, a `Lost` claim phase, a `Pending` address phase, the provisioner stamp -annotation, the finalizers. The implementation's design document is the reference for +the claim's `conditions` field with `ClassResolved` and `Bound`, a `Pending` address +phase, the provisioner stamp annotation, the finalizers. The implementation's design document is the reference for these. None changes the model. ## Context @@ -240,9 +243,9 @@ Five findings shape the design: is the IP, with a generic `spec.parentRef`) and named "any generalization onto something like an IPAM API" a non-goal. Nothing upstream will provide this. -2. **Every backend pins a specific address through an annotation.** - `Service.spec.loadBalancerIP` was deprecated in Kubernetes 1.24 with no replacement, so - each implementation defined its own: +2. **Every backend can pin a specific address.** `Service.spec.loadBalancerIP` was + deprecated in Kubernetes 1.24 with no replacement, so each implementation defined its + own annotation, and one still reads the field: | backend | pin mechanism | |---|---| @@ -252,7 +255,7 @@ Five findings shape the design: | OpenELB | `eip.openelb.kubesphere.io/v1alpha1` | | AWS LB Controller | `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` | | GCP / GKE | `networking.gke.io/load-balancer-ip-addresses` | - | PureLB | **none found** — see *Open questions* | + | PureLB | `spec.loadBalancerIP`, the deprecated core field; no annotation ([purelb#132](https://gitlab.com/purelb/purelb/-/issues/132)) | The semantics are identical. This is the seam the design rests on: a backend that can pin can bind a reserved address to a workload. @@ -299,7 +302,7 @@ spec: family: IPv4 # IPv4 | IPv6 | Dual # addressName: ip-203-0-113-7 # optional: bind this specific address (see below) status: - phase: Bound + phase: Bound # Pending | Bound | Lost addresses: # a list, so a Dual claim can report v4 + v6 - name: ip-203-0-113-7 # the IPAddress object address: 203.0.113.7 # what the tenant reads, and puts in DNS @@ -377,7 +380,14 @@ is not ours. The claim does not expose the difference. ### 4. The provisioner contract `IPAddressClass.spec.provisioner` names a controller, as `StorageClass` names a CSI driver. -Each provisioner declares capabilities, and the core refuses what a backend cannot do: +The contract as implemented is two things. The core stamps the provisioner name onto each +claim as an annotation, and a driver selects its claims by it. The driver fills the +`IPAddress` source union with `fromClass` or `providerRef`. Nothing declares capabilities; +the core has no registry to consult, and a driver that cannot serve a claim leaves it +`Pending` and reports why on the claim. + +The intended shape is a declaration per provisioner, with the driver refusing what it +cannot do: | capability | meaning | consequence if absent | |---|---|---| @@ -387,7 +397,8 @@ Each provisioner declares capabilities, and the core refuses what a backend cann `Pin` is the capability the model depends on. A backend that cannot be told which address to use cannot attach a reserved one, so a class over such a backend rejects claims instead -of allocating an address it cannot bind. PureLB may be this case (*Open questions*). +of allocating an address it cannot bind. No surveyed backend lacks it (§1.2). The +declaration is deferred with the registration object it would live on (*Open questions*). Whether provisioners are in-tree (a switch in one controller) or out-of-tree (separate deployments with a CRD contract, as CSI did) is open. In-tree suffices for three @@ -486,6 +497,12 @@ placeholders carry the pool annotation, so this rule excludes them without the o annotation and without knowledge of the driver's namespace. The auto-claim class is independent of the default class. +An auto-claimed Service has no `loadBalancerClass`, and the core does not set one: the +field is immutable and the Service already exists. The Service is therefore served by the +implementation that handles unclassed Services, the cluster default (§1.5), and an +auto-claim class must have an empty `loadBalancerClass`. The core ignores the auto-claim +annotation on a class with a non-empty value and reports it with an event. + Two rules follow: - **A claim with a controller owner follows its owner, not the class reclaim policy.** @@ -785,7 +802,7 @@ Sequencing is open: 2. Gateway as an association target (§5), with `Gateway.spec.addresses` fenced. The accepted external-database-exposure proposal puts a tenant's database traffic on the tenant Gateway's single address, so this is the consumer tenants meet first. Which - Cilium release honours `spec.addresses` is open question 4. + Cilium release honours `spec.addresses` is open question 3. 3. The Cilium provisioner. A second provisioner tests the allocation seam; a second consumer has tested the association seam (*Implementation status*). 4. A cloud provisioner (`source.providerRef`). Tests the source union. @@ -795,17 +812,15 @@ Sequencing is open: 1. **Provisioners: in-tree or out-of-tree?** In-tree suffices for three backends; CSI's history says it stops scaling. Where is the line, and is the CRD contract needed on day one? -2. **PureLB.** Does it have any "request this exact IP" mechanism? If not, it cannot be a - backend. -3. **Sharing one address across Services on disjoint ports.** GCP and Cilium permit it. +2. **Sharing one address across Services on disjoint ports.** GCP and Cilium permit it. Today one consumer holds a claim at a time (§5). This is the mirror of the several-addresses-on-one-consumer case in *Non-goals*; both are the cardinality between a claim and its consumers. -4. **Gateway.** `Gateway.spec.addresses` is the natural pin for a Gateway consumer, and +3. **Gateway.** `Gateway.spec.addresses` is the natural pin for a Gateway consumer, and Cilium supports it with LB-IPAM. Whether the shipped Cilium release honours it, and whether the infrastructure-annotation path is needed as a fallback, is unverified. Which object a tenant annotates needs settling here. -5. **The entitlement object's shape** (*Security*). One object per namespace or per class, +4. **The entitlement object's shape** (*Security*). One object per namespace or per class, who may read it, and whether the reference admission policy consumes it as its `paramRef`. diff --git a/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md b/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md index a9c2fda6..bf1a4197 100644 --- a/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md +++ b/design-proposals/ip-address-claims/decisions/0001-the-backend-allocates-and-the-ledger-records.md @@ -23,10 +23,11 @@ This had two defects. MetalLB did not know about the reservation: a held, unatta address was free in its accounting, and only the admission policy, which impersonation defeats, stopped a pin from taking it. And the driver allocated independently of MetalLB, so it could pick an address a plain Service already held from an overlapping pool and -detect the collision only at pin time. @lexfrei's review of the proposal +detect the collision only at pin time. Three days after the change, @lexfrei's review of +the proposal ([2026-07-23](https://github.com/cozystack/community/pull/35#pullrequestreview-4763639339)) -asked for the same change from the other side: enforce the reservation in the allocator's -own accounting, with detection as the backstop. +made the same argument independently: enforce the reservation in the allocator's own +accounting, with detection as the backstop. ## Decision From 759dd6c2539acdd821010fc9da3d7c28d9593935 Mon Sep 17 00:00:00 2001 From: Timofei Larkin Date: Tue, 3 Nov 2026 14:21:42 +0100 Subject: [PATCH 16/16] design-proposal(ip-address-claims): auto-claim into a non-default implementation is deferred An auto-claimed Service has no loadBalancerClass and MetalLB reads no annotation in its place, so the default implementation serves it. State that as the field's semantics rather than a rule of this design, and defer auto-claim into a class of another implementation to a mutating admission webhook that sets the field at creation. Signed-off-by: Timofei Larkin Assisted-by: LLM --- design-proposals/ip-address-claims/README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/design-proposals/ip-address-claims/README.md b/design-proposals/ip-address-claims/README.md index b5a008fc..5cdc9dd2 100644 --- a/design-proposals/ip-address-claims/README.md +++ b/design-proposals/ip-address-claims/README.md @@ -497,11 +497,15 @@ placeholders carry the pool annotation, so this rule excludes them without the o annotation and without knowledge of the driver's namespace. The auto-claim class is independent of the default class. -An auto-claimed Service has no `loadBalancerClass`, and the core does not set one: the -field is immutable and the Service already exists. The Service is therefore served by the -implementation that handles unclassed Services, the cluster default (§1.5), and an -auto-claim class must have an empty `loadBalancerClass`. The core ignores the auto-claim -annotation on a class with a non-empty value and reports it with an event. +An auto-claimed Service has no `loadBalancerClass`, so the cluster's default +implementation serves it (§1.5). An auto-claim class is therefore a class of the default +implementation, with an empty `loadBalancerClass`. This is the semantics of the field, +not a rule of this design: the field is immutable once the Service is +`type: LoadBalancer`, and MetalLB reads no annotation in its place, so no component can +route an unclassed Service to another implementation after creation. The core reports an +event on a class marked for auto-claim whose `loadBalancerClass` is non-empty, since no +Service it mints for can be served. Auto-claim into such a class needs the field set at +creation, which a mutating admission webhook could do from the class; that is deferred. Two rules follow: