Repository navigation
design-proposal: IP addresses as a first-class resource (IPAddress / IPAddressClaim / IPAddressClass) - #35
Conversation
Walkthrough
Pre-merge checks:
|
There was a problem hiding this comment.
Code Review
This pull request introduces a design proposal to treat public IPs as first-class resources in Cozystack, modeling them after the Kubernetes PersistentVolume pattern with PublicIP, PublicIPClaim, and PublicIPClass resources. The feedback highlights three key areas for improvement: updating the PublicIPClaim status schema to support dual-stack scenarios by using a list of IP objects, configuring MetalLB's IPAddressPool with autoAssign: false to prevent IP conflicts with standard services, and clarifying how the ValidatingAdmissionPolicy will securely identify the controller principal.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| status: | ||
| phase: Bound | ||
| publicIPName: pip-203-0-113-7 | ||
| address: 203.0.113.7 # what the tenant reads, and puts in DNS |
There was a problem hiding this comment.
If a Dual stack claim binds to two separate PublicIP objects (as suggested in Open Question 5), the current status schema for PublicIPClaim is insufficient because publicIPName and address are single scalar strings.
To support dual-stack where one claim maps to multiple PublicIP resources, the status should be designed to accommodate multiple names and addresses (e.g., using a list of IP status objects).
| status: | |
| phase: Bound | |
| publicIPName: pip-203-0-113-7 | |
| address: 203.0.113.7 # what the tenant reads, and puts in DNS | |
| status: | |
| phase: Bound | |
| publicIPs: | |
| - name: pip-203-0-113-7 | |
| address: 203.0.113.7 |
| For MetalLB, forever, for the whole cluster: | ||
|
|
||
| - **one** `IPAddressPool` covering the range, and | ||
| - **one** `L2Advertisement` (or `BGPAdvertisement`) selecting it. | ||
|
|
||
| That is all. No pool per tenant, no `/32`s. The `PublicIPClass` carries the range; the | ||
| provisioner renders the backend objects. |
There was a problem hiding this comment.
There is a potential conflict between the Cozystack provisioner and MetalLB's automatic allocation. If a single IPAddressPool covering the entire range is configured in MetalLB (with autoAssign: true by default), MetalLB will automatically allocate any free IP from this pool to standard Services that do not use claims.
Since MetalLB is unaware of unassociated PublicIP reservations, it could allocate a reserved (but currently unattached) IP to a standard Service, leading to IP conflicts or binding failures when the claim owner later attempts to associate it.
To prevent this, the proposal should specify that the MetalLB IPAddressPool managed by the provisioner must have autoAssign: false configured, or clarify how standard Services are prevented from stealing reserved IPs.
| 3. A `ValidatingAdmissionPolicy` **rejects any principal but the controller** writing | ||
| *any* backend pin annotation (§1.2's list) on a Service. |
There was a problem hiding this comment.
Using a ValidatingAdmissionPolicy to reject any principal except the controller from writing the pin annotations is a robust security measure. However, identifying "the controller" securely in a ValidatingAdmissionPolicy can be challenging and brittle if based solely on hardcoded usernames or namespaces.
Consider specifying how the policy will securely identify the controller principal (e.g., by matching a specific ServiceAccount name and namespace, or utilizing a specific SubjectGroup) to ensure it is both secure and maintainable across different deployment environments.
…esses either Tenet 3 says the CNI does not announce, and increment 3 deleted the announcer. But it left the ALLOCATOR: cozyplane still picks external addresses out of an ExternalPool (firstFreeAddress, for both a VPCGateway's NAT identity and every FloatingIP) and NOTHING ATTRACTS what it allocates. An ExternalPool is a hand-written CIDR list that nothing routes; the e2e admits it -- 'here we do what a CCM would do and simply configure it on one' -- and that CCM does not exist. Allocation without attraction is an address that exists in etcd and nowhere on the wire. Pinning the address onto a Service does not bridge the gap: MetalLB's autoAssign:false does NOT reserve, so MetalLB stays the allocator of record and can hand the address cozyplane chose to some other Service. Two IPAMs, one range, no arbitration. And an LB implementation only ever announces an address that is the live loadBalancer.ingress of a Service it allocated -- so an external address can only be attracted if some OBJECT holds it. Resolved in north-south.md §9: cozyplane does not allocate either. It consumes a platform-allocated, platform-attracted PublicIPClaim (cozystack/community#35). Three consequences, each fixing something currently papered over: - The grant moves onto the claim. It must: a bare Service would let anyone who can create a Service mint a public address, which is the R10 hole. - A pod references a CLAIM, not an address. FloatingIP.spec.target is a VPC IP today -- address-thinking, against tenet 4 -- and a claim reference on the pod makes 'one target, one address' structural instead of a controller tiebreak. - Only a claim can hold an EGRESS identity. A Service is an ingress object; VPCGateway.status.natAddress is many-to-one and has no Service to hang on, yet still needs attracting because the replies come back to it. The VPC case is what proves the claim object has to exist. FloatingIP as a kind is likely superseded by a pod annotation naming a claim. The eBPF does not change at all -- the maps key on an address that arrives from somewhere and never cared who picked it, which is the strongest evidence the boundary was drawn in the right place. Answers north-south.md §7's standing questions on who attracts an EIP and whether its ingress half survives. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
|
A few reactions grounded in a production Cozystack deployment that is squarely the Strong +1 on the shape, and especially on the security move as a standalone winThe PV/PVC/StorageClass framing is the right altitude, and the Security section The "regression, not a wishlist" framing overstates it for at least one adopterkube-ovn is installed for us and the Suggested additional open question: multiple public addresses per workloadFine to defer the design, but I'd like it captured explicitly, because it
Two API observations that this open question should keep in mind:
None of this blocks the core question — "is an address a resource?" is a clear yes, |
0140e8d to
bfc1675
Compare
|
Thanks — this is the production grounding the proposal needed, and all three points landed. The revision just pushed reflects them. VAP allowlist — agreed, and it's now explicit, but admission is only half of it. You're right that a naive "reject everyone but the controller" breaks reconciliation the moment a GitOps reconciler or a system Service writes a pool annotation. The policy now identifies the controller by its ServiceAccount (configured at install, not a hardcoded username) and denies the write only for principals outside an explicit, admin-configurable allowlist (controller SA + GitOps/system SAs and groups). It also covers the pool-select annotation ( But identity-keyed admission has a ceiling: anyone who can impersonate an allowlisted ServiceAccount — a cluster-admin can — sails past it. Rather than pretend otherwise, the design now stands on two layers. Layer 1 is that admission policy, which reduces the blast radius for tenants and namespaced principals — and it's still worth landing on its own, since the theft window it closes is live in any cluster running a shared auto-assign pool. Layer 2 is reconciliation: the Regression framing — dropped entirely, and further than "for you it's new." In Cozystack it isn't a regression at all: the platform uses kube-ovn only for subnets and VPCs, never its EIP objects, so there's no reserve-then-associate capability being lost. Anyone driving Multiple addresses per workload — captured, but as explicitly deferred scope, not part of the core model. It's a step beyond "an address is a resource," and it's not clear the platform-owned case is even real. The hard sub-case you name — N owned same-family addresses, each needing its own internal address plus in-guest config the platform can't perform — is the one that looks orthogonal to reservation rather than an extension of it, since those extra addresses tend to live on interfaces the platform neither owns nor sees. So the model keeps 1 claim ↔ 1 Service, and the multi-attach case waits for a concrete platform-owned use case to shape it. Dual-stack (one v4 + one v6) is the one narrow multiplicity the model does admit — the claim status is a list from the start — but that's kept separate from the harder "N owned addresses" case. Two changes worth flagging for your setup specifically. The kinds are renamed |
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Overall this is cleaner than what I had — I'll close my additive-service PR in favor of this direction. The object model and the two-layer security story look right, and the controller/backend logic is fine.
The structural decision I want to confirm we're aligned on: this proposal deliberately covers only the allocation half (address as a resource — reserve / hold / move / quota / enumerate) and pushes the exposure half (what/how to publish, ports, RO vs RW, dropping external: bool) onto #29. That split is the right answer to "does this object own the IP or the fact of exposure": ExternalIP/Claim own the address, #29 + the app operators own the target. Let's make that boundary explicit in both docs — a reader will otherwise expect one object to do both.
What I'd like Timofei Larkin (@lllamnyp) to expand — the API surface for existing multi-service apps. The concrete driver is Postgres, which exposes three endpoints (-rw primary, -ro replicas, -r any); today external: true publishes only the primary.
- Exposing specific endpoints. Replace
external: boolwith anexpose: []list, each entry{target, class, [claimRef], ...}, wheretargetis an app-defined vocabulary (rw|ro|rfor Postgres; the chart owns thetarget → selector/portsmapping). An omitted target stays ClusterIP. - Same endpoint on multiple classes. Because it's a list, not a map keyed by target, the same
targetcan appear twice with differentclass— e.g.rwon both a public range and the internal GPN range. Please define deterministic naming for the resulting Services/claims. - Order-first or not? Two binding modes, distinguished by whether the entry carries a
claimRef: dynamic (the operator mints an ephemeral claim fromclass— the default, no pre-ordering, backend sugar only) vs static (claimRefto a pre-reserved held address, for the Elastic-IP hold/move case). State the default per app type (VM → static/hold, managed services → dynamic). The answer to "must a user reserve an IP before exposing anything" should be no by default. - Whole-IP / Service ownership.
ExternalIPmust not create or own the Service — the app stays the sole owner. Whole-IP vs port-list, ports and ICMP are fields of the app'sexpose[]entry (only VM allowsWholeIP), and the chart keeps rendering the existingservice.kubernetes.io/service-proxy-name+networking.cozystack.io/wholeIPcontract with cozy-proxy unchanged. The association controller writes only the backend pin annotation (metallb.io/loadBalancerIPs) via SSA under its own field manager, so app and controller own disjoint fields and never fight. A held-but-unattached address is kept alive by a separate holder Service in the ipam namespace, never a second owner on the app's Service. external: truedeprecation. Additive migration with auto-translation:external: true+ emptyexposesynthesizes the legacy entry (for Postgres a single{target: rw, class: <default>}, matching today's primary-only external Service);external+ non-emptyexposeis a validation error. This needs a per-app table, since the legacy meaning ofexternaldiffers across apps (vpn, tcp-balancer, vm-instance).
Two things to correct in the current draft:
- Open question 3 (ExposureClass) is stale.
ExposureClass/ServiceExposureand thenetwork.cozystack.iogroup no longer exist — they were dropped in favor of nativeloadBalancerClass(#3164 / #3218). The real question is howExternalIPClassrelates toloadBalancerClassand to #29'sclass, not whether it's a second class kind next toExposureClass. - The "one claim per workload / one address per workload" non-goal conflicts with a live requirement: a managed service reachable on a public address and an internal (GPN) address at the same time. Either drop that non-goal or state explicitly that multi-network is expressed as multiple
expose[]entries on the #29 side (one external Service per address).
Naming nit: I'd lean toward ExternalIP / ExternalIPClaim / ExternalIPClass over IPAddress*. It's narrower and accurate (external to the cluster — true for both the public and the internal-routable ranges), and it sidesteps the networking.k8s.io/IPAddress (KEP-1880) and Cluster-API IPAddress collisions the doc already flags. One caveat: mild overlap with the Service.spec.externalIPs field, worth a sentence in the doc. Group stays ipam.cozystack.io.
…review - #29 (ServiceExposure/ExposureClass) was removed by cozystack/cozystack#3218 in favor of native Service type: LoadBalancer + loadBalancerClass; retarget the tunnel entry point and drop the Phase 1 blocker - record initiator-vs-responder as an explicit decision: Phase 1 responder-only (native LB), dial-out/either-end deferred to Phase 4 gated on stable per-tenant egress IPs (#35) - reframe Phase 2 (site-gateway/NAT) as a deliberate design re-open - add portal-side image/cloud-init synchronization requirement - record NETMAP-style 1:1 subnet mapping as NAT future work Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…review - #29 (ServiceExposure/ExposureClass) was removed by cozystack/cozystack#3218 in favor of native Service type: LoadBalancer + loadBalancerClass; retarget the tunnel entry point and drop the Phase 1 blocker - record initiator-vs-responder as an explicit decision: Phase 1 responder-only (native LB), dial-out/either-end deferred to Phase 4 gated on stable per-tenant egress IPs (#35) - reframe Phase 2 (site-gateway/NAT) as a deliberate design re-open - add portal-side image/cloud-init synchronization requirement - record NETMAP-style 1:1 subnet mapping as NAT future work Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
The shape is right and I want this to land, but three findings hit the shape itself, so requesting changes. All three can be resolved as design changes or as explicit open questions.
-
Gateway is missing from the association surface. The accepted external-database-exposure proposal puts a tenant's external traffic on the single LB IP of the tenant Gateway, so the address a tenant most wants to keep across a rebuild belongs to a Service rendered by Cilium, and a Service annotation can't reach it. Gateway API already has
Gateway.spec.addressesfor this (Cilium supports it with LB-IPAM;io.cilium/lb-ipam-ipsin infrastructure annotations is the legacy path). Which Cilium release supports it needs checking against the deployed 1.19.x. Please add Gateway as an association target, or at least an open question. -
Reservation can be enforced in the allocator's own accounting, with detection kept as backstop. Neither MetalLB nor Cilium will give one IP to a second Service without an explicit sharing key: the second Service just stays Pending. A controller-owned holder Service pinning each reserved-but-unattached address makes the §6 "eager allocator got there first" theft impossible on both backends (the keep-alive mechanism already sketched in this thread). Also §6 is more pessimistic than §7 allows: with
autoAssign: falsea plain Service can never draw a reserved address automatically, so the remaining gap exists only through annotation writers, which the two layers already govern. I'd make holder-based reservation the primary mechanism and rewrite §6. -
The allowlist model breaks on confused deputies, and one is in-tree today. Tenant HelmReleases are emitted without
spec.serviceAccountName, so helm-controller applies tenant-driven content under its own privileged identity; allowlisting Flux allowlists whatever a chart renders from tenant values. Sharper: cloud-provider-kubevirt copies tenant-cluster Service annotations verbatim into the infra Service it creates (pkg/provider/loadbalancer.go, no filter, no config knob), and every Kubernetes-app tenant is admin of their tenant cluster, so any tenant can pushmetallb.io/loadBalancerIPsthrough the CCM. Allowlist the CCM and layer 1 is bypassed; deny it and tenant-cluster LB Services with any gated annotation break. The Security section should treat "trusted SA applying tenant-authored content" as a first-class threat and make annotation filtering at the deputy (a strip-list in the CCM, Flux impersonation for tenant releases) a precondition of layer 1.
Minor, non-blocking:
- "Two Services reference one claim, the second is rejected": name the mechanism. A VAP can't read the claim (CEL sees only the request object and params), so admission-time rejection needs a webhook. I'd borrow the Gateway API attachment model instead: attachment declared on one side, conflicts resolved deterministically (oldest creationTimestamp wins), outcome reported as Accepted/Conflicted status conditions. It also scales to open question 6 (several Services on disjoint ports is "many routes, one listener") and leaves room for weights later.
- §6 implies inventorying every LB address in the cluster, foreign auto-assign pools included. I'd scope the authoritative ledger to class-managed ranges and keep the rest observe-only, otherwise Conflict noise on unmanaged pools is guaranteed.
- PR title and body still say PublicIP* and still make the regression argument the doc has dropped; the directory is still public-ip-claims.
- +1 to ExternalIP* naming. Since KEP-1880 went GA in 1.33,
networking.k8s.io/v1IPAddress exists in every cluster by default, sokubectl get ipaddressambiguity is daily. Worth the one caveat sentence onService.spec.externalIPs(CVE-2020-8554). - There is no analogue of
PVC.spec.volumeName: a tenant can't ask for a specific previously-held address, pre-binding is admin-side only via claimRef. Fine if intentional, worth one line. - "bill" appears in the motivation but not in goals or the model; a Dual claim against a single-stack Service isn't in the failure cases.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
There was a problem hiding this comment.
🧹 Nitpick comments (3)
design-proposals/public-ip-claims/README.md (3)
508-514: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winClarify the parenthetical example of an allowlisted system principal.
"system controllers (system Istio Services ship with pool annotations already set)" is confusing — it's unclear whether "system Istio Services" is the principal or the annotation source. Consider rephrasing, e.g. "system controllers (e.g. an Istio installation whose Services ship with pool annotations pre-set)".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@design-proposals/public-ip-claims/README.md` around lines 508 - 514, Clarify the allowlist example in the parenthetical within the ServiceAccount policy description: rephrase “system Istio Services” to make clear that an Istio installation is the system controller or annotation source, while its Services ship with pool annotations pre-set. Preserve the surrounding explanation of admin-configured allowlisted principals.
300-317: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider naming the multi-address capability gap in the capability table itself.
The review feedback asked to consider capability declarations for backends that cannot attach multiple reserved addresses to one workload. The doc defers "multiple platform-owned addresses per workload" as a non-goal (lines 162-169), but the capability table (lines 303-308) doesn't foreshadow a future
MultiAttach-style capability the way it does forPin. Since dual-stack is explicitly in scope (open question 4) and would need twoIPAddressobjects bound to one Service, a note here that a backend's ability (or not) to attach more than one reserved address is a capability question for later would tighten the connection between §4 and the dual-stack open question.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@design-proposals/public-ip-claims/README.md` around lines 300 - 317, Add a brief note to the capability table or immediately surrounding discussion introducing a future multi-address attachment capability, such as MultiAttach. Connect it to backends attaching multiple reserved addresses to one workload, particularly dual-stack Services requiring two IPAddress objects, while preserving the current non-goal and deferring the capability’s final design.
113-116: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueBlank lines inside blockquote break rendering (MD028).
Static analysis flags blank lines inside the blockquote block spanning these pull-quotes; some renderers will terminate the blockquote early. Separate the quotes with
>continuation or a blockquote marker on the blank line instead of a bare blank line.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@design-proposals/public-ip-claims/README.md` around lines 113 - 116, Update the pull-quote block in the README so the blank line separating the two quoted paragraphs retains the blockquote marker, preventing the blockquote from being terminated while preserving both quotes.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@design-proposals/public-ip-claims/README.md`:
- Around line 508-514: Clarify the allowlist example in the parenthetical within
the ServiceAccount policy description: rephrase “system Istio Services” to make
clear that an Istio installation is the system controller or annotation source,
while its Services ship with pool annotations pre-set. Preserve the surrounding
explanation of admin-configured allowlisted principals.
- Around line 300-317: Add a brief note to the capability table or immediately
surrounding discussion introducing a future multi-address attachment capability,
such as MultiAttach. Connect it to backends attaching multiple reserved
addresses to one workload, particularly dual-stack Services requiring two
IPAddress objects, while preserving the current non-goal and deferring the
capability’s final design.
- Around line 113-116: Update the pull-quote block in the README so the blank
line separating the two quoted paragraphs retains the blockquote marker,
preventing the blockquote from being terminated while preserving both quotes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 92f6ae23-9417-427d-bba0-ee0da03e51ba
📒 Files selected for processing (1)
design-proposals/public-ip-claims/README.md
|
Two updates that bear on the open naming question, plus the argument for the API group the implementation ships. The document has just been updated to match both. The contract has a second consumer, and it isn't MetalLB-shapedThe model is implemented and running: address-controller is the class-agnostic core (the PVC/PV binding analog), metallb-iad the reference per-class driver. One backend proves little on its own, so the evidence worth putting in front of reviewers is the second consumer. Cozyplane consumes reservations through this contract, and does it through one annotation and nothing else: no module import, no CRD dependency, no informer on the claim kinds. A Validated end to end on a three-node dev cluster with the full stack (address-controller + metallb-iad + MetalLB + cozyplane): a To be clear about the dependency direction: this proposal does not wait on cozyplane, and cozyplane is not the reason to land it. Attachable/detachable addresses are a prerequisite for #45, and the MetalLB path serves stock Cozystack today. Cozyplane is corroboration, not motivation — it happens to be the sharpest available test of one specific claim, below. That claim is narrower than the whole design. Rollout §2 names a Cilium provisioner as the phase that validates or falsifies this — that's the allocation seam, and it's still outstanding. What cozyplane exercises is the association seam: whether a backend with an entirely different datapath can wear a reserved address while knowing nothing about the ledger beyond one annotation. It can, and the coupling turned out looser than this document assumed it would need to be. The API group:
|
|
Aleksei Sviridkin (@lexfrei) — all three land. Two became corrections to the document, just pushed; the third moves into implementation. Where each went: 1. Gateway is missing from the association surfaceAgreed, and the underlying problem is more basic than a missing target: §5 assumed the address consumer is a Service somebody wrote. Under Gateway API the implementation renders the data-plane Service from the What the document now says is the part that's design rather than mechanism: the contract is an annotated object that resolves to an address consumer, with One consequence is not provisioner business and is now in Security: Which Cilium release honours 2. Reservation enforced in the allocator's own accountingAgreed, and this was the document being behind its own implementation. The MetalLB provisioner already holds every reserved-but-unattached address with a placeholder Service — selectorless, in a provisioner-owned namespace, assigned by MetalLB and never announced. §6 is rewritten with that as the primary guarantee and detection as the backstop, and your §7 point is taken: with Two things the rewrite keeps rather than smooths over: the handoff (releasing the holder and pinning the workload's Service isn't atomic, and the address is briefly unheld), and your scoping point — the ledger is authoritative only over ranges a class manages, with everything else observed rather than adjudicated, so foreign auto-assign pools don't generate 3. The allowlist breaks on confused deputiesAgreed, and this is the one that changed what the document is allowed to claim. An identity allowlist assumes a trusted principal writes content it authored, and that fails wherever a privileged controller applies material a user supplied — a reconciler that applies user-authored releases under its own identity instead of impersonating, a cloud-controller-manager copying Service annotations verbatim out of a user-controlled cluster. Allowlist the deputy and everyone behind it inherits the gated write; deny it and reconciliation breaks. Security now states annotation filtering at the deputy as a precondition of layer 1, not a hardening extra, and says plainly that layer 1 is partial for the users behind an unfiltered deputy, with layer 2 covering them meanwhile. That's a weaker claim than the section made before, and the honest one. The minor points
The changes-requested is yours to clear when you've had a look. |
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
The implementation evidence changes what this document is. A class-agnostic core plus a reference driver, and a second consumer that touches the contract through one annotation and no imports, is the strongest possible answer to "is this shape bent around MetalLB" — much stronger than the argument alone. The address-source union is the right spine, and the Security section is unusually honest: naming the confused-deputy case, saying layer 1 is partial for users behind an unfiltered deputy, and framing cluster-admin as a trust boundary rather than pretending to close it. The response to Aleksei Sviridkin (@lexfrei) is thorough — generalising Gateway into "an annotated object that resolves to an address consumer" is a better answer than adding a second target.
Four things.
1. addressName is tenant-writable and entirely ungoverned. It appears exactly once in 738 lines — commented out in the claim example, described as "the PVC spec.volumeName analogue" — and nowhere else. Not in Security, not in Failure and edge cases, not in RBAC.
That matters because IPAddress is cluster-scoped and IPAddressClaim is the tenant-facing object. The security model rests on the invariant stated in Security §1: a tenant may only name an object in their own namespace, because that is what RBAC can gate. addressName breaks exactly that invariant — it names a cluster-scoped object by name, so a tenant could bind a specific Available address in a class they were never granted, or name an address whose class contradicts their own className. Layer 1 gates the association annotation and the backend pins and never sees this field.
Note Aleksei Sviridkin (@lexfrei) read the document as having no volumeName analogue at all ("pre-binding is admin-side only via claimRef"). One of the two readings is wrong, which is itself the signal. Either drop the field and say pre-binding is admin-side via claimRef, or specify the checks: class match, class-level authorization, and the behaviour when it names a Bound or Released address.
2. ExposureClass is stated as existing, and it does not. Scope says "an ExposureClass kind exists today in network.cozystack.io", and open question 3 is anchored to "the fate of ExposureClass/ServiceExposure". On main today, ExposureClass appears only in ROADMAP.md and docs/changelogs/v1.6.0.md, and network.cozystack.io has zero occurrences under packages/ or api/. The kind was never shipped — the direction went to native loadBalancerClass (cozystack/cozystack#3164, cozystack/cozystack#3218). As written, open question 3 cannot be answered, because its subject does not exist. The real question is how IPAddressClass relates to loadBalancerClass.
3. The one-claim-per-workload non-goal still excludes a live case. The added rebuttal answers the VM-as-router example — extra addresses on a tunnel interface the platform never sees — and that answer is right. It does not reach the case I raised in July: a managed service reachable on a public address and an internal GPN address at the same time. Both are platform-owned, both are real Services, and "a workload holds one claim" forbids it. One sentence settles it: multi-network is expressed as several Services, each with its own claim.
4. Group name, non-blocking. local.sdn.cozystack.io files an IPAM primitive under the SDN module, but its consumers include Gateways and cloud providers that are not SDN. ipam.cozystack.io reads closer to what the object is. Your call.
On naming: you and I disagree about IPAddress* versus ExternalIP*, the Naming section states your reasoning openly, and that is enough — I am not going to press it further.
None of these blocks the document. Point 1 is the one to settle before it turns into code — as a commented-out line with no spec text around it, it is currently ambiguous enough that two reviewers read it differently, and the invariant it touches is the one the whole security model rests on. Points 2 and 3 are a few lines each.
Approving. Note this does not clear Aleksei Sviridkin (@lexfrei)'s outstanding review.
…#20; add VM modes Resolve the review feedback on community#45 and finalize the proposal: - applicationRef gains group (default apps.cozystack.io); the resolver compares all three lineage labels AND requires the tenant-facing marker internal.cozystack.io/tenantresource=true — lineage labels alone cover operator-internal Services too. - Address-source fields (className/claimName/family) are immutable via CEL; render knobs stay mutable. Phase enum defined (Detached is the endpoint-absent state); conditions are standard metav1.Condition, phase a derived summary. - The rendered Service's identity is the controller ownerReference; the ownership label is index/diagnostic only. The controller sets loadBalancerClass from the claim's class at creation, per #35 §5. - GC identity is UID-anchored by design: attachments belong to one application incarnation; same-name recreate collects them. - New section positioning attachments beside the accepted SNI consolidation design (#20): dedicated addresses are the exception, the gateway union member is #20's inherited tenant-facing trigger. - VMs are a second first-class consumer: ports filter, method WholeIP|PortList rendering the existing cozy-proxy contract, with an at-most-one-whole-IP rule; the multi-address egress model is the one open question. - Quota/group coupling, per-address ACL granularity, and the mirroring contract (static selector, pod-label roles) stated explicitly. Assisted-By: Claude Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Add the exposure half to the IP-addresses-as-a-resource substrate (#35): a tenant-created, namespaced EndpointAttachment that attaches an external address to one endpoint (tenant-facing Service) of a managed application. Attachments are additive and engine-agnostic (the controller mirrors the endpoint Service's selector and ports), mint or reference IPAddressClaims, and are garbage-collected with the application. Succeeds the closed structured-external-exposure proposal (#29); external: true coexists until an address-preserving migration exists. Assisted-by: LLM Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Align the document with what is built and argue the group choice on the record rather than leaving it in a commit message of another repository. - API group: ipam.cozystack.io -> local.sdn.cozystack.io throughout, including the association annotation, the default-class annotation and the example provisioner name, which now match what the controllers actually ship. Positioning carries the argument: Cozystack networking is already sdn.cozystack.io (SecurityGroup, served by the Cozystack API), cozyplane serves its own SecurityGroup there plus local.sdn.cozystack.io for its CRD-served kinds, and addresses are the third networking feature to want a home. One sdn family, with the local. prefix marking CRD-served kinds and the bare group the aggregated ones; ipam.cozystack.io would add a fourth name to a subject area that should have one. Stated plainly as a proposal, not a precedent: nothing serves local.sdn.cozystack.io today. - New Implementation status section: the core controller and the MetalLB reference driver exist, and cozyplane consumes reservations through the contract with no module import, no CRD dependency and no informer on the claim kinds -- the association seam is exercised by a second consumer, while Rollout phase 2, a second provisioner, remains outstanding. The ValidatingAdmissionPolicy is recorded as implemented nowhere and still part of phase 1. - The stub banner no longer says the proposal predates any code, and Rollout phases 1 and 2 say what exists. - Scope records that community #45 depends on this proposal, which does not wait on it. - Directory renamed public-ip-claims -> ip-address-claims; the kinds lost the "public" prefix two revisions ago. Assisted-by: LLM Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Review found several places where the document asserts something that is not true or not achievable. Each is corrected rather than annotated. - Section 6 described the reserved-but-unattached theft window as an acknowledged gap with no clean remedy. Holders close it: for every reserved address not currently attached the provisioner keeps a placeholder consumer, so the backend's own accounting refuses to double-assign, and neither MetalLB nor Cilium will give one address to a second Service without an explicit sharing key. The section now leads with that as the primary guarantee and keeps detection as the backstop, naming what genuinely remains: the non-atomic handoff, and explicit annotation writers. The backwards inventory is scoped to ranges a class manages, so foreign auto-assign pools are observed rather than adjudicated. - Security claimed more for layer 1 than an identity allowlist can deliver. A privileged controller applying user-authored content -- a GitOps reconciler that does not impersonate, a cloud-controller-manager copying annotations verbatim out of a user-controlled cluster -- hands every user behind it the gated write. Annotation filtering at the deputy is now stated as a precondition of layer 1, which is partial until it holds, with layer 2 covering those users meanwhile. - Two failure cases named admission as the mechanism where admission cannot be one. A ValidatingAdmissionPolicy sees the request object and its params, not arbitrary cluster state, so it can read neither the claim nor a backend's capabilities: the second Service on a claim is resolved by reconciliation with a deterministic winner and a Conflicted loser, and a class over a Pin-less backend is refused by its provisioner. - Section 5 assumed the annotated object is always a Service. Under Gateway API the implementation renders the data-plane Service from the Gateway, so the contract is an annotated object that resolves to an address consumer. Where a provisioner lands the pin is its own business; the part that is not is that Gateway.spec.addresses names an address by value and therefore falls under the same admission policy as a backend pin annotation. - The claim carries spec.addressName, the PVC spec.volumeName analogue, which the document never mentioned. - A Dual claim against a single-stack Service was missing from the failure cases. - Dropped "bill" from the motivation: the object model delivers enumeration and quota, and billing is something a platform builds on the inventory. Assisted-by: LLM Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Review approved the shape and left four specifics. Three change the document. - spec.addressName appeared once, commented out in the claim example, with no prose anywhere. Two reviewers read it opposite ways -- one as absent entirely, one as an ungoverned way for a tenant to name a cluster-scoped object. It is now specified: it narrows the candidate set and never widens it, honoured only for an address that is Available, unclaimed, not being deleted, and of the claim's own resolved class and family, with a name that matches nothing leaving the claim Pending rather than failing. Failure cases cover the absent, Bound, Released and wrong-class names. Security now states where this component stops. It secures the surface a reservation can be stolen through; it does not decide which class a namespace may draw from, and neither addressName nor className can be gated by namespace RBAC. That is deliberate: the capability is reserve/hold/attach/release, and encoding a tenancy model this controller cannot see belongs above it -- an admission policy over those two fields, or machinery that mints claims on a tenant's behalf. Both fields are named so a deployment hardening this does not have to find them. - ExposureClass was described as a kind that exists in network.cozystack.io. It was proposed alongside community #29 and never shipped, and that group does not exist; the direction went to loadBalancerClass. Open question 3 was anchored to its fate and so could not be answered. It now asks the live question -- whether a class should declare the loadBalancerClass it corresponds to, so an association can check the pair -- and records that loadBalancerClass is immutable on an existing Service, so a mismatch has to be a refused association rather than a rewrite. - The non-goal said a workload holds one claim, which forbids a case that is not ours to forbid and that community #45 depends on: a managed service reachable on a public address and an internal routable one is two Services with two claims. The 1:1 rule that is real -- one claim, one consumer at a time -- is kept, and what stays deferred is N owned addresses on a single consumer. Also records, in Positioning, why the group is not ipam.cozystack.io: this is not all of IPAM and cannot become it, since allocation stays with the allocator. Assisted-by: LLM Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…inality question Two corrections from review. - IPAddressClass and Service.spec.loadBalancerClass were left as an open question about whether they should be related. They should not be merely related: a class names the pool and the provisioner that carves it, which determines the implementation that announces anything drawn from it, so the class states that implementation in spec.loadBalancerClass and a consuming Service must match it, empty matching empty. A Service selecting one implementation while its claim's address is held by another asks two backends for one address and both are wrong about who owns it; the provisioner's holder Services carry the class's value for the same reason. The check is a refusal and never a repair, since loadBalancerClass is immutable on an existing Service, so whoever creates the Service sets it at creation -- and a controller that both creates the Service and names the claim can read the class first. The open question is dropped and the rest renumbered. - The non-goal claimed a claim binds one consumer and left it there, which reads as though the cardinality is settled in both directions. It is not. Gateway.spec.addresses is a list, so one Gateway legitimately wears several addresses, and nothing here says how several claims attach to one consumer or how a provisioner renders that. What stays settled is the half that is enforced: a claim is consumed by one consumer at a time. How many claims one consumer may hold is acknowledged as open and deferred to future work, with the tunnel-interface and platform-owned variants deferred alongside it. The open question on sharing one address across Services is reworded to match: it is the mirror of that case, and the two want answering together. Assisted-by: LLM Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
… local.sdn family Community #45 now homes its EndpointAttachment kind in local.sdn.cozystack.io rather than consuming the substrate from another group. Record it as another CRD-served resident of the family, which exercises the local.-prefix quota guarantee the Positioning section argues for. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Drop the Status and Date header fields, which community #111 removed from proposals in favour of deriving both from PR and Git history, and add the empty Decisions section the template now carries. Join the three pull-quotes in The problem into one blockquote so the lint does not flag bare blank lines inside it. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
…mentation The code moved while the document did not, in both directions. Record where the document runs ahead of the binaries, and correct what it got wrong about them. - Implementation status points at the cozystack organisation, the packaging PR and the driver's admission-policy and auto-claim PRs, and lists every rule specified here and not yet implemented, plus the detail the code carries that the model does not need. - Section 6 no longer reconciles foreign Service assignments into the ledger. Nothing justified catch-all adoption: with a class pool at autoAssign false a plain Service can only reach a class address by a pin, which is a conflict to surface. The dropped design is recorded under Alternatives. In its place, the opt-in auto-claim the core now has, and why: a cloud-controller-manager renders host-side LoadBalancer Services that have never heard of claims. Two rules follow: a claim with a controller owner follows its owner rather than the class reclaim policy, and a namespace's held claims are attached before a new one is minted. - Section 5: the class determines the loadBalancerClass, which an explicit field is the simplest but not the only way to express; the check is marked specified, not implemented. Dropping the annotation does not make MetalLB withdraw an assignment from a live Service, so say so and say what detaching from a live Service means. - Section 3: the reference driver lets MetalLB pick the address through the holder and records the result, after a driver-side allocator was tried and replaced. The ledger is the record of reservation, not the allocator. - The principal that writes pins is the driver's ServiceAccount, never the core's; the examples said otherwise. Conflict adjudication is per class, so two drivers do not judge each other's handoff windows. - Two Services on one claim: the live holder keeps it, the loser gets an event and the claim an Associated condition. IPAddress gains conditions with transition times, so the ledger answers when as well as who. Dual is settled as one claim binding two objects, because a dual-stack Service names one claim; the driver pins only the families the Service carries. - Rollout gains Gateway as its second phase, ahead of the Cilium provisioner, since the accepted database-exposure proposal puts tenant traffic on the Gateway's one address. Open questions on Cluster API reuse and dual-stack close; Gateway and the entitlement object open. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
…itlement and lifecycle Review on this PR, on the packaging PR and on the sibling proposal asked for the same handful of things, which the text now carries. - The admission fence covers every input the backend honours: MetalLB's legacy annotation prefix and Service.spec.loadBalancerIP, which the controller still parses, are gated beside the current keys, and the comparison is old against new. Not every platform controller is allowlisted: a deputy forwarding user content is denied, and its Services reach the ledger through the claim annotation or auto-claim. The policy ships with Warn and Audit modes, and already-pinned Services are adopted by pre-binding, not broken. - Who may draw from which class is stated as a contract rather than handed to a sibling: the check keys on the namespace so that machinery-minted claims are covered, the empty className is an explicit decision, and addressName is refused to tenants or checked in a webhook, since CEL cannot resolve it. The natural home is an optional per-namespace, per-class count in the core, the per-class quota scoping Kubernetes applies to storage; its shape is an open question and the distribution writes it. addressName itself is exactly volumeName: unauthorized by design, with pre-binding as the admin's lever. - Two operating modes fall out of RBAC alone: self-service, where tenants create claims, and brokered, where platform machinery does and tenants attach and detach but never release. Metering follows the claim, since a held address is held whether or not a Service wears it. Both are stated under user-facing changes. - Lifecycle cases the implementation surfaced: a namespace deleted while the core is down waits on the finalizer, a class with bound addresses refuses deletion rather than withdrawing them all, and an optional releaseDelay on the class returns Released addresses to Available after a cooling-off period, with a hold marker for addresses an operator does not want reissued. - One sentence on Service.spec.externalIPs, the field behind CVE-2020-8554, as a further reason the kinds are not ExternalIP*. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
aeff65f to
71e0383
Compare
…tation forced Two changes in the preceding revision were not review feedback but implementation findings that changed the design's course, and the repository's process asks for a decision record in that case so the reasoning outlives the pull-request thread. 0001 records that the reference driver reserves through placeholder Services and lets MetalLB allocate, after a driver-side allocator proved invisible to the backend and able to split-brain against its live state, and what that costs: the handoff gap, and sticky assignments that keep a live Service announcing after its pin is withdrawn. 0002 records that the ledger adopts nothing and that the core instead claims, opt-in per class, on behalf of Services that cannot name a claim, after tenant-cluster Services rendered by a cloud-controller- manager sat pending under autoAssign false; and why such claims follow their owner rather than the class reclaim policy. Both are linked from the proposal's Decisions section. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
|
Four replies in one, after today's push ( Aleksei Sviridkin (@lexfrei) — since your review the document has changed in four pushes, the last two today, and I owe you a summary rather than a diff. Your three findings: Gateway is an association target as "an annotated object that resolves to an address consumer", with Two things the implementation taught us are recorded as decision records under the proposal rather than silently amended: the reference driver lets MetalLB allocate through placeholder Services after a driver-side allocator proved invisible to the backend (0001), and the backwards inventory is dropped in favour of auto-claim for Services that cannot name a claim (0002). Also from the implementation: the fence must cover Would you take another look? Andrei Kvapil (@kvaps) — your approval of 2026-08-31 was dismissed automatically by the push that followed it four seconds later, so I am asking for it again. Of your four points: IvanHunters — thanks, and sorry for the delay. All four findings hold, and the answer to your question is: the entitlement policy belongs in the distribution, because its parameter is the What the proposal now states, so the two sides stop pointing at each other, is the contract any such policy meets: it keys on the request namespace rather than the requesting user, because platform machinery mints claims under its own identity; the empty myasnikovdaniil — done in the latest push: |
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
LGTM as of d6e35ac. All four from my last review are in: addressName filters and never authorizes, ExposureClass is gone, two Services with two claims covers public plus internal, and the group choice is written down. The per-class entitlement as quota, zero meaning not entitled, is the same shape we use for GPU budgets in #99.
IvanHunters
left a comment
There was a problem hiding this comment.
Architectural review (design + reference implementation at address-controller#7 / metallb-iad#5)
Reviewed this as a network/platform architect, with the reference controllers checked out alongside the document because the Implementation status section makes claims about them. The model is sound: the PV pattern fits, the allocation/association split is the right seam, and the two decision records close real defects from the first iteration. Most of what follows is about one thing, the association handoff, plus a couple of places where the document and the code have drifted apart.
[CRITICAL] Association has no serviceability precondition, so the handoff silently black-holes the reserved address
associate() acts on the presence of the claim annotation alone; it never checks svc.Spec.Type or svc.Spec.LoadBalancerClass (metallb-iad internal/controller/service_controller.go). The handoff then runs unconditionally: set associatedTo, delete the hold placeholder (which frees the address in MetalLB's books), write the pin.
If the annotated Service is one MetalLB will not serve, a non-empty loadBalancerClass while the speaker runs with an empty --lb-class, or a non-LoadBalancer type, the address is never announced. And because associatedTo now names a live Service, the driver's IPAddress reconciler takes the "associated" branch and deletes placeholders rather than re-arming the hold (internal/controller/ipaddress_controller.go). The end state: the address is unheld in MetalLB, announced nowhere, available for an eager allocator to hand to someone else, while the ledger reports associatedTo set and the claim Bound. That is a silent failure of the exact invariant this proposal exists to provide, and it hits the headline use case, the address a customer has in a firewall allow-list.
The document flags the §5 loadBalancerClass match check as unimplemented, but not this active consequence: the handoff tears down the hold before confirming the target can be served. The contract should require association to refuse a consumer it cannot serve, with an event, before it releases the hold.
[HIGH] As shipped there is no admission layer, though the document says there is
Rollout states the admission policy "exists as the driver's opt-in template." In the reviewed branches there is no webhook and no ValidatingAdmissionPolicy manifest: the chart ships only role.yaml and controller.yaml, and the only references to the policy are in README/design.md. It lives in metallb-iad#6, not here. So in this state Security's layer 1 is absent and the reserved-but-unattached theft window the proposal calls its sharp edge is open. The driver ClusterRole also grants services: update/patch cluster-wide, which is the escalation surface layer 1 is meant to fence. The document should say layer 1 is not yet in the controllers rather than imply it ships with them.
[HIGH] The association handoff is the fragile core: non-atomic and detection-only
Three facets of one seam:
- Release-then-pin is not atomic. If the pin write fails or the controller restarts after the placeholder is deleted,
associatedTois already set to the live Service, so the hold is not re-armed and the address stays unheld until the first successful pin, not merely "bounded by reconcile latency" as decision 0001 states. - Detaching from a live Service is not clean.
unpinclearsassociatedToand removes the pin, but MetalLB keeps the IP on the live Service (sticky, confirmed against MetalLBcontroller/service.goand the docs), so the nextdetectConflictsdrives that address toConflict. A routine detach cannot settle without recreating the Service. - Conflict is detected after assignment, not prevented (
detectConflictskeys onstatus.loadBalancer.ingress). For the impersonation case that is the stated design; for the two cases above it means a real-traffic window.
None of these is fatal on its own, but together the association seam carries most of the design's risk. It may be worth treating as one unit rather than guarding case by case.
[MEDIUM] Auto-claim has no structural guard against the driver's own holders
The core's wantsAutoClaim has no namespace or placeholder-label exclusion and no predicate, and the manager cache is cluster-wide, so it sees the driver's placeholder Services. Protection rests on the opt-out annotation (metallb-iad#5) plus the status.loadBalancer.ingress == 0 race. The driver's own association reconciler already skips placeholders structurally by namespace and label; the same guard in wantsAutoClaim would make the invariant independent of deploy order and of whether #5 is merged (both #5 and #7 are still open). Related: the single-default-class and single-auto-claim-class rules are enforced only by a runtime event, not admission or CRD validation, so two of either fails silently, two defaults leave every pending claim stuck, two auto-claim classes turn auto-claim off.
[MEDIUM] Controller-side scaling
A cluster-wide Service informer runs regardless of whether auto-claim is enabled anywhere, and several hot paths do unindexed full lists: withdrawAssociations lists every IPAddress on every Service deletion, missingFamilies/bindAvailableAddress list every IPAddress per claim reconcile, and all unbound claims are re-queued on any IPAddress or class event. Fine at hundreds of objects, degrades around 10k+. The placeholder ClusterIP cost is a related ceiling: one ClusterIP per reserved address ties reservation capacity to the host Service CIDR (a /24 is 254, a /16 is 65k), not to the external pool. The decision record notes the placeholder is "cheap but not free" without naming that ceiling.
Smaller items
- A
Lostaddress stays inclaimRefand inclaim.status.addressesand accrues; nothing clears a fromClassLostledger entry. - No entitlement object: a tenant may name any
className, and quota bounds count, not source. (Acknowledged as deferred.) releaseDelayis unimplemented, soReleasedaddresses accumulate underRetainuntil an admin clearsclaimRef. (Acknowledged.)- The driver ClusterRole could scope
services: create/deleteto the placeholder namespace and keep onlyupdate/patchcluster-wide. - The BGP path is not discussed: prefix aggregation and route-table growth under scattered reservations.
What looks right
Conflict is recoverable rather than terminal, allocation is idempotent through deterministic placeholder names so the driver and MetalLB cannot split-brain, claimRef UID matching drops stale bindings, address immutability and the union source are in place, and the finalizer plus namespace-deletion handling mirror PVC protection correctly.
…cal statements A reading pass over the proposal and both decision records, with no change of substance. Every sentence now names a component, says what it does, and gives the consequence in a sentence of its own. Evaluative asides, closing restatements, and reassurances that the rule already implied are deleted. A fact that was derived in several sections is now stated once in the section that owns it and referenced from the others. The document is a third shorter with no fact removed. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
Name cloud-provider-kubevirt where the text described its behaviour as that of deputies in general: it copies guest Service annotations once, at creation, and that constraint is its own, not a property of every controller that forwards user content. State in §7 what the MetalLB driver does with the class's address list: it renders one IPAddressPool from the CIDRs and does not adopt an existing pool, and MetalLB rejects overlapping pools. Fix the order of operations in §5: the driver reads the annotation, looks up the claim in the Service's namespace, then resolves it. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
… handoff; correct the policy's status From the architectural review of 2026-11-02. The driver releases the hold on the annotation alone, so a Service that MetalLB will not serve ends up with the address unheld, announced nowhere, and reported as attached. §5 now requires the driver to check type and loadBalancerClass and refuse with an event before it sets associatedTo and deletes the holder. A failure case states the outcome. The admission policy is in an open PR, not in the driver's chart, so Implementation status and Rollout say layer 1 is absent on main. Auto-claim skips any Service carrying a backend pool or pin input, which excludes the driver's placeholders structurally. Duplicate default or auto-claim classes are a stated runtime condition. The handoff window lasts until the first successful pin, not one reconcile interval, in §6 and in record 0001. Record 0001 names the ClusterIP-per-placeholder ceiling. A Lost address keeps its claimRef until an admin acts. One sentence on BGP: reservations do not change route-table growth. Community #99 is cited for the per-class quota shape. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
|
IvanHunters, thank you for reading the code beside the document. Each finding, and where it landed, as of Critical, association without a serviceability check. Confirmed in the driver. The document shares the blame: §5 specified the No admission layer as shipped. Correct, and a wording error on my side: "ships" described an open PR. Implementation status and Rollout now say the chart on The handoff. The window lasts until the first successful pin, not one reconcile interval; §6 and record 0001 say so now. Detach from a live Service ending in Auto-claim guards. The core now skips any Service carrying a backend pool or pin input, which excludes the driver's placeholders without the opt-out annotation and without knowing the driver's namespace. Duplicate default or auto-claim classes are stated as a runtime condition, as Kubernetes does for duplicate default StorageClasses. Scaling. The ClusterIP-per-placeholder ceiling is now in record 0001. The controller findings are filed against the implementation: cozystack/address-controller#8 (full ledger lists and the unbound-claim requeue), cozystack/address-controller#9 (the always-on Service informer in #7), cozystack/metallb-iad#7 ( Smaller items. A Would you take another look? |
Re-review of a07dd3eThe previous round is fully addressed, and well. For the record: §5 now requires the serviceability and A fresh read of the rewritten revision turned up a few new things, most of them introduced by the prose reflow. [HIGH] The banner claims phase 1 is implemented; the document's own status says it is notThe opening note states "phase 1 of Rollout is implemented." Rollout defines phase 1 as "CRDs, core controller, the MetalLB provisioner, and the admission policy," and adds that "the §5 [HIGH] §4 presents the capability contract as modelled and enforced; elsewhere the document says it is neither§4 reads "Each provisioner declares capabilities, and the core refuses what a backend cannot do," with a table of hard consequences. The banner says the opposite, "the provisioner contract's own API (capabilities are not modelled)," Failure and edge cases says "There is no capability registry for admission to consult," and there are no capability types in [MEDIUM] Auto-claim and the new serviceability check exclude each other for a class with a non-empty
|
…ioner contract are today From the re-review of 2026-11-03. The banner said phase 1 is implemented; by Rollout's own definition it is not, since the policy is in an open PR and two §5 and §6 rules are unimplemented. It now says the controllers exist and are pending review. §4 described capabilities as declared and enforced by the core. Nothing declares them: the contract as implemented is the provisioner name stamped onto claims and the source union on the address, and a driver that cannot serve a claim leaves it Pending. The table is now the intended shape, with the refusal the driver's. An auto-claimed Service has no loadBalancerClass and the core cannot set one after creation, so an auto-claim class must have an empty loadBalancerClass; the core ignores the annotation otherwise. PureLB pins through the deprecated spec.loadBalancerIP field, so it is a backend after all; the §1.2 table says so and the open question is dropped. The status lists no longer contradict the text on the Lost claim phase and the claim's conditions. Record 0001 says lexfrei's review came three days after the change and made the same argument independently. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
…lementation is deferred An auto-claimed Service has no loadBalancerClass and MetalLB reads no annotation in its place, so the default implementation serves it. State that as the field's semantics rather than a rule of this design, and defer auto-claim into a class of another implementation to a mutating admission webhook that sets the field at creation. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> Assisted-by: LLM
|
IvanHunters, as of The banner now says the controllers of phase 1 exist and are pending review; Implementation status lists what phase 1 still lacks. §4 is rewritten as what exists and what is intended. As implemented the contract is the provisioner name stamped onto claims and the source union on the address; nothing declares capabilities, and a driver that cannot serve a claim leaves it On auto-claim and The status lists no longer contradict the text: PureLB, checked: it honours |
IvanHunters
left a comment
There was a problem hiding this comment.
Re-review of 759dd6c
The round-2 items are resolved. The banner no longer claims phase 1 is implemented and now matches Rollout. §4 separates the implemented contract (the provisioner-name stamp and the source union, with no capability registry) from the intended capability shape, and the refusal is the driver's. §6 states that an auto-claim class is a class of the default implementation and reports an event when its loadBalancerClass is non-empty, with cross-implementation auto-claim deferred to a mutating webhook. The status lists and the prose now agree on the Lost claim phase and on claim conditions. PureLB is corrected to spec.loadBalancerIP and its open question is dropped.
What remains is genuinely open: the entitlement object, in-tree versus out-of-tree provisioners, Gateway, and the claim-to-consumer cardinality. The specified-versus-implemented split reads honestly throughout.
Approving the design. The implementation-level items from the earlier round are tracked separately in cozystack/address-controller#8, cozystack/address-controller#9, and cozystack/metallb-iad#7.
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
Timofei Larkin (@lllamnyp) LGTM for the design. My previous blocking points are addressed, and the document separates the intended contract from what is implemented.
What
Adds a design proposal under
design-proposals/ip-address-claims/introducing IP addresses as a first-class resource: a cluster-scopedIPAddress, a namespacedIPAddressClaim, anIPAddressClass, and a provisioner contract in the shape of CSI. The kinds live inlocal.sdn.cozystack.io.Why
A tenant cannot own an address. One appears as a side effect of creating a
Service type: LoadBalancer, and evaporates when that Service is deleted. So an address cannot be:The experience we want is the AWS one: allocate an address, keep it, attach it, detach it, attach it to something else, release it when done.
The model
The
PersistentVolumepattern, applied to addresses:StorageClassIPAddressClassPersistentVolumeIPAddress(cluster-scoped)claimRefand a reclaim policyPersistentVolumeClaimIPAddressClaim(namespaced)Two things fall out of the primitives survey and are worth a reviewer's attention:
fromClassvsproviderRef), directly analogous to PV's volume-source union. Self-allocating backends (MetalLB, Cilium, kube-vip) have no reservation concept anywhere, so the ledger is the record of reservation. Cloud backends (AWS, GCP) already hold the reservation with a stable handle and their own IAM, so the provisioner must adopt, not allocate. Surviving a backend where allocation isn't ours is the evidence the shape isn't bent around MetalLB.Pincapability, which the entire model depends on: a backend that cannot be told which address to use can never attach a reserved one, so a class over it must reject claims outright rather than allocate an address it can never bind.The sharp edge (please read the Security section)
The per-backend "pin this IP" annotation (
metallb.io/loadBalancerIPs,lbipam.cilium.io/ips, …) is a privilege-escalation surface: a self-allocating backend has no concept of a reservation, so it will happily hand one tenant an address another tenant has reserved but not yet attached.The design answers this in two layers rather than one invariant, because a single one cannot hold:
local.sdn.cozystack.io/ip-address-claim); the driver writes the raw pin. AValidatingAdmissionPolicyidentifies the driver by ServiceAccount and denies every pool-select and pin input the backend honours — legacy spellings andspec.loadBalancerIPincluded — to principals outside an admin-configurable allowlist. A deputy that forwards user content is denied, not allowlisted.IPAddressinventory is authoritative: any collision surfaces asstatus.phase: Conflictnaming the offending Service, and a reservation is never silently overwritten.Who may draw from which class is stated as a contract rather than solved here: namespace-keyed, so machinery-minted claims are covered; the empty
classNamean explicit decision;addressNamerefused to tenants or checked in a webhook. The natural home is an optional per-namespace, per-class count in the core, whose shape is an open question.Implementation status
The class-agnostic core (cozystack/address-controller) and a reference per-class driver for MetalLB (cozystack/metallb-iad) exist, with reservations held as placeholder Services so the backend's own accounting refuses to double-assign. The driver's main branch has no admission policy. The open PR cozystack/metallb-iad#6 adds it as an opt-in template; cozystack/cozystack#3382 packages both for Cozystack and carries a distribution-specific policy. cozystack/address-controller#7 adds the opt-in auto-claim path of §6.
The load-bearing evidence is a second consumer: cozyplane wears reserved addresses through this contract with no module import, no CRD dependency and no informer on the claim kinds — one annotation on a Service it already owns. That exercises the association seam. The allocation seam (a second provisioner) remains outstanding.
The document lists, under Implementation status, every rule it specifies that the code does not yet have.
Decisions
Two implementation findings changed the design rather than filling it in, and are recorded under
decisions/:Notes for reviewers
IPAddress*: at the ledger levelExternalIPbeside aFloatingIPwould be two names for one thing, andExternalIP*would invite confusion withService.spec.externalIPs, the field behind CVE-2020-8554. The group islocal.sdn.cozystack.io; the argument is in Positioning, and design-proposal: application endpoints and EndpointAttachment #45 now shares the group.EndpointAttachment), which attaches an application's endpoint to an address and consumes the claim and the association annotation defined here. This proposal does not wait on it.ServiceCIDR/IPAddressand explicitly named "any generalization onto something like an IPAM API" a non-goal;loadBalancerIPwas deprecated in 1.24 with no core replacement. Cluster API'sIPAddressClaim/IPAddressis literally this pattern but has only ever been wired to Machine addressing, with no reclaim policy and no association.The test that is the proposal
Claim → bind → associate → delete the Service → the address is still held → associate it to a different workload → the same address comes back.
That single integration test is precisely what cannot pass today.