[vvoytovych]

Every network claims its fabric identity as an IPClaim in the platform's one project, under that project's quota on claims. collect() removed a gone network's identity and kept its claim, so each gone network held a quota unit forever. Once the quota is full, no new network anywhere gets an identity, while existing networks keep working. In our production this filled the platform project's claim quota, and new networks waited for an identity.

Summary

 collect(network is gone, as the hub's cache reads it)
+  read the network past the cache; if it lives, or the read fails, collect nothing
   delete the NetworkFabricIdentity
   unplace it
+  fabricidentity.Release: read the claim and its bound IPAllocation
+    delete the claim only if the allocation is recorded Retain; NotFound counts as released

Releasing the claim keeps what keeping it protected. Under Retain the address service keeps the allocation under the claim's name, so the block is never reissued. The policy that decides is the one recorded on the allocation when the claim bound, so Release checks that one. A network of the same name that returns adopts it, through the existing RetainedAllocation path.

Test plan

• Reproduced first: the fake address service now models the platform quota and Retain on delete. TestAGoneNetworkGivesItsQuotaUnitBack fails on main, where the next network is refused for quota, and passes with the fix.
• Edge cases, through the reconciler. A returning name adopts its block. A release refused once is retried by the next reconcile. A network the cache reads as gone while it lives keeps its identity, its placement and its claim. So does one whose direct read fails. A claim whose allocation is recorded other than Retain is kept, with an error naming it. Each case fails on its mutant, and the fake now records each allocation's policy as the address service does.
• go test ./internal/cloud/... passes, also under -race, and golangci-lint v2.12.2 reports 0 issues on both changed packages. The full lint, with its eBPF build, runs on this draft.
• On staging: the identity claims follow the live networks, and the quota's allocated count falls back after networks are deleted.

Merge danger

The direct read closes one window: a network deleted and recreated under one name can read as gone in the cache while it lives. Collected then, it would lose its identity and placement for a reconcile. Its block would also stay held with no claim, uncounted by the quota.

The platform identity needs delete on ipclaims. Without it, collect() returns the refusal and retries; the identity and its placement are already gone.

A two-way door: a revert keeps claims again. A claim released meanwhile is not restored. Release deletes only a claim whose allocation is recorded Retain, so its block stays held and no identity is lost or reissued. The blast radius is the fabric identity controller.

[ecv]

Approve.

Releasing the claim last, behind a direct read of the network, gives the quota unit back while ipam keeps the block retained. Checked against milo-os/ipam v0.5.5, the version galactic pins: a Retain delete nulls the claim key and keeps the allocation, so the block is not reissued. All checks pass at 480b741, and removing the direct-read guard fails TestALiveNetworkThatReadsGoneKeepsItsClaim and TestAnUnconfirmedGoneNetworkKeepsItsClaim.

• The direct read uses hub.GetAPIReader() under the existing get on networks. Any error other than NotFound fails the reconcile before an ipam call, so the claim is kept.
• A same-name network re-created mid-release gets 409 AllocationRetained from ipam and adopts the same block. No second block is claimed.
• A refused release returns an error and requeues on per-key backoff. It does not hot-loop, block other networks or recreate the collected identity.

Follow-ups:

• Production's Milo RBAC grants only get, create on ipclaims; infra 9217 adds delete in staging only. Grant it in production before the galactic pin moves, or every deleted network retries forever and the leak stays.
• A restart while a release is failing leaks the quota unit for good. The controller watches only Network and NetworkContext, and a deleted network is not in the initial list, so nothing re-enqueues it. A startup sweep that releases fabric-identity.* claims with no network, like sweepLegacyPlacements, would close it.
• A network that adopts its retained block seems to hold it with no IPClaim, so quota does not count it while it lives. Worth confirming and recreating the claim on adopt.
• Quota admission runs before the adopt path, so a returning network at the limit may be refused even though it adds no block. TestAReturningNetworkAdoptsTheBlockItGaveUp cannot show this. Decide the expected behavior and test it.

[vvoytovych]

Each follow-up, and where it lives:

1. Production RBAC. Datum's deployment grants delete on ipclaims in production before production moves to a build with this change. That promotion gates the production pin.
2. A restart while a release fails. A startup sweep, run once on the leader, releases the fabric-identity.* claims whose network is gone: Release fabric identity claims of gone networks at startup #1382.
3. An adopted block without a claim. IPAM refuses a claim over a retained allocation by design, so the fix belongs there, as an opt-in adopt on create: Let a claim adopt the allocation its name retained milo-os/ipam#176.
4. Quota before adopt. Decided behaviour: a returning network at a full quota is refused like any network, retried, and adopts once a unit frees. A follow-up PR adds its test. Reading the retained allocation first would tie galactic to IPAM's private allocation naming.

[ecv]

Approve.

The delta since 480b741 tightens both ends. The direct read now gates the whole collect, so nothing is withdrawn on a stale cache. Release deletes only a claim whose bound allocation is recorded Retain, under a UID precondition. A freed block can't be reissued, and a claim re-created mid-release isn't the one deleted.

• The new read of the allocation needs get on ipallocations, which Milo's RBAC already grants.

Follow-ups:

• A claim with no bound allocation, or one bound to a non-Retain allocation, now errors on every reconcile and retries on backoff forever. If either can happen in practice, treat it as terminal: log it once, emit a metric and stop requeueing.
• The earlier follow-ups still stand: production's delete grant, the restart leak, and the adopted block with no claim.

[vvoytovych]

On the follow-up: neither case arises through IPAM's API, so I keep the retry.

• IPAM v0.5.6 creates a claim and its allocation in one transaction. A claim it stores is bound.
• A bound allocation refuses a delete (ValidateDelete), so a claim cannot lose its allocation that way.
• galactic asks for Retain on every identity claim.
• Read today in production: every identity claim is Bound, each to an allocation recorded Retain. Staging read the same before today's cleanup.

The paths left are a hand edit of ipclaims/status, a defect in IPAM, or a restore of its storage. In each, an error logged on every retry is the signal we want, and stopping would hide it. Before production, we scrape this controller's metrics or add a log rule, so that a release refused on every retry raises an alert.

[scotwells]

I need a bit more time to review this. When I built out the network identity stuff originally, I intentionally had IP claims be retained so that a network identity would never be re-used across VPCs.

[scotwells]

Thought through this one a bit more and I think I'd be comfortable with a Release behavior after some sort of cooldown so that VPC IDs are released immediately by the IPAM system. I'd want this capability built into the IPAM system though so it can be a generic capability that's re-used across all services.

I'd also like to wait for @privateip's thoughts here so he has a chance to chime in.