Releasing a kept interface leaves its network stuck deleting #664

Activity

1. drewr commented on Oct 17, 2026

   

   drewr

   on Oct 17, 2026

   ContributorAuthor

   More actions

   The project's NetworkInterface is written by the interface projector, which copies from the hub into the project and never back. The release command deletes that copy, so the cell's interface and its finalizer, which deletes the IPClaim and the retained IPAllocation, never run. The network's location then waits on a subnet with one allocation still inside it, and the network waits on the location.

   The fix for retained addresses in d050316 covers deleting the cell's interface. Release never reaches it.

   The repro network retain-repro-net in staging project project-xx4hh is still stuck. Its location retain-repro-net-us-central-1 reports one allocation inside its IPv6 subnet. Deleting the cell's interface retain-repro-default-us-central-1-0-eth0 should let it finish.
2. vvoytovych

   self-assigned this

   on Oct 19, 2026
3. vvoytovych commented on Oct 19, 2026

   

   vvoytovych

   on Oct 19, 2026

   Contributor

   More actions

   @drewr, I'll check/fix it.
4. vvoytovych commented on Oct 19, 2026

   

   vvoytovych

   on Oct 19, 2026

   Contributor

   More actions

   I looked into this, and the release cannot work as built, because two sound decisions collide.

   The interface a user sees in their project is a display copy, restorable on purpose since 602795a. A deleted copy comes back, and nothing carries the deletion toward the edge.

   The release command from #634 (667780f) deletes that copy and tells the user the addresses are given up. The edge never hears of it, so the addresses stay held and the network cannot finish deleting.

   One project in production has had 7 networks stuck deleting since 2026-10-17, held by their kept interfaces. I'll clean it up separately.

   I'd fix it in three steps, base first:

   1. Stop the harm. datumctl network interface release refuses with a clear message until a real release exists. Today it reports success for something that does not happen, and it ships in the published network plugin.
   2. Make the base hold. Users can read the interface copies in their project but not delete them, through the project's permissions or an admission policy. A copy is a view, so a delete from kubectl or the portal misleads the same way.
   3. Add a real release. A release becomes a request that travels to the edge the same way a project's other intent does. The edge deletes the interface itself, and its existing cleanup (d050316) gives the addresses back.

   One product question: should deleting a network release its kept interfaces? Today a network cannot finish deleting while one of them still holds addresses.

   Nothing is built yet. I'd like your view on the order and on the question first.
5. vvoytovych commented on Oct 19, 2026

   

   vvoytovych

   on Oct 19, 2026

   Contributor

   More actions

   The stuck project is cleaned up: its 29 kept interfaces were deleted at their edge locations, and its 7 networks finished deleting at 14:09 UTC.
6. vvoytovych commented on Oct 21, 2026

   

   vvoytovych

   on Oct 21, 2026

   Contributor

   More actions

   @scotwells @mattdjenkinson, a nudge on my plan above. It was meant for you, and I did not say so. You wrote the two parts involved. The project's copy of a network interface mirrors the cell's interface and comes back when deleted (602795a). datumctl network interface release deletes that copy (667780f). So release tells the user the addresses are given up, but the cell's interface keeps them, and the network cannot finish deleting.

   1. Is this order right? First, release refuses with a clear message until a real release exists. Then users can no longer delete the project's copy. Then a release reaches the cell's interface, whose cleanup (d050316) gives the addresses back.
   2. Should deleting a network also release its kept interfaces? The answer changes the first acceptance criterion in the issue description.

   Step 1 matters most, because the published plugin still reports a release that does not happen ("Network interface %q released."). If step 1 is fine on its own, I can start it now while we settle the rest.
7. scotwells commented on Oct 22, 2026

   

   scotwells

   on Oct 22, 2026

   Contributor

   More actions

   @vvoytovych hey viktor, sorry missed this before.

   1. Yes, this is the correct order.
   2. Yes, deleting a network should delete any kept interfaces.

   👍React with 👍1Reacted by Victor Voytovych (private)
8. vvoytovych

   added a parent issue on Nov 3, 2026

   • Make network services settle and finish deleting #854
9. vvoytovych

   added the

   Bug

   issue type on Nov 3, 2026
10. vvoytovych commented on Nov 9, 2026

    

    vvoytovych

    on Nov 9, 2026

    Contributor

    More actions

    The first box holds in production, so I'm closing this.

    • On 2026-11-09 13:31 UTC I ran the test in my own production project, on NSO v0.35.0. A workload's interface was kept (reclaimPolicy: Retain). The workload was deleted, and its interface stayed Available. Then I deleted the network.
    • The network was gone within a minute of its delete, with no interface or network context left. A network waits until its location has given back its addresses, so this also shows the kept interface's address was returned.
    • fix: Refuse to release a kept interface until release reaches the edge #743 makes release refuse until a release reaches the edge. fix: Give up a network's kept interfaces when the network is deleted #828 gives up a network's kept interfaces when the network is deleted.

    Releasing one kept interface while its network stays moves to the second box of #854, as the description now says.
11. vvoytovych

    closed this as completedon Nov 9, 2026