Skip to content

fix: Bring back every network presence when the controller starts - #844

Merged
vvoytovych merged 4 commits into
mainfrom
fix/presence-reconciles-every-presence-on-start
Nov 3, 2026
Merged

vvoytovych merged 4 commits into
mainfrom
fix/presence-reconciles-every-presence-on-start

Conversation

@vvoytovych

@vvoytovych vvoytovych commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Since #828, a deleted Network waits until the presence controller ends each of its contexts, but the controller heard of a presence only through the hub bindings and the interface copies that cells publish (SetupWithManager). A presence with neither, in its grace or declared only in its project, came back only when its wait ran out, and a restart or a change of leader takes that wait with it. So its grace never ended, and a Network deleted later stayed deleting: v0.34.0 deleted the context in the Network's finalizer, so this is new in #828 and not released yet. Staging holds two such contexts today, unclaimed since 2026-10-30 and three days past the 24-hour grace.

Changes

change file
The presence controller reconciles the hub's copies of the contexts as its primary type (For). The bindings and the interface copies stay as secondary watches networkpresence_controller.go

A presence's key is its hub copy's namespace and name. The replicator keeps that copy on the hub while the context exists and is not being deleted. So the initial list on start brings back every presence, and RequeueAfter goes back to re-checking a wait, as the kubebuilder book recommends. A standby that takes the lease over gets the list too: client-go sends every cached object to a handler added after its informer started (shared_informer.go). The grace still comes from the unclaimed-since annotation, so a restart neither restarts it nor skips it.

Corner cases

case what happens proof
A presence in its grace, with nothing at its location. The controller restarts, then the Network is deleted The new process lists the copy and takes up the wait. The deletion reaches the presence at its next resync, the context ends, and the Network goes TestANetworkDeletedAfterARestartReachesAPresenceNothingDeclares
The same presence, and its grace ends after the restart The presence ends on time TestAPresenceNothingDeclaresEndsAtItsGraceAfterARestart
A presence declared only in its project, written by the presence controller or by the binding controller. The controller restarts, then the Network is deleted As in the first case TestANetworkDeletedAfterARestartReachesAPresenceDeclaredInItsProject
A standby takes the lease over, with its cache already full Nothing acts while another replica holds the lease. The new leader brings every presence back TestANewLeaderBringsBackEveryPresence
The presence's own write changes its copy The change brings the presence back once. A presence with nothing to change writes nothing TestAPresenceWritesNothingWhenOnlyItsHubCopyChanges
A copy goes while its presence waits Nothing ends. The presence reads its project, never the copy TestAPresenceOutlivesItsHubCopyGoing
A copy in a namespace that is not a hub namespace, where one cluster plays both roles Left alone, as a binding there is TestNetworkPresenceLeavesProjectPlaneBindingsAlone
A reconcile keyed on a copy Never writes or deletes the copy TestNetworkPresenceLeavesTheReplicatedHubCopyAlone

Cost

  • One informer of NetworkContexts on the hub, in the presence controller's manager: one object per presence. The replicator's own cache already holds the same list.
  • At start, one reconcile per presence, from cached reads.
  • No new permission. The role already allows get, list and watch on NetworkContexts, and the replicator watches them on the hub with the same identity.
  • SetupWithManager: gocyclo 3 → 3, gocognit 3 → 3.

Test plan

  • On main, the restart, grace, declared-in-project, standby and copy-change tests fail, and the one-process control passes. All pass with the change, and the restart test passes on v0.34.0, whose finalizer deletes the context.
  • A planted defect per mechanism turns its tests red, 4 of 4. golangci-lint v2.15.0 and CI's suites pass.
  • On kind, with the NSO manager restarted through its lease in each trial, a presence that nothing declares and no interface holds ended at its grace in the new process, and its Network, deleted after the restart, then went through its own finalizer: 2 of 2 each. On main both stayed, 1 of 1 each. The e2e grace is 15 s, about one restart, so a deletion inside the grace after a restart is the envtest test's here, and staging's below.
  • On staging, the bug before this change: after a restart on main's build, a network in our test project whose presence nothing declared was deleted at 09:36:29Z and was still deleting 90 s later. After the merge, its build started on the hub at 10:12:58Z, the presence controller ended that presence at 10:13:01Z, and the network went.

Related to #664

A presence that no hub binding declares and whose location publishes no
interface comes back only when the wait its process set runs out. These
tests run the presence controller under a real manager and replace it,
or hand its lease to a standby, before the network is deleted or before
the grace ends. On main nothing brings such a presence back: its grace
never ends, and the deleted network's finalizer waits for its context
for good. The same steps in one process pass.

They also pin what reconciling the hub's copies must not change: a copy
that changes alone brings no write, and a copy that goes ends nothing.
A presence that no hub binding declares and whose location publishes no
interface came back only when the wait its process set ran out. A
restart or a change of leader took that wait with it, so nothing looked
at the presence again: its grace never ended, and since #828 a network
deleted later waited for its context for good.

The presence's key is the key of the hub's copy of its context, and the
copy lasts as long as the presence does. So the controller now
reconciles those copies as its primary type, with the bindings and the
interface copies as secondary watches, and the list of copies on start
brings back every presence.
Such a presence has no hub binding either, and may never have had one:
with internal DNS on, the binding controller writes the context for a
project-plane binding, network UID label included. Both fail on main
after a restart and pass when the controller reconciles the hub's copies.
@vvoytovych
vvoytovych marked this pull request as ready for review November 3, 2026 09:51
@vvoytovych
vvoytovych requested a review from a team as a code owner November 3, 2026 09:51
@vvoytovych
vvoytovych merged commit d80150e into main Nov 3, 2026
11 checks passed
@vvoytovych
vvoytovych deleted the fix/presence-reconciles-every-presence-on-start branch November 3, 2026 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants