Skip to content

fix: Let a context settle when its network's DNS is off - #847

Merged
vvoytovych merged 1 commit into
mainfrom
fix/internal-dns-disabled-region-loop
Nov 3, 2026
Merged

vvoytovych merged 1 commit into
mainfrom
fix/internal-dns-disabled-region-loop

Conversation

@vvoytovych

Copy link
Copy Markdown
Contributor

Since internal DNS was turned on in production on 2026-10-30, the hub's project reconciler rewrites a NetworkContext several times a second when the context's Network has DNS off and its location has an internal-DNS region. Each such context today is the internal DNS producer network's: one in production (us-central-1), two in staging. The reconcile stamps the region annotation, then disable clears it with the rest of the intent. Each write is an event on the context it reconciles, so it never settles, and the replicator and Karmada carry every change to the cells.

Changes

change file
The reconcile decides that the Network's DNS is off before it stamps the region internaldns_controller.go

disable then cleans up with the region the context last carried, which is the region of any access binding it made.

Corner cases

case what happens proof
DNS off, in a location with a region The second reconcile writes nothing TestInternalDNSProjectSettlesWhenDNSIsOff/in_a_location_with_a_region
DNS off, in a location without a region The same, as before TestInternalDNSProjectSettlesWhenDNSIsOff/in_a_location_without_a_region
DNS turned off after it was on The first reconcile removes the intent, the finalizer and the regional access. The second writes nothing TestInternalDNSProjectSettlesWhenDNSIsOff/after_DNS_was_on
DNS on Unchanged TestInternalDNSProjectLifecycleFencesAccess

Cost of the loop, per minute, read on 2026-11-03

where writes
production, Milo: NetworkContext PATCH 434
production, Karmada: NetworkContext PUT and status PATCH, ResourceBinding PUT, Work PUT 387, 774, 564, 370
production, us-central-1's cell manager: PUT 2100
staging, Milo: NetworkContext PATCH 259

The change adds no read and no write. reconcile: gocyclo 30 → 30, gocognit 39 → 39.

Test plan

  • On main, both cases in a location with a region fail: the second reconcile writes the context twice. All pass with the change.
  • A planted defect turns the test red, 4 of 4: the old order, and disable without its cleanup, its finalizer removal or the region in the intent. golangci-lint v2.15.0 and internal/controller's tests pass.
  • CI passes. On staging after the merge, whose hub follows main's builds, the two producer contexts stop changing and Milo's NetworkContext PATCH rate falls near 0.

The project reconciler stamped the internal-dns-region annotation and then,
because the network's DNS is off, cleared it again with the rest of the
intent. Each write is an event on the context it reconciles, so it never
settled. Decide that DNS is off before the region is stamped.
@vvoytovych vvoytovych self-assigned this Nov 3, 2026
@vvoytovych
vvoytovych marked this pull request as ready for review November 3, 2026 13:00
@vvoytovych
vvoytovych requested a review from a team as a code owner November 3, 2026 13:00
@vvoytovych
vvoytovych merged commit 23342fd into main Nov 3, 2026
11 checks passed
@vvoytovych
vvoytovych deleted the fix/internal-dns-disabled-region-loop branch November 3, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants