Skip to content

Publish VPC backend records as soon as a proxy needs them - #896

Merged
scotwells merged 5 commits into
mainfrom
claude/project-thread-nageha
Nov 10, 2026
Merged

scotwells merged 5 commits into
mainfrom
claude/project-thread-nageha

Conversation

@scotwells

@scotwells scotwells commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

A new HTTP proxy whose workload runs in another location can wait up to a minute before the edge can reach it, because the cell only notices that a workload has been put behind a proxy on a once-a-minute pass. This change drops every timer and makes the cell react to the events it was waiting on: the hub's reachability record changing, a published copy being deleted or edited on the hub, the cell's location being delivered, and the hub namespace appearing for a project's reachability record. Failed writes are still retried with the framework's normal backoff, and the cleanup of leftover copies runs once at startup and again whenever the cell's location changes. No new permissions are needed.

Test plan

  • Controller test suite passes locally, with new tests for the reachability, hub copy and hub namespace watches
  • New end-to-end scenario: a record is published, restored after deletion and withdrawn, each within 10 seconds, against the real cell and hub
  • CI lint and e2e suites pass
  • After merge, a new cross-location proxy in staging answers faster than the 34s measured after the last fix

Related to datum-cloud/enhancements#884

@scotwells
scotwells requested a review from a team as a code owner November 10, 2026 12:12
@scotwells
scotwells requested review from ecv and kevwilliams November 10, 2026 12:12
@cla-assistant

cla-assistant Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@cla-assistant

cla-assistant Bot commented Nov 10, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding. Removing the one-minute resync drops the only retry for several paths where publish returns nil without a requeue, so a slice can stay unpublished, or a hub copy stay missing, until the pod changes or the operator restarts.

  • The hub namespace not there yet (vpcendpointslice_writeback.go:182-186) logs and returns nil. A new namespace appearing on the hub is no event for either controller, and the EdgeReachability write swallows the same NotFound (edgereachability_controller.go:101-107), so its create event may never come.
  • An unresolved location (:109-111) returns nil, and the startup sweep then no-ops, returns nil and never runs again.
  • A hub copy that Karmada or a person deletes or edits is never re-applied, since nothing watches the hub's EndpointSlices. The edge then has no backend for that pod.
  • collect reads the copy through the hub cache (:362). A slice deleted before the cache sees its fresh copy leaves the copy orphaned, which the ten-minute sweep used to collect.

Ask: requeue with a delay on those nil returns, or keep a slow periodic republish and sweep as a backstop next to the new watch.

@scotwells
scotwells requested a review from ecv November 10, 2026 13:17

scotwells commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

@ecv thanks, all four were real gaps. We went with watches rather than a timed requeue or a periodic backstop, so every wait is tied to the event it waits on; failed writes still retry through the controller's normal backoff.

  • Hub namespace not there yet: the EdgeReachability writer now watches namespaces being created on the hub, so a record skipped for a missing namespace is written as soon as federation makes it. In the cell write-back this case can't happen while a project is live: the cell's project namespace is propagated from the hub namespace, so the hub one always exists first, and a missing one means the project is being torn down. It stays a no-op, which also avoids an infra RBAC change, since the cell's hub identity has no namespace access.
  • Unresolved location: the cell watches ServingLocation (when the kind is served). Once the location resolves or changes, it republishes every slice and reruns the sweep.
  • Hub copy deleted or edited: the cell watches its own copies on the hub (filtered to its location) and republishes the source slice. Writes are server-side apply, so an unchanged republish writes nothing.
  • Orphaned by a stale cache: covered by the same watch. When the cache sees the copy, the event maps back to a source that's gone, and publish collects it.

Controller tests cover the reachability, hub copy and hub namespace paths, and a new e2e scenario against the real cell and hub checks that a record is published, restored after deletion, and withdrawn, each within 10s. The ServingLocation watch has no test of its own yet.


Generated by Claude Code

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Each gap from my hold now has an event behind it:

  • A hub namespace's creation enqueues its reachability record, whose write then republishes the namespace's slices.
  • A ServingLocation arriving or changing republishes everything and sweeps.
  • A copy deleted or edited on the hub maps back to its source slice and is put back, or collected when the source is gone.

The hub cache already held EndpointSlices for the copy reads, so the new copy watch adds no informer.

The cell write-back learned that a pod had been put behind a proxy only on a
one-minute resync, so a new cross-location proxy waited up to a minute before
the edge could reach its backend. It now watches the hub's edge reachability
records and republishes that namespace's slices the moment one changes.

Key changes:
- watch EdgeReachability on the hub and republish only its namespace
- drop the one-minute resync of every slice
- run the sweep for orphaned copies once at startup, with backoff, instead
  of every ten minutes; a deletion seen while the hub is down is retried by
  the workqueue, so only a restart can miss one
The one-minute resync also restored a hub copy deleted or edited out from
under the cell. Watch the cell's own copies on the hub so that is now an
event too, and add an e2e scenario against the real cell and hub that fails
if any of these waits for a periodic pass.

Key changes:
- watch this location's copies on the hub and republish their source
- e2e: a slice is published, restored and withdrawn within 10s of the
  reachability record or the hub copy changing
Every case where the write-back held a slice back used to be picked up again
by the one-minute pass. Without it, each needs a watch on what it waits for.

Key changes:
- watch namespaces appearing on the hub, in the cell write-back and in the
  edge reachability writer, so a record held for a missing hub namespace
  is written once federation makes it
- watch the cell's ServingLocation, when served, and republish and sweep
  once the location is known or changes
- watch project labels on the cell's namespaces
A cell's project namespace is propagated from its hub namespace, so a slice
cannot reach the cell before the hub namespace exists. A missing one means
the project is being torn down and there is nothing to wait for.

Key changes:
- remove the cell's watch on hub namespaces, which needed a hub
  permission the cell does not hold
- remove the cell namespace label watch; labels arrive with the namespace
@scotwells
scotwells force-pushed the claude/project-thread-nageha branch from 5669c6f to 3b8b494 Compare November 10, 2026 14:41
@scotwells
scotwells requested a review from ecv November 10, 2026 14:41

@kevwilliams kevwilliams left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified concurrent publishes from the four controllers self-heal through the copy watch, and the e2e's 10s deadlines fail against the old one-minute resync.

@scotwells
scotwells merged commit 59dac55 into main Nov 10, 2026
10 checks passed
@scotwells
scotwells deleted the claude/project-thread-nageha branch November 10, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants