Repository navigation
Publish VPC backend records as soon as a proxy needs them - #896
Conversation
|
|
ecv
left a comment
There was a problem hiding this comment.
Holding. Removing the one-minute resync drops the only retry for several paths where publish returns nil without a requeue, so a slice can stay unpublished, or a hub copy stay missing, until the pod changes or the operator restarts.
- The hub namespace not there yet (
vpcendpointslice_writeback.go:182-186) logs and returns nil. A new namespace appearing on the hub is no event for either controller, and the EdgeReachability write swallows the same NotFound (edgereachability_controller.go:101-107), so its create event may never come. - An unresolved location (
:109-111) returns nil, and the startup sweep then no-ops, returns nil and never runs again. - A hub copy that Karmada or a person deletes or edits is never re-applied, since nothing watches the hub's EndpointSlices. The edge then has no backend for that pod.
collectreads the copy through the hub cache (:362). A slice deleted before the cache sees its fresh copy leaves the copy orphaned, which the ten-minute sweep used to collect.
Ask: requeue with a delay on those nil returns, or keep a slow periodic republish and sweep as a backstop next to the new watch.
|
@ecv thanks, all four were real gaps. We went with watches rather than a timed requeue or a periodic backstop, so every wait is tied to the event it waits on; failed writes still retry through the controller's normal backoff.
Controller tests cover the reachability, hub copy and hub namespace paths, and a new e2e scenario against the real cell and hub checks that a record is published, restored after deletion, and withdrawn, each within 10s. The ServingLocation watch has no test of its own yet. Generated by Claude Code |
ecv
left a comment
There was a problem hiding this comment.
Approve. Each gap from my hold now has an event behind it:
- A hub namespace's creation enqueues its reachability record, whose write then republishes the namespace's slices.
- A ServingLocation arriving or changing republishes everything and sweeps.
- A copy deleted or edited on the hub maps back to its source slice and is put back, or collected when the source is gone.
The hub cache already held EndpointSlices for the copy reads, so the new copy watch adds no informer.
The cell write-back learned that a pod had been put behind a proxy only on a one-minute resync, so a new cross-location proxy waited up to a minute before the edge could reach its backend. It now watches the hub's edge reachability records and republishes that namespace's slices the moment one changes. Key changes: - watch EdgeReachability on the hub and republish only its namespace - drop the one-minute resync of every slice - run the sweep for orphaned copies once at startup, with backoff, instead of every ten minutes; a deletion seen while the hub is down is retried by the workqueue, so only a restart can miss one
The one-minute resync also restored a hub copy deleted or edited out from under the cell. Watch the cell's own copies on the hub so that is now an event too, and add an e2e scenario against the real cell and hub that fails if any of these waits for a periodic pass. Key changes: - watch this location's copies on the hub and republish their source - e2e: a slice is published, restored and withdrawn within 10s of the reachability record or the hub copy changing
Every case where the write-back held a slice back used to be picked up again by the one-minute pass. Without it, each needs a watch on what it waits for. Key changes: - watch namespaces appearing on the hub, in the cell write-back and in the edge reachability writer, so a record held for a missing hub namespace is written once federation makes it - watch the cell's ServingLocation, when served, and republish and sweep once the location is known or changes - watch project labels on the cell's namespaces
A cell's project namespace is propagated from its hub namespace, so a slice cannot reach the cell before the hub namespace exists. A missing one means the project is being torn down and there is nothing to wait for. Key changes: - remove the cell's watch on hub namespaces, which needed a hub permission the cell does not hold - remove the cell namespace label watch; labels arrive with the namespace
5669c6f to
3b8b494
Compare
kevwilliams
left a comment
There was a problem hiding this comment.
Verified concurrent publishes from the four controllers self-heal through the copy watch, and the e2e's 10s deadlines fail against the old one-minute resync.
A new HTTP proxy whose workload runs in another location can wait up to a minute before the edge can reach it, because the cell only notices that a workload has been put behind a proxy on a once-a-minute pass. This change drops every timer and makes the cell react to the events it was waiting on: the hub's reachability record changing, a published copy being deleted or edited on the hub, the cell's location being delivered, and the hub namespace appearing for a project's reachability record. Failed writes are still retried with the framework's normal backoff, and the cleanup of leftover copies runs once at startup and again whenever the cell's location changes. No new permissions are needed.
Test plan
Related to datum-cloud/enhancements#884