You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Warn when a Network or Gateway stays deleting #830
A deletion that one of NSO's finalizers holds can wait for good, and nothing warns. A Network waits until its locations hold none of its interfaces: #828 gives up kept interfaces through the copies each location publishes to the hub, and if a copy is missing the kept interface is never given up, so the Network waits as it did before #828. A Gateway waits for its finalizer to detach its routes and remove its DNS records and its copies on the edge, and a step that keeps failing holds it the same way. Each kind's Ready condition or log names the step, but no alert reads it.
Proposed scope
One metric for what NSO's finalizers hold: the time each held object was deleted, with its kind, project, namespace and name. It is set while the finalizer holds the object and dropped when the finalizer goes or the object is found gone. One series per object, because the alert must name the object to act on, and series exist only while a deletion is held, so their number is the deletions in flight, not the fleet.
A replica drops a project's series when it stops reconciling the project, so a hand-over leaves none behind.
One alert in the shape of Milo's ProjectStuckDeleting (milo-os/milo#884): a warning once an object has waited more than an hour, held for 15 minutes.
Network and Gateway first. The network-deletion-stuck skill names the alert, and a location whose copies are missing on the hub as one cause to check.
Non-goals
No change to how a deletion proceeds.
Acceptance criteria
In promtool tests, a Network or a Gateway held for more than an hour raises one warning, and a deletion that finishes raises none
The series goes when the finalizer goes, and when the object is found gone
A replica that stops reconciling a project drops that project's series
Activity