Repository navigation
Alert when a cell can't tell which location it serves - #903
Conversation
|
|
|
|
A cell that cannot work out its location allocates no interface addresses and publishes no workload records, but the write-back said nothing about it. It now logs an error and reports the waiting metric, and a new alert fires when the metric stays set for 10 minutes. Key changes: - vpc write-back reports location identity and logs while unresolved - CellLocationUnresolved alert with promtool tests and a runbook
5150238 to
0ef4da8
Compare
kevwilliams
left a comment
kevwilliams
left a comment
There was a problem hiding this comment.
Verified reportLocationIdentity resets the gauge each call, so a recovered cell clears its old reason series and the alert resolves.
ecv
left a comment
ecv
left a comment
There was a problem hiding this comment.
Approve. The alert reads a gauge the code emits (nso_cell_location_identity_waiting, internal/controller/metrics.go:175, set by reportLocationIdentity), and both callers resolve the same identity, so the shared reset does not flap. The reason values match the locations module constants, and the runbook's fix matches the resolver.
Follow-ups:
- The test rules file is a hand copy of the shipped alert, byte-identical today, with nothing keeping them equal. A check across all alerts under test/prometheus-rules would close that.
A cell that can't work out its location allocates no interface addresses and publishes no workload records to the edge, so HTTP proxies from other locations can't reach its workloads. Until now the record publishing side waited on this silently. It now logs an error and reports the existing waiting metric, and a new warning alert fires when a cell has been unable to name its location for 10 minutes, with a runbook explaining the two causes and the fix. This is the follow-up promised on #896.
Test plan
Related to datum-cloud/enhancements#884