Skip to content

fix: Give policy alerts without an owner activity's owner - #352

Merged
vvoytovych merged 1 commit into
mainfrom
fix/dlq-alerts-owner-fallback
Nov 12, 2026
Merged

vvoytovych merged 1 commit into
mainfrom
fix/dlq-alerts-owner-fallback

Conversation

@vvoytovych

Copy link
Copy Markdown
Contributor

A dead-letter alert takes its service and team from the failing policy's owner, so a policy that declares no owner alerts with no service, which no route reaches, and the alert lands in the catch-all channel, as identity policies did in both environments over the last two weeks.

Such an alert now falls back to activity's own owner, the control plane service and the platform team, as the platform's catalog names the owner of the namespace activity runs in.

A policy that declares an owner keeps it, so its failures still reach its own team.

Changes

Change Effect
The five policy alerts fall back to the control plane service and the platform team when the policy declares no owner
The alert for failed dead-letter publishes names the same owner
The alert tests expect the fallback for policies without an owner, and keep the declared owner for those with one

Test plan

  • The alert tests fail on the previous fallback and pass on this one, and a declared owner is kept
  • Routed through both environments' live alert trees, the fallback reaches the platform team's channel and a declared owner reaches its own
  • After the release reaches production, the next alert for a policy without an owner posts to the platform team's channel
  • For a week after the release, no activity policy alert reaches the catch-all channel in either environment

A DLQ alert takes its service and team from the failing policy's owner. A
policy that declares no owner fired with no service and team sre, which no
route keys on, so the alert fell to the catch-all channel. It now falls back
to activity's own owner, service control-plane and team platform, as the
platform's catalog names the owner of activity-system. A policy that
declares an owner keeps it.

Key changes:
- The five policy alerts fall back to service control-plane and team
  platform when the policy declares no owner
- The processor alert for failed DLQ publishes names the same owner
- The alert tests expect the fallback for policies without an owner
@vvoytovych vvoytovych self-assigned this Nov 12, 2026
@vvoytovych
vvoytovych marked this pull request as ready for review November 12, 2026 11:02
@vvoytovych
vvoytovych requested review from ecv and scotwells November 12, 2026 11:03

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

Dead-letter alerts for a policy with no declared owner now go to the platform team's channel instead of the catch-all. A policy with an owner keeps routing to its own team.

  • In the unless branch the series has no policy_service, so label_replace(..., "service", "$1", "policy_service", "(.*)") sets service to empty. That drops any scrape-time service label, so the template's control-plane fallback is what lands, not the Kubernetes Service name.
  • The tests cover both the fallback and a declared owner.

@vvoytovych
vvoytovych merged commit 3938d30 into main Nov 12, 2026
10 checks passed
@vvoytovych
vvoytovych deleted the fix/dlq-alerts-owner-fallback branch November 12, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants