Skip to content

Add native AWS CloudWatch and Azure Monitor alert ingress - #4535

Open
sarora-eightfold wants to merge 2 commits into
target:masterfrom
EightfoldAI:native-cloudwatch-azuremonitor-ingress
Open

Add native AWS CloudWatch and Azure Monitor alert ingress#4535
sarora-eightfold wants to merge 2 commits into
target:masterfrom
EightfoldAI:native-cloudwatch-azuremonitor-ingress

Conversation

@sarora-eightfold

Copy link
Copy Markdown

Summary

Adds two native alert-ingress integrations, cloudwatch and azuremonitor, so CloudWatch alarms and Azure Monitor alerts can reach GoAlert directly with no forwarder in between. They're siblings: same handler shape, same payload-mapper structure, same dedup discipline, same smoke-test layout.

Note on size: this is two related integrations landing together and comes in well over the repo's 500-line PR-size guideline (the labeler check will flag it XL). I'd rather be upfront about that than have it be a surprise. Happy to split into smaller sequential PRs (key-type + migration, then handler, then payload mapper, then tests, per integration) if that's preferred — just say so and I'll re-split.

Why not the Universal Integration Key

Since both of these are "just point a webhook at GoAlert," the natural question is why not build them on UIK (@experimental(flagName: "univ-keys")) instead of new key types. Two independent reasons:

  • CloudWatch specifically cannot work behind UIK. SNS requires the endpoint to fetch a SubscribeURL to confirm the subscription — without that handshake, the subscription stays PendingConfirmation forever and zero messages are ever delivered. UIK has no mechanism to make that outbound confirmation GET. This is the actual blocker, not a stylistic preference.
  • The integration-key URL is generated server-side from the key type alone (IntegrationKey.Href). A UIK-based approach would still need the UI to know which Expr rules to scaffold per vendor, and would still leave CloudWatch's handshake problem unsolved. A first-class key type is a strict subset of that complexity.

Everything else about parsing, UIK could arguably do — the handshake is the reason CloudWatch needs its own type.

CloudWatch (via SNS)

CloudWatch alarms commonly reach on-call tooling through an SNS topic. GoAlert can't be subscribed to one directly today, for three independent reasons:

  1. SNS requires the endpoint to fetch a SubscribeURL to confirm the subscription, and nothing does that — so the subscription never leaves PendingConfirmation and zero messages are delivered.
  2. The SNS envelope has no summary field; the alarm is a JSON string inside Message.
  3. SNS always sends Content-Type: text/plain, which genericapi.ServeCreateAlert ignores because it only unmarshals application/json. This is the root cause of Messages from AWS SNS are blank #4463 ("Messages from AWS SNS are blank") — that issue was self-closed on a content-type theory that doesn't actually hold, since SNS can't be configured to send a different content type.

This adds a cloudwatch integration key type and POST /api/v2/cloudwatch/incoming, which performs the subscription handshake and verifies the SNS message signature, so a topic can be subscribed directly.

Notable implementation details:

  • Body is parsed regardless of Content-Type — that's the bug this whole feature exists to fix.
  • MaxBytesReader rather than io.LimitReader, so an oversized body yields a clean 413 instead of silently truncating into a misleading 400.
  • Signature verification is pure functions (canonical string, verify, cert parse) — no I/O, fully unit-testable. Subject is a *string because AWS omits the field from the string-to-sign entirely when absent, which a plain string can't distinguish from an empty value.
  • Both outbound fetches (signing cert, SubscribeURL) are host-allowlisted with an anchored pattern, and the client blocks redirects — the allowlist only covers the first hop, so a single 302 from an allowlisted host would otherwise reach link-local addresses.
  • The signing-cert cache is bounded (32 entries, FIFO) since the cert URL path is attacker-influenced and would otherwise grow without limit.
  • A freshness window on the signed Timestamp bounds replay of a captured envelope (reject >1h old or >5min future).
  • Dedup is hex(sha256(AlarmName)) — stable and derived from the one field guaranteed present and unique per alarm rule, so the same alarm produces the same key regardless of which delivery reaches GoAlert. Never nil: a nil dedup silently falls back to a content hash that changes on every state transition, which would break both idempotency and the OK-closes-the-alert path.
  • NewStateReason is capped before assembling details so a verbose reason can't push AlarmDescription (which carries the runbook URL) past the length limit.
  • INSUFFICIENT_DATA and a stray OK with no open alert both create nothing and return 2xx; non-2xx is reserved for infrastructure failure, so SNS only retries when a retry could actually help.

Security note, anticipating a static-analysis flag: a taint-analysis pass may flag the outbound http.Client.Do calls (cert fetch, SubscribeURL fetch) as "uncontrolled data in a network request," because the URL originates from the request body. This is a false positive in context — both call sites dial only the *url.URL returned by an anchored host-allowlist check (^sns\.[a-z0-9-]+\.amazonaws\.com(\.cn)?$, HTTPS-only, no userinfo, no port), never the raw string, and redirects are disabled via http.ErrUseLastResponse. Adversarial tests cover the unanchored-suffix bypass (sns.us-west-2.amazonaws.com.evil.com), scheme downgrade, and userinfo confusion — all rejected before any dial.

Verified against live AWS SNS: subscription confirms, a real alarm creates one alert with the runbook URL preserved, and re-delivery is suppressed as a duplicate.

Azure Monitor

Azure Monitor delivers alerts by having an action group POST a webhook. The generic endpoint can't serve this either: Azure sends application/json so it clears the content-type gate, but ServeCreateAlert expects a flat body while Azure nests everything under data.essentials / data.alertContext with no top-level summary — every delivery would create an alert with a blank summary rather than an error. Nothing maps Azure's alertId onto the dedup field either.

This adds an azureMonitor integration key type and POST /api/v2/azuremonitor/incoming.

Unlike SNS, Azure has no subscription handshake and no signature, so the handler makes no outbound requests at all — no analogue of cloudwatch's host allowlist or cert cache is needed. The integration key in the URL is the only credential, so the docs mark the webhook URL as credential-grade for this key type specifically.

Parsing:

  • Only the common alert schema (schemaId: "azureMonitorCommonAlertSchema") is accepted. A legacy-schema payload is rejected with a message naming the fix (enable the common alert schema on the action group's webhook receiver) rather than silently degraded, which would produce content-free alerts with no indication why. 400, not 5xx — Azure doesn't retry 4xx, and a misconfigured receiver is a permanent condition, not a transient one.
  • Dispatch is on the presence of condition.allOf, not a conditionType allowlist. Every metric and log criteria shape shares that envelope, so one code path covers SingleResourceMultipleMetricCriteria, MultipleResourceMultipleMetricCriteria (multi-resource / resource-group-scoped rules), DynamicThresholdCriteria, and WebtestLocationAvailabilityCriteria. conditionType still selects the two behaviours that genuinely differ: suppressing the dynamic threshold (a sensitivity artifact, not a limit) and the log query/link rendering.
  • Prometheus rule groups get their own branch — that shape has no conditionType and no condition, only expression/labels/annotations.
  • Service Health / activity-log payloads render from properties, with HTML stripped and string-containing-JSON fields left opaque rather than (failingly) parsed as JSON.
  • Anything unrecognised builds a best-effort alert from essentials alone (present on every payload regardless of type) and logs the signalType/monitoringService/conditionType triple, so a newly-encountered alert type is visible in logs instead of silently producing a thin alert forever.

signalType is deliberately not the discriminator: Platform and Prometheus metric alerts both report signalType: "Metric", and Log Alerts V2, Azure Backup, and ActivityLog Administrative all report "Log". Branching on it would feed unrelated payloads to the wrong renderer.

Dedup is sha256(essentials.alertId). Azure alerts are stateful — one alert object carries the whole lifecycle, and the Fired/Resolved deliveries of one firing share an alertId — which is what lets the Resolved delivery close the alert its Fired delivery opened. originAlertId is deliberately not used: it's per-rule for metric alerts, so a single missed close would hold the dedup key and mute that rule permanently. Status comes from essentials.monitorCondition, never alertContext.status, which can disagree with it (the underlying incident can resolve while the alert itself is still reported as fired).

A json.UnmarshalTypeError on an individual field is tolerated rather than fatal — Azure documents threshold and dimension values as strings but isn't consistent about it across shapes, and a hard failure means a 400 (which Azure won't retry), so one oddly-typed field would otherwise cost the whole page instead of just that one value.

Verified against Microsoft's documented common alert schema, a real tenant's alert-rule inventory spanning metric, log-v2, and Prometheus rule groups, and table-driven unit and smoke tests.

Tests

Both packages have table-driven unit tests (canonical string, allowlist, signature/freshness, and payload mapping for cloudwatch; schema gate, condition-shape dispatch, and payload mapping for azuremonitor) plus a smoke test each — the first unit tests in any GoAlert ingress package. go build ./..., go vet, and gofmt are clean; golangci-lint reports 0 issues on this diff.

Out of scope

Neither integration adds an outbound notification channel (that's #552, the opposite direction). No new dependency for crypto, HTTP, or cloud SDKs — both handlers are stdlib-only; cloudwatch never calls an AWS API and needs no AWS SDK.

CloudWatch alarms commonly reach on-call tools through an SNS topic, but
GoAlert cannot be subscribed to one directly today, for three independent
reasons: SNS requires the endpoint to fetch a SubscribeURL to confirm the
subscription and nothing does that, so zero messages are ever delivered; the
SNS envelope has no `summary` field and carries the alarm as a JSON string
inside `Message`; and SNS always sends `text/plain`, which
genericapi.ServeCreateAlert ignores because it only unmarshals
`application/json`. That last point is target#4463.

Add a `cloudwatch` integration key type and a handler at
POST /api/v2/cloudwatch/incoming that performs the subscription handshake and
verifies the SNS message signature, so a topic can be subscribed directly with
no Lambda or forwarder in between.

The key type is a full integration rather than a reuse of TypeGeneric because
the webhook URL shown in the UI is generated server-side from the key type
alone (IntegrationKey.Href). Reusing TypeGeneric would hand the user a
/api/v2/generic/incoming URL that silently fails to confirm as an SNS
subscription -- reproducing the exact bug this change fixes. Because the UI is
driven entirely by IntegrationKeyTypes and Href, no frontend changes are
needed.

Notable implementation details:

- The body is parsed regardless of Content-Type, and bounded with
  MaxBytesReader rather than io.LimitReader so an oversized body yields a 413
  instead of silently truncating into a misleading 400.
- Signature verification is split into pure functions (canonical string,
  verify, cert parse) so they are testable with no I/O. `Subject` is a *string
  because AWS omits the field from the string-to-sign entirely when absent,
  which a plain string cannot distinguish from an empty value.
- Both outbound fetches are host-allowlisted with an anchored pattern, and the
  client blocks redirects: the allowlist only covers the first hop, so a single
  302 from an allowlisted host would otherwise reach link-local addresses.
- The signing-cert cache is bounded with FIFO eviction because the cert URL
  path is attacker-supplied and would otherwise grow without limit.
- A freshness window on the signed Timestamp bounds replay of a captured
  envelope. Tradeoff: retries arriving over an hour late are rejected.
- Dedup is hex sha256(AlarmName). This is a stable, human-meaningful key
  derived from the one field guaranteed present and unique per alarm rule, so
  a given alarm produces the same key across redeliveries regardless of which
  ingress path receives it. It is never nil, since a nil dedup silently falls
  back to a content hash that changes on every state transition and would
  break both idempotency and the OK close.
- NewStateReason is capped before assembling details so a verbose reason cannot
  push AlarmDescription, which carries the runbook URL, past the length limit.
- INSUFFICIENT_DATA and a stray OK with no open alert both create nothing and
  return 2xx; non-2xx is reserved for infrastructure failure so SNS retries
  only when a retry could help.

Tests: table-driven unit tests for the canonical string, allowlist (including
the unanchored-suffix bypass), signature and freshness, and the alarm mapping;
plus a smoke test that generates an RSA key, serves a self-signed cert from an
httptest server, and drives the real crypto and allowlist paths end to end.

Verified against live AWS SNS: subscription confirms, an alarm creates one
alert with the runbook URL preserved, and re-delivery is suppressed as a
duplicate.

Signed-off-by: Sarthak Arora <sarora@eightfold.ai>
Azure Monitor delivers alerts by having an action group POST a webhook. This
adds a native GoAlert destination for that webhook contract, as a sibling to
the cloudwatch integration in the previous commit.

The generic endpoint cannot serve this: Azure sends application/json so it
clears the content-type gate, but ServeCreateAlert expects a flat
{summary, details, action, dedup, meta} body while Azure nests everything under
data.essentials / data.alertContext with no top-level summary -- so every
delivery would create an alert with a blank summary rather than an error. There
is also nothing mapping Azure's alertId onto the dedup field.

Unlike SNS there is no subscription handshake and no signature, so the handler
makes no outbound requests at all -- there is no analogue of cloudwatch's host
allowlist or certificate cache. The integration key in the URL is the only
credential, which makes the webhook URL credential-grade; the docs say so
explicitly for this key type.

Parsing:

- Only the common alert schema is accepted. A legacy-schema payload is rejected
  with a message naming the fix (enable the common alert schema on the receiver)
  rather than being degraded to the fallback, which would produce content-free
  alerts with no indication why. 400 rather than 5xx, since Azure does not retry
  4xx and a misconfigured receiver is a permanent condition.
- Dispatch is on the presence of condition.allOf, not on a conditionType
  allowlist. Every metric and log criteria shape shares that envelope, so this
  covers SingleResource, MultipleResource (used by multi-resource and
  resource-group-scoped rules), DynamicThreshold and WebtestLocationAvailability
  with one code path. conditionType still selects the two behaviours that
  genuinely differ: suppressing the dynamic threshold, which is a sensitivity
  artifact rather than a limit, and the log query/link lines.
- Prometheus rule groups get their own branch: the shape carries no
  conditionType and no condition, only expression/labels/annotations.
- Service Health and activity-log payloads render from properties, with HTML
  stripped and string-containing-JSON fields left opaque.
- Anything unrecognised builds a best-effort alert from essentials, which is
  present on every payload regardless of type, and logs the
  signalType/monitorService/conditionType triple so a newly-routed alert type
  announces itself instead of silently producing thin alerts.

signalType is deliberately not the discriminator: Platform and Prometheus metric
alerts share signalType "Metric", and Log Alerts V2, Azure Backup and
ActivityLog Administrative all share "Log". Branching on it would feed unrelated
payloads to the wrong renderer.

Dedup is sha256(essentials.alertId). Azure alerts are stateful, so one alert
object carries the whole lifecycle and the Fired and Resolved deliveries share an
alertId -- which is what lets the Resolved delivery close the alert its Fired
delivery opened. originAlertId is deliberately not used: it is per-rule for
metric alerts, so a single missed close would hold the dedup key and mute that
rule permanently. Status comes from essentials.monitorCondition, never
alertContext.status, which can disagree with it because the underlying incident
resolved while the alert fired.

A json.UnmarshalTypeError on an individual field is tolerated rather than fatal.
Azure documents threshold and dimension values as strings but is not consistent
across shapes, and a hard failure means a 400, which Azure does not retry -- so
one oddly-typed field would lose the page instead of one value.

Verified against Microsoft's documented common alert schema, a real tenant's
alert-rule inventory spanning metric, log-v2, and Prometheus rule groups, and
table-driven unit and smoke tests. All three shapes are natively parsed.

Signed-off-by: Sarthak Arora <sarora@eightfold.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant