Add native AWS CloudWatch and Azure Monitor alert ingress - #4535
Open
sarora-eightfold wants to merge 2 commits into
Open
Add native AWS CloudWatch and Azure Monitor alert ingress#4535sarora-eightfold wants to merge 2 commits into
sarora-eightfold wants to merge 2 commits into
Conversation
CloudWatch alarms commonly reach on-call tools through an SNS topic, but GoAlert cannot be subscribed to one directly today, for three independent reasons: SNS requires the endpoint to fetch a SubscribeURL to confirm the subscription and nothing does that, so zero messages are ever delivered; the SNS envelope has no `summary` field and carries the alarm as a JSON string inside `Message`; and SNS always sends `text/plain`, which genericapi.ServeCreateAlert ignores because it only unmarshals `application/json`. That last point is target#4463. Add a `cloudwatch` integration key type and a handler at POST /api/v2/cloudwatch/incoming that performs the subscription handshake and verifies the SNS message signature, so a topic can be subscribed directly with no Lambda or forwarder in between. The key type is a full integration rather than a reuse of TypeGeneric because the webhook URL shown in the UI is generated server-side from the key type alone (IntegrationKey.Href). Reusing TypeGeneric would hand the user a /api/v2/generic/incoming URL that silently fails to confirm as an SNS subscription -- reproducing the exact bug this change fixes. Because the UI is driven entirely by IntegrationKeyTypes and Href, no frontend changes are needed. Notable implementation details: - The body is parsed regardless of Content-Type, and bounded with MaxBytesReader rather than io.LimitReader so an oversized body yields a 413 instead of silently truncating into a misleading 400. - Signature verification is split into pure functions (canonical string, verify, cert parse) so they are testable with no I/O. `Subject` is a *string because AWS omits the field from the string-to-sign entirely when absent, which a plain string cannot distinguish from an empty value. - Both outbound fetches are host-allowlisted with an anchored pattern, and the client blocks redirects: the allowlist only covers the first hop, so a single 302 from an allowlisted host would otherwise reach link-local addresses. - The signing-cert cache is bounded with FIFO eviction because the cert URL path is attacker-supplied and would otherwise grow without limit. - A freshness window on the signed Timestamp bounds replay of a captured envelope. Tradeoff: retries arriving over an hour late are rejected. - Dedup is hex sha256(AlarmName). This is a stable, human-meaningful key derived from the one field guaranteed present and unique per alarm rule, so a given alarm produces the same key across redeliveries regardless of which ingress path receives it. It is never nil, since a nil dedup silently falls back to a content hash that changes on every state transition and would break both idempotency and the OK close. - NewStateReason is capped before assembling details so a verbose reason cannot push AlarmDescription, which carries the runbook URL, past the length limit. - INSUFFICIENT_DATA and a stray OK with no open alert both create nothing and return 2xx; non-2xx is reserved for infrastructure failure so SNS retries only when a retry could help. Tests: table-driven unit tests for the canonical string, allowlist (including the unanchored-suffix bypass), signature and freshness, and the alarm mapping; plus a smoke test that generates an RSA key, serves a self-signed cert from an httptest server, and drives the real crypto and allowlist paths end to end. Verified against live AWS SNS: subscription confirms, an alarm creates one alert with the runbook URL preserved, and re-delivery is suppressed as a duplicate. Signed-off-by: Sarthak Arora <sarora@eightfold.ai>
Azure Monitor delivers alerts by having an action group POST a webhook. This
adds a native GoAlert destination for that webhook contract, as a sibling to
the cloudwatch integration in the previous commit.
The generic endpoint cannot serve this: Azure sends application/json so it
clears the content-type gate, but ServeCreateAlert expects a flat
{summary, details, action, dedup, meta} body while Azure nests everything under
data.essentials / data.alertContext with no top-level summary -- so every
delivery would create an alert with a blank summary rather than an error. There
is also nothing mapping Azure's alertId onto the dedup field.
Unlike SNS there is no subscription handshake and no signature, so the handler
makes no outbound requests at all -- there is no analogue of cloudwatch's host
allowlist or certificate cache. The integration key in the URL is the only
credential, which makes the webhook URL credential-grade; the docs say so
explicitly for this key type.
Parsing:
- Only the common alert schema is accepted. A legacy-schema payload is rejected
with a message naming the fix (enable the common alert schema on the receiver)
rather than being degraded to the fallback, which would produce content-free
alerts with no indication why. 400 rather than 5xx, since Azure does not retry
4xx and a misconfigured receiver is a permanent condition.
- Dispatch is on the presence of condition.allOf, not on a conditionType
allowlist. Every metric and log criteria shape shares that envelope, so this
covers SingleResource, MultipleResource (used by multi-resource and
resource-group-scoped rules), DynamicThreshold and WebtestLocationAvailability
with one code path. conditionType still selects the two behaviours that
genuinely differ: suppressing the dynamic threshold, which is a sensitivity
artifact rather than a limit, and the log query/link lines.
- Prometheus rule groups get their own branch: the shape carries no
conditionType and no condition, only expression/labels/annotations.
- Service Health and activity-log payloads render from properties, with HTML
stripped and string-containing-JSON fields left opaque.
- Anything unrecognised builds a best-effort alert from essentials, which is
present on every payload regardless of type, and logs the
signalType/monitorService/conditionType triple so a newly-routed alert type
announces itself instead of silently producing thin alerts.
signalType is deliberately not the discriminator: Platform and Prometheus metric
alerts share signalType "Metric", and Log Alerts V2, Azure Backup and
ActivityLog Administrative all share "Log". Branching on it would feed unrelated
payloads to the wrong renderer.
Dedup is sha256(essentials.alertId). Azure alerts are stateful, so one alert
object carries the whole lifecycle and the Fired and Resolved deliveries share an
alertId -- which is what lets the Resolved delivery close the alert its Fired
delivery opened. originAlertId is deliberately not used: it is per-rule for
metric alerts, so a single missed close would hold the dedup key and mute that
rule permanently. Status comes from essentials.monitorCondition, never
alertContext.status, which can disagree with it because the underlying incident
resolved while the alert fired.
A json.UnmarshalTypeError on an individual field is tolerated rather than fatal.
Azure documents threshold and dimension values as strings but is not consistent
across shapes, and a hard failure means a 400, which Azure does not retry -- so
one oddly-typed field would lose the page instead of one value.
Verified against Microsoft's documented common alert schema, a real tenant's
alert-rule inventory spanning metric, log-v2, and Prometheus rule groups, and
table-driven unit and smoke tests. All three shapes are natively parsed.
Signed-off-by: Sarthak Arora <sarora@eightfold.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two native alert-ingress integrations,
cloudwatchandazuremonitor, so CloudWatch alarms and Azure Monitor alerts can reach GoAlert directly with no forwarder in between. They're siblings: same handler shape, same payload-mapper structure, same dedup discipline, same smoke-test layout.Note on size: this is two related integrations landing together and comes in well over the repo's 500-line PR-size guideline (the labeler check will flag it XL). I'd rather be upfront about that than have it be a surprise. Happy to split into smaller sequential PRs (key-type + migration, then handler, then payload mapper, then tests, per integration) if that's preferred — just say so and I'll re-split.
Why not the Universal Integration Key
Since both of these are "just point a webhook at GoAlert," the natural question is why not build them on UIK (
@experimental(flagName: "univ-keys")) instead of new key types. Two independent reasons:SubscribeURLto confirm the subscription — without that handshake, the subscription staysPendingConfirmationforever and zero messages are ever delivered. UIK has no mechanism to make that outbound confirmation GET. This is the actual blocker, not a stylistic preference.IntegrationKey.Href). A UIK-based approach would still need the UI to know which Expr rules to scaffold per vendor, and would still leave CloudWatch's handshake problem unsolved. A first-class key type is a strict subset of that complexity.Everything else about parsing, UIK could arguably do — the handshake is the reason CloudWatch needs its own type.
CloudWatch (via SNS)
CloudWatch alarms commonly reach on-call tooling through an SNS topic. GoAlert can't be subscribed to one directly today, for three independent reasons:
SubscribeURLto confirm the subscription, and nothing does that — so the subscription never leavesPendingConfirmationand zero messages are delivered.summaryfield; the alarm is a JSON string insideMessage.Content-Type: text/plain, whichgenericapi.ServeCreateAlertignores because it only unmarshalsapplication/json. This is the root cause of Messages from AWS SNS are blank #4463 ("Messages from AWS SNS are blank") — that issue was self-closed on a content-type theory that doesn't actually hold, since SNS can't be configured to send a different content type.This adds a
cloudwatchintegration key type andPOST /api/v2/cloudwatch/incoming, which performs the subscription handshake and verifies the SNS message signature, so a topic can be subscribed directly.Notable implementation details:
Content-Type— that's the bug this whole feature exists to fix.MaxBytesReaderrather thanio.LimitReader, so an oversized body yields a clean 413 instead of silently truncating into a misleading 400.Subjectis a*stringbecause AWS omits the field from the string-to-sign entirely when absent, which a plainstringcan't distinguish from an empty value.SubscribeURL) are host-allowlisted with an anchored pattern, and the client blocks redirects — the allowlist only covers the first hop, so a single 302 from an allowlisted host would otherwise reach link-local addresses.Timestampbounds replay of a captured envelope (reject >1h old or >5min future).hex(sha256(AlarmName))— stable and derived from the one field guaranteed present and unique per alarm rule, so the same alarm produces the same key regardless of which delivery reaches GoAlert. Never nil: a nil dedup silently falls back to a content hash that changes on every state transition, which would break both idempotency and theOK-closes-the-alert path.NewStateReasonis capped before assembling details so a verbose reason can't pushAlarmDescription(which carries the runbook URL) past the length limit.INSUFFICIENT_DATAand a strayOKwith no open alert both create nothing and return 2xx; non-2xx is reserved for infrastructure failure, so SNS only retries when a retry could actually help.Security note, anticipating a static-analysis flag: a taint-analysis pass may flag the outbound
http.Client.Docalls (cert fetch,SubscribeURLfetch) as "uncontrolled data in a network request," because the URL originates from the request body. This is a false positive in context — both call sites dial only the*url.URLreturned by an anchored host-allowlist check (^sns\.[a-z0-9-]+\.amazonaws\.com(\.cn)?$, HTTPS-only, no userinfo, no port), never the raw string, and redirects are disabled viahttp.ErrUseLastResponse. Adversarial tests cover the unanchored-suffix bypass (sns.us-west-2.amazonaws.com.evil.com), scheme downgrade, and userinfo confusion — all rejected before any dial.Verified against live AWS SNS: subscription confirms, a real alarm creates one alert with the runbook URL preserved, and re-delivery is suppressed as a duplicate.
Azure Monitor
Azure Monitor delivers alerts by having an action group POST a webhook. The generic endpoint can't serve this either: Azure sends
application/jsonso it clears the content-type gate, butServeCreateAlertexpects a flat body while Azure nests everything underdata.essentials/data.alertContextwith no top-levelsummary— every delivery would create an alert with a blank summary rather than an error. Nothing maps Azure'salertIdonto the dedup field either.This adds an
azureMonitorintegration key type andPOST /api/v2/azuremonitor/incoming.Unlike SNS, Azure has no subscription handshake and no signature, so the handler makes no outbound requests at all — no analogue of
cloudwatch's host allowlist or cert cache is needed. The integration key in the URL is the only credential, so the docs mark the webhook URL as credential-grade for this key type specifically.Parsing:
schemaId: "azureMonitorCommonAlertSchema") is accepted. A legacy-schema payload is rejected with a message naming the fix (enable the common alert schema on the action group's webhook receiver) rather than silently degraded, which would produce content-free alerts with no indication why. 400, not 5xx — Azure doesn't retry 4xx, and a misconfigured receiver is a permanent condition, not a transient one.condition.allOf, not aconditionTypeallowlist. Every metric and log criteria shape shares that envelope, so one code path coversSingleResourceMultipleMetricCriteria,MultipleResourceMultipleMetricCriteria(multi-resource / resource-group-scoped rules),DynamicThresholdCriteria, andWebtestLocationAvailabilityCriteria.conditionTypestill selects the two behaviours that genuinely differ: suppressing the dynamic threshold (a sensitivity artifact, not a limit) and the log query/link rendering.conditionTypeand nocondition, onlyexpression/labels/annotations.properties, with HTML stripped and string-containing-JSON fields left opaque rather than (failingly) parsed as JSON.essentialsalone (present on every payload regardless of type) and logs thesignalType/monitoringService/conditionTypetriple, so a newly-encountered alert type is visible in logs instead of silently producing a thin alert forever.signalTypeis deliberately not the discriminator: Platform and Prometheus metric alerts both reportsignalType: "Metric", and Log Alerts V2, Azure Backup, and ActivityLog Administrative all report"Log". Branching on it would feed unrelated payloads to the wrong renderer.Dedup is
sha256(essentials.alertId). Azure alerts are stateful — one alert object carries the whole lifecycle, and theFired/Resolveddeliveries of one firing share analertId— which is what lets theResolveddelivery close the alert itsFireddelivery opened.originAlertIdis deliberately not used: it's per-rule for metric alerts, so a single missed close would hold the dedup key and mute that rule permanently. Status comes fromessentials.monitorCondition, neveralertContext.status, which can disagree with it (the underlying incident can resolve while the alert itself is still reported as fired).A
json.UnmarshalTypeErroron an individual field is tolerated rather than fatal — Azure documentsthresholdand dimension values as strings but isn't consistent about it across shapes, and a hard failure means a 400 (which Azure won't retry), so one oddly-typed field would otherwise cost the whole page instead of just that one value.Verified against Microsoft's documented common alert schema, a real tenant's alert-rule inventory spanning metric, log-v2, and Prometheus rule groups, and table-driven unit and smoke tests.
Tests
Both packages have table-driven unit tests (canonical string, allowlist, signature/freshness, and payload mapping for
cloudwatch; schema gate, condition-shape dispatch, and payload mapping forazuremonitor) plus a smoke test each — the first unit tests in any GoAlert ingress package.go build ./...,go vet, andgofmtare clean;golangci-lintreports 0 issues on this diff.Out of scope
Neither integration adds an outbound notification channel (that's #552, the opposite direction). No new dependency for crypto, HTTP, or cloud SDKs — both handlers are stdlib-only;
cloudwatchnever calls an AWS API and needs no AWS SDK.