Skip to content

Commit ccd5c3d

Browse files
feat(monitoring): alert on ECS services that cannot start tasks (#160)
## Why `testteam-platform-test-app` has been crash-looping for weeks with desired=1 and running=0, because of a `CannotPullContainerError` (its ECR repo is empty). That is about 80 failed task starts a day, and nobody noticed. Each attempt creates and deletes a Fargate ENI, which AWS Config records. This was the main driver of about $29/mo in Config charges found in the 2026-10 cost audit. PR #159 reduces the Config side. This PR makes the crash loop itself visible. ## What - `monitoring/main.tf`: new EventBridge rule `javabin-ecs-task-start-impaired`. It matches `source=aws.ecs`, `detail-type=ECS Service Action` and `eventName=SERVICE_TASK_START_IMPAIRED`, for the `javabin-platform` cluster only (legacy CDK clusters are excluded). The target is the `javabin-security` SNS topic, whose policy already allows `events.amazonaws.com`. slack-alert sends non-finding events from that topic to #javabin-infra-alerts. I chose that channel over the cost channel because this is an operational problem. - `slack_alert/handler.py`: - New `format_ecs_service_impaired` formatter, routed before the generic EventBridge branch. Without it, the event would fall through to the IAM-change formatter. - The message shows the service, the cluster and a link to the service's events page in the ECS console. - Dedup uses the existing persistent `javabin-alert-dedup` table, keyed per service ARN per UTC day. ECS keeps emitting the event while the service is impaired. - New test `terraform/lambda-src/tests/test_slack_alert_ecs.py` checks the formatting, once-per-day dedup per service, and that other ECS service actions are ignored. - Docs: added a line to the CLAUDE.md alert routing section and to `docs/lambda-functions.md`. ## Expected plan 2 to add: `aws_cloudwatch_event_rule.ecs_task_start_impaired` and `aws_cloudwatch_event_target.ecs_task_start_impaired_sns`. 1 in-place update: the slack-alert Lambda code. No IAM changes, because slack-alert already has GetItem/PutItem on the dedup table. ## How verified - `python3 -m unittest discover -s terraform/lambda-src/tests` (67 OK, 3 new) and `-s scripts/tests` pass. - `terraform fmt -recursive -check`, then `terraform init -backend=false` and `terraform validate` pass. ## Follow-ups - Once this is applied, the first alert should be for `testteam-platform-test-app`. Scale it to 0 or push its image. - It is independent of #159. Both touch `monitoring/main.tf`, but in different blocks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 8b6ab4c commit ccd5c3d

5 files changed

Lines changed: 181 additions & 0 deletions

File tree

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,7 @@ EventBridge ──► javabin-security SNS ──► slack-alert Lambda:
224224
Security Hub findings (NEW only) ──► #platform-security-alerts
225225
GuardDuty findings ──► #platform-security-alerts
226226
IAM / resource / login events ──► #javabin-infra-alerts
227+
ECS service can't start tasks ──► #javabin-infra-alerts (once per service per day)
227228
Cost Anomaly ──► javabin-alerts SNS ──► slack-alert Lambda ──► #javabin-cost-alerts
228229
229230
Account budget (FORECASTED >100%, ACTUAL >100%) ──► javabin-alerts SNS ──► #javabin-cost-alerts

‎docs/lambda-functions.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ Log groups `/aws/lambda/javabin-{function}` are declared in `lambdas/log-groups.
1818

1919
Features: GitHub OIDC attribution (extracts actor/repo/SHA from session tags), cost estimation via shared pricing module, Security Hub finding formatting.
2020

21+
ECS `SERVICE_TASK_START_IMPAIRED` events from the `javabin-platform` cluster only (a crash-looping service) are posted to #javabin-infra-alerts at most once per service per UTC day. Legacy clusters (vaultwarden, cakeredux, moresleep-test, dennis-test) are not covered. As with the other deduplicated alerts, the dedup key is written before the Slack post, so a failed post suppresses that service's alert for the rest of the day; if the dedup table is unreadable, dedup fails open.
22+
2123
Uses the `javabin-alert-dedup` DynamoDB table for deduplication of alerts (prevents repeated notifications for the same finding).
2224

2325
## cost-report

‎terraform/lambda-src/slack_alert/handler.py‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1172,6 +1172,57 @@ def format_budget_alert(parsed):
11721172
return {"blocks": blocks, "text": fallback}
11731173

11741174

1175+
def format_ecs_service_impaired(parsed):
1176+
"""ECS service cannot start tasks (SERVICE_TASK_START_IMPAIRED) — Block Kit.
1177+
1178+
ECS keeps emitting this event while a service crash-loops, so alert at most
1179+
once per service per UTC day (persistent DynamoDB dedup). A crash loop
1180+
burns Fargate starts, image pulls and AWS Config items until someone acts.
1181+
"""
1182+
detail = parsed.get("detail", {})
1183+
if detail.get("eventName") != "SERVICE_TASK_START_IMPAIRED":
1184+
return None
1185+
1186+
resources = parsed.get("resources") or []
1187+
service_arn = resources[0] if resources else "unknown"
1188+
service = service_arn.rsplit("/", 1)[-1]
1189+
cluster = detail.get("clusterArn", "").rsplit("/", 1)[-1] or "unknown"
1190+
region = parsed.get("region", "")
1191+
1192+
day = (parsed.get("time") or datetime.now(timezone.utc).isoformat())[:10]
1193+
dedup_key = f"ecs-start-impaired:{service_arn}:{day}"
1194+
if is_finding_already_alerted(dedup_key):
1195+
logger.info("Suppressed repeat ECS start-impaired alert: %s", dedup_key)
1196+
return None
1197+
record_finding_alert(dedup_key)
1198+
1199+
events_url = (f"https://{region}.console.aws.amazon.com/ecs/v2/clusters/{cluster}"
1200+
f"/services/{service}/events?region={region}")
1201+
1202+
blocks = [
1203+
{"type": "header",
1204+
"text": {"type": "plain_text", "text": ":warning: ECS service cannot start tasks",
1205+
"emoji": True}},
1206+
{"type": "section",
1207+
"text": {"type": "mrkdwn",
1208+
"text": (f"*{service}* keeps failing to start tasks and ECS is backing off. "
1209+
"Usual causes: missing image, failing health check, bad secret. "
1210+
"Fix the deploy or scale the service to 0.")}},
1211+
{"type": "section", "fields": [
1212+
{"type": "mrkdwn", "text": f"*Service*\n`{service}`"},
1213+
{"type": "mrkdwn", "text": f"*Cluster*\n`{cluster}`"},
1214+
]},
1215+
{"type": "section",
1216+
"text": {"type": "mrkdwn", "text": f"<{events_url}|Service events in the ECS console>"}},
1217+
{"type": "divider"},
1218+
{"type": "context", "elements": [
1219+
{"type": "mrkdwn", "text": "Source: ECS Service Action (alerts once per service per day)"}
1220+
]},
1221+
]
1222+
1223+
return {"blocks": blocks, "text": f"ECS service {service} cannot start tasks"}
1224+
1225+
11751226
# ---------------------------------------------------------------------------
11761227
# Weekly Security Hub summary
11771228
# ---------------------------------------------------------------------------
@@ -1382,6 +1433,10 @@ def format_structured_alert(subject, parsed):
13821433
if "anomalyId" in parsed or "rootCauses" in parsed:
13831434
return format_cost_anomaly(parsed)
13841435

1436+
# ECS service crash loop (EventBridge "ECS Service Action")
1437+
if parsed.get("detail-type") == "ECS Service Action":
1438+
return format_ecs_service_impaired(parsed)
1439+
13851440
# EventBridge / CloudTrail events
13861441
if "detail-type" in parsed:
13871442
detail = parsed.get("detail", {})
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
"""Tests for slack_alert ECS start-impaired alerts — run: python3 -m unittest discover -s terraform/lambda-src/tests"""
2+
3+
import importlib.util
4+
import os
5+
import sys
6+
import types
7+
import unittest
8+
9+
LAMBDA_SRC = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")
10+
11+
12+
class FakeTable:
13+
def __init__(self):
14+
self.items = {}
15+
16+
def get_item(self, Key):
17+
item = self.items.get(Key["finding_key"])
18+
return {"Item": item} if item else {}
19+
20+
def put_item(self, Item):
21+
self.items[Item["finding_key"]] = Item
22+
23+
24+
def load(table):
25+
boto3 = types.ModuleType("boto3")
26+
boto3.client = lambda *a, **kw: object()
27+
boto3.resource = lambda *a, **kw: types.SimpleNamespace(Table=lambda name: table)
28+
botocore = types.ModuleType("botocore")
29+
botocore_config = types.ModuleType("botocore.config")
30+
botocore_config.Config = lambda **kw: None
31+
sys.modules.update({"boto3": boto3, "botocore": botocore, "botocore.config": botocore_config})
32+
os.environ.update({
33+
"INFRA_WEBHOOK_PARAM": "/x/infra",
34+
"COST_WEBHOOK_PARAM": "/x/cost",
35+
"SECURITY_TOPIC_ARN": "arn:aws:sns:eu-central-1:111111111111:javabin-security",
36+
})
37+
if LAMBDA_SRC not in sys.path:
38+
sys.path.insert(0, LAMBDA_SRC)
39+
for name in [m for m in sys.modules if m == "shared" or m.startswith("shared.")]:
40+
del sys.modules[name]
41+
spec = importlib.util.spec_from_file_location(
42+
"slack_alert_handler", os.path.join(LAMBDA_SRC, "slack_alert", "handler.py"))
43+
mod = importlib.util.module_from_spec(spec)
44+
spec.loader.exec_module(mod)
45+
return mod
46+
47+
48+
def event(service="testteam-platform-test-app", name="SERVICE_TASK_START_IMPAIRED",
49+
time="2026-10-24T10:00:00Z"):
50+
return {
51+
"version": "0",
52+
"detail-type": "ECS Service Action",
53+
"source": "aws.ecs",
54+
"account": "111111111111",
55+
"time": time,
56+
"region": "eu-central-1",
57+
"resources": [f"arn:aws:ecs:eu-central-1:111111111111:service/javabin-platform/{service}"],
58+
"detail": {
59+
"eventType": "WARN",
60+
"eventName": name,
61+
"clusterArn": "arn:aws:ecs:eu-central-1:111111111111:cluster/javabin-platform",
62+
},
63+
}
64+
65+
66+
class EcsStartImpairedTests(unittest.TestCase):
67+
def setUp(self):
68+
self.table = FakeTable()
69+
self.mod = load(self.table)
70+
71+
def test_formats_service_and_console_link(self):
72+
msg = self.mod.format_structured_alert("subject", event())
73+
self.assertEqual(msg["text"], "ECS service testteam-platform-test-app cannot start tasks")
74+
rendered = str(msg["blocks"])
75+
self.assertIn("`javabin-platform`", rendered)
76+
self.assertIn(
77+
"https://eu-central-1.console.aws.amazon.com/ecs/v2/clusters/javabin-platform"
78+
"/services/testteam-platform-test-app/events?region=eu-central-1",
79+
rendered,
80+
)
81+
82+
def test_alerts_once_per_service_per_day(self):
83+
self.assertIsNotNone(self.mod.format_structured_alert("s", event()))
84+
self.assertIsNone(self.mod.format_structured_alert("s", event(time="2026-10-24T18:00:00Z")))
85+
self.assertIsNotNone(self.mod.format_structured_alert("s", event(service="other-app")))
86+
self.assertIsNotNone(self.mod.format_structured_alert("s", event(time="2026-10-25T08:00:00Z")))
87+
88+
def test_ignores_other_ecs_service_actions(self):
89+
self.assertIsNone(self.mod.format_structured_alert("s", event(name="SERVICE_STEADY_STATE")))
90+
self.assertEqual(self.table.items, {})
91+
92+
93+
if __name__ == "__main__":
94+
unittest.main()

‎terraform/platform/monitoring/main.tf‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -561,6 +561,35 @@ resource "aws_dynamodb_table" "alert_dedup" {
561561
}
562562
}
563563

564+
################################################################################
565+
# ECS services that cannot start tasks → javabin-security (#javabin-infra-alerts)
566+
#
567+
# A service stuck failing task starts (missing image, failing health check)
568+
# crash-loops until someone notices: wasted Fargate starts, image pulls and
569+
# AWS Config items for every ENI. slack-alert posts once per service per day.
570+
# Scoped to the platform cluster (compute/: "${var.project}-platform").
571+
################################################################################
572+
573+
resource "aws_cloudwatch_event_rule" "ecs_task_start_impaired" {
574+
name = "${var.project}-ecs-task-start-impaired"
575+
description = "ECS services in the platform cluster that cannot start tasks"
576+
577+
event_pattern = jsonencode({
578+
source = ["aws.ecs"]
579+
detail-type = ["ECS Service Action"]
580+
detail = {
581+
eventName = ["SERVICE_TASK_START_IMPAIRED"]
582+
clusterArn = ["arn:aws:ecs:${var.region}:${var.aws_account_id}:cluster/${var.project}-platform"]
583+
}
584+
})
585+
}
586+
587+
resource "aws_cloudwatch_event_target" "ecs_task_start_impaired_sns" {
588+
rule = aws_cloudwatch_event_rule.ecs_task_start_impaired.name
589+
target_id = "send-to-security-sns"
590+
arn = aws_sns_topic.security.arn
591+
}
592+
564593
################################################################################
565594
# Cost Allocation Tags
566595
################################################################################

0 commit comments

Comments
 (0)