Make Discord case alerts route-aware and durable

[Svaag]

Summary

• classify alert delivery deterministically into network, AI, or CI routes, with unknown production alerts defaulting to network
• replace Discord lifecycle spam with one persistent case card that is edited on investigation, acknowledgement, and recovery
• persist Discord message IDs, retry transient failures, replace deleted cards once, and elect one outbox worker across the two API workers
• remind only on unacknowledged high-severity network/AI cases every six hours
• separate model readiness from transient provider/runtime degradation and bypass model invocation for model/loop self-health alerts
• persist proactive digest deduplication across restarts

Validation

• uv run pytest -q — 502 passed, 3 skipped
• uv run ruff check app tests
• uv run mypy app

Rollout

Keep this draft until the route-specific Discord destinations are present in Vault. The infrastructure cutover is coordinated in the network-operations routing PR.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🏅 Score: 85

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Global State Mutation The test uses install_case_notifier(notifier) to set the module-level global CASE_BOT_NOTIFIER, then restores it to None in a finally block. While the cleanup is present, this pattern makes the test fragile under concurrent execution or if the test body raises an exception before the finally runs. Polluted global state can cause cascading failures in other tests that depend on the default None value. This is a testing concern that violates the guidance to avoid mutating module-global agents in tests. calls = [] async def notifier(**kwargs): calls.append(kwargs) return SimpleNamespace(message_id="bot-message", channel_id="ai", action="updated") install_case_notifier(notifier) try: result = await send_case_notification( case_id="case-ai", title="Model degraded", description="Fallback active", route="ai", message_id="existing-message", ) finally: install_case_notifier(None) Self-Health Triage Bypass The function _uses_deterministic_self_health_triage determines whether to skip model invocation by checking for substring matches in alert labels (e.g., "noc-agent-model", "allmodels", "gemini"). If an attacker can inject crafted labels into an alert payload (e.g., via a malicious Alertmanager webhook or Icinga notification), they could cause the system to bypass AI triage for a non‑self‑health alert, potentially hiding an incident from the investigation pipeline. The trust boundary of alert labels should be assumed untrusted, but this function uses them directly without sanitization beyond the substring check. def _uses_deterministic_self_health_triage(alert_payload: dict) -> bool: """Self-health checks must not invoke the dependency they are checking.""" labels = _labels(alert_payload) subject = " ".join( str(labels.get(key) or "") for key in ("alertname", "service", "check_command", "job") ).lower() return any( term in subject for term in ( "engineering-loop", "engineering_loop", "noc-agent-model", "noc_agent_model", "noc-agent-mcp", "noc_agent_mcp", "noc-agent-config", "noc-agent-health", "model-fallback", "allmodels", "gemini", "openrouter", "venice", ) )

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Use existing message ID for reminders The reminder handler always passes an empty message_id to the notifier, which will create a new Discord message each time instead of updating the existing card. This will result in duplicate messages for each reminder cycle. Pass the existing case.discord_message_id to the notifier so it updates the original card. app/cases/handlers.py [133-145] -async def handle(intent: OutboxIntent) -> OutboxHandlerResult: - if not intent.case_id or intent.payload.get("action") != "reminder": - raise ValueError("discord_update intent requires a reminder case") - case = await case_service.store.get_case(intent.case_id) - if not isinstance(case, AtomicCaseProjection): - raise KeyError(f"atomic case not found for reminder intent: {intent.case_id}") - if case.acknowledged_at or case.status in {"resolved", "closed", "expired", "linked"}: - return OutboxHandlerResult(payload_updates={"skipped": "no_longer_due"}) - delivery = await notifier( - case_id=f"{case.case_id}:reminder", - title=f"🔁 Unacknowledged critical: {case.title or case.detector or case.rule_id}", - description=f"{case.case_number or case.case_id} remains critical and unacknowledged.", - color=_severity_color(case.severity), - fields=[ - {"name": "Case", "value": _case_url(case, control_public_url) or case.case_number or case.case_id}, - {"name": "Route", "value": case.notification_route, "inline": True}, - ], - level=Verbosity.WARNING, - route=case.notification_route, - message_id="", - ) - await case_service.mark_discord_reminder(case.case_id) - return OutboxHandlerResult( - external_id=str(delivery.message_id if delivery is not None else ""), - payload_updates={"notification_route": case.notification_route, "discord_action": "reminder"}, - ) +delivery = await notifier( + case_id=f"{case.case_id}:reminder", + title=f"🔁 Unacknowledged critical: {case.title or case.detector or case.rule_id}", + description=f"{case.case_number or case.case_id} remains critical and unacknowledged.", + color=_severity_color(case.severity), + fields=[ + {"name": "Case", "value": _case_url(case, control_public_url) or case.case_number or case.case_id}, + {"name": "Route", "value": case.notification_route, "inline": True}, + ], + level=Verbosity.WARNING, + route=case.notification_route, + message_id=case.discord_message_id, +) Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies that always passing an empty message_id for reminders will create duplicate Discord messages. Passing case.discord_message_id is critical to update the existing card. The 'existing_code' snippet accurately matches the new hunk lines (133-145), and the 'improved_code' correctly applies the fix.	Medium
	Use dedicated intent for acknowledgements The ack method enqueues a report intent after acknowledging a case, but the report handler will attempt to send a notification to Discord. Since the case is already acknowledged, this will create a duplicate notification or attempt to update a card that may no longer be relevant. Consider using a dedicated "acknowledgement update" intent type instead of reusing the report handler. app/cases/service.py [401-406] if case.discord_message_id: - await self.request_report( - case, - state_signature=self.report_state_signature(case), - payload={"schema": "case_acknowledgement_v1", "model_consumption_allowed": False}, + await self.store.enqueue_outbox( + OutboxIntent( + case_id=case.case_id, + intent_type="discord_update", + idempotency_key=f"discord-ack:{case.case_id}", + state_signature=self.report_state_signature(case), + payload={"action": "acknowledgement", "schema": "case_acknowledgement_v1", "model_consumption_allowed": False}, + ) ) Suggestion importance[1-10]: 7 __ Why: The suggestion raises a valid concern about reusing the 'report' intent for acknowledgements, which could lead to unintended behavior. However, the proposed 'discord_update' intent with action 'acknowledgement' may not be handled by any current handler, and the PR might already have correct logic in place. The score reflects the moderate impact and potential overcomplication.	Medium

[Svaag]

Addressed the two hardening observations in bbe6ad0:

• the bot-notifier test now uses pytest monkeypatch, so module state is restored automatically;
• self-health triage now uses an exact allowlist of known detector/service identifiers rather than substring matching, with a crafted-label regression test.

The two code suggestions are intentionally unchanged:

• a six-hour critical reminder is a new post because editing the existing card does not generate a new unread notification;
• acknowledgement reuses the report intent specifically because the report handler passes the persisted message ID and edits the existing card rather than creating a duplicate.