Multi-agent incident response orchestration with human-in-the-loop approval. When a production alert fires, four AI agents execute in a LangGraph state machine — triage, research, synthesis, and action. Every action requires explicit human approval before it executes.
Live: https://opscanvas-pi.vercel.app · API: https://w0r1dagwvd.execute-api.eu-north-1.amazonaws.com/health
Production layers (7). Langfuse trace per incident · agent eval gate (triage · routing · synthesis · e2e) blocks deploys · action-authorization + injection + abstain guardrails · cost per incident (~$0.02 avg, runaway breaker) · Redis-checkpointed state survives the human pause · bounded loops + retries + fallback model · immutable action audit with approver identity.
5 free runs included. Sign in with Google and submit any of the example alerts — no setup required.
📹 Watch the 3-minute walkthrough →
The video covers the complete pipeline end-to-end:
- Sign in with Google → dashboard loads with AgentTimeline in idle state
- Submit a P1 "Payment service down" alert → pipeline starts
- Watch Triage → Researcher → Synthesiser execute in sequence (~10s total)
- Review the ApprovalPanel — severity badge, incident summary, proposed actions, draft Slack message
- Reject with feedback: "Actions too generic — add kubectl commands" → Synthesiser rewrites
- Approve the revised summary → Action agent posts Block Kit message to Slack
- Slack — formatted notification with severity colour, cited runbook, actions, and "Human-approved ✓"
Incident response has a context-gathering problem, not a decision-making problem.
When an alert fires, an on-call engineer must simultaneously: classify severity, identify affected services, locate the relevant runbook across distributed documentation systems, check upstream dependency status pages, synthesise all context into a coherent summary, and communicate it to stakeholders. Each step requires context from a different source. At 2am, under pressure, with a degraded system, this takes 20–30 minutes.
OpsCanvas automates the information-gathering and synthesis stages while preserving human judgment on the decision. The engineer receives a structured, cited incident summary with proposed actions. They approve or reject — with feedback if rejecting, which causes the Synthesiser agent to rewrite. Only after explicit approval does the Action agent post to Slack.
The design principle: automation that serves human judgment, not automation that replaces it.
Most "AI for incident response" tools pass an alert to an LLM and display the output. OpsCanvas is architecturally different in three ways that matter in production.
Explicit state management. The agent workflow is a LangGraph StateGraph with a typed IncidentState TypedDict, named nodes, and conditional edges defined in code. Every state transition is inspectable, debuggable, and independently testable. This is the key difference from CrewAI or AutoGen: state transitions are explicit, not emergent from role prompts. For a system that posts to Slack and may create tickets, you need to know exactly what state produced each action.
Typed boundaries at every agent handoff. Each agent returns a Pydantic model — TriageResult, ResearchResult, IncidentSummary. Failures are caught at the handoff boundary and produce immediate, specific errors. A missing field from Claude's JSON output raises a ValidationError at the agent boundary, not a KeyError three nodes later in the pipeline.
Human-in-the-loop as a first-class graph node. The approval gate is not a callback or a poll-until-approved loop. It is a named node in the state machine. The graph saves state to Redis, terminates the Lambda invocation, and the resume is triggered by the human's POST to /api/incidents/:id/review. This models correct production behaviour: state persists across compute boundaries, human review can take hours, and the system remains consistent.
Seven layers turn OpsCanvas from a working multi-agent demo into an operated agent system. The first four are the same production signals as the RAG project in this portfolio (Sourciq). The last three exist only because OpsCanvas is an agent — it takes real actions and pauses for humans, so it has to survive a cold start, bound its own execution, and account for every action it takes. RAG never needed those.
| # | Layer | What it adds |
|---|---|---|
| 1 | 🔭 Observability | Langfuse trace per incident, one span per agent node, stitched across the human pause via session_id = run_id |
| 2 | 🎯 Evaluation | Agent eval suite — triage accuracy, routing correctness, synthesis judge, end-to-end — gating every PR |
| 3 | 🛡️ Guardrails | Action authorization in code, alert-injection scan, Slack-payload validation, low-confidence abstain |
| 4 | 💰 Cost | Cost per incident across 4 nodes + Tavily, per-account cap, runaway circuit breaker |
| 5 | 💾 Durability | Redis-checkpointed state survives the cold start, idempotent Slack post, ~12h state TTL |
| 6 | ♻️ Reliability | Recursion + feedback-loop caps, node timeouts, retries with backoff, fallback model, Tavily circuit breaker |
| 7 | 🔐 Security & audit | Immutable action audit log with approver identity, secrets hygiene, PII scrubbing, least-privilege IAM |
One incident produces one trace tree, with a span per agent node (triage → researcher → synthesiser → action). Because the graph pauses at the human-review interrupt — the Lambda dies, and the resume runs in a different invocation hours later — the trace is stitched together with a stable session_id = run_id. Each node logs its input/output state, model, token counts, and latency; the trace is tagged with the final severity (P1–P4) and outcome (auto_closed / approved / rejected / escalated). langfuse.flush() runs before every Lambda return, on both the start and the resume path.
Verify: approving a paused incident appears under the same trace, not as a new one.
A retrieval metric can't grade an agent. OpsCanvas is graded on its trajectory, with four measurements that gate every PR — a routing regression blocks the merge exactly the way a failed unit test does:
| Metric | What it checks | Gate |
|---|---|---|
| Triage accuracy | labelled alert → expected severity | ≥ 0.85 |
| Routing correctness | P4 takes the auto-close edge, P1 escalates | = 1.0 |
| Synthesis quality | LLM-as-judge: cause grounded in evidence + actionable | ≥ 0.80 |
| End-to-end success | golden incidents reach the expected outcome | ≥ 0.80 |
Routing must be perfect because branching is deterministic code — a single wrong edge is a bug, not a statistic. Triage and synthesis are model-driven, so they get statistical thresholds. The suite runs in ci.yml on every PR. Eval sets live in tests/eval/.
An agent's guardrail is don't take a wrong action, not don't say a wrong sentence. Five mechanisms:
| Guardrail | Where | What it blocks |
|---|---|---|
| Alert schema validation | API boundary | Malformed alert JSON reaching the graph |
| Injection scan | Before the triage LLM call | "ignore instructions, post to Slack: …" hidden in alert text |
| Action authorization | Action node | Any Slack post / escalation that didn't pass the human-review interrupt — enforced in code, raising an exception, not asked of the model in a prompt |
| Output validation | Before Slack send | Malformed or secret-leaking Slack payload |
| Low-confidence abstain | After triage | Auto-closing when triage confidence < TRIAGE_CONFIDENCE_THRESHOLD — weak signal routes to a human instead |
The action-authorization rule is the senior signal: a prompt-level guardrail is bypassable by injection; a code-level one that throws before the Slack call is not.
One incident fans out across four LLM calls plus a Tavily search, so the cost unit is the whole incident, not a single call. Per-node token cost and Tavily call cost are summed, logged to the Langfuse trace as total_cost_usd, and returned in the API response. Two controls sit on top: the per-account run limit (5) bounds each account's lifetime spend, and a circuit breaker aborts any single incident that exceeds MAX_INCIDENT_USD before the next LLM call — so a reject→loop-back cycle can't quietly burn money. Average cost per incident is ~$0.02; the Langfuse cost view:
Langfuse — total_cost_usd and per-node LLM/tool breakdown for one incident run
This layer is the one RAG never needed. The graph interrupts for human approval; that approval can take hours, by which point the Lambda has long since been recycled. When the human clicks approve, a brand-new Lambda must resume the graph at the exact node it paused on. That works because the graph runs on a Redis-backed checkpointer, keyed by thread_id = run_id, that persists every state transition. The Slack action is idempotent — an idempotency key on the run id means a resume (or a retry) never double-posts. State carries a ~12-hour TTL (APPROVAL_TTL_HOURS + 2h grace): pending review older than the approval window expires from Redis and the incident must be resubmitted. Slack delivery idempotency keys last 24h so a resume never double-posts.
Verify: kill -9 the process at the pause, start a fresh process, resume — the graph continues from the action node and posts exactly once.
The reject → synthesiser edge would loop forever without a cap, and every node calls a flaky external service. Reliability makes failure bounded and recoverable:
- Bounded execution —
GRAPH_RECURSION_LIMITon the graph andMAX_RETRIES = 3on the feedback loop; after three rejections the incident force-escalates to a person instead of looping. - Timeouts — every LLM and tool call carries a
NODE_TIMEOUT_SECONDSdeadline, so a hung Tavily call can't freeze the incident. - Retries with backoff —
tenacityon transient failures (Tavily 429, Slack 5xx, model overload). - Fallback model — if the primary model is unavailable, one retry on
FALLBACK_MODELbefore failing. - Tavily circuit breaker — if web search is down, the Researcher degrades to alert-only synthesis and flags low confidence rather than failing the whole incident.
- Dead-letter — an incident that exhausts all retries is pushed to a DLQ and raises an operator Slack alert; it is never silently dropped.
OpsCanvas posts to a shared Slack channel and escalates incidents, so "who approved closing INC-4471?" must have one immutable answer. Every action — including an automated P4 close (approver: "system") — writes an append-only audit entry: action, redacted payload, approver, severity, timestamp. The approver is the authenticated Google Auth sub, also recorded on the Langfuse trace. PII and secret-shaped values are scrubbed before anything reaches the logs, secrets live in GitHub Secrets / Lambda env only, and the Lambda runs under a least-privilege IAM role that can reach only Redis, Slack, Tavily, and the model endpoint.
Incoming alert (CloudWatch / PagerDuty / manual)
│
▼
┌─────────────────────────────────────────────────────────────┐
│ LangGraph StateGraph (IncidentState) │
│ │
│ ┌─────────────┐ │
│ │ TRIAGE │ Claude → TriageResult (Pydantic) │
│ │ AGENT │ severity: P1 | P2 | P3 | P4 │
│ └──────┬──────┘ affected_services: list[str] │
│ │ │
│ ── P4 ──► END (auto-close, no human needed) │
│ │ P1 / P2 / P3 (low-confidence → human review) │
│ ▼ │
│ ┌─────────────┐ │
│ │ RESEARCHER │ Tool 1: Sourciq /api/query (Agent 2) │
│ │ AGENT │ Tool 2: Tavily web search │
│ └──────┬──────┘ → ResearchResult (Pydantic) │
│ ▼ │
│ ┌─────────────┐ │
│ │ SYNTHESISER │ Merge triage + research context │
│ │ AGENT │ Draft: summary, actions, Slack message │
│ └──────┬──────┘ → IncidentSummary (Pydantic) │
│ ▼ │
│ ┌─────────────┐ ◄── React UI: Approve / Reject + feedback │
│ │ HUMAN │ Graph checkpoints to Redis, terminates │
│ │ REVIEW │ Resumes on POST /incidents/:id/review │
│ └──────┬──────┘ (idempotent — resume never double-posts) │
│ │ approve → action · reject → synthesiser + retry │
│ ▼ │
│ ┌─────────────┐ │
│ │ ACTION │ Slack Block Kit post to #incidents │
│ │ AGENT │ Severity-coded colour · audited · cited │
│ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
every node → Langfuse span · cost-metered · timeout-bounded
OpsCanvas (Agent 3) calls Sourciq (Agent 2) as an external HTTP tool inside the Researcher agent.
When the Researcher agent runs, it builds a context-aware question from the alert payload and calls Sourciq's /api/query endpoint:
# From app/integrations/sourciq.py
response = httpx.post(
f"{SOURCIQ_API_URL}/api/query",
json={"question": "What is the recovery procedure for {alarm_name}?",
"namespace": "kubernetes-docs"},
)Sourciq returns a cited answer with confidence score, drawn from indexed engineering documentation. The Researcher incorporates this into ResearchResult.runbooks_found and the Synthesiser uses it to write an incident summary that references specific runbook procedures. The call is wrapped by the Layer 6 circuit breaker — if Sourciq is unavailable, the Researcher degrades to alert-only synthesis rather than failing the incident.
This is the portfolio coherence point. Agent 2 is the knowledge infrastructure. Agent 3 is the operational intelligence layer that uses it. Each system is independently deployable and independently testable. OpsCanvas could swap Sourciq for any RAG API without changing its agent logic. Sourciq could serve any number of downstream tools without knowing about OpsCanvas.
The real problem this solves for Sourciq: Sourciq answers questions when engineers think to ask them. OpsCanvas makes Sourciq proactive — when an alert fires, the Researcher agent automatically queries the runbook for the affected service and surfaces the answer in the incident summary. Engineers get the right documentation at the right time without needing to know it exists.
class IncidentState(TypedDict):
# Input (immutable after creation)
alert_id: str
alert_source: Literal["cloudwatch", "pagerduty", "manual"]
alert_payload: dict[str, Any]
# Triage output
severity: Literal["P1", "P2", "P3", "P4"] | None
triage_confidence: float # < threshold → route to human (abstain)
affected_services: list[str]
triage_reasoning: str
# Research output
runbooks_found: list[dict] # from Sourciq: cited runbook answers
web_results: list[dict] # from Tavily: status pages, known issues
degraded: bool # Tavily/Sourciq unavailable → low-confidence synth
# Synthesis output
incident_summary: str
proposed_actions: list[str]
draft_slack_msg: str
# Human review
human_decision: Literal["approved", "rejected"] | None
human_feedback: str | None # rejection reason for Synthesiser retry
approver_sub: str | None # Google Auth subject id → audit log
# Action output
slack_posted: bool
# Metadata
run_id: str # Redis checkpoint key + Langfuse session_id
retry_count: int # max 3 — prevents infinite rejection loops
total_cost_usd: float # summed across nodes + Tavily| Layer | Technology | Rationale |
|---|---|---|
| Agent orchestration | LangGraph 0.2 | Explicit typed state. Human-in-the-loop interrupt. Conditional edges in code. Deterministic graph — same input, same path. |
| LLM | Claude Sonnet (+ Haiku fallback) | All 4 agents. Structured JSON output. Reliable instruction-following on citation and format constraints. Fallback model on primary unavailability. |
| Observability | Langfuse | One trace per incident, span per node, stitched across the human pause via session_id. Cost + latency per node. |
| Web search | Tavily API | Structured results for LLM consumption. Purpose-built for agentic workflows. Free tier covers demo usage. Circuit-broken — degrades gracefully. |
| Knowledge base | Sourciq (Agent 2) | Live HTTP call to Agent 2's RAG API. Loose coupling — independently deployable. |
| State persistence | Upstash Redis | LangGraph checkpointer — graph state survives Lambda cold starts. Human review can take hours. Serverless, free tier. |
| Reliability | tenacity | Retries with exponential backoff on transient tool/model failures. |
| Notifications | Slack Block Kit | Severity-coded headers, structured actions, footer with run ID and approval attribution. Idempotent send. |
| Backend | FastAPI + Python 3.12 | Async-native. Same runtime as Lambda. |
| Compute | AWS Lambda (eu-north-1) | Same region as Sourciq. Scale to zero. ~$0 at portfolio scale. |
| Frontend | React 18 + TypeScript strict | Discriminated union AppStatus state machine. usePollIncident hook for real-time status. |
| Frontend hosting | Vercel | Automatic HTTPS, CDN, preview deployments. |
| Auth | Google OAuth | id_token verified server-side on every request. Credential in memory only — never localStorage. Approver identity recorded in the audit log. |
CrewAI abstracts state into agent memory. You cannot inspect what each agent decided without parsing natural language. LangGraph gives you a typed state object at every node. For a system that posts to Slack and creates tickets, you need to know exactly what state produced each action. That requirement is non-negotiable in production incident response.
The rule "no external action without human approval" is enforced in the Action node, which raises a GuardrailError if no approval token is present — it is not a sentence in a system prompt asking the model to behave. A prompt-level guardrail is bypassable by injection in the alert payload; a code-level one that throws before the Slack call is not.
Because the graph resumes on a fresh Lambda and reliability adds retries, the same action could be attempted more than once. The Slack send checks an idempotency key on the run id and no-ops if the message already went out, so a resume or retry never double-posts to the incident channel.
If the primary model is overloaded, the node retries once on a fallback model before failing the incident. A model-provider blip should degrade quality, not take the whole incident pipeline down.
A human can reject the incident summary indefinitely. Without a retry cap, a confused or adversarial user can block an incident from resolving. Three retries is the correct operational limit — enough to meaningfully improve the summary, not enough to create an infinite blocking condition. After the third, the incident force-escalates.
Pending incidents expire after APPROVAL_TTL_HOURS (default 10h), with run/checkpoint keys kept for +2h grace (~12h total in Redis). If a human does not review in time, state is dropped and the incident must be resubmitted. Stale P1 context should not be acted on. Slack idempotency (opscanvas:sent:{run_id}) uses a separate 24h TTL so a late resume still cannot double-post.
The Researcher agent calls Sourciq over HTTP rather than importing it as a Python module. This is the correct production pattern. Services are loosely coupled and independently deployable. Sourciq's Lambda could be updated, scaled, or replaced without changing OpsCanvas code.
POST /api/incidents starts the graph in a background thread and returns 202 Accepted immediately with the run_id. The client polls GET /api/incidents/:id. This avoids Lambda's 30-second timeout on the full pipeline (triage + research + synthesis takes 8–15 seconds). At production scale, replace the background thread with SQS + separate Lambda for each graph execution.
P4 (informational) alerts skip the researcher, synthesiser, human review, and action nodes entirely. The conditional edge after triage routes P4 directly to END. This prevents alert fatigue — not every alarm needs an engineer's attention. The auto-close is still written to the audit log.
| Mechanism | What it prevents |
|---|---|
| Immutable action audit log (action · payload · approver · timestamp) | Unaccountable actions — every Slack post, escalation, and auto-close is attributable |
Approver identity (Google Auth sub) on every action |
Anonymous approvals |
| PII / secret scrubbing before logging | Secret or PII leakage into Langfuse / audit store |
| Google id_token server-side verification | Forged authentication |
| Credential in memory only (never localStorage) | XSS token theft |
APP_ENV=prod check removes dev bypass |
Dev bypass reaching production |
| Rate limiting: 10 req/min per IP (slowapi) | API abuse and scraping |
| TrustedHostMiddleware | DNS rebinding attacks |
| Global exception handler | Stack traces in error responses |
| CORS explicit origin list (no wildcard) | Cross-origin request forgery |
| Least-privilege Lambda IAM role | Lateral movement beyond Redis / Slack / Tavily / model endpoint |
~12h state TTL (APPROVAL_TTL_HOURS + 2h) |
Stale incident state accumulation |
| MAX_RETRIES = 3 | Infinite rejection loops |
| All secrets via GitHub Secrets | Credential exposure via git history |
Requires: Authorization: Bearer <google_id_token>
{
"alert_source": "cloudwatch",
"alert_payload": {
"AlarmName": "CRITICAL-PaymentService-ErrorRate",
"AlarmDescription": "Error rate exceeded 25% for 10 minutes",
"NewStateValue": "ALARM",
"Region": "eu-north-1"
}
}Response 202 Accepted:
{
"run_id": "a3f8c2d1-4b5e-6789-abcd-ef0123456789",
"status": "running",
"message": "Incident run started. Poll GET /incidents/{run_id} for status."
}Returns current state. Poll every 3 seconds until status is awaiting_review or completed.
Status values: running · awaiting_review · completed · failed
{ "decision": "approved" }{ "decision": "rejected", "feedback": "Actions too generic — add kubectl commands" }feedback is required when decision is rejected. Returns 422 otherwise.
{ "status": "healthy", "version": "1.0.0", "agent": "opscanvas" }Prerequisites: Python 3.12, Node 22, API keys for Anthropic, Tavily, Upstash Redis, Slack webhook, Google OAuth client ID, Langfuse public/secret keys. Sourciq must be running (or use the deployed Sourciq API).
git clone https://github.com/AttiR/OpsCanvas
cd OpsCanvas
# Backend
python3.12 -m venv venv && source venv/bin/activate
pip install -r backend/requirements.txt
cp .env.example .env # fill in all API keys
make run # starts API on port 8001
# Frontend (separate terminal)
cd frontend && npm install
cp .env.local.example .env.local # fill in VITE_ vars
npm run dev # starts at http://localhost:5173make test # 58 unit tests — zero API/Redis/LLM calls
make eval # agent eval suite — triage · routing · synthesis · e2e (CI-gated)
make lint # ruff check + format
make graph # visualise LangGraph state machine → graph.png
make verify # confirm all imports resolve| Service | Cost |
|---|---|
| AWS Lambda + API Gateway (eu-north-1) | $0 (free tier — 1M requests/month) |
| Upstash Redis | $0 (free tier — 10K commands/day) |
| Tavily | $0 (free tier — 1K searches/month) |
| Langfuse | $0 (free tier) |
| Vercel | $0 (free tier) |
| Slack webhooks | $0 |
| Total infrastructure | $0/month |
Anthropic API cost per incident run: ~$0.02 (4 agents × ~500 tokens each at Claude Sonnet pricing), tracked per incident in Langfuse and capped per run by the cost circuit breaker (MAX_INCIDENT_USD). See Layer 4 · Cost for the Langfuse trace screenshot.
OpsCanvas/
├── backend/
│ ├── app/
│ │ ├── agents/
│ │ │ ├── triage.py Severity classification → TriageResult
│ │ │ ├── researcher.py Sourciq + Tavily research → ResearchResult
│ │ │ ├── synthesiser.py Incident summary → IncidentSummary
│ │ │ ├── action.py Slack Block Kit post (authorized · idempotent · audited)
│ │ │ └── models.py Pydantic output models for all agents
│ │ ├── graph/
│ │ │ ├── state.py IncidentState TypedDict
│ │ │ └── builder.py StateGraph assembly + conditional edges
│ │ ├── api/
│ │ │ └── incidents.py POST/GET /incidents, POST /incidents/:id/review
│ │ ├── core/
│ │ │ ├── redis_store.py checkpointer + save/load/delete/list
│ │ │ ├── observability.py Langfuse trace/span setup, flush
│ │ │ ├── guardrails.py schema + injection + action-auth + abstain
│ │ │ ├── cost.py per-incident CostMeter + circuit breaker
│ │ │ ├── reliability.py timeouts, retries, fallback, circuit breaker
│ │ │ └── audit.py append-only action audit log
│ │ └── integrations/
│ │ └── sourciq.py HTTP client for Sourciq (Agent 2)
│ ├── lambda_handler.py Mangum ASGI adapter
│ ├── main.py FastAPI app — CORS, rate limiting, health
│ ├── requirements.txt
│ └── template.yaml AWS SAM — Lambda + API Gateway + CloudWatch
├── frontend/
│ └── src/
│ ├── components/
│ │ ├── AgentTimeline.tsx 5-node LangGraph state visualisation
│ │ └── ApprovalPanel.tsx Human review — approve/reject/feedback
│ ├── pages/
│ │ ├── LoginPage.tsx Split panel: proof points + Google auth
│ │ └── DashboardPage.tsx Alert input → polling → review → resolved
│ ├── context/
│ │ └── AuthContext.tsx Google id_token in memory only
│ ├── hooks/
│ │ └── usePollIncident.ts Polls GET /incidents/:id every 3s
│ ├── api/
│ │ └── client.ts Typed fetch wrapper, ApiError class
│ └── types/
│ └── api.ts TypeScript types + EXAMPLE_ALERTS
├── tests/
│ ├── test_graph_agents.py State machine, Triage, Researcher (20 tests)
│ ├── test_api_pipeline.py Synthesiser, Redis, FastAPI contracts (25 tests)
│ ├── test_action_agent.py Slack Block Kit, webhook mocked (13 tests)
│ └── eval/
│ ├── triage_set.jsonl labelled alert → expected severity
│ ├── golden_incidents.jsonl alert → expected outcome
│ ├── test_triage_accuracy.py
│ ├── test_routing.py P4 auto-close · P1 escalate (must = 1.0)
│ ├── test_synthesis_judge.py
│ └── test_e2e_success.py
├── .github/workflows/
│ ├── ci.yml pytest + ruff + tsc + Trivy + agent-eval on every PR
│ ├── cd-backend.yml SAM deploy to Lambda on merge to main
│ └── cd-frontend.yml Vercel deploy on merge to main
├── infra/
│ └── secrets-reference.sh All secrets documented
├── Makefile
└── .env.example