Skip to content

Add azmon-alert-noise-filter-snow recipe + deploy script fixes - #251

Open
Deepthi Chelupati (dm-chelupati) wants to merge 1 commit into
mainfrom
feature/azmon-alert-noise-filter-snow
Open

Add azmon-alert-noise-filter-snow recipe + deploy script fixes#251
Deepthi Chelupati (dm-chelupati) wants to merge 1 commit into
mainfrom
feature/azmon-alert-noise-filter-snow

Conversation

@dm-chelupati

Copy link
Copy Markdown
Collaborator

New Recipe: Azure Monitor Alert Noise Filtering with ServiceNow MCP

This recipe deploys an SRE Agent configured to triage Azure Monitor alerts, filter noise, and create ServiceNow tickets for real issues.

What the recipe includes

  • Triage subagent with alert-noise-filter skill — transient detection (observation window + re-check), dependency deduplication (correlate alerts across resource groups), repeat alert detection
  • PythonFunctionTool timertime.sleep in subprocess for observation window, no LLM tokens consumed during wait
  • ServiceNow MCP connector (stdio, @onlyflows/servicenow-mcp) — create incidents, add work notes, query records
  • Response plan — Azure Monitor Sev0-3, merge window 1 hour
  • Hooks — verify-alert-state-before-action (command), deny-prod-deletes (prompt), require-approval-for-restarts (prompt)
  • Tool permissions — allow RunInTerminal/ExecutePythonCode, ask RunAzCliWriteCommands, deny FetchWebpage/SearchWebpage
  • Alert simulation scripts — fire-noisy-alert.sh and fire-real-alert.sh for testing

Deploy script fixes

  • new-agent.sh: Add snowPasswordSERVICENOW_PASSWORD and gitlabPatGITLAB_PAT secret mappings
  • apply-extras.sh: Add GitLab repo support (PAT auth via /api/v2/gitlab/auth), tool deployment via PUT /api/v1/extendedAgent/apply with Content-Type: application/yaml and kind: ToolList
  • assemble-agent.sh: Pass tools through Bicep params

Tested

  • Deployed to rg-noise-filter (swedencentral) monitoring rg-contoso-swe (contoso trading app)
  • Transient alerts: agent waits through observation window, confirms self-resolved, closes as noise
  • Persistent alerts: agent investigates via App Insights + LAW, creates ServiceNow ticket with RCA and remediation steps
  • ServiceNow MCP: confirmed ticket creation on dev432295.service-now.com

Blog

Zero Alert Fatigue: Let SRE Agent Separate Real Alerts from Noise

@dm-chelupati
Deepthi Chelupati (dm-chelupati) force-pushed the feature/azmon-alert-noise-filter-snow branch 3 times, most recently from fed4115 to cdcad99 Compare July 16, 2026 00:29
Recipe: Azure Monitor alert noise filtering with ServiceNow MCP integration
- Triage subagent with alert-noise-filter skill for transient detection,
  dependency deduplication, and repeat alert detection
- PythonFunctionTool timer for observation window (time.sleep in subprocess)
- ServiceNow MCP connector (stdio, @onlyflows/servicenow-mcp) for ticket creation
- Response plan: Azure Monitor Sev0-3, merge window 1hr
- Hooks: verify-alert-state-before-action, deny-prod-deletes, require-approval-for-restarts
- Tool permissions: allow RunInTerminal/ExecutePythonCode, ask RunAzCliWriteCommands, deny FetchWebpage

Deploy script fixes:
- new-agent.sh: Add snowPassword -> SERVICENOW_PASSWORD and gitlabPat -> GITLAB_PAT secret mappings
- apply-extras.sh: Add GitLab repo support (PAT auth via /api/v2/gitlab/auth),
  tool deployment via PUT /api/v1/extendedAgent/apply with Content-Type: application/yaml
- assemble-agent.sh: Pass tools through Bicep params
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant