Skip to content

Update SRE Agent VNet subnet examples to /28 #269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Deepthi Chelupati (dm-chelupati) merged 1 commit into from
Jul 24, 2026
Merged

Update SRE Agent VNet subnet examples to /28 #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion labs/zava-aks-postgres/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ These are gotchas for someone editing this repo's IaC or Bicep — they're *not*
- **Scenario 2 NetworkPolicy is named `database-tier-isolation`, not `block-postgres`.** This is deliberate de-spoon-feeding: the old name let the agent reach the diagnosis purely from string-matching. The new name reads like a security-architect zero-trust attempt that accidentally over-blocks; the agent has to read the egress rules and reason about the destination subnet to find it.
- **NSG deny rule installed by `break-network.ps1` is a red herring, not a "breadcrumb".** PG Flexible Server private access uses a delegated subnet whose routing/policy is platform-managed (see [subnet-delegation-overview](https://learn.microsoft.com/azure/virtual-network/subnet-delegation-overview) and [PG private networking](https://learn.microsoft.com/azure/postgresql/network/concepts-networking-private)), so a user-added NSG rule on 5432 looks like the smoking gun in config but isn't the active enforcement point. The agent is expected to cross-reference the KB and discount it. Don't reintroduce framing that calls it a "cosmetic clue" — that telegraphs the answer.
- **No `SendOutlookEmail` tool wired into the skills.** Email-out requires an OAuth consent flow that has to be completed by an interactive user in the agent's portal — there is no Bicep/ARM verb to provision it on the agent's behalf. Adding it to `skill.tools[]` without that consent makes the skill *fail to load*. If you want post-remediation email for your own deployment, follow the public docs ([Microsoft 365 connector for SRE Agent](https://learn.microsoft.com/azure/sre-agent/)) to grant consent in the portal, then re-add `'SendOutlookEmail'` to the relevant skill's `tools` array and a "send a summary email" line to the runbook.
- **The network is hub-and-spoke (three VNets), not one flat VNet.** `vnet.bicep` deploys a **hub** (`vnet-Zava-hub-*`, 10.10.0.0/22 — `AzureFirewallSubnet` + the Azure Firewall, a reserved `GatewaySubnet` for a future ExpressRoute/VPN gateway, and `pe-subnet` for the AMPLS private endpoint), a **platform spoke** (`vnet-Zava-platform-*`, 10.20.0.0/16 — `aks-subnet` + delegated `db-subnet`), and an **agent spoke** (`vnet-Zava-agent-*`, 10.30.0.0/24 — the delegated `agent-subnet`). The firewall lives in the **hub**; the agent subnet force-tunnels to it via a UDR (`0.0.0.0/0` → firewall private IP `10.10.0.4`) over VNet peering, and the firewall policy's `sourceAddresses` is the agent subnet (`10.30.0.0/27`) — update both if you renumber. This is safe because the agent reaches AKS via native `kubectl` over the private API-server path and PG through the in-cluster helper, never raw DB sockets, so it never needed to share a VNet with them. Consequence for scripts: anything that picks "the VNet" must select the one containing its target subnet — `break-network.ps1` now queries `[?subnets[?name=='aks-subnet']]`, not `[0]`.
- **The network is hub-and-spoke (three VNets), not one flat VNet.** `vnet.bicep` deploys a **hub** (`vnet-Zava-hub-*`, 10.10.0.0/22 — `AzureFirewallSubnet` + the Azure Firewall, a reserved `GatewaySubnet` for a future ExpressRoute/VPN gateway, and `pe-subnet` for the AMPLS private endpoint), a **platform spoke** (`vnet-Zava-platform-*`, 10.20.0.0/16 — `aks-subnet` + delegated `db-subnet`), and an **agent spoke** (`vnet-Zava-agent-*`, 10.30.0.0/24 — the delegated `agent-subnet`). The firewall lives in the **hub**; the agent subnet force-tunnels to it via a UDR (`0.0.0.0/0` → firewall private IP `10.10.0.4`) over VNet peering, and the firewall policy's `sourceAddresses` is the agent subnet (`10.30.0.0/28`) — update both if you renumber. This is safe because the agent reaches AKS via native `kubectl` over the private API-server path and PG through the in-cluster helper, never raw DB sockets, so it never needed to share a VNet with them. Consequence for scripts: anything that picks "the VNet" must select the one containing its target subnet — `break-network.ps1` now queries `[?subnets[?name=='aks-subnet']]`, not `[0]`.
- **Agent VNet injection is REGIONAL; cross-region reach is via PEERING.** The `agent-subnet` (delegated to `Microsoft.App/environments`) **must be in the same region as the `Microsoft.App/agents` resource** — VNet injection is regional, not a tuning knob ([SRE Agent subnet requirements](https://learn.microsoft.com/azure/sre-agent/network-integration#configure-azure-vnet-mode): *"The subnet must be in the same region as your SRE Agent resource"*). A single-region `azd up` satisfies this automatically — `vnet.bicep` deploys all three VNets and `sre-agent.bicep` deploys the agent with the same `location: location`; don't move the agent subnet to another region expecting injection to work. The agent's **reach is NOT regional**, though: peered to the hub, it routes to anything the hub peers to — **other Azure regions over global VNet peering**, **on-prem over the reserved `GatewaySubnet` gateway** (*"as long as your network routes and rules allow it"*). The on-prem example is just one instance of this. `vnet.bicep` ships a **commented `remote-region` global-peering example** (after the local peerings) and the README's *"Reaching other regions and on-premises"* section is the narrative. Because the agent is force-tunneled to the hub firewall, reaching a new peered range also needs a firewall network rule (`agent-subnet → that range`), not just the peering.
- **Native kubectl requires the AKS private-DNS link, and that link is now codified.** `vnet.bicep` already adds the agent-subnet → AKS API :443 firewall rule and SNAT. The remaining requirement is linking the AKS-managed private-DNS zone (`<guid>.privatelink.<region>.azmk8s.io`, created in the node resource group after cluster creation) to the agent VNet. That zone name is dynamic, so it cannot be a static Bicep resource; `scripts/post-provision.ps1` **Step 4b** now discovers the node RG, azmk8s.io zone, and agent VNet, then idempotently creates the `agent-link` virtual-network link on every deploy. Before this was codified, fresh deploys could spend minutes floundering on private AKS DNS before falling back/proceeding.
- **The Microsoft Learn MCP connector routes entirely through the hub Azure Firewall (no platform bypass) — `raw.githubusercontent.com` must be allow-listed or it silently surfaces zero tools.** The `microsoft-learn` connector is a *Streamable-HTTP* MCP server (`endpoint: https://learn.microsoft.com/api/mcp`). We keep **`allowHttpMcpServerNetworkAccess: false`** (the default) on purpose: when `true`, the platform routes the MCP runtime endpoint as `Rewrite{RoutingMode=Platform}` — a broker that egresses *outside* the customer VNet, bypassing our firewall (an egress escape hatch that contradicts the lab's "every connection gated by our firewall" thesis; empirically, with it on, the MCP runtime to `learn.microsoft.com` never appears in our `AZFWApplicationRule` logs). With it `false`, the MCP host falls under AzureVNet's **default-Allow** and egresses via the VNet → forced-tunnel → hub Azure Firewall. So both the in-sandbox `mcp-broker`'s **server-bits fetch from `raw.githubusercontent.com`** (the `microsoftdocs/mcp` repo, during the `tools/list` handshake) AND the **runtime stream to `learn.microsoft.com`** are gated by our firewall. `vnet.bicep`'s `allow-microsoft-learn` collection allows `learn.microsoft.com` + an `allow-github-raw-mcp-bits` rule scoped to **`raw.githubusercontent.com`** only (the single host the agent hits — verified in `AZFWApplicationRule` denials; no `*.githubusercontent.com` wildcard). Without the GitHub-raw allow the connector provisions `Succeeded` yet shows **"no active connection"** with **zero** tools (a raw GET to `/api/mcp` returns `405` "use a streamable HTTP transport", so the endpoint is reachable — it's the bits fetch that's blocked). This is a **Standard** firewall, so L7 matching is FQDN/SNI only — pinning the exact repo path would need Azure Firewall **Premium** + TLS inspection (`targetUrls`). (The only true pod-side bypass is the platform `ExperimentalSettings.HttpMcpInSandbox` flag, which defaults to the locked-down in-sandbox broker and isn't exposed in our bicep.) **Separately**, MCP connector tools ship `defaultMode: disabled` (skill-gated — they only surface once a skill like `database-incidents` is active). To make the 3 Learn tools part of the **global** tool roster, `setup-sre-agent.ps1` Step 2b enables them via `POST /api/v2/agent/tools/configure` (`{overrides:[{name,enabled}]}`, merge semantics) — there is **no ARM/Bicep property** for per-tool state (the agent's `permissions` stays `null`; Microsoft's `srectl tool config set` CLI exists for exactly this).
Expand Down
2 changes: 1 addition & 1 deletion labs/zava-aks-postgres/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ This lab is configured so the agent uses **native `kubectl`** against the privat
ZONE=$(az network private-dns zone list -g <MC_rg> --query "[?contains(name,'azmk8s')].name|[0]" -o tsv)
az network private-dns link vnet create -g <MC_rg> -z $ZONE -n agent-link -v <agentVnetId> -e false
```
2. **Firewall** — `vnet.bicep` adds an allow rule (`agent-subnet 10.30.0.0/27 → aks-subnet 10.20.0.0/20 :443`) **and SNATs** all traffic (`snat.privateRanges = 255.255.255.255/32`). SNAT is essential: the API server's NSG only admits the `VirtualNetwork` tag and the agent spoke isn't *directly* peered to the platform spoke, so the agent's source IP is rewritten to the firewall's hub IP (which *is* in the tag) — that also makes the return path symmetric without touching the AKS subnet's routing.
2. **Firewall** — `vnet.bicep` adds an allow rule (`agent-subnet 10.30.0.0/28 → aks-subnet 10.20.0.0/20 :443`) **and SNATs** all traffic (`snat.privateRanges = 255.255.255.255/32`). SNAT is essential: the API server's NSG only admits the `VirtualNetwork` tag and the agent spoke isn't *directly* peered to the platform spoke, so the agent's source IP is rewritten to the firewall's hub IP (which *is* in the tag) — that also makes the return path symmetric without touching the AKS subnet's routing.

**Agent in-session setup — encoded in the skill runbook:**
1. `az aks get-credentials -g <rg> -n <aks> --overwrite-existing`
Expand Down
20 changes: 10 additions & 10 deletions labs/zava-aks-postgres/infra/main.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"_generator": {
"name": "bicep",
"version": "0.43.1.21952",
"templateHash": "3184786289939969259"
"templateHash": "17454459993975253304"
}
},
"parameters": {
Expand Down Expand Up @@ -83,7 +83,7 @@
"_generator": {
"name": "bicep",
"version": "0.43.1.21952",
"templateHash": "14346587368849512345"
"templateHash": "7676869149456753884"
}
},
"parameters": {
Expand Down Expand Up @@ -252,7 +252,7 @@
{
"name": "agent-subnet",
"properties": {
"addressPrefix": "10.30.0.0/27",
"addressPrefix": "10.30.0.0/28",
"routeTable": {
"id": "[resourceId('Microsoft.Network/routeTables', format('rt-agent-{0}', parameters('uniqueSuffix')))]"
},
Expand Down Expand Up @@ -401,7 +401,7 @@
"TCP"
],
"sourceAddresses": [
"10.30.0.0/27"
"10.30.0.0/28"
],
"destinationAddresses": [
"168.63.129.16"
Expand All @@ -428,7 +428,7 @@
"TCP"
],
"sourceAddresses": [
"10.30.0.0/27"
"10.30.0.0/28"
],
"destinationAddresses": [
"10.20.0.0/20"
Expand All @@ -455,7 +455,7 @@
"TCP"
],
"sourceAddresses": [
"10.30.0.0/27"
"10.30.0.0/28"
],
"destinationAddresses": [
"10.10.2.0/27"
Expand All @@ -482,7 +482,7 @@
"TCP"
],
"sourceAddresses": [
"10.30.0.0/27"
"10.30.0.0/28"
],
"destinationAddresses": "[if(parameters('lockAgentToPrivateMonitor'), createArray('AzureResourceManager', 'AzureActiveDirectory'), createArray('AzureResourceManager', 'AzureActiveDirectory', 'AzureMonitor'))]",
"destinationPorts": [
Expand All @@ -504,7 +504,7 @@
"name": "allow-arm-aad-graph",
"description": "FQDN access to ARM, Entra ID, and Microsoft Graph",
"sourceAddresses": [
"10.30.0.0/27"
"10.30.0.0/28"
],
"protocols": [
{
Expand Down Expand Up @@ -533,7 +533,7 @@
"name": "allow-learn-microsoft-com",
"description": "Microsoft Learn docs + MCP runtime endpoint (the agent looks up Azure/AKS/PostgreSQL guidance here)",
"sourceAddresses": [
"10.30.0.0/27"
"10.30.0.0/28"
],
"protocols": [
{
Expand All @@ -551,7 +551,7 @@
"name": "allow-github-raw-mcp-bits",
"description": "GitHub raw content — the Microsoft Learn MCP connector fetches its server bits here to complete the tool-discovery handshake",
"sourceAddresses": [
"10.30.0.0/27"
"10.30.0.0/28"
],
"protocols": [
{
Expand Down
24 changes: 12 additions & 12 deletions labs/zava-aks-postgres/infra/modules/vnet.bicep
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ param lockAgentToPrivateMonitor bool = true
// └─ db-subnet 10.20.16.0/24 PostgreSQL Flexible Server delegation.
//
// SPOKE 2 — agent vnet-Zava-agent-* 10.30.0.0/24 (the SRE Agent)
// └─ agent-subnet 10.30.0.0/27 Microsoft.App/environments delegation;
// └─ agent-subnet 10.30.0.0/28 Microsoft.App/environments delegation;
// ALL egress forced to the hub firewall
// via a UDR (0.0.0.0/0 → firewall private
// IP) over peering.
Expand Down Expand Up @@ -226,10 +226,10 @@ resource agentVnet 'Microsoft.Network/virtualNetworks@2024-01-01' = {
{
// SRE Agent workload subnet — delegated to Microsoft.App/environments so
// the agent's sandbox is injected here, with all egress forced through
// the hub Azure Firewall (route table above). Minimum size is /27.
// the hub Azure Firewall (route table above). Minimum size is /28.
name: 'agent-subnet'
properties: {
addressPrefix: '10.30.0.0/27'
addressPrefix: '10.30.0.0/28'
routeTable: { id: agentRouteTable.id }
delegations: [
{
Expand Down Expand Up @@ -372,7 +372,7 @@ resource peerAgentToHub 'Microsoft.Network/virtualNetworks/virtualNetworkPeering
// // range to the ruleCollectionGroup below (the module already SNATs all egress,
// // so the return path stays symmetric):
// // { name: 'allow-agent-to-remote-region', ruleType: 'NetworkRule',
// // sourceAddresses: ['10.30.0.0/27'], destinationAddresses: ['10.40.0.0/24'],
// // sourceAddresses: ['10.30.0.0/28'], destinationAddresses: ['10.40.0.0/24'],
// // destinationPorts: ['*'], ipProtocols: ['Any'] }
// // Cross-region peering alone is enough for VNet-to-VNet traffic that ISN'T
// // force-tunneled; this lab force-tunnels the agent, hence the extra firewall rule.
Expand Down Expand Up @@ -426,7 +426,7 @@ resource firewallPolicy 'Microsoft.Network/firewallPolicies@2024-05-01' = {
// exposing the cluster API server publicly.
// AzureCloud is deliberately NOT used (it covers ~65k prefixes including
// third-party SaaS); precise service tags are used instead. The source is the
// agent spoke's subnet (10.30.0.0/27).
// agent spoke's subnet (10.30.0.0/28).
//
// To let the agent reach a NETWORK DEVICE or other private service DIRECTLY, its
// management endpoint must be HTTPS and its FQDN added BOTH here (an application
Expand All @@ -451,7 +451,7 @@ resource ruleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionG
name: 'allow-azure-dns'
description: 'DNS resolution via Azure DNS (required for the firewall DNS proxy)'
ipProtocols: ['UDP', 'TCP']
sourceAddresses: ['10.30.0.0/27']
sourceAddresses: ['10.30.0.0/28']
destinationAddresses: ['168.63.129.16']
destinationPorts: ['53']
}
Expand All @@ -475,7 +475,7 @@ resource ruleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionG
name: 'agent-to-apiserver'
description: 'Agent subnet -> AKS API server (enables native kubectl)'
ipProtocols: ['TCP']
sourceAddresses: ['10.30.0.0/27']
sourceAddresses: ['10.30.0.0/28']
destinationAddresses: ['10.20.0.0/20']
destinationPorts: ['443']
}
Expand All @@ -500,7 +500,7 @@ resource ruleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionG
name: 'agent-to-ampls-pe'
description: 'Agent subnet -> AMPLS private endpoint (private Azure Monitor)'
ipProtocols: ['TCP']
sourceAddresses: ['10.30.0.0/27']
sourceAddresses: ['10.30.0.0/28']
destinationAddresses: ['10.10.2.0/27']
destinationPorts: ['443']
}
Expand All @@ -517,7 +517,7 @@ resource ruleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionG
name: 'allow-azure-services-l4'
description: 'L4 access to Azure services via precise service tags (NOT AzureCloud)'
ipProtocols: ['TCP']
sourceAddresses: ['10.30.0.0/27']
sourceAddresses: ['10.30.0.0/28']
// AzureMonitor is dropped by DEFAULT (lockAgentToPrivateMonitor=true)
// so the agent reaches Monitor only over the AMPLS private endpoint —
// private-only / maximum restraint; the agent remains fully functional
Expand All @@ -544,7 +544,7 @@ resource ruleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionG
ruleType: 'ApplicationRule'
name: 'allow-arm-aad-graph'
description: 'FQDN access to ARM, Entra ID, and Microsoft Graph'
sourceAddresses: ['10.30.0.0/27']
sourceAddresses: ['10.30.0.0/28']
protocols: [{ protocolType: 'Https', port: 443 }]
targetFqdns: [
'management.azure.com'
Expand All @@ -564,7 +564,7 @@ resource ruleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionG
ruleType: 'ApplicationRule'
name: 'allow-learn-microsoft-com'
description: 'Microsoft Learn docs + MCP runtime endpoint (the agent looks up Azure/AKS/PostgreSQL guidance here)'
sourceAddresses: ['10.30.0.0/27']
sourceAddresses: ['10.30.0.0/28']
protocols: [{ protocolType: 'Https', port: 443 }]
targetFqdns: [
'learn.microsoft.com'
Expand All @@ -585,7 +585,7 @@ resource ruleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionG
// (raw.githubusercontent.com/microsoftdocs/mcp/*) you'd need Azure
// Firewall Premium + TLS inspection (targetUrls). See README caveats.
description: 'GitHub raw content — the Microsoft Learn MCP connector fetches its server bits here to complete the tool-discovery handshake'
sourceAddresses: ['10.30.0.0/27']
sourceAddresses: ['10.30.0.0/28']
protocols: [{ protocolType: 'Https', port: 443 }]
targetFqdns: [
'raw.githubusercontent.com'
Expand Down
Loading
Loading