From 30312ce1ebc44c4fce7338754a35dde340cd6456 Mon Sep 17 00:00:00 2001 From: Paulo Lacerda Date: Mon, 1 Jun 2026 16:02:52 -0300 Subject: [PATCH 1/2] docs(tutorials): document data-plane RBAC step missing from Foundry portal Creating a Foundry project through the portal only assigns the user 'Foundry User' at the project scope. That role does not cover OpenAI data-plane actions on the parent AI Services account, where chat completions actually live - so every AI-assisted evaluator and every cloud-eval grader fails with PermissionDenied the first time a fresh workspace tries to run eval. Subscription Owner is also insufficient because the built-in Owner role has actions: ['*'] but dataActions: []. All three tutorials (prompt-agent quickstart, hosted-agent quickstart, end-to-end) now document the one-time 'az role assignment create' that grants 'Cognitive Services OpenAI User' at the resource-group scope of the Foundry account, with the exact error signature so future readers can self-diagnose if they skipped it. A future AgentOps Doctor check will detect the missing assignment pre-run; until then, this step is a documented manual prerequisite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/tutorial-end-to-end.md | 28 +++++++++++++++++++ docs/tutorial-hosted-agent-quickstart.md | 26 ++++++++++++++++++ docs/tutorial-prompt-agent-quickstart.md | 34 ++++++++++++++++++++++++ 3 files changed, 88 insertions(+) diff --git a/docs/tutorial-end-to-end.md b/docs/tutorial-end-to-end.md index 27fdbe5..4074a9d 100644 --- a/docs/tutorial-end-to-end.md +++ b/docs/tutorial-end-to-end.md @@ -286,6 +286,34 @@ for creating agents, tools, tracing, evaluation, and red-team scans: https://github.com/Azure-Samples/microsoft-foundry-e2e-agent-observability-workshop/tree/2026-04-aie-europe ``` +### Grant your identity data-plane access to the AI Services account + +Both options above (prompt agent and hosted HTTP agent) eventually drive +an `agentops eval run` that calls chat-completions on the AI Services +account behind your Foundry project — either through Foundry's cloud +graders or through the local AI-assisted evaluators. Creating a project +through the portal assigns you `Foundry User` **only at the project +scope**, which does not cover OpenAI data-plane actions on the parent +account. Subscription `Owner` is also insufficient: its built-in role +definition has `actions: ["*"]` but `dataActions: []`. Skipping this is +what causes the eval to fail later with `PermissionDenied` on +`Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/ +completions/action`. + +Run the assignment once per resource group that hosts a Foundry account +you will evaluate against. Replace ``, +``, and `` with your own values (use +`az ad signed-in-user show --query id -o tsv` to get the object ID): + +```powershell +az role assignment create ` + --assignee ` + --role "Cognitive Services OpenAI User" ` + --scope /subscriptions//resourceGroups/ +``` + +Propagation usually completes within 30–120 seconds. + ## 2. Create the travel eval dataset ```powershell diff --git a/docs/tutorial-hosted-agent-quickstart.md b/docs/tutorial-hosted-agent-quickstart.md index fffdbae..9c7ae2e 100644 --- a/docs/tutorial-hosted-agent-quickstart.md +++ b/docs/tutorial-hosted-agent-quickstart.md @@ -310,6 +310,32 @@ If the deployed endpoint needs a bearer token: $env:HOSTED_AGENT_TOKEN = "" ``` +### Grant your identity data-plane access to the AI Services account + +The local AI-assisted evaluators that AgentOps runs in step 8 call +chat-completions on the AI Services account that backs your Foundry +project. Creating a project through the portal only assigns you +`Foundry User` **at the project scope**, which does not cover the +OpenAI data-plane action on the parent account. Even subscription +`Owner` is insufficient: the built-in `Owner` role has `actions: ["*"]` +but `dataActions: []`. Skipping this once causes the eval to fail with +`PermissionDenied` on `Microsoft.CognitiveServices/accounts/OpenAI/ +deployments/chat/completions/action`. + +Run the assignment once per resource group hosting a Foundry account +you will evaluate against (replace ``, +``, and `` with your values; get the +object ID with `az ad signed-in-user show --query id -o tsv`): + +```powershell +az role assignment create ` + --assignee ` + --role "Cognitive Services OpenAI User" ` + --scope /subscriptions//resourceGroups/ +``` + +Propagation usually completes within 30–120 seconds. + ## 5. Initialize AgentOps interactively ```powershell diff --git a/docs/tutorial-prompt-agent-quickstart.md b/docs/tutorial-prompt-agent-quickstart.md index 8126f67..d622b25 100644 --- a/docs/tutorial-prompt-agent-quickstart.md +++ b/docs/tutorial-prompt-agent-quickstart.md @@ -241,6 +241,40 @@ Show me the planned changes and the resulting endpoints before applying. If the skill is not available, use Path A. +### Grant your identity data-plane access to the AI Services account + +Creating a project through the portal only assigns you `Foundry User` **at +the project scope**. That role does not cover the OpenAI data-plane actions +that live on the parent AI Services *account* — the chat-completions call +that backs every AI-assisted evaluator and every cloud-eval grader. Even +`Owner` on the subscription is not enough: the built-in `Owner` role +definition has `actions: ["*"]` but `dataActions: []`, so it grants full +control plane and zero data plane on Cognitive Services accounts. + +Skipping this step is what causes the eval grader to fail later with:: + + PermissionDenied: The principal `` lacks the required + data action `Microsoft.CognitiveServices/accounts/OpenAI/deployments/ + chat/completions/action` to perform `POST /openai/deployments/...` + +Run the assignment once per resource group that hosts a Foundry account +you will evaluate against. Replace ``, ``, +and `` with your own values (you can get the object ID +with `az ad signed-in-user show --query id -o tsv`): + +```powershell +az role assignment create ` + --assignee ` + --role "Cognitive Services OpenAI User" ` + --scope /subscriptions//resourceGroups/ +``` + +Repeat the command with the `travel-agent-dev` resource group if the dev +project lives in a different RG. The assignment usually propagates within +30–120 seconds. AgentOps Doctor will detect the missing assignment in a +future release, but until then this is a manual one-time setup step per +new environment. + ## 4. Seed `travel-agent` in the sandbox project You only author the agent in **one place**: your sandbox Foundry From 7b650d61a2bd16311a8f191b2674be74144945f4 Mon Sep 17 00:00:00 2001 From: Paulo Lacerda Date: Mon, 1 Jun 2026 16:13:22 -0300 Subject: [PATCH 2/2] feat(skills): preflight data-plane RBAC in agentops-eval skill The agentops-eval coding-agent skill now resolves the Foundry project endpoint from .azure//.env or .agentops/.env, looks up the backing AI Services account + resource group with az cognitiveservices account list, fetches the signed-in object ID, and runs an idempotent az role assignment create for 'Cognitive Services OpenAI User' at the resource-group scope BEFORE 'agentops eval analyze' / 'agentops eval run'. This mirrors the new manual step added in the same PR to all three tutorials and keeps the skill experience aligned: users running the skill against a fresh Foundry project no longer hit the 401 PermissionDenied that the portal's default 'Foundry User'-at-project assignment leaves behind. CHANGELOG entry added under [Unreleased]. Plugin skills mirror under plugins/agentops/skills/ regenerated via scripts/sync-skills.ps1 to keep the VS Code extension copy identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 24 ++++++++++ .../agentops/skills/agentops-eval/SKILL.md | 48 +++++++++++++++++++ .../templates/skills/agentops-eval/SKILL.md | 48 +++++++++++++++++++ 3 files changed, 120 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a5e4392..c0b0334 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,30 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres ## [Unreleased] +### Changed +- **`agentops-eval` coding-agent skill now preflights the data-plane RBAC + step that the Foundry portal does not assign by default.** Creating a + Foundry project through the portal only grants the user `Foundry User` + at the *project* scope, which does not cover + `Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action` + on the parent AI Services account where chat completions actually live. + Subscription `Owner` is also insufficient because the built-in `Owner` + role definition has `actions: ["*"]` but `dataActions: []`. The first + `agentops eval run` against a fresh workspace therefore failed with + `PermissionDenied` on every AI-assisted evaluator and every cloud-eval + grader. The skill's new **Step 0.5 - Ensure data-plane RBAC on the AI + Services account** resolves the Foundry project endpoint from + `.azure//.env` or `.agentops/.env`, looks up the backing AI + Services account + resource group with + `az cognitiveservices account list`, fetches the signed-in object ID + with `az ad signed-in-user show`, and runs an idempotent + `az role assignment create` for `Cognitive Services OpenAI User` at + the resource-group scope before handing off to `agentops eval analyze`. + This keeps the skill experience consistent with the new manual + instructions added to the prompt-agent, hosted-agent, and end-to-end + tutorials, so users running the skill against a fresh Foundry project + no longer hit the same 401 the manual tutorials previously hid. + ## [0.3.4] - 2026-06-01 ### Fixed diff --git a/plugins/agentops/skills/agentops-eval/SKILL.md b/plugins/agentops/skills/agentops-eval/SKILL.md index 093a0ea..662fb53 100644 --- a/plugins/agentops/skills/agentops-eval/SKILL.md +++ b/plugins/agentops/skills/agentops-eval/SKILL.md @@ -25,6 +25,54 @@ with a `name:version` or URL. (`--project-endpoint`, `--agent`, `--dataset`, …) for non-interactive runs. Run `agentops init show` later to inspect the resolved config. +## Step 0.5 - Ensure data-plane RBAC on the AI Services account + +AgentOps eval (cloud graders **and** local AI-assisted evaluators) calls +`/openai/deployments/.../chat/completions` on the AI Services account +that backs the Foundry project. Creating a project through the Foundry +portal only assigns the user `Foundry User` at the *project* scope, +which does **not** cover OpenAI data-plane actions on the parent +account. Subscription `Owner` is also insufficient because the built-in +`Owner` role has `actions: ["*"]` but `dataActions: []`. The first +`agentops eval run` against a fresh workspace will otherwise fail with: + +``` +PermissionDenied … lacks the required data action +'Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action' +``` + +Run this preflight before Step 1 - it is idempotent (Azure returns +`RoleAssignmentExists` if already granted) and takes ~5 seconds: + +```bash +# 1. Resolve the AI Services account from agentops.yaml / .azure//.env +PROJECT_ENDPOINT=$(grep -h '^AZURE_AI_FOUNDRY_PROJECT_ENDPOINT' .azure/*/.env .agentops/.env 2>/dev/null | tail -1 | cut -d= -f2- | tr -d '"') +ACCOUNT_HOST=$(echo "$PROJECT_ENDPOINT" | awk -F[/:] '{print $4}') +ACCOUNT_NAME=$(echo "$ACCOUNT_HOST" | cut -d. -f1) + +# 2. Resolve subscription, resource group, and signed-in object ID +SUB_ID=$(az account show --query id -o tsv) +RG=$(az cognitiveservices account list --subscription "$SUB_ID" --query "[?name=='$ACCOUNT_NAME'].resourceGroup | [0]" -o tsv) +OBJ_ID=$(az ad signed-in-user show --query id -o tsv) + +# 3. Grant data-plane access at the RG scope (covers sandbox + future evals) +az role assignment create \ + --assignee "$OBJ_ID" \ + --role "Cognitive Services OpenAI User" \ + --scope "/subscriptions/$SUB_ID/resourceGroups/$RG" +``` + +PowerShell equivalent: replace `$(...)` with the PowerShell variable +assignments shown in `docs/tutorial-prompt-agent-quickstart.md`. + +If the user has not run `az login` yet, do that first. If +`az cognitiveservices account list` returns an empty RG, the AI Services +account lives in a different subscription - ask the user which one. + +Skip this step only if the user explicitly says the role is already +assigned, or if a previous `agentops eval run` succeeded against the +same Foundry account. + ## Step 1 - Analyze evaluation setup Run the deterministic local triage first: diff --git a/src/agentops/templates/skills/agentops-eval/SKILL.md b/src/agentops/templates/skills/agentops-eval/SKILL.md index 093a0ea..662fb53 100644 --- a/src/agentops/templates/skills/agentops-eval/SKILL.md +++ b/src/agentops/templates/skills/agentops-eval/SKILL.md @@ -25,6 +25,54 @@ with a `name:version` or URL. (`--project-endpoint`, `--agent`, `--dataset`, …) for non-interactive runs. Run `agentops init show` later to inspect the resolved config. +## Step 0.5 - Ensure data-plane RBAC on the AI Services account + +AgentOps eval (cloud graders **and** local AI-assisted evaluators) calls +`/openai/deployments/.../chat/completions` on the AI Services account +that backs the Foundry project. Creating a project through the Foundry +portal only assigns the user `Foundry User` at the *project* scope, +which does **not** cover OpenAI data-plane actions on the parent +account. Subscription `Owner` is also insufficient because the built-in +`Owner` role has `actions: ["*"]` but `dataActions: []`. The first +`agentops eval run` against a fresh workspace will otherwise fail with: + +``` +PermissionDenied … lacks the required data action +'Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action' +``` + +Run this preflight before Step 1 - it is idempotent (Azure returns +`RoleAssignmentExists` if already granted) and takes ~5 seconds: + +```bash +# 1. Resolve the AI Services account from agentops.yaml / .azure//.env +PROJECT_ENDPOINT=$(grep -h '^AZURE_AI_FOUNDRY_PROJECT_ENDPOINT' .azure/*/.env .agentops/.env 2>/dev/null | tail -1 | cut -d= -f2- | tr -d '"') +ACCOUNT_HOST=$(echo "$PROJECT_ENDPOINT" | awk -F[/:] '{print $4}') +ACCOUNT_NAME=$(echo "$ACCOUNT_HOST" | cut -d. -f1) + +# 2. Resolve subscription, resource group, and signed-in object ID +SUB_ID=$(az account show --query id -o tsv) +RG=$(az cognitiveservices account list --subscription "$SUB_ID" --query "[?name=='$ACCOUNT_NAME'].resourceGroup | [0]" -o tsv) +OBJ_ID=$(az ad signed-in-user show --query id -o tsv) + +# 3. Grant data-plane access at the RG scope (covers sandbox + future evals) +az role assignment create \ + --assignee "$OBJ_ID" \ + --role "Cognitive Services OpenAI User" \ + --scope "/subscriptions/$SUB_ID/resourceGroups/$RG" +``` + +PowerShell equivalent: replace `$(...)` with the PowerShell variable +assignments shown in `docs/tutorial-prompt-agent-quickstart.md`. + +If the user has not run `az login` yet, do that first. If +`az cognitiveservices account list` returns an empty RG, the AI Services +account lives in a different subscription - ask the user which one. + +Skip this step only if the user explicitly says the role is already +assigned, or if a previous `agentops eval run` succeeded against the +same Foundry account. + ## Step 1 - Analyze evaluation setup Run the deterministic local triage first: