diff --git a/docs/mcp/README.md b/docs/mcp/README.md index 87d07ce04..e95569d61 100644 --- a/docs/mcp/README.md +++ b/docs/mcp/README.md @@ -19,6 +19,8 @@ them. The [Toolkit index](../README.md) also links the CLI and shared guides. [remote](../testing/remote.md) — tests and transport QA. - [HTTP Code Mode](../testing/code-mode.md) — runtime integration, permissions, and safety validation. +- [Code Mode evaluations](../testing/code-mode-evals.md) — agent discovery, + stateful API fixtures, task verification and historical results. ## Run and Release diff --git a/docs/testing/code-mode-evals.md b/docs/testing/code-mode-evals.md new file mode 100644 index 000000000..d18229d97 --- /dev/null +++ b/docs/testing/code-mode-evals.md @@ -0,0 +1,304 @@ +# HTTP Code Mode evaluations + +The agent evaluation harness tests tasks missing from the normal MCP catalog +through the real MCP transport and Dynamic Worker runtime. Sentry API responses +come from stateful local fixtures; an independent verifier checks persisted +state rather than accepting the agent's success claim. + +See [HTTP Code Mode](code-mode.md) for the runtime, permissions and deployment +boundary. These evaluations do not exercise hosted OAuth or authorize production +writes. Historical successes are not fresh evaluations of every later runtime +change. SDK comparisons remain future work. + +## Surfaces and fixtures + +`packages/mcp-server-evals/src/code-mode/run-dashboard.ts` starts an authenticated +loopback API fixture and a local Wrangler Worker, then creates a fresh external +agent session. It never supplies a task algorithm. Each repetition resets state. + +| Surface | Available tools and agent instructions | +| ------------------ | ------------------------------------------------------------------------------------------------------------- | +| `code-mode` | The two HTTP tools, with explicit instructions to use them; execution baseline only | +| `catalog-and-code` | Normal top-level tools/catalog plus Code Mode; Claude's default system prompt and an outcome-only user prompt | +| `catalog` | The same normal catalog without Code Mode; negative control | + +Natural discovery prompts do not mention Code Mode, JavaScript, HTTP, schemas or +tool names. The same MCP is named `sentry`; release examples in tool descriptions +avoid scaffolding dashboard copying. Discovery searches the full pinned OpenAPI +snapshot (159 paths / 254 operations), although only fixture routes execute. +No catalog-first sequence is required: selection is the behavior being measured. + +All surfaces use an empty temporary directory and disable other MCP servers, +built-in tools, hooks, skills and persistence where supported by the client. +This is a controlled MCP selection test, not a test with every client capability +enabled. `discovery-worker.ts` uses the real core server and rejects non-loopback +API configuration. It is an eval-only composition, not hosted authorization. + +The isolated entrypoint uses a direct bearer token held outside generated code. +Server-owned `API_BASE_URL`, `OPENAPI_URL` and `CODE_MODE_ALLOW_WRITES` configure +its API origin/prefix, trusted dereferenced schema and writes (false by default). +Evaluations supply local URLs, synthetic data, a dummy token and explicit writes +where needed. They do not call a real Sentry account. + +| Task | Missing catalog capability | Independent verification | +| --------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `dashboard` | Create dashboard copies | Exactly two copies for two projects with environment `staging`; source, widgets, queries, order, layout, other filters and time range preserved; real IDs/links | +| `release` | Release/deploy writes | Exactly one release for two projects and one staging deploy; requested metadata/timestamps; original resources unchanged; actual deploy ID | +| `repositories` | Repository listing | All 107 active records retrieved across pages; four matching GitHub repositories with exact IDs/names/URLs and count; sorted, no disabled results or writes | +| `alert` | Issue alert editing | Workflow disabled with frequency 60; conditions, actions, owner, environment, detector associations and another alert preserved at every write | +| `monitor` | Cron monitor editing | Requested weekday schedule/timezone; name, slug, owner, enabled state, thresholds, notifications, per-environment mute state and another monitor preserved at every write | +| `release-recovery` | Safe recovery after an ambiguous write | A deploy persists but its first POST response is 502; exactly one deploy must remain, with its real ID | +| `repositories-retry` | Read recovery | First inventory GET is 503; complete paginated result, no writes | +| `repositories-denied` | Honest permission failure handling | Persistent 403 yields JSON `blocked` with a reason; no invented inventory/writes and at most two denied collection attempts | + +The repository page cap is 100 and matching results exist on page two. Release +grading inspects all persisted deployments even though the API lists the latest +per project/environment. Alert/monitor graders inspect every successful write, +so later repairs cannot erase collateral changes. A blocked 403 outcome is a +successful evaluation of limitation handling, not a completed inventory. +Recovery variants reuse the original prompts; injected faults are not revealed. + +Fixtures model CLI task semantics, not a benchmark run of the native CLI: +`sentry release create/finalize/deploy/deploys`, `sentry repo list`, +`sentry alert issues edit`, and `sentry api` for monitor changes. Routes were +checked against CLI revision `b625111aef9cc3a44abbed634a043fce8505366d` and Sentry +revision `b4b356554798716ceaeab74efdc095782be9ca16`. Auxiliary alert project, +detector and workflow reads were checked against Sentry revision +`c4b7b5f3502b7f7c2ae8edc9ae1da36c8a1c98a4`. Reports record provenance and source +hashes. Dashboard schema provenance is in `dashboard-fixture-data.json`. + +Fixtures do not emulate query-language validation, quotas, feature flags, all +valid auxiliary routes or real authorization. Unimplemented fixture routes may +remain discoverable. Their 404s are not automatically product failures. + +## Running evaluations + +Use Node 24 for this checkout. After `pnpm install --frozen-lockfile`: + +```bash +pnpm --filter '@sentry/mcp-core...' --if-present run build +pnpm --filter @sentry/mcp-cloudflare test:code-mode +pnpm --filter @sentry/mcp-cloudflare tsc:code-mode +pnpm --filter @sentry/mcp-server-evals test:code-mode +pnpm --filter @sentry/mcp-server-evals tsc:code-mode + +# Natural selection, followed by a catalog-only control: +pnpm --filter @sentry/mcp-server-evals eval:code-mode:discovery --runs 3 +pnpm --filter @sentry/mcp-server-evals eval:code-mode --client claude --surface catalog --runs 1 + +# Additional operations and failure recovery: +pnpm --filter @sentry/mcp-server-evals eval:code-mode:flows --runs 3 +pnpm --filter @sentry/mcp-server-evals eval:code-mode:resilience --runs 5 +pnpm --filter @sentry/mcp-server-evals eval:code-mode --client claude --surface catalog-and-code --task alert --runs 3 + +# Isolated execution baseline: +pnpm --filter @sentry/mcp-server-evals eval:code-mode --client claude --runs 3 +pnpm --filter @sentry/mcp-server-evals eval:code-mode --client codex --model '' --runs 3 +``` + +Fixture tests and typechecks require no model credentials and run through the +package's `sentryCi` configuration. Model-based evals require an installed, +authenticated CLI session and use that account's model access. The server does +not call a model. The default Codex client requires an explicit available model; +discovery/control surfaces require Claude. + +Codex uses `exec --ignore-user-config` with medium reasoning. Some built-in tools +and global `CODEX_HOME/AGENTS.md` may still be included. Claude uses its default +model unless overridden, records the resolved model, and sets medium effort. +The isolated baseline gives both clients the same task/instructions and rejects +outside tools. Claude discovery surfaces retain the default system prompt and +allow the selected MCP inventory. Keep client, model, settings and budgets fixed +when comparing changes; these are client benchmarks, not bare-model benchmarks. + +Reports default to `.cache/code-mode/http---.json`. +`--report` selects another destination for one task only. Multi-task suites keep +separate reports. Failed verification still writes evidence and exits nonzero; +that includes catalog-only controls that cannot complete their task. + +Reports retain requested/resolved model, client version, prompts, full inventory, +tool sequence, generated code, usage, API outcomes, state and implementation +hashes. Synthetic reports are local artifacts, not committed generated files. +Claude usage is taken once from its final aggregate: uncached input, cache +creation, cache reads and output remain separate. Do not count cached tokens as +free or sum aggregate and per-message usage. Subscription billing is not inferred +from client-reported dollar estimates. + +## Historical dashboard evidence + +The following pilot results predate the hosted authorization boundary and RPC +dispatcher. They establish fixture-backed behavior at their recorded revisions, +not production readiness or current success rates. + +### Natural selection, 2026-10-26 + +Claude Code 2.1.263, resolved `claude-opus-5`, medium effort: + +| Surface | Result | Actual selection | MCP calls | API calls | Wall time | +| -------------------------- | --------------- | --------------------------------------------- | --------- | --------- | --------- | +| Catalog + Code Mode, run 1 | Pass | Execute → schema search → execute | 3 | 5 | 23.269 s | +| Catalog + Code Mode, run 2 | Pass | Execute → schema search → execute | 3 | 7 | 21.889 s | +| Catalog + Code Mode, run 3 | Pass | Execute → schema search → execute | 3 | 5 | 18.983 s | +| Catalog-only control | Cannot complete | Five catalog searches, two catalog executions | 7 | 2 | 31.338 s | + +Each combined run created two correct copies, preserved source/content/filters +and returned real IDs/links without API errors. The agent selected Code Mode +directly, read the source, inspected the create schema, then wrote and verified +copies. This demonstrates unprompted selection, not catalog-first fallback. +The control read the original, created nothing and reported its limitation. + +Evidence: `.cache/code-mode/http-dashboard-natural-claude.json` and +`.cache/code-mode/http-dashboard-catalog-control-claude.json`. These four runs +cover one synthetic task, not broad reliability or savings. Per-user favorites +and dashboard permissions are outside the preservation grader. + +### Isolated baseline, 2026-10-26 + +The same Claude client/model completed 3/3 repetitions with explicit Code Mode +instructions. Every run used two schema searches and two executions. Its final +execution read the source, created both copies and re-read the source: four HTTP +requests inside one MCP call. Generated payloads removed source widget/query IDs. + +| Run | Result | Wall time | MCP calls | API calls | Uncached input | Cache creation | Cache reads | Output | +| --- | ------ | --------- | --------- | --------- | -------------- | -------------- | ----------- | ------ | +| 1 | Pass | 24.625 s | 4 | 5 | 8 | 9,234 | 11,060 | 1,944 | +| 2 | Pass | 20.312 s | 4 | 5 | 8 | 6,054 | 11,635 | 1,495 | +| 3 | Pass | 19.206 s | 4 | 5 | 8 | 6,564 | 12,221 | 1,514 | + +Each run made three GETs/two POSTs with no API or MCP errors; schema downloads +are excluded. Total input including caches was 20,302 / 17,697 / 18,793 tokens. +Fresh conversations can still reuse provider caches. These programs returned +complete dashboards; compact projection was not optimized. Evidence: +`.cache/code-mode/http-dashboard-claude.json`. + +An earlier session-driven proof on 2026-10-25 passed with three MCP calls and +seven API calls. Generated calls were relayed unchanged; a large schema response +was transferred verbatim through a file. No token usage is available, and relay +latency is not comparable. Initial automatic attempts failed before inference +because Claude authentication expired and Codex rejected requested models. +Those are infrastructure failures. Evidence: +`.cache/code-mode/http-dashboard-session.json`. + +## Historical operation and recovery evidence + +All reported agent runs used synthetic APIs. Natural prompts did not name tools +or algorithms; tool descriptions included a release example, so prompts do not +imply an unprimed model or a statistically neutral benchmark. + +### Discovery guidance comparison, 2026-10-26 + +Baseline runs used Claude Code 2.1.263, `claude-opus-5`, medium effort, +Wrangler 4.80.0 and workerd 1.20260401.1. Nine normal top-level tools were +available, plus two Code Mode tools on the combined surface. The same fixtures, +prompts and preservation graders were used before/after general discovery +guidance; OpenAPI, API behavior and permissions were unchanged. + +| Task | Success before → after | MCP calls before → after | API calls before → after | API errors before → after | +| -------------------- | ---------------------- | ------------------------ | ------------------------ | ------------------------- | +| Release/deploy | 3/3 → 3/3 | 4–5 → 4 | 5–7 → 6–7 | 0–1 → 1–2 | +| Repository inventory | 3/3 → 3/3 | 2–4 → 3–4 | 2–4 → 2–4 | 0 → 0 | +| Disable alert | 3/3 → 3/3 | 8–9 → 5 | 10–12 → 4 | 1–4 → 0 | + +Catalog-only controls completed 0/1 for each task and left state unchanged. +Baseline durations were 18.5–24.2 s for release, 14.3–23.7 s for repositories +and 26.3–31.7 s for alerts. Release 404s checked existence before creation; +alert 404s came from legacy route guesses. Searching operation metadata rather +than paths alone found workflows without those guesses in all three later runs. +Each later alert run performed one PUT and preserved IDs/monitor connections, +but still resent full nested configurations and fetched the resource twice. +This measured less discovery friction, not improved preservation success. + +Across three alert runs, tool-result bytes fell 214,374 → 100,024 and output +tokens 6,146 → 5,164. Release bytes rose 27,350 → 50,251; repository bytes rose +8,892 → 30,513 as the agent retrieved more contract information. Descriptions +grew 2,420 → 4,452 UTF-8 bytes. Cache variation and three samples cannot establish +general token/cost savings. An additional dashboard regression passed 3/3 with +4–6 MCP calls/seven API calls per run, but used a different historical runner. + +Evidence: `.cache/code-mode/guidance-before/` and +`.cache/code-mode/guidance-after/`. The frozen baseline excludes earlier +diagnostic runs with incomplete fixtures or weaker grading; those remain below. + +### Further operations and failure recovery + +This expansion used 31 fresh Claude Code 2.1.263 / `claude-opus-5` conversations +at medium effort, with the normal catalog plus Code Mode. + +| Scenario | Initial results | After generic write guidance | +| ------------------------------- | ----------------------------------- | ---------------------------- | +| Monitor schedule/timezone | 0/5 | 5/5 | +| Persisted deploy, lost response | 5/5 | 3/3 | +| Repository GET initially 503 | 5/5 | Not rerun | +| Repository GET always 403 | 5/5 blocked after grader correction | Not rerun | +| Alert regression | Earlier 3/3 baseline | 3/3 after parser correction | + +All five initial monitor programs copied GET's aggregate `isMuted` into PUT, +changing a child environment the user asked to preserve. Later repairs could +not erase the violation. Generic guidance now explains that GET is not a write +template: use required fields/requested changes and include unchanged fields +only when the update contract requires them. No monitor fields, routes or SDK +were added to the tool description. + +With the same prompt, fixture, grader, model and settings, five later runs each +used one successful write and preserved child mute state (initially 2–9 writes). +Some unchanged fields were still included. This does not prove minimal payloads +or safe writes across all endpoints. No new catalog-only controls were run. + +Two grading corrections are separate from agent behavior: + +- The original 403 grade was 4/5 because harmless `readOnly: true` answer + metadata was rejected. Regrading unchanged evidence yields 5/5; fabricated + inventory, writes and repeated denials still fail. +- The alert grade was 2/3 because a parser included explanatory braces after + valid fenced JSON. The shared parser now accepts one explicit fence and + rejects multiple/malformed fences. Reconstructing exact saved write boundaries + and regrading unchanged answers yields 3/3, with preservation checks intact. + +These corrections replayed evidence, not model inference or generated code. +Raw reports remain under `.cache/code-mode/expanded-diagnostic/` and +`.cache/code-mode/expanded-after/`; `repositories-denied-regraded.json` and +`alert-regraded.json` record original report hashes, grader hashes and both +verdicts. New reports hash the shared JSON parser. + +The monitor fixture omits valid experimental repair routes. Some diagnostic +404s reflect fixture limits; they do not explain first-write violations or prove +repair is impossible in Sentry. The 403 grader checks honest blockage, not the +accuracy of suggestions about missing OAuth scopes. + +### Diagnostic and later smoke evidence + +Earlier alert diagnostics were 2/3 under final-state-only grading. One program +removed condition IDs, recreating three conditions; another detached then +restored `detectorIds`. Final-state grading hid the temporary detachment. The +current grader checks exact state after every successful write. Original traces +remain under `.cache/code-mode/diagnostic-http-*` without retroactive relabeling. + +Initial fixtures omitted valid auxiliary alert and organization/project reads; +those routes were later added. One release catalog control called a catalog-only +tool directly instead of through `execute_sentry_tool`, and the client rejected +it. `http-release-claude-catalog.json` retains that failure; +`http-release-claude-catalog-repeat.json` records a valid control that read +projects/releases, made no changes and reported missing write capabilities. + +Two later smoke checks on 2026-10-26 used `catalog-and-code` without instructions +to choose Code Mode: dashboard copying passed in one run (five MCP/six API calls), +and repository 403 handling produced an honest blocked result in one run. Both +resolved to `claude-opus-5`. Evidence: `.cache/code-mode/ready-dashboard.json` +and `.cache/code-mode/ready-repositories-denied.json`. They do not establish a +success rate or hosted authorization. + +## Interpretation and next comparisons + +The [Cloudflare MCP implementation](https://github.com/cloudflare/mcp) at +`550e1be82b2227818e6b98fc32a313a59403ae3f` tests the real MCP/Worker path with +external API fixtures. This suite follows that boundary and measures capabilities +beyond existing catalog tools. [Stainless](https://www.stainless.com/blog/sdk-code-mode/) +separates completeness, efficiency and factual accuracy; state verification and +second-page sentinels make completeness explicit here before comparing usage. +[Armin Ronacher's article](https://lucumr.pocoo.org/2026/10/6/codemode/) discusses +host-side composition; client-side Code Mode wrapping this server was not tested. + +Measure success before optimizing tokens or latency. These small samples from +one client/model do not establish production compatibility, complete API +coverage, general token savings or an HTTP-versus-SDK winner. For a later SDK +comparison, keep task, fixture, verifier, client, model settings and budgets +identical. SDK work is not required for the HTTP deployment. diff --git a/packages/mcp-cloudflare/src/code-mode/discovery-worker.test.ts b/packages/mcp-cloudflare/src/code-mode/discovery-worker.test.ts new file mode 100644 index 000000000..c2a28b84b --- /dev/null +++ b/packages/mcp-cloudflare/src/code-mode/discovery-worker.test.ts @@ -0,0 +1,224 @@ +import { execFile } from "node:child_process"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { promisify } from "node:util"; +import { convertV4MiniflareOptions, Miniflare } from "miniflare"; +import { afterAll, beforeAll, beforeEach, expect, it } from "vitest"; +import { TOP_LEVEL_TOOL_NAMES } from "../../../mcp-core/src/tools/surfaces"; +import { createDashboardFixture } from "../../../mcp-server-evals/src/code-mode/dashboard-fixture"; + +let outputDirectory: string; +let worker: Miniflare; +let fixture: Awaited>; + +async function rpc( + method: string, + params: Record = {}, + route = "/mcp", +) { + const response = await worker.dispatchFetch(`http://localhost${route}`, { + method: "POST", + headers: { + authorization: "Bearer code-mode-eval-token", + "content-type": "application/json", + accept: "application/json, text/event-stream", + "mcp-protocol-version": "2026-07-28", + "mcp-method": method, + ...(typeof params.name === "string" ? { "mcp-name": params.name } : {}), + }, + body: JSON.stringify({ + jsonrpc: "2.0", + id: 1, + method, + params: { + ...params, + _meta: { + "io.modelcontextprotocol/protocolVersion": "2026-07-28", + "io.modelcontextprotocol/clientInfo": { + name: "discovery-tests", + version: "1.0.0", + }, + "io.modelcontextprotocol/clientCapabilities": {}, + }, + }, + }), + }); + const text = await response.text(); + if (!response.ok) throw new Error(`MCP HTTP ${response.status}: ${text}`); + const data = text.startsWith("event:") + ? text + .split("\n") + .find((line) => line.startsWith("data: ")) + ?.slice(6) + : text; + const envelope = JSON.parse(data ?? "{}"); + if (envelope.error) throw new Error(JSON.stringify(envelope.error)); + return envelope.result; +} + +beforeAll(async () => { + outputDirectory = await mkdtemp(join(tmpdir(), "sentry-discovery-test-")); + await promisify(execFile)( + process.execPath, + [ + resolve("node_modules/wrangler/bin/wrangler.js"), + "deploy", + "src/code-mode/discovery-worker.ts", + "--dry-run", + "--config", + "wrangler.code-mode.jsonc", + "--outdir", + outputDirectory, + ], + { + timeout: 60_000, + env: { ...process.env, WRANGLER_SEND_METRICS: "false" }, + }, + ); + fixture = await createDashboardFixture(); + worker = new Miniflare( + convertV4MiniflareOptions({ + modules: true, + modulesRoot: outputDirectory, + scriptPath: join(outputDirectory, "discovery-worker.js"), + compatibilityDate: "2026-01-12", + compatibilityFlags: ["nodejs_compat"], + workerLoaders: { LOADER: {} }, + bindings: { + API_BASE_URL: `${fixture.baseUrl}/api/0`, + OPENAPI_URL: fixture.openapiUrl, + CODE_MODE_ALLOW_WRITES: "true", + }, + }), + ); +}); + +beforeEach(() => fixture.reset()); + +afterAll(async () => { + await worker?.dispose(); + await fixture?.close(); + if (outputDirectory) + await rm(outputDirectory, { recursive: true, force: true }); +}); + +it("offers the real catalog and Code Mode together without replacing normal tools", async () => { + const combined = await rpc("tools/list"); + const control = await rpc("tools/list", {}, "/catalog/mcp"); + expect( + combined.tools.map((tool: { name: string }) => tool.name).sort(), + ).toEqual( + [ + ...TOP_LEVEL_TOOL_NAMES, + "search_sentry_api", + "execute_sentry_code", + ].sort(), + ); + expect( + control.tools.map((tool: { name: string }) => tool.name).sort(), + ).toEqual([...TOP_LEVEL_TOOL_NAMES].sort()); + await expect( + rpc( + "tools/call", + { name: "execute_sentry_code", arguments: { code: "async () => 1" } }, + "/catalog/mcp", + ), + ).rejects.toThrow(); + + const catalog = await rpc("tools/call", { + name: "search_sentry_tools", + arguments: { query: "dashboard" }, + }); + expect(catalog.isError).not.toBe(true); + expect( + catalog.structuredContent.results.map( + (tool: { name: string }) => tool.name, + ), + ).toEqual( + expect.arrayContaining(["get_dashboard_details", "find_dashboards"]), + ); + expect( + catalog.structuredContent.results.every( + (tool: { annotations: { readOnlyHint: boolean } }) => + tool.annotations.readOnlyHint, + ), + ).toBe(true); + + const details = await rpc("tools/call", { + name: "execute_sentry_tool", + arguments: { + name: "get_dashboard_details", + arguments: { + organizationSlug: fixture.organizationSlug, + dashboardIdOrTitle: "101", + }, + }, + }); + expect(details.isError).not.toBe(true); + expect(details.content[0].text).toContain("Errors Overview"); + expect(fixture.snapshot().requests).toEqual([ + { + method: "GET", + path: `/api/0/organizations/${fixture.organizationSlug}/dashboards/101/`, + status: 200, + }, + ]); +}); + +it("executes the existing sandbox and broker alongside the normal catalog", async () => { + const discovery = await rpc("tools/call", { + name: "search_sentry_api", + arguments: { code: "async () => Object.keys(spec.paths)" }, + }); + expect(discovery.isError).not.toBe(true); + expect(discovery.structuredContent.result.length).toBeGreaterThan(0); + const result = await rpc("tools/call", { + name: "execute_sentry_code", + arguments: { + code: `async () => (await sentry.request({path: '/organizations/${fixture.organizationSlug}/dashboards/101/'})).body.id`, + }, + }); + expect(result.isError).not.toBe(true); + expect(result.structuredContent).toEqual({ + result: "101", + stats: { apiCalls: 1, writesAttempted: 0 }, + }); +}); + +it("requires authentication before exposing either eval surface", async () => { + for (const route of ["/mcp", "/catalog/mcp"]) { + const response = await worker.dispatchFetch(`http://localhost${route}`, { + method: "POST", + }); + expect(response.status).toBe(401); + } +}); + +it("refuses non-fixture API configuration", async () => { + const misconfigured = new Miniflare( + convertV4MiniflareOptions({ + modules: true, + modulesRoot: outputDirectory, + scriptPath: join(outputDirectory, "discovery-worker.js"), + compatibilityDate: "2026-01-12", + compatibilityFlags: ["nodejs_compat"], + workerLoaders: { LOADER: {} }, + bindings: { + API_BASE_URL: "https://sentry.io/api/0", + OPENAPI_URL: fixture.openapiUrl, + }, + }), + ); + try { + const response = await misconfigured.dispatchFetch("http://localhost/mcp", { + method: "POST", + headers: { authorization: "Bearer code-mode-eval-token" }, + }); + expect(response.status).toBe(500); + expect(await response.text()).toContain("loopback HTTP fixtures"); + expect(fixture.snapshot().requests).toEqual([]); + } finally { + await misconfigured.dispose(); + } +}); diff --git a/packages/mcp-cloudflare/src/code-mode/discovery-worker.ts b/packages/mcp-cloudflare/src/code-mode/discovery-worker.ts new file mode 100644 index 000000000..340861145 --- /dev/null +++ b/packages/mcp-cloudflare/src/code-mode/discovery-worker.ts @@ -0,0 +1,57 @@ +import { createMcpHandler } from "agents/mcp/server"; +import { MAX_MCP_REQUEST_BODY_SIZE } from "../../../mcp-core/src/constants"; +import { buildServer } from "../../../mcp-core/src/server"; +import { ALL_SKILLS } from "../../../mcp-core/src/skills"; +import type { CodeModeEnv } from "./broker"; +import { configuredUrl, limitMcpRequestBody } from "./http"; +import { registerCodeModeTools } from "./worker"; + +/** Eval-only composition of the real catalog and HTTP Code Mode on one MCP. */ +export default { + async fetch( + request: Request, + env: CodeModeEnv, + ctx: ExecutionContext, + ): Promise { + const route = new URL(request.url).pathname; + if (route !== "/mcp" && route !== "/catalog/mcp") { + return new Response("Not found", { status: 404 }); + } + const token = request.headers + .get("authorization") + ?.match(/^Bearer\s+(\S+)$/i)?.[1]; + if (!token) return new Response("Bearer token required", { status: 401 }); + + const api = configuredUrl(env.API_BASE_URL); + if ( + api.protocol !== "http:" || + !["localhost", "127.0.0.1", "[::1]"].includes(api.hostname) || + api.pathname.replace(/\/$/, "") !== "/api/0" + ) { + return new Response("Discovery evals require loopback HTTP fixtures", { + status: 500, + }); + } + + const boundedRequest = await limitMcpRequestBody(request); + if (boundedRequest instanceof Response) return boundedRequest; + return await createMcpHandler( + () => { + const server = buildServer({ + sdkVersion: "v2", + context: { + accessToken: token, + sentryHost: api.host, + sentryProtocol: "http", + grantedSkills: new Set(ALL_SKILLS), + constraints: {}, + transport: "http", + }, + }); + if (route === "/mcp") registerCodeModeTools(server, env, token); + return server; + }, + { route, maxRequestBodySize: MAX_MCP_REQUEST_BODY_SIZE }, + )(boundedRequest, env, ctx); + }, +} satisfies ExportedHandler; diff --git a/packages/mcp-cloudflare/src/code-mode/tsconfig.json b/packages/mcp-cloudflare/src/code-mode/tsconfig.json index f1c824202..65fa72c7b 100644 --- a/packages/mcp-cloudflare/src/code-mode/tsconfig.json +++ b/packages/mcp-cloudflare/src/code-mode/tsconfig.json @@ -5,5 +5,5 @@ "rootDir": "../../../..", "tsBuildInfoFile": "../../node_modules/.tmp/tsconfig.code-mode.tsbuildinfo" }, - "include": ["./worker.ts", "./broker.ts", "./sandbox.ts"] + "include": ["./worker.ts", "./broker.ts", "./sandbox.ts", "./discovery-worker.ts"] } diff --git a/packages/mcp-server-evals/package.json b/packages/mcp-server-evals/package.json index 393716cff..d2aee9996 100644 --- a/packages/mcp-server-evals/package.json +++ b/packages/mcp-server-evals/package.json @@ -12,8 +12,17 @@ "dev": "tsc -w", "start": "tsx src/bin/start-mock-stdio.ts", "eval": "vitest --config=vitest.config.ts", + "eval:code-mode": "tsx src/code-mode/run-dashboard.ts", + "eval:code-mode:discovery": "tsx src/code-mode/run-dashboard.ts --client claude --surface catalog-and-code", + "eval:code-mode:flows": "tsx src/code-mode/run-dashboard.ts --client claude --surface catalog-and-code --task release --task repositories --task alert", + "eval:code-mode:resilience": "tsx src/code-mode/run-dashboard.ts --client claude --surface catalog-and-code --task monitor --task release-recovery --task repositories-retry --task repositories-denied", + "test:code-mode": "vitest run --config=vitest.code-mode.config.ts", + "tsc:code-mode": "tsc --project tsconfig.code-mode.json --noEmit", "eval:ci": "vitest run --reporter=vitest-evals/reporter --reporter=junit --reporter=json --outputFile.json=eval-results.json --outputFile.junit=eval.junit.xml" }, + "devDependencies": { + "wrangler": "catalog:" + }, "dependencies": { "@ai-sdk/mcp": "catalog:", "@ai-sdk/openai": "catalog:", @@ -29,5 +38,9 @@ "vitest": "catalog:", "vitest-evals": "catalog:", "zod": "catalog:" + }, + "sentryCi": { + "typecheck": "tsc:code-mode", + "test": "test:code-mode" } } diff --git a/packages/mcp-server-evals/src/code-mode/alert-fixture.test.ts b/packages/mcp-server-evals/src/code-mode/alert-fixture.test.ts new file mode 100644 index 000000000..d67198b63 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/alert-fixture.test.ts @@ -0,0 +1,241 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { z } from "zod"; +import { createAlertFixture } from "./alert-fixture"; + +type Fixture = Awaited>; +const active: Fixture[] = []; +const accessToken = "local-alert-test-token"; +const finalAnswer = JSON.stringify({ + id: "701", + enabled: false, + frequency: 60, +}); + +afterEach(async () => { + await Promise.all(active.splice(0).map((fixture) => fixture.close())); +}); + +async function fixture() { + const instance = await createAlertFixture({ accessToken }); + active.push(instance); + return instance; +} + +function request(instance: Fixture, id = "701", body?: unknown) { + return fetch( + `${instance.baseUrl}/api/0/organizations/${instance.organizationSlug}/workflows/${id}/`, + { + method: body === undefined ? "GET" : "PUT", + headers: { + authorization: `Bearer ${accessToken}`, + "content-type": "application/json", + }, + body: body === undefined ? undefined : JSON.stringify(body), + }, + ); +} + +async function editBody(instance: Fixture) { + const response = await request(instance); + expect(response.status).toBe(200); + const source = z.record(z.string(), z.unknown()).parse(await response.json()); + return { ...source, enabled: false, config: { frequency: 60 } }; +} + +describe("issue alert Code Mode eval fixture", () => { + it("serves project scope and detector reads used by the ordinary alert tool", async () => { + const instance = await fixture(); + const paths = [ + "/api/0/projects/alert-evals/frontend/", + "/api/0/organizations/alert-evals/workflows/701/project-scope/", + "/api/0/organizations/alert-evals/detectors/901/", + "/api/0/organizations/alert-evals/workflows/", + "/api/0/organizations/alert-evals/detectors/", + ]; + const responses = await Promise.all( + paths.map((path) => + fetch(`${instance.baseUrl}${path}`, { + headers: { authorization: `Bearer ${accessToken}` }, + }), + ), + ); + expect(responses.map(({ status }) => status)).toEqual([ + 200, 200, 200, 200, 200, + ]); + const [project, scope, detector, workflows, detectors] = await Promise.all( + responses.map((response) => response.json()), + ); + expect(project).toEqual({ + id: "101", + slug: "frontend", + name: "Frontend", + platform: "javascript", + }); + expect(scope).toEqual({ projectIds: ["101"], includesAllProjects: false }); + expect(detector).toMatchObject({ + id: "901", + projectId: "101", + type: "error", + enabled: true, + workflowIds: ["701", "702"], + config: {}, + conditionGroup: null, + dataSources: [], + dateCreated: "2026-10-25T10:00:00.000Z", + dateUpdated: "2026-10-25T10:00:00.000Z", + }); + expect(workflows).toEqual(instance.snapshot().workflows); + expect(detectors).toEqual([detector]); + expect( + instance.snapshot().workflows.every((workflow) => workflow.enabled), + ).toBe(true); + }); + + it("persists the CLI-equivalent edit and preserves nested alert configuration", async () => { + const instance = await fixture(); + const before = instance.snapshot(); + const body = await editBody(instance); + const updated = await request(instance, "701", body); + expect(updated.status).toBe(200); + const fetched = await request(instance); + expect(await fetched.json()).toMatchObject({ + id: "701", + enabled: false, + config: { frequency: 60 }, + environment: "production", + owner: "team:42", + detectorIds: ["901"], + triggers: before.workflows[0].triggers, + actionFilters: before.workflows[0].actionFilters, + }); + expect(instance.verify(finalAnswer)).toEqual({ + success: true, + workflowId: "701", + enabled: false, + frequency: 60, + }); + expect(instance.snapshot().workflows[1]).toEqual(before.workflows[1]); + expect(instance.snapshot().writes).toEqual([ + { workflowId: "701", workflows: instance.snapshot().workflows }, + ]); + expect(instance.snapshot().requests.map(({ method }) => method)).toEqual([ + "GET", + "PUT", + "GET", + ]); + instance.reset(); + expect(instance.snapshot()).toEqual(before); + }); + + it("preserves omitted optional fields and requires name as WorkflowValidator does", async () => { + const instance = await fixture(); + const missingName = await request(instance, "701", { + enabled: false, + config: { frequency: 60 }, + }); + expect(missingName.status).toBe(400); + expect(instance.snapshot().workflows[0].enabled).toBe(true); + const response = await request(instance, "701", { + name: "Frontend production errors", + enabled: false, + config: { frequency: 60 }, + }); + expect(response.status).toBe(200); + expect(instance.verify(finalAnswer).success).toBe(true); + }); + + it("accepts snake_case nested fields but rejects alias collisions", async () => { + const instance = await fixture(); + const body = await editBody(instance); + const snakeBody = JSON.parse( + JSON.stringify(body).replace( + /"([A-Za-z]+)":/g, + (_, key: string) => + `"${key.replace(/[A-Z]/g, (letter) => `_${letter.toLowerCase()}`)}":`, + ), + ) as unknown; + expect((await request(instance, "701", snakeBody)).status).toBe(200); + expect(instance.verify(finalAnswer).success).toBe(true); + const collision = await request(instance, "701", { + ...body, + detector_ids: [902], + }); + expect(collision.status).toBe(400); + expect(instance.verify(finalAnswer).success).toBe(true); + }); + + it("rejects a claimed success when the edit dropped notification actions", async () => { + const instance = await fixture(); + const body = await editBody(instance); + expect( + (await request(instance, "701", { ...body, actionFilters: [] })).status, + ).toBe(200); + expect(() => instance.verify(finalAnswer)).toThrow( + "Alert settings other than enabled/frequency changed", + ); + }); + + it("rejects success after an unrelated alert was changed", async () => { + const instance = await fixture(); + expect( + (await request(instance, "701", await editBody(instance))).status, + ).toBe(200); + expect( + ( + await request(instance, "702", { + name: "Backend production errors", + enabled: false, + }) + ).status, + ).toBe(200); + expect(() => instance.verify(finalAnswer)).toThrow( + "Unrelated alert changed", + ); + }); + + it("rejects temporarily clearing monitor connections even after restoring them", async () => { + const instance = await fixture(); + const body = await editBody(instance); + expect( + (await request(instance, "701", { ...body, detectorIds: [] })).status, + ).toBe(200); + expect((await request(instance, "701", body)).status).toBe(200); + const state = instance.snapshot(); + expect(state.workflows[0].detectorIds).toEqual(["901"]); + expect(state.writes.map((write) => write.workflows[0].detectorIds)).toEqual( + [[], ["901"]], + ); + expect(() => instance.verify(finalAnswer)).toThrow( + "Write 1: unrelated alert settings changed during the operation", + ); + }); + + it("rejects wrong reported values even when the expected tokens appear elsewhere", async () => { + const instance = await fixture(); + expect( + (await request(instance, "701", await editBody(instance))).status, + ).toBe(200); + const wrongAnswer = JSON.stringify({ + id: "999", + enabled: true, + frequency: 15, + note: "The requested values were alert 701, false (disabled), 60 minutes.", + }); + expect(() => instance.verify(wrongAnswer)).toThrow( + "Final answer does not match the saved alert values", + ); + }); + + it("rejects an invented success and an answer that omits the actual result", async () => { + const instance = await fixture(); + expect(() => instance.verify(finalAnswer)).toThrow( + "Target alert is still enabled", + ); + expect( + (await request(instance, "701", await editBody(instance))).status, + ).toBe(200); + expect(() => instance.verify("Done.")).toThrow( + "Final answer must include the saved values as JSON", + ); + }); +}); diff --git a/packages/mcp-server-evals/src/code-mode/alert-fixture.ts b/packages/mcp-server-evals/src/code-mode/alert-fixture.ts new file mode 100644 index 000000000..2a8df3efe --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/alert-fixture.ts @@ -0,0 +1,450 @@ +import assert from "node:assert/strict"; +import { z } from "zod"; +import { startApiFixture } from "./api-fixture"; +import { parseEvalJsonAnswer } from "./json-answer"; + +const ORGANIZATION = "alert-evals"; +const WORKFLOW_ID = "701"; +const OTHER_WORKFLOW_ID = "702"; +const DATE_CREATED = "2026-10-25T10:00:00.000Z"; +const DATE_UPDATED = "2026-10-26T10:00:00.000Z"; + +const answerSchema = z.object({ + id: z.union([z.string(), z.number().int().positive()]).transform(String), + enabled: z.boolean(), + frequency: z.number().int(), +}); +const conditionSchema = z.object({ + id: z.coerce.string().optional(), + type: z.string(), + comparison: z.unknown(), + conditionResult: z.unknown(), +}); +const actionSchema = z.object({ + id: z.coerce.string().optional(), + type: z.string(), + integrationId: z.coerce.string().nullable().optional(), + data: z.record(z.string(), z.unknown()), + config: z.record(z.string(), z.unknown()), + status: z.string().default("active"), +}); +const groupSchema = z.object({ + id: z.coerce.string().optional(), + logicType: z.enum(["all", "any", "any-short"]), + conditions: z.array(conditionSchema).default([]), +}); +const actionFilterSchema = groupSchema.extend({ + actions: z.array(actionSchema), +}); +const updateSchema = z.object({ + name: z.string().min(1).max(256), + enabled: z.boolean().default(true), + config: z + .object({ frequency: z.number().int().min(0).optional() }) + .strict() + .optional(), + environment: z.string().nullable().optional(), + owner: z.string().nullable().optional(), + detectorIds: z.array(z.coerce.string()).optional(), + triggers: groupSchema.optional(), + actionFilters: z.array(actionFilterSchema).optional(), +}); + +type Group = z.infer & { + id: string; + organizationId: string; + actions: z.infer[]; +}; +interface Workflow { + id: string; + name: string; + organizationId: string; + createdBy: string; + dateCreated: string; + dateUpdated: string; + triggers: Group; + actionFilters: Group[]; + environment: string | null; + config: { frequency?: number }; + detectorIds: string[]; + enabled: boolean; + lastTriggered: string | null; + owner: string | null; +} + +// CamelSnakeSerializer accepts either spelling recursively. Detect collisions +// before normalizing so the fixture cannot silently accept contradictory fields. +function camelKeys(value: unknown): unknown { + if (Array.isArray(value)) return value.map(camelKeys); + if (value === null || typeof value !== "object") return value; + const result: Record = {}; + for (const [key, item] of Object.entries(value)) { + const converted = key.replace(/_([a-z])/g, (_, letter: string) => + letter.toUpperCase(), + ); + if (Object.hasOwn(result, converted)) { + throw new SyntaxError(`${key} collides with ${converted}`); + } + result[converted] = camelKeys(item); + } + return result; +} + +// Sanitized shape from WorkflowEngineExamples.GET_WORKFLOW and its serializers. +function sourceWorkflow(id: string, name: string): Workflow { + return { + id, + name, + organizationId: "1", + createdBy: "10", + dateCreated: DATE_CREATED, + dateUpdated: DATE_CREATED, + triggers: { + id: `${id}1`, + organizationId: "1", + logicType: "any-short", + conditions: [ + { + id: `${id}11`, + type: "first_seen_event", + comparison: true, + conditionResult: true, + }, + { + id: `${id}12`, + type: "regression_event", + comparison: true, + conditionResult: true, + }, + ], + actions: [], + }, + actionFilters: [ + { + id: `${id}2`, + organizationId: "1", + logicType: "all", + conditions: [ + { + id: `${id}21`, + type: "issue_priority_deescalating", + comparison: true, + conditionResult: true, + }, + ], + actions: [ + { + id: `${id}31`, + type: "slack", + integrationId: "20", + data: {}, + config: { + targetType: "specific", + targetDisplay: "#frontend-alerts", + targetIdentifier: "C0000000001", + }, + status: "active", + }, + { + id: `${id}32`, + type: "slack", + integrationId: "20", + data: {}, + config: { + targetType: "specific", + targetDisplay: "#oncall-alerts", + targetIdentifier: "C0000000002", + }, + status: "active", + }, + ], + }, + ], + environment: "production", + config: { frequency: 15 }, + detectorIds: ["901"], + enabled: true, + lastTriggered: "2026-10-25T09:00:00.000Z", + owner: "team:42", + }; +} + +/** + * A narrow stateful substitute for workflow GET/PUT, based on Sentry's + * WorkflowValidator. Omitted nested fields are preserved; supplied groups + * replace their child lists. It does not emulate integration validation, + * permissions, feature flags, detector membership or database side effects. + */ +export async function createAlertFixture({ + accessToken = "code-mode-eval-token", +}: { + accessToken?: string; +} = {}) { + const original = sourceWorkflow(WORKFLOW_ID, "Frontend production errors"); + const other = sourceWorkflow(OTHER_WORKFLOW_ID, "Backend production errors"); + const initial = [original, other]; + let workflows = new Map( + initial.map((item) => [item.id, structuredClone(item)]), + ); + const writes: Array<{ workflowId: string; workflows: Workflow[] }> = []; + let nextId = 8000; + const collectionPath = `/api/0/organizations/${ORGANIZATION}/workflows/`; + const detectorPath = `/api/0/organizations/${ORGANIZATION}/detectors/`; + const detector = () => ({ + id: "901", + projectId: "101", + name: "Frontend error monitor", + type: "error", + workflowIds: [...workflows.values()] + .filter((item) => item.detectorIds.includes("901")) + .map((item) => item.id), + dateCreated: DATE_CREATED, + dateUpdated: DATE_CREATED, + dataSources: [], + conditionGroup: null, + config: {}, + enabled: true, + }); + const api = await startApiFixture(accessToken, ({ method, url, body }) => { + // These reads keep the ordinary get_alert_rule tool usable in the control + // arm. It checks project membership and expands the connected detector. + if ( + method === "GET" && + ["frontend", "101"].some( + (project) => + url.pathname === `/api/0/projects/${ORGANIZATION}/${project}/`, + ) + ) { + return { + body: { + id: "101", + slug: "frontend", + name: "Frontend", + platform: "javascript", + }, + }; + } + if ( + method === "GET" && + (url.pathname === detectorPath || url.pathname === `${detectorPath}901/`) + ) { + return { + body: url.pathname === detectorPath ? [detector()] : detector(), + }; + } + if (!url.pathname.startsWith(collectionPath)) return undefined; + if (method === "GET" && url.pathname === collectionPath) { + return { body: [...workflows.values()] }; + } + if (method === "GET" && url.pathname.endsWith("/project-scope/")) { + const workflowId = url.pathname.slice( + collectionPath.length, + -"/project-scope/".length, + ); + const workflow = workflows.get(workflowId); + return workflow + ? { + body: { + projectIds: workflow.detectorIds.includes("901") ? ["101"] : [], + includesAllProjects: false, + }, + } + : { status: 404, body: { detail: "Alert not found" } }; + } + const id = url.pathname.slice(collectionPath.length).replace(/\/$/, ""); + const workflow = workflows.get(id); + if (!workflow) return { status: 404, body: { detail: "Alert not found" } }; + if (method === "GET") return { body: workflow }; + if (method !== "PUT") return undefined; + const input = updateSchema.parse(camelKeys(body)); + const { triggers, actionFilters, ...fields } = input; + const existingFilterIds = new Set( + workflow.actionFilters.map((item) => item.id), + ); + const existingActionIds = new Set( + workflow.actionFilters.flatMap((item) => + item.actions.map((action) => action.id), + ), + ); + if ( + (triggers?.id && triggers.id !== workflow.triggers.id) || + actionFilters?.some( + (item) => + (item.id && !existingFilterIds.has(item.id)) || + item.actions.some( + (action) => action.id && !existingActionIds.has(action.id), + ), + ) + ) { + return { + status: 400, + body: { + detail: "Condition group or action does not belong to this workflow", + }, + }; + } + // Backend updates the rows whose IDs are supplied and creates new rows + // when IDs are omitted; copying only condition values changes identity. + const serializeGroup = ( + group: z.infer & { + actions?: z.infer[]; + }, + existingId?: string, + ): Group => ({ + ...group, + id: group.id ?? existingId ?? String(nextId++), + organizationId: workflow.organizationId, + conditions: group.conditions.map((condition) => ({ + ...condition, + id: condition.id ?? String(nextId++), + })), + actions: (group.actions ?? []).map((action) => ({ + ...action, + id: action.id ?? String(nextId++), + integrationId: action.integrationId ?? null, + })), + }); + const updated: Workflow = { + ...workflow, + ...fields, + dateUpdated: DATE_UPDATED, + triggers: triggers + ? serializeGroup(triggers, workflow.triggers.id) + : workflow.triggers, + actionFilters: actionFilters + ? actionFilters.map((item) => serializeGroup(item)) + : workflow.actionFilters, + }; + workflows.set(id, updated); + writes.push( + structuredClone({ workflowId: id, workflows: [...workflows.values()] }), + ); + return { body: updated }; + }); + + return { + baseUrl: api.baseUrl, + organizationSlug: ORGANIZATION, + prompt: [ + `In organization ${ORGANIZATION}, disable the issue alert "${original.name}" (alert ID ${WORKFLOW_ID}, for project frontend) and set its notification frequency to 60 minutes.`, + "Preserve its name, environment, owner, monitor connections, trigger conditions, filters and notification actions. Leave all other alerts unchanged.", + "Keep those settings unchanged throughout the operation.", + "Return JSON with the alert's saved values: {id, enabled, frequency}.", + ].join("\n"), + cliCommands: [ + `sentry alert issues edit ${ORGANIZATION}/frontend/${WORKFLOW_ID} --status disabled --frequency 60`, + ], + provenance: { + cliRevision: "b625111aef9cc3a44abbed634a043fce8505366d", + cliSource: "packages/cli/src/commands/alert/issues/edit.ts", + backendRevision: "b4b356554798716ceaeab74efdc095782be9ca16", + backendSources: [ + "src/sentry/workflow_engine/endpoints/organization_workflow_details.py", + "src/sentry/workflow_engine/endpoints/validators/base/workflow.py", + "src/sentry/workflow_engine/endpoints/validators/base/data_condition_group.py", + "src/sentry/workflow_engine/endpoints/serializers/workflow_serializer.py", + "src/sentry/apidocs/examples/workflow_engine_examples.py", + ], + catalogReadSupport: { + // This private endpoint exists in the local origin/master ref, which + // is newer than the backend checkout used for the mutation contract. + backendRevision: "c4b7b5f3502b7f7c2ae8edc9ae1da36c8a1c98a4", + backendRef: "origin/master", + backendSources: [ + "src/sentry/workflow_engine/endpoints/organization_workflow_project_scope.py", + "src/sentry/workflow_engine/endpoints/organization_workflow_index.py", + "src/sentry/workflow_engine/endpoints/organization_detector_index.py", + "src/sentry/workflow_engine/endpoints/organization_detector_details.py", + "src/sentry/workflow_engine/endpoints/serializers/detector_serializer.py", + "src/sentry/core/endpoints/project_details.py", + "src/sentry/api/serializers/models/project.py", + ], + }, + endpoints: [ + "GET /organizations/{organization_id_or_slug}/workflows/{workflow_id}/", + "PUT /organizations/{organization_id_or_slug}/workflows/{workflow_id}/", + "GET /organizations/{organization_id_or_slug}/workflows/", + "GET /organizations/{organization_id_or_slug}/workflows/{workflow_id}/project-scope/", + "GET /organizations/{organization_id_or_slug}/detectors/{detector_id}/", + "GET /organizations/{organization_id_or_slug}/detectors/", + "GET /projects/{organization_id_or_slug}/{project_id_or_slug}/", + ], + }, + reset() { + workflows = new Map( + initial.map((item) => [item.id, structuredClone(item)]), + ); + nextId = 8000; + api.requests.length = 0; + writes.length = 0; + }, + snapshot() { + return structuredClone({ + workflows: [...workflows.values()], + requests: api.requests, + writes, + }); + }, + verify(resultText: string) { + const updated = workflows.get(WORKFLOW_ID); + assert(updated, "Target alert is missing"); + assert.equal(updated.enabled, false, "Target alert is still enabled"); + assert.equal( + updated.config.frequency, + 60, + "Target alert frequency is not 60 minutes", + ); + assert.deepEqual( + updated, + { + ...original, + enabled: false, + config: { frequency: 60 }, + dateUpdated: DATE_UPDATED, + }, + "Alert settings other than enabled/frequency changed", + ); + assert.deepEqual( + workflows.get(OTHER_WORKFLOW_ID), + other, + "Unrelated alert changed", + ); + for (const [index, write] of writes.entries()) { + const target = write.workflows.find((item) => item.id === WORKFLOW_ID); + assert(target, `Write ${index + 1}: target alert is missing`); + assert.deepEqual( + target, + { + ...original, + enabled: target.enabled, + config: { ...original.config, frequency: target.config.frequency }, + dateUpdated: target.dateUpdated, + }, + `Write ${index + 1}: unrelated alert settings changed during the operation`, + ); + assert.deepEqual( + write.workflows.find((item) => item.id === OTHER_WORKFLOW_ID), + other, + `Write ${index + 1}: another alert changed during the operation`, + ); + } + const answer = answerSchema.parse(parseEvalJsonAnswer(resultText)); + assert.deepEqual( + answer, + { + id: updated.id, + enabled: updated.enabled, + frequency: updated.config.frequency, + }, + "Final answer does not match the saved alert values", + ); + return { + success: true as const, + workflowId: WORKFLOW_ID, + enabled: false, + frequency: 60, + }; + }, + close: api.close, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/api-fixture.ts b/packages/mcp-server-evals/src/code-mode/api-fixture.ts new file mode 100644 index 000000000..1a62f08ab --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/api-fixture.ts @@ -0,0 +1,98 @@ +import assert from "node:assert/strict"; +import { createServer, type IncomingHttpHeaders } from "node:http"; +import { z } from "zod"; + +export interface FixtureRequest { + method: string; + path: string; + query: string; + status: number; +} + +interface ApiRequest { + method: string; + url: URL; + headers: IncomingHttpHeaders; + body: unknown; +} + +interface ApiReply { + status?: number; + body: unknown; + headers?: Record; +} + +/** Authenticated loopback HTTP boundary; scenario state and grading stay with the fixture. */ +export async function startApiFixture( + accessToken: string, + handle: ( + request: ApiRequest, + ) => ApiReply | undefined | Promise, +) { + const requests: FixtureRequest[] = []; + const server = createServer(async (request, response) => { + const url = new URL(request.url ?? "/", `http://${request.headers.host}`); + const method = request.method ?? "GET"; + let reply: ApiReply; + try { + if (request.headers.authorization !== `Bearer ${accessToken}`) { + reply = { status: 401, body: { detail: "Invalid token" } }; + } else { + const chunks: Buffer[] = []; + let size = 0; + for await (const chunk of request) { + size += chunk.length; + if (size > 256_000) + throw new Error("Fixture request body exceeds its budget"); + chunks.push(chunk); + } + const raw = Buffer.concat(chunks).toString("utf8"); + reply = (await handle({ + method, + url, + headers: request.headers, + body: raw ? JSON.parse(raw) : undefined, + })) ?? { + status: 404, + body: { detail: "Endpoint not implemented in this eval fixture" }, + }; + } + } catch (error) { + reply = + error instanceof z.ZodError || error instanceof SyntaxError + ? { + status: 400, + body: { + detail: "Invalid request", + ...(error instanceof z.ZodError + ? { errors: error.issues } + : {}), + }, + } + : { status: 500, body: { detail: "Fixture internal error" } }; + } + const status = reply.status ?? 200; + requests.push({ method, path: url.pathname, query: url.search, status }); + response.writeHead(status, { + "content-type": "application/json", + ...reply.headers, + }); + response.end(JSON.stringify(reply.body)); + }); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", resolve); + }); + const address = server.address(); + assert(address && typeof address !== "string"); + return { + baseUrl: `http://127.0.0.1:${address.port}`, + requests, + async close(this: void) { + await new Promise((resolve, reject) => { + server.close((error) => (error ? reject(error) : resolve())); + server.closeAllConnections(); + }); + }, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/claude-agent.test.ts b/packages/mcp-server-evals/src/code-mode/claude-agent.test.ts new file mode 100644 index 000000000..ebb56d7a9 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/claude-agent.test.ts @@ -0,0 +1,281 @@ +import { chmod, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { runClaudeAgent } from "./claude-agent"; + +const tools = [ + "mcp__sentry_code_mode__search_sentry_api", + "mcp__sentry_code_mode__execute_sentry_code", +]; +const options = { + serverName: "sentry_code_mode", + toolNames: ["search_sentry_api", "execute_sentry_code"], +}; +const init = { + type: "system", + subtype: "init", + model: "claude-test-alias", + tools, + session_id: "private-session-metadata", + apiKeySource: "private-account-metadata", +}; +const directories: string[] = []; + +async function fakeClaude(events: unknown[], exitCode = 0) { + const directory = await mkdtemp(path.join(tmpdir(), "claude-eval-test-")); + directories.push(directory); + const executable = path.join(directory, "claude"); + // A real process exercises argument boundaries, stdin and JSONL parsing. + // Deliberately omit the final newline to cover the stream's trailing event. + const payload = events.map((event) => JSON.stringify(event)).join("\n"); + await writeFile( + executable, + `#!${process.execPath} +const fs = require("node:fs"); +let input = ""; +process.stdin.setEncoding("utf8"); +process.stdin.on("data", chunk => { input += chunk; }); +process.stdin.on("end", () => { + fs.writeFileSync("capture.json", JSON.stringify({ args: process.argv.slice(2), input })); + const payload = ${JSON.stringify(payload)}; + process.stdout.write(payload.slice(0, 37)); + process.stdout.write(payload.slice(37)); + process.exitCode = ${exitCode}; +}); +`, + ); + await chmod(executable, 0o755); + vi.stubEnv("PATH", `${directory}${path.delimiter}${process.env.PATH ?? ""}`); + return directory; +} + +afterEach(async () => { + vi.unstubAllEnvs(); + await Promise.all( + directories + .splice(0) + .map((directory) => rm(directory, { recursive: true })), + ); +}); + +describe("Claude Code Mode adapter", () => { + it("uses only MCP tools and reports aggregate usage, resolved model and task evidence", async () => { + const usage = { + input_tokens: 100, + output_tokens: 30, + cache_creation_input_tokens: 60, + cache_read_input_tokens: 400, + }; + const resultContent = [{ type: "text", text: '{"copied":2}' }]; + const directory = await fakeClaude([ + init, + { + type: "assistant", + session_id: "private-session-metadata", + message: { + model: "claude-test-resolved", + usage, + content: [ + { + type: "tool_use", + id: "call-1", + name: tools[1], + input: { code: "async () => 2" }, + }, + ], + }, + }, + { + type: "user", + message: { + content: [ + { + type: "tool_result", + tool_use_id: "call-1", + content: resultContent, + is_error: false, + }, + ], + }, + }, + { + type: "assistant", + message: { + model: "claude-test-resolved", + usage, + content: [{ type: "text", text: "Copied both dashboards." }], + }, + }, + { + type: "result", + subtype: "success", + is_error: false, + result: "Copied both dashboards.", + usage, + total_cost_usd: 0.01, + num_turns: 2, + }, + ]); + const result = await runClaudeAgent( + directory, + "http://127.0.0.1:4321/mcp", + "dummy-token", + "Copy two dashboards", + { + ...options, + model: "requested-alias", + systemPrompt: "System instructions", + }, + ); + expect(result).toMatchObject({ + exitCode: 0, + error: undefined, + model: "claude-test-resolved", + advertisedModel: "claude-test-alias", + text: "Copied both dashboards.", + usage, + reportedCostUsd: 0.01, + turns: 2, + mcpCalls: [ + { + id: "call-1", + name: "execute_sentry_code", + input: { code: "async () => 2" }, + }, + ], + mcpErrors: 0, + toolResultBytes: Buffer.byteLength(JSON.stringify(resultContent)), + }); + expect(JSON.stringify(result.events)).not.toContain("private-"); + const capture = JSON.parse( + await readFile(path.join(directory, "capture.json"), "utf8"), + ); + expect(capture.input).toBe("Copy two dashboards"); + const argument = (name: string) => + capture.args[capture.args.indexOf(name) + 1]; + expect(argument("--tools")).toBe(""); + expect(argument("--allowedTools")).toBe(tools.join(",")); + expect(argument("--setting-sources")).toBe(""); + expect(argument("--settings")).toBe('{"disableAllHooks":true}'); + expect(argument("--permission-mode")).toBe("dontAsk"); + expect(argument("--model")).toBe("requested-alias"); + expect(argument("--effort")).toBe("medium"); + expect(argument("--system-prompt")).toBe("System instructions"); + expect(capture.args).toEqual( + expect.arrayContaining([ + "--strict-mcp-config", + "--disable-slash-commands", + "--no-session-persistence", + ]), + ); + expect(capture.args).not.toContain("--bare"); + expect(JSON.parse(argument("--mcp-config"))).toEqual({ + mcpServers: { + sentry_code_mode: { + type: "http", + url: "http://127.0.0.1:4321/mcp", + headers: { Authorization: "Bearer dummy-token" }, + }, + }, + }); + }); + + it("lets Claude select among the full MCP inventory with its default system prompt", async () => { + const names = [ + "search_sentry_tools", + "execute_sentry_tool", + "get_issue_details", + ...options.toolNames, + ]; + const directory = await fakeClaude([ + { ...init, tools: names.map((name) => `mcp__sentry__${name}`) }, + { type: "result", subtype: "success", is_error: false, result: "Done" }, + ]); + const result = await runClaudeAgent( + directory, + "http://localhost/mcp", + "dummy", + "Copy my dashboard", + { + serverName: "sentry", + toolNames: names, + }, + ); + expect(result.error).toBeUndefined(); + const capture = JSON.parse( + await readFile(path.join(directory, "capture.json"), "utf8"), + ); + expect(capture.input).toBe("Copy my dashboard"); + expect(capture.args).not.toContain("--system-prompt"); + expect(capture.args).not.toContain("--append-system-prompt"); + expect(capture.args[capture.args.indexOf("--allowedTools") + 1]).toBe( + names.map((name) => `mcp__sentry__${name}`).join(","), + ); + }); + + it("fails an OAuth error even when the result subtype says success", async () => { + const error = + "Failed to authenticate: OAuth session expired and could not be refreshed"; + const directory = await fakeClaude([ + init, + { + type: "assistant", + message: { + model: "", + content: [{ type: "text", text: error }], + }, + }, + { type: "result", subtype: "success", is_error: true, result: error }, + ]); + const result = await runClaudeAgent( + directory, + "http://localhost/mcp", + "dummy", + "Copy", + options, + ); + expect(result.error).toBe(error); + expect(result.model).toBe("claude-test-alias"); + expect(result.mcpCalls).toEqual([]); + }); + + it.each([ + { label: "advertised", events: [{ ...init, tools: [...tools, "Bash"] }] }, + { + label: "used", + events: [ + init, + { + type: "assistant", + message: { + content: [ + { + type: "tool_use", + id: "outside", + name: "Bash", + input: { command: "echo unexpected" }, + }, + ], + }, + }, + ], + }, + ])( + "rejects tools $label outside the two-tool experiment", + async ({ events }) => { + const directory = await fakeClaude([ + ...events, + { type: "result", subtype: "success", is_error: false, result: "Done" }, + ]); + const result = await runClaudeAgent( + directory, + "http://localhost/mcp", + "dummy", + "Copy", + options, + ); + expect(result.error).toContain("outside the experiment"); + }, + ); +}); diff --git a/packages/mcp-server-evals/src/code-mode/claude-agent.ts b/packages/mcp-server-evals/src/code-mode/claude-agent.ts new file mode 100644 index 000000000..f8b82d630 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/claude-agent.ts @@ -0,0 +1,236 @@ +import { spawn } from "node:child_process"; +import { z } from "zod"; + +interface ClaudeAgentOptions { + toolNames: string[]; + serverName: string; + model?: string; + /** Omit to keep Claude Code's default instructions and let it select tools. */ + systemPrompt?: string; +} + +const usageSchema = z.object({ + input_tokens: z.number(), + output_tokens: z.number(), + cache_creation_input_tokens: z.number().optional(), + cache_read_input_tokens: z.number().optional(), +}); +const contentSchema = z.object({ + type: z.string(), + text: z.string().optional(), + id: z.string().optional(), + name: z.string().optional(), + input: z.unknown().optional(), + tool_use_id: z.string().optional(), + content: z.unknown().optional(), + is_error: z.boolean().optional(), +}); +// Select task evidence without persisting account, session, or machine metadata. +const eventSchema = z.object({ + type: z.string(), + subtype: z.string().optional(), + model: z.string().optional(), + tools: z.array(z.string()).optional(), + message: z + .object({ + model: z.string().optional(), + content: z.array(contentSchema), + }) + .optional(), + is_error: z.boolean().optional(), + result: z.string().optional(), + errors: z.array(z.string()).optional(), + usage: usageSchema.optional(), + total_cost_usd: z.number().optional(), + num_turns: z.number().optional(), + duration_api_ms: z.number().optional(), +}); +type TraceEvent = z.infer; + +/** Fresh authenticated Claude session with the supplied MCP inventory. */ +export async function runClaudeAgent( + directory: string, + mcpUrl: string, + accessToken: string, + prompt: string, + { toolNames, serverName, model, systemPrompt }: ClaudeAgentOptions, +) { + const toolPrefix = `mcp__${serverName}__`; + const allowedTools = toolNames.map((name) => `${toolPrefix}${name}`); + const events: TraceEvent[] = []; + let buffer = ""; + let bytes = 0; + let protocolError: string | undefined; + let processError: string | undefined; + const args = [ + "--print", + "--output-format", + "stream-json", + "--verbose", + "--tools", + "", + "--allowedTools", + allowedTools.join(","), + "--strict-mcp-config", + "--mcp-config", + JSON.stringify({ + mcpServers: { + [serverName]: { + type: "http", + url: mcpUrl, + headers: { Authorization: `Bearer ${accessToken}` }, + }, + }, + }), + "--settings", + JSON.stringify({ disableAllHooks: true }), + "--setting-sources", + "", + "--disable-slash-commands", + "--no-session-persistence", + "--permission-mode", + "dontAsk", + "--effort", + "medium", + "--no-chrome", + ...(systemPrompt === undefined ? [] : ["--system-prompt", systemPrompt]), + ...(model ? ["--model", model] : []), + ]; + const started = performance.now(); + const child = spawn("claude", args, { + cwd: directory, + stdio: ["pipe", "pipe", "pipe"], + signal: AbortSignal.timeout(300_000), + }); + const fail = (error: string) => { + protocolError ??= error; + child.kill(); + }; + const consumeLine = (line: string) => { + if (!line.trim()) return; + let event: TraceEvent; + try { + event = eventSchema.parse(JSON.parse(line)); + } catch { + fail("Claude emitted an invalid JSON event"); + return; + } + events.push(event); + if ( + event.type === "system" && + event.subtype === "init" && + (event.tools?.length !== allowedTools.length || + allowedTools.some((name) => !event.tools?.includes(name))) + ) { + fail( + "Claude advertised tools outside the experiment or omitted its MCP tools", + ); + } + if ( + event.type === "assistant" && + event.message?.content.some( + (block) => + block.type === "tool_use" && !allowedTools.includes(block.name ?? ""), + ) + ) { + fail("Claude used a tool outside the experiment"); + } + }; + child.stdout.setEncoding("utf8"); + child.stdout.on("data", (chunk: string) => { + bytes += Buffer.byteLength(chunk); + if (bytes > 10_000_000) { + fail("Agent trace exceeded 10 MB"); + return; + } + buffer += chunk; + const lines = buffer.split("\n"); + buffer = lines.pop() ?? ""; + for (const line of lines) consumeLine(line); + }); + // Drain diagnostics, but do not save potentially identifying CLI metadata. + child.stderr.resume(); + child.stdin.on("error", () => { + processError ??= "Claude closed its input before accepting the prompt"; + }); + child.stdin.end(prompt); + const exitCode = await new Promise((resolve) => { + child.once("error", (error) => { + processError = error.message; + }); + child.once("close", resolve); + }); + consumeLine(buffer); + + const init = events.find( + (event) => event.type === "system" && event.subtype === "init", + ); + const completion = events.findLast((event) => event.type === "result"); + const assistantMessages = events.filter( + (event) => event.type === "assistant", + ); + const calls = new Map( + assistantMessages.flatMap((event) => + (event.message?.content ?? []) + .filter((block) => block.type === "tool_use" && block.id) + .map((block) => [block.id, block] as const), + ), + ); + const results = new Map( + events + .filter((event) => event.type === "user") + .flatMap((event) => + (event.message?.content ?? []) + .filter((block) => block.type === "tool_result" && block.tool_use_id) + .map((block) => [block.tool_use_id, block] as const), + ), + ); + const finalMessage = assistantMessages.findLast((event) => + event.message?.content.some((block) => block.type === "text"), + ); + const resolvedModel = assistantMessages.findLast( + (event) => event.message?.model && event.message.model !== "", + )?.message?.model; + return { + durationMs: Math.round(performance.now() - started), + model: resolvedModel ?? init?.model, + advertisedModel: init?.model, + exitCode, + error: + processError ?? + protocolError ?? + (!init ? "Claude did not advertise its tool inventory" : undefined) ?? + (exitCode !== 0 || + !completion || + completion.subtype !== "success" || + completion.is_error + ? completion?.errors?.join("; ") || + completion?.result || + "Claude did not finish successfully" + : undefined), + text: + completion?.result ?? + finalMessage?.message?.content + .filter((block) => block.type === "text") + .map((block) => block.text) + .join("\n") ?? + "", + // The result is already aggregated; summing message usage double-counts it. + usage: completion?.usage, + reportedCostUsd: completion?.total_cost_usd, + turns: completion?.num_turns, + modelDurationMs: completion?.duration_api_ms, + mcpCalls: [...calls.values()].map(({ id, name, input }) => ({ + id, + name: name?.startsWith(toolPrefix) ? name.slice(toolPrefix.length) : name, + input, + })), + mcpErrors: [...results.values()].filter((result) => result.is_error).length, + toolResultBytes: [...results.values()].reduce( + (sum, result) => + sum + Buffer.byteLength(JSON.stringify(result.content ?? null)), + 0, + ), + events, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/codex-agent.test.ts b/packages/mcp-server-evals/src/code-mode/codex-agent.test.ts new file mode 100644 index 000000000..bd83b6246 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/codex-agent.test.ts @@ -0,0 +1,133 @@ +import { chmod, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { runCodexAgent } from "./codex-agent"; + +const directories: string[] = []; +const options = { model: "test-model", systemPrompt: "Use the MCP tools." }; + +async function fakeCodex(events: unknown[], closeInput = false) { + const directory = await mkdtemp(path.join(tmpdir(), "codex-eval-test-")); + directories.push(directory); + const executable = path.join(directory, "codex"); + // Exercise the real process boundary, including a final event without a newline. + const payload = events.map((event) => JSON.stringify(event)).join("\n"); + await writeFile( + executable, + `#!${process.execPath} +const fs = require("node:fs"); +if (${closeInput}) { + fs.closeSync(0); + setTimeout(() => process.exit(1), 20); +} else { + let input = ""; + process.stdin.setEncoding("utf8"); + process.stdin.on("data", chunk => { input += chunk; }); + process.stdin.on("end", () => { + fs.writeFileSync("capture.json", JSON.stringify({ args: process.argv.slice(2), input })); + const payload = ${JSON.stringify(payload)}; + process.stdout.write(payload.slice(0, 37)); + process.stdout.write(payload.slice(37)); + }); +} +`, + ); + await chmod(executable, 0o755); + vi.stubEnv("PATH", `${directory}${path.delimiter}${process.env.PATH ?? ""}`); + return directory; +} + +afterEach(async () => { + vi.unstubAllEnvs(); + await Promise.all( + directories + .splice(0) + .map((directory) => rm(directory, { recursive: true })), + ); +}); + +describe("Codex Code Mode adapter", () => { + it("keeps the final event and aggregate usage without a trailing newline", async () => { + const usage = { input_tokens: 100, output_tokens: 20 }; + const directory = await fakeCodex([ + { + type: "item.completed", + session_id: "private-session-metadata", + item: { + id: "call-1", + type: "mcp_tool_call", + server: "sentry_code_mode", + tool: "execute_sentry_code", + arguments: { code: "async () => 2" }, + status: "completed", + result: { content: [{ type: "text", text: "2" }] }, + }, + }, + { + type: "item.completed", + item: { + id: "answer", + type: "agent_message", + text: "Copied both dashboards.", + }, + }, + { type: "turn.completed", usage }, + ]); + const result = await runCodexAgent( + directory, + "http://127.0.0.1:4321/mcp", + "dummy-token", + "Copy two dashboards", + options, + ); + expect(result).toMatchObject({ + exitCode: 0, + error: undefined, + text: "Copied both dashboards.", + usage, + mcpCalls: [{ id: "call-1", name: "execute_sentry_code" }], + }); + expect(JSON.stringify(result.events)).not.toContain("private-"); + const capture = JSON.parse( + await readFile(path.join(directory, "capture.json"), "utf8"), + ); + expect(capture.input).toBe("Copy two dashboards"); + expect(capture.args).toContain( + 'developer_instructions="Use the MCP tools."', + ); + expect(capture.args).toContain("--ignore-user-config"); + }); + + it("reports a closed input instead of crashing the evaluation process", async () => { + const directory = await fakeCodex([], true); + const result = await runCodexAgent( + directory, + "http://localhost/mcp", + "dummy-token", + "x".repeat(2_000_000), + options, + ); + expect(result.error).toBe( + "Codex closed its input before accepting the prompt", + ); + }); + + it("rejects an out-of-scope tool even when it is the final event", async () => { + const directory = await fakeCodex([ + { type: "turn.completed", usage: { input_tokens: 1, output_tokens: 1 } }, + { + type: "item.completed", + item: { id: "shell", type: "command_execution" }, + }, + ]); + const result = await runCodexAgent( + directory, + "http://localhost/mcp", + "dummy-token", + "Copy", + options, + ); + expect(result.error).toBe("Agent used a tool outside the experiment"); + }); +}); diff --git a/packages/mcp-server-evals/src/code-mode/codex-agent.ts b/packages/mcp-server-evals/src/code-mode/codex-agent.ts new file mode 100644 index 000000000..38cfe86f3 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/codex-agent.ts @@ -0,0 +1,188 @@ +import { spawn } from "node:child_process"; +import { z } from "zod"; + +const toolNames = ["search_sentry_api", "execute_sentry_code"]; + +const usageSchema = z.object({ + input_tokens: z.number(), + output_tokens: z.number(), + cached_input_tokens: z.number().optional(), + cache_write_input_tokens: z.number().optional(), + reasoning_output_tokens: z.number().optional(), +}); +const itemSchema = z.object({ + id: z.string(), + type: z.string(), + server: z.string().optional(), + tool: z.string().optional(), + arguments: z.unknown().optional(), + result: z + .object({ + content: z.array(z.unknown()), + structured_content: z.unknown().optional(), + }) + .nullable() + .optional(), + error: z.object({ message: z.string() }).nullable().optional(), + status: z.string().optional(), + text: z.string().optional(), +}); +const eventSchema = z.object({ + type: z.string(), + item: itemSchema.optional(), + usage: usageSchema.optional(), + error: z.object({ message: z.string() }).optional(), + message: z.string().optional(), +}); +type TraceEvent = z.infer; + +/** Fresh CLI session; reject runs that use anything besides the two MCP tools. */ +export async function runCodexAgent( + directory: string, + mcpUrl: string, + accessToken: string, + prompt: string, + { model, systemPrompt }: { model: string; systemPrompt: string }, +) { + const events: TraceEvent[] = []; + let buffer = ""; + let bytes = 0; + let stderr = ""; + let protocolError: string | undefined; + let processError: string | undefined; + const config = { + approval_policy: "never", + developer_instructions: systemPrompt, + project_doc_max_bytes: 0, + "skills.include_instructions": false, + web_search: "disabled", + model_reasoning_effort: "medium", + "tools.experimental_request_user_input.enabled": false, + "mcp_servers.sentry_code_mode.url": mcpUrl, + "mcp_servers.sentry_code_mode.http_headers.Authorization": `Bearer ${accessToken}`, + "mcp_servers.sentry_code_mode.enabled_tools": toolNames, + "mcp_servers.sentry_code_mode.required": true, + "mcp_servers.sentry_code_mode.default_tools_approval_mode": "approve", + }; + const disabled = [ + "shell_tool", + "apps", + "plugins", + "hooks", + "memories", + "browser_use", + "computer_use", + "image_generation", + "multi_agent", + "code_mode", + "tool_suggest", + ]; + const args = [ + "exec", + "--json", + "--ephemeral", + "--ignore-user-config", + "--ignore-rules", + "--skip-git-repo-check", + "--sandbox", + "read-only", + "--model", + model, + ...disabled.flatMap((feature) => ["--disable", feature]), + ...Object.entries(config).flatMap(([key, value]) => [ + "-c", + `${key}=${JSON.stringify(value)}`, + ]), + "-", + ]; + const started = performance.now(); + const child = spawn("codex", args, { + cwd: directory, + stdio: ["pipe", "pipe", "pipe"], + signal: AbortSignal.timeout(300_000), + }); + const consumeLine = (line: string) => { + if (!line.trim()) return; + try { + // Keep task evidence, not session/account/machine metadata. + events.push(eventSchema.parse(JSON.parse(line))); + } catch { + protocolError = "Agent emitted an invalid JSON event"; + } + }; + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk: string) => { + bytes += Buffer.byteLength(chunk); + if (bytes > 10_000_000) { + protocolError = "Agent trace exceeded 10 MB"; + child.kill(); + return; + } + buffer += chunk; + const lines = buffer.split("\n"); + buffer = lines.pop() ?? ""; + for (const line of lines) consumeLine(line); + }); + child.stderr.on("data", (chunk: string) => { + stderr = (stderr + chunk).slice(-4_000); + }); + child.stdin.on("error", () => { + processError ??= "Codex closed its input before accepting the prompt"; + }); + child.stdin.end(prompt); + const exitCode = await new Promise((resolve) => { + child.once("error", (error) => { + processError = error.message; + }); + child.once("close", resolve); + }); + consumeLine(buffer); + const completion = events.findLast( + (event) => event.type === "turn.completed", + ); + const items = events.flatMap((event) => + event.type === "item.completed" && event.item ? [event.item] : [], + ); + const calls = items.filter((item) => item.type === "mcp_tool_call"); + const unexpected = items.filter( + (item) => + !["mcp_tool_call", "agent_message", "reasoning", "error"].includes( + item.type, + ) || + (item.type === "mcp_tool_call" && + (item.server !== "sentry_code_mode" || + !toolNames.includes(item.tool ?? ""))), + ); + return { + durationMs: Math.round(performance.now() - started), + model, + exitCode, + error: + processError ?? + protocolError ?? + (unexpected.length + ? "Agent used a tool outside the experiment" + : undefined) ?? + (exitCode !== 0 || !completion + ? events.findLast((event) => event.error)?.error?.message || + stderr || + "Agent did not finish successfully" + : undefined), + text: items.findLast((item) => item.type === "agent_message")?.text ?? "", + usage: completion?.usage, + mcpCalls: calls.map(({ id, tool, arguments: input }) => ({ + id, + name: tool, + input, + })), + mcpErrors: calls.filter((call) => call.error || call.status === "failed") + .length, + toolResultBytes: calls.reduce( + (sum, call) => + sum + Buffer.byteLength(JSON.stringify(call.result ?? null)), + 0, + ), + events, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/dashboard-fixture-data.json b/packages/mcp-server-evals/src/code-mode/dashboard-fixture-data.json new file mode 100644 index 000000000..361dcc629 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/dashboard-fixture-data.json @@ -0,0 +1,2307 @@ +{ + "provenance": { + "openapiCommit": "803d6de98c9156c15e9ddc1bf9accbac6e24cebc", + "backendCommit": "c4b7b5f3502b7f7c2ae8edc9ae1da36c8a1c98a4", + "sourceDashboard": "packages/mcp-server-mocks/src/fixtures/dashboard-details.json" + }, + "sourceDashboard": { + "id": "101", + "title": "Errors Overview", + "dateCreated": "2025-04-14T10:15:00.000Z", + "createdBy": { + "id": "1", + "name": "Test User", + "email": "user@example.com" + }, + "widgets": [ + { + "id": "201", + "title": "Handled Errors", + "description": "Handled errors over time", + "displayType": "line", + "thresholds": null, + "interval": "5m", + "dateCreated": "2025-04-14T10:16:00.000Z", + "dashboardId": "101", + "queries": [ + { + "id": "301", + "name": "", + "fields": ["count()"], + "aggregates": ["count()"], + "columns": [], + "fieldAliases": [], + "conditions": "error.handled:true", + "orderby": "-count()", + "widgetId": "201", + "onDemand": [], + "isHidden": false, + "selectedAggregate": 0, + "linkedDashboards": [] + } + ], + "limit": null, + "widgetType": "error-events", + "layout": { + "x": 0, + "y": 0, + "w": 6, + "h": 2, + "minH": 2 + }, + "axisRange": null, + "legendType": null, + "datasetSource": "user", + "changedReason": null + }, + { + "id": "202", + "title": "Top Issues", + "description": null, + "displayType": "table", + "thresholds": null, + "interval": "5m", + "dateCreated": "2025-04-14T10:18:00.000Z", + "dashboardId": "101", + "queries": [ + { + "id": "302", + "name": "Top Issues", + "fields": ["issue", "count()"], + "aggregates": ["count()"], + "columns": ["issue"], + "fieldAliases": [], + "conditions": "is:unresolved", + "orderby": "-count()", + "widgetId": "202", + "onDemand": [], + "isHidden": false, + "selectedAggregate": 0, + "linkedDashboards": [] + } + ], + "limit": 5, + "widgetType": "error-events", + "layout": { + "x": 0, + "y": 2, + "w": 6, + "h": 3, + "minH": 2 + }, + "axisRange": null, + "legendType": null, + "datasetSource": "user", + "changedReason": null + } + ], + "filters": { + "release": ["frontend@1.2.3"] + }, + "permissions": { + "isEditableByEveryone": true, + "teamsWithEditAccess": [] + }, + "isFavorited": true, + "projects": [101, 102], + "environment": ["production"], + "period": "24h", + "utc": true, + "expired": false, + "prebuiltId": null + }, + "openapi": { + "openapi": "3.0.3", + "info": { + "title": "Sentry Dashboard API eval fixture", + "version": "v0" + }, + "paths": { + "/api/0/organizations/{organization_id_or_slug}/dashboards/": { + "get": { + "operationId": "listOrganizationDashboards", + "description": "Retrieve a list of custom dashboards that are associated with the given organization.", + "summary": "List an Organization's Custom Dashboards", + "parameters": [ + { + "in": "path", + "name": "organization_id_or_slug", + "schema": { + "type": "string" + }, + "description": "The ID or slug of the organization the resource belongs to.", + "required": true + }, + { + "in": "query", + "name": "filter", + "schema": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "excludeFavorites", + "excludePrebuilt", + "onlyFavorites", + "onlyPrebuilt", + "owned", + "shared", + "showHidden" + ] + } + }, + "description": "Filter the dashboards returned. Repeat this parameter to apply multiple filters." + }, + { + "in": "query", + "name": "query", + "schema": { + "type": "string" + }, + "description": "Filter dashboards by title." + }, + { + "in": "query", + "name": "sort", + "schema": { + "type": "string" + }, + "description": "The property to sort results by. Prefix the value with `-` to sort in descending order.\n\nAvailable fields are:\n- `title`\n- `dateCreated`\n- `mostPopular`\n- `recentlyViewed`\n- `mydashboards`\n- `myDashboardsAndRecentlyViewed`\n" + }, + { + "in": "query", + "name": "pin", + "schema": { + "type": "string", + "enum": ["favorites"] + }, + "description": "Pin favorited dashboards to the top of the results." + }, + { + "in": "query", + "name": "per_page", + "schema": { + "type": "integer" + }, + "description": "Limit the number of rows to return in the result. Default and maximum allowed is 100." + }, + { + "in": "query", + "name": "cursor", + "schema": { + "type": "string", + "minLength": 1 + }, + "description": "A pointer to the last object fetched and its sort order; used to retrieve the next or previous results.", + "x-learn-more": "https://docs.sentry.io/api/pagination/" + } + ], + "tags": ["Dashboards"], + "security": [ + { + "auth_token": ["org:admin", "org:read", "org:write"] + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "title": { + "type": "string" + }, + "dateCreated": { + "type": "string" + }, + "createdBy": { + "type": "object", + "properties": { + "identities": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "organization": { + "type": "object", + "properties": { + "slug": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": ["name", "slug"] + }, + "provider": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": ["id", "name"] + }, + "dateVerified": { + "type": "string", + "format": "date-time" + }, + "dateSynced": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "dateSynced", + "dateVerified", + "id", + "name", + "organization", + "provider" + ] + } + }, + "avatar": { + "type": "object", + "properties": { + "avatarType": { + "type": "string" + }, + "avatarUuid": { + "type": "string", + "nullable": true + }, + "avatarUrl": { + "type": "string", + "nullable": true + } + } + }, + "authenticators": { + "type": "array", + "items": {} + }, + "canReset2fa": { + "type": "boolean" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "username": { + "type": "string" + }, + "email": { + "type": "string" + }, + "avatarUrl": { + "type": "string" + }, + "isActive": { + "type": "boolean" + }, + "isSuspended": { + "type": "boolean" + }, + "hasPasswordAuth": { + "type": "boolean" + }, + "isManaged": { + "type": "boolean" + }, + "dateJoined": { + "type": "string", + "format": "date-time" + }, + "lastLogin": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "has2fa": { + "type": "boolean" + }, + "lastActive": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "isSuperuser": { + "type": "boolean" + }, + "isStaff": { + "type": "boolean" + }, + "experiments": { + "type": "object", + "additionalProperties": {} + }, + "emails": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "email": { + "type": "string" + }, + "is_verified": { + "type": "boolean" + } + }, + "required": ["email", "id", "is_verified"] + } + } + }, + "required": [ + "avatarUrl", + "dateJoined", + "email", + "emails", + "experiments", + "has2fa", + "hasPasswordAuth", + "id", + "isActive", + "isManaged", + "isStaff", + "isSuperuser", + "isSuspended", + "lastActive", + "lastLogin", + "name", + "username" + ] + }, + "environment": { + "type": "array", + "items": { + "type": "string" + } + }, + "filters": { + "type": "object", + "properties": { + "release": { + "type": "array", + "items": { + "type": "string" + } + }, + "releaseId": { + "type": "array", + "items": { + "type": "string" + } + }, + "globalFilter": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": {} + } + } + } + }, + "lastVisited": { + "type": "string", + "nullable": true + }, + "widgetDisplay": { + "type": "array", + "items": { + "type": "string" + } + }, + "widgetPreview": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "permissions": { + "type": "object", + "properties": { + "isEditableByEveryone": { + "type": "boolean" + }, + "teamsWithEditAccess": { + "type": "array", + "items": { + "type": "integer" + } + } + }, + "required": [ + "isEditableByEveryone", + "teamsWithEditAccess" + ], + "nullable": true + }, + "isFavorited": { + "type": "boolean" + }, + "projects": { + "type": "array", + "items": { + "type": "integer" + } + }, + "prebuiltId": { + "type": "integer", + "nullable": true + } + }, + "required": [ + "createdBy", + "dateCreated", + "environment", + "filters", + "id", + "isFavorited", + "lastVisited", + "permissions", + "prebuiltId", + "projects", + "title", + "widgetDisplay", + "widgetPreview" + ] + } + }, + "examples": { + "QueryDashboards": { + "value": [ + { + "id": "1", + "title": "Dashboard", + "dateCreated": "2024-06-20T14:38:03.498574Z", + "lastVisited": "2024-06-20T14:38:03.498574Z", + "projects": [1], + "environment": ["alpha"], + "filters": { + "release": [ + "frontend@a02311a400636ff9640b3e4ca2991ee153dbbdcc", + "frontend@36934c05140c16df93aa8ebf671f9386e916b501" + ] + }, + "createdBy": { + "id": "1", + "name": "Admin", + "username": "admin", + "email": "admin@example.com", + "avatarUrl": "www.example.com", + "isActive": true, + "isSuspended": false, + "hasPasswordAuth": true, + "isManaged": false, + "dateJoined": "2021-10-25T17:07:33.190596Z", + "lastLogin": "2024-07-16T15:28:39.261659Z", + "has2fa": true, + "lastActive": "2024-07-16T20:45:49.364197Z", + "isSuperuser": false, + "isStaff": false, + "experiments": {}, + "emails": [ + { + "id": "1", + "email": "admin@example.com", + "is_verified": true + } + ], + "avatar": { + "avatarType": "letter_avatar", + "avatarUuid": null, + "avatarUrl": "www.example.com" + } + }, + "widgetDisplay": [], + "widgetPreview": [], + "permissions": { + "isEditableByEveryone": true, + "teamsWithEditAccess": [] + }, + "isFavorited": false, + "prebuiltId": null + }, + { + "id": "2", + "title": "Dashboard", + "dateCreated": "2024-06-20T14:38:03.498574Z", + "lastVisited": "2024-06-20T14:38:03.498574Z", + "projects": [], + "environment": ["alpha"], + "filters": { + "release": [ + "frontend@a02311a400636ff9640b3e4ca2991ee153dbbdcc", + "frontend@36934c05140c16df93aa8ebf671f9386e916b501" + ] + }, + "createdBy": { + "id": "1", + "name": "Admin", + "username": "admin", + "email": "admin@example.com", + "avatarUrl": "www.example.com", + "isActive": true, + "isSuspended": false, + "hasPasswordAuth": true, + "isManaged": false, + "dateJoined": "2021-10-25T17:07:33.190596Z", + "lastLogin": "2024-07-16T15:28:39.261659Z", + "has2fa": true, + "lastActive": "2024-07-16T20:45:49.364197Z", + "isSuperuser": false, + "isStaff": false, + "experiments": {}, + "emails": [ + { + "id": "1", + "email": "admin@example.com", + "is_verified": true + } + ], + "avatar": { + "avatarType": "letter_avatar", + "avatarUuid": null, + "avatarUrl": "www.example.com" + } + }, + "widgetDisplay": [], + "widgetPreview": [], + "permissions": null, + "isFavorited": false, + "prebuiltId": null + } + ], + "summary": "Query Dashboards" + } + } + } + }, + "description": "" + }, + "400": { + "description": "Bad Request" + }, + "403": { + "description": "Forbidden" + }, + "404": { + "description": "Not Found" + } + } + }, + "post": { + "operationId": "createOrganizationDashboard", + "description": "Create a new dashboard for the given Organization", + "summary": "Create a New Dashboard for an Organization", + "parameters": [ + { + "in": "path", + "name": "organization_id_or_slug", + "schema": { + "type": "string" + }, + "description": "The ID or slug of the organization the resource belongs to.", + "required": true + } + ], + "tags": ["Dashboards"], + "requestBody": { + "content": { + "application/json": { + "schema": { + "type": "object", + "description": "Allows parameters to be defined in snake case, but passed as camel case.\n\nErrors are output in camel case.", + "properties": { + "title": { + "type": "string", + "description": "The user defined title for this dashboard.", + "maxLength": 255 + }, + "id": { + "type": "string", + "description": "A dashboard's unique id." + }, + "widgets": { + "type": "array", + "items": { + "type": "object", + "description": "Allows parameters to be defined in snake case, but passed as camel case.\n\nErrors are output in camel case.", + "properties": { + "id": { + "type": "string" + }, + "title": { + "type": "string", + "maxLength": 255 + }, + "description": { + "type": "string", + "nullable": true + }, + "thresholds": { + "type": "object", + "additionalProperties": {}, + "nullable": true + }, + "display_type": { + "enum": [ + "line", + "area", + "bar", + "table", + "big_number", + "details", + "categorical_bar", + "wheel", + "rage_and_dead_clicks", + "server_tree", + "text", + "agents_traces_table", + "heatmap" + ], + "type": "string", + "description": "* `line`\n* `area`\n* `bar`\n* `table`\n* `big_number`\n* `details`\n* `categorical_bar`\n* `wheel`\n* `rage_and_dead_clicks`\n* `server_tree`\n* `text`\n* `agents_traces_table`\n* `heatmap`" + }, + "interval": { + "type": "string", + "maxLength": 10 + }, + "queries": { + "type": "array", + "items": { + "type": "object", + "description": "Allows parameters to be defined in snake case, but passed as camel case.\n\nErrors are output in camel case.", + "properties": { + "id": { + "type": "string" + }, + "fields": { + "type": "array", + "items": { + "type": "string" + } + }, + "aggregates": { + "type": "array", + "items": { + "type": "string" + }, + "nullable": true + }, + "columns": { + "type": "array", + "items": { + "type": "string" + }, + "nullable": true + }, + "field_aliases": { + "type": "array", + "items": { + "type": "string" + }, + "nullable": true + }, + "name": { + "type": "string" + }, + "conditions": { + "type": "string" + }, + "orderby": { + "type": "string" + }, + "is_hidden": { + "type": "boolean" + }, + "on_demand_extraction": { + "type": "object", + "description": "Allows parameters to be defined in snake case, but passed as camel case.\n\nErrors are output in camel case.", + "properties": { + "extraction_state": { + "type": "string" + }, + "enabled": { + "type": "boolean" + } + } + }, + "on_demand_extraction_disabled": { + "type": "boolean" + }, + "selected_aggregate": { + "type": "integer", + "nullable": true + }, + "linked_dashboards": { + "type": "array", + "items": { + "type": "object", + "description": "Allows parameters to be defined in snake case, but passed as camel case.\n\nErrors are output in camel case.", + "properties": { + "field": { + "type": "string" + }, + "dashboard_id": { + "type": "string" + } + }, + "required": ["dashboard_id", "field"] + }, + "nullable": true + } + } + } + }, + "widget_type": { + "enum": [ + "discover", + "issue", + "metrics", + "error-events", + "transaction-like", + "spans", + "logs", + "tracemetrics", + "preprod-app-size", + null + ], + "type": "string", + "description": "* `discover`\n* `issue`\n* `metrics`\n* `error-events`\n* `transaction-like`\n* `spans`\n* `logs`\n* `tracemetrics`\n* `preprod-app-size`", + "nullable": true + }, + "limit": { + "type": "integer", + "minimum": 1, + "nullable": true + }, + "layout": { + "allOf": [ + { + "type": "object", + "description": "Widget grid layout position and dimensions for dashboard creation.\n\nThe dashboard uses a 6-column grid. Required keys: x, y, w, h.\nConstraints: x (0-5), y (>= 0), w (1-6), h (>= 1), and x + w <= 6.", + "properties": { + "x": { + "type": "integer", + "maximum": 5, + "minimum": 0, + "description": "Column position (0-indexed)." + }, + "y": { + "type": "integer", + "minimum": 0, + "description": "Row position (0-indexed)." + }, + "w": { + "type": "integer", + "maximum": 6, + "minimum": 1, + "description": "Width in grid columns (1-6)." + }, + "h": { + "type": "integer", + "minimum": 1, + "description": "Height in grid rows." + } + }, + "required": ["h", "w", "x", "y"] + } + ], + "nullable": true + }, + "axis_range": { + "enum": ["auto", "dataMin", null], + "type": "string", + "description": "* `auto`\n* `dataMin`", + "nullable": true + }, + "legend_type": { + "enum": ["default", "breakdown", null], + "type": "string", + "description": "* `default`\n* `breakdown`", + "nullable": true + } + } + }, + "description": "A json list of widgets saved in this dashboard." + }, + "projects": { + "type": "array", + "items": { + "type": "integer" + }, + "description": "The saved projects filter for this dashboard." + }, + "environment": { + "type": "array", + "items": { + "type": "string" + }, + "nullable": true, + "description": "The saved environment filter for this dashboard." + }, + "period": { + "type": "string", + "nullable": true, + "description": "The saved time range period for this dashboard." + }, + "start": { + "type": "string", + "format": "date-time", + "nullable": true, + "description": "The saved start time for this dashboard." + }, + "end": { + "type": "string", + "format": "date-time", + "nullable": true, + "description": "The saved end time for this dashboard." + }, + "filters": { + "type": "object", + "additionalProperties": {}, + "description": "The saved filters for this dashboard." + }, + "utc": { + "type": "boolean", + "description": "Setting that lets you display saved time range for this dashboard in UTC." + }, + "permissions": { + "allOf": [ + { + "type": "object", + "description": "Allows parameters to be defined in snake case, but passed as camel case.\n\nErrors are output in camel case.", + "properties": { + "is_editable_by_everyone": { + "type": "boolean", + "description": "Whether the dashboard is editable by everyone." + }, + "teams_with_edit_access": { + "type": "array", + "items": { + "type": "integer" + }, + "description": "List of team IDs that have edit access to a dashboard." + } + }, + "required": ["is_editable_by_everyone"] + } + ], + "nullable": true, + "description": "Permissions that restrict users from editing dashboards" + }, + "is_favorited": { + "type": "boolean", + "default": false, + "description": "Favorite the dashboard automatically for the request user" + } + }, + "required": ["title"] + } + } + }, + "required": true + }, + "security": [ + { + "auth_token": ["org:admin", "org:read", "org:write"] + } + ], + "responses": { + "201": { + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "environment": { + "type": "array", + "items": { + "type": "string" + } + }, + "period": { + "type": "string" + }, + "utc": { + "type": "string" + }, + "expired": { + "type": "boolean" + }, + "start": { + "type": "string", + "format": "date-time" + }, + "end": { + "type": "string", + "format": "date-time" + }, + "id": { + "type": "string" + }, + "title": { + "type": "string" + }, + "dateCreated": { + "type": "string" + }, + "createdBy": { + "type": "object", + "properties": { + "identities": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "organization": { + "type": "object", + "properties": { + "slug": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": ["name", "slug"] + }, + "provider": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": ["id", "name"] + }, + "dateVerified": { + "type": "string", + "format": "date-time" + }, + "dateSynced": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "dateSynced", + "dateVerified", + "id", + "name", + "organization", + "provider" + ] + } + }, + "avatar": { + "type": "object", + "properties": { + "avatarType": { + "type": "string" + }, + "avatarUuid": { + "type": "string", + "nullable": true + }, + "avatarUrl": { + "type": "string", + "nullable": true + } + } + }, + "authenticators": { + "type": "array", + "items": {} + }, + "canReset2fa": { + "type": "boolean" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "username": { + "type": "string" + }, + "email": { + "type": "string" + }, + "avatarUrl": { + "type": "string" + }, + "isActive": { + "type": "boolean" + }, + "isSuspended": { + "type": "boolean" + }, + "hasPasswordAuth": { + "type": "boolean" + }, + "isManaged": { + "type": "boolean" + }, + "dateJoined": { + "type": "string", + "format": "date-time" + }, + "lastLogin": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "has2fa": { + "type": "boolean" + }, + "lastActive": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "isSuperuser": { + "type": "boolean" + }, + "isStaff": { + "type": "boolean" + }, + "experiments": { + "type": "object", + "additionalProperties": {} + }, + "emails": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "email": { + "type": "string" + }, + "is_verified": { + "type": "boolean" + } + }, + "required": ["email", "id", "is_verified"] + } + } + }, + "required": [ + "avatarUrl", + "dateJoined", + "email", + "emails", + "experiments", + "has2fa", + "hasPasswordAuth", + "id", + "isActive", + "isManaged", + "isStaff", + "isSuperuser", + "isSuspended", + "lastActive", + "lastLogin", + "name", + "username" + ], + "nullable": true + }, + "widgets": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "title": { + "type": "string" + }, + "description": { + "type": "string", + "nullable": true + }, + "displayType": { + "type": "string" + }, + "thresholds": { + "type": "object", + "properties": { + "preferredPolarity": { + "type": "string" + }, + "timeWindow": { + "type": "string" + }, + "max_values": { + "type": "object", + "additionalProperties": { + "type": "integer" + } + }, + "unit": { + "type": "string" + } + }, + "required": ["max_values", "unit"], + "nullable": true + }, + "interval": { + "type": "string" + }, + "dateCreated": { + "type": "string" + }, + "dashboardId": { + "type": "string" + }, + "queries": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "fields": { + "type": "array", + "items": { + "type": "string" + } + }, + "aggregates": { + "type": "array", + "items": { + "type": "string" + } + }, + "columns": { + "type": "array", + "items": { + "type": "string" + } + }, + "fieldAliases": { + "type": "array", + "items": { + "type": "string" + } + }, + "conditions": { + "type": "string" + }, + "orderby": { + "type": "string" + }, + "widgetId": { + "type": "string" + }, + "onDemand": { + "type": "array", + "items": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "extractionState": { + "type": "string" + }, + "dashboardWidgetQueryId": { + "type": "integer" + } + }, + "required": [ + "dashboardWidgetQueryId", + "enabled", + "extractionState" + ] + } + }, + "isHidden": { + "type": "boolean" + }, + "selectedAggregate": { + "type": "integer", + "nullable": true + }, + "linkedDashboards": { + "type": "array", + "items": { + "type": "object", + "properties": { + "field": { + "type": "string" + }, + "dashboardId": { + "type": "integer" + } + }, + "required": ["dashboardId", "field"] + } + } + }, + "required": [ + "aggregates", + "columns", + "conditions", + "fieldAliases", + "fields", + "id", + "isHidden", + "linkedDashboards", + "name", + "onDemand", + "orderby", + "selectedAggregate", + "widgetId" + ] + } + }, + "limit": { + "type": "integer", + "nullable": true + }, + "widgetType": { + "type": "string", + "nullable": true + }, + "layout": { + "type": "object", + "additionalProperties": { + "type": "integer" + }, + "nullable": true + }, + "axisRange": { + "type": "string", + "nullable": true + }, + "legendType": { + "enum": ["default", "breakdown", null], + "type": "string", + "nullable": true + }, + "datasetSource": { + "type": "string", + "nullable": true + }, + "exploreUrls": { + "type": "array", + "items": { + "type": "string" + }, + "nullable": true + }, + "changedReason": { + "type": "array", + "items": { + "type": "object", + "properties": { + "orderby": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "nullable": true + }, + "equations": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + } + }, + "nullable": true + }, + "selected_columns": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "equations", + "orderby", + "selected_columns" + ] + }, + "nullable": true + } + }, + "required": [ + "axisRange", + "changedReason", + "dashboardId", + "datasetSource", + "dateCreated", + "description", + "displayType", + "exploreUrls", + "id", + "interval", + "layout", + "legendType", + "limit", + "queries", + "thresholds", + "title", + "widgetType" + ] + } + }, + "projects": { + "type": "array", + "items": { + "type": "integer" + } + }, + "filters": { + "type": "object", + "properties": { + "release": { + "type": "array", + "items": { + "type": "string" + } + }, + "releaseId": { + "type": "array", + "items": { + "type": "string" + } + }, + "globalFilter": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": {} + } + } + } + }, + "permissions": { + "type": "object", + "properties": { + "isEditableByEveryone": { + "type": "boolean" + }, + "teamsWithEditAccess": { + "type": "array", + "items": { + "type": "integer" + } + } + }, + "required": [ + "isEditableByEveryone", + "teamsWithEditAccess" + ], + "nullable": true + }, + "isFavorited": { + "type": "boolean" + }, + "prebuiltId": { + "type": "integer", + "nullable": true + } + }, + "required": [ + "createdBy", + "dateCreated", + "filters", + "id", + "isFavorited", + "permissions", + "prebuiltId", + "projects", + "title", + "widgets" + ] + }, + "examples": { + "CreateDashboard": { + "value": { + "id": "1", + "title": "Dashboard", + "dateCreated": "2024-06-20T14:38:03.498574Z", + "createdBy": { + "id": "1", + "name": "Admin", + "username": "admin", + "email": "admin@example.com", + "avatarUrl": "www.example.com", + "isActive": true, + "isSuspended": false, + "hasPasswordAuth": true, + "isManaged": false, + "dateJoined": "2021-10-25T17:07:33.190596Z", + "lastLogin": "2024-07-16T15:28:39.261659Z", + "has2fa": true, + "lastActive": "2024-07-16T20:45:49.364197Z", + "isSuperuser": false, + "isStaff": false, + "experiments": {}, + "emails": [ + { + "id": "1", + "email": "admin@example.com", + "is_verified": true + } + ], + "avatar": { + "avatarType": "letter_avatar", + "avatarUuid": null, + "avatarUrl": "www.example.com" + } + }, + "widgets": [ + { + "id": "658714", + "title": "Custom Widget", + "description": null, + "displayType": "table", + "thresholds": null, + "interval": "5m", + "dateCreated": "2024-07-16T15:36:46.048343Z", + "dashboardId": "1", + "datasetSource": "user", + "queries": [ + { + "id": "1", + "name": "", + "fields": [ + "avg(transaction.duration)", + "transaction" + ], + "aggregates": ["avg(transaction.duration)"], + "columns": ["transaction"], + "fieldAliases": ["", ""], + "conditions": "", + "orderby": "-avg(transaction.duration)", + "widgetId": "1", + "onDemand": [ + { + "enabled": false, + "extractionState": "disabled:not-applicable", + "dashboardWidgetQueryId": 1 + } + ], + "isHidden": false, + "selectedAggregate": null, + "linkedDashboards": [] + } + ], + "limit": null, + "widgetType": "transaction-like", + "layout": { + "w": 2, + "y": 0, + "h": 2, + "minH": 2, + "x": 0 + }, + "exploreUrls": null, + "changedReason": null, + "axisRange": null, + "legendType": "default" + } + ], + "projects": [1], + "filters": {}, + "environment": ["alpha"], + "period": "7d", + "permissions": { + "isEditableByEveryone": true, + "teamsWithEditAccess": [] + }, + "isFavorited": false, + "prebuiltId": null + }, + "summary": "Create Dashboard" + } + } + } + }, + "description": "" + }, + "400": { + "description": "Bad Request" + }, + "403": { + "description": "Forbidden" + }, + "404": { + "description": "Not Found" + }, + "409": { + "description": "Conflict" + } + } + } + }, + "/api/0/organizations/{organization_id_or_slug}/dashboards/{dashboard_id}/": { + "get": { + "operationId": "getOrganizationDashboard", + "description": "Return details about an organization's custom dashboard.", + "summary": "Retrieve an Organization's Custom Dashboard", + "parameters": [ + { + "in": "path", + "name": "organization_id_or_slug", + "schema": { + "type": "string" + }, + "description": "The ID or slug of the organization the resource belongs to.", + "required": true + }, + { + "in": "path", + "name": "dashboard_id", + "schema": { + "type": "integer" + }, + "description": "The ID of the dashboard you'd like to retrieve.", + "required": true + } + ], + "tags": ["Dashboards"], + "security": [ + { + "auth_token": ["org:admin", "org:read", "org:write"] + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "environment": { + "type": "array", + "items": { + "type": "string" + } + }, + "period": { + "type": "string" + }, + "utc": { + "type": "string" + }, + "expired": { + "type": "boolean" + }, + "start": { + "type": "string", + "format": "date-time" + }, + "end": { + "type": "string", + "format": "date-time" + }, + "id": { + "type": "string" + }, + "title": { + "type": "string" + }, + "dateCreated": { + "type": "string" + }, + "createdBy": { + "type": "object", + "properties": { + "identities": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "organization": { + "type": "object", + "properties": { + "slug": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": ["name", "slug"] + }, + "provider": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + } + }, + "required": ["id", "name"] + }, + "dateVerified": { + "type": "string", + "format": "date-time" + }, + "dateSynced": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "dateSynced", + "dateVerified", + "id", + "name", + "organization", + "provider" + ] + } + }, + "avatar": { + "type": "object", + "properties": { + "avatarType": { + "type": "string" + }, + "avatarUuid": { + "type": "string", + "nullable": true + }, + "avatarUrl": { + "type": "string", + "nullable": true + } + } + }, + "authenticators": { + "type": "array", + "items": {} + }, + "canReset2fa": { + "type": "boolean" + }, + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "username": { + "type": "string" + }, + "email": { + "type": "string" + }, + "avatarUrl": { + "type": "string" + }, + "isActive": { + "type": "boolean" + }, + "isSuspended": { + "type": "boolean" + }, + "hasPasswordAuth": { + "type": "boolean" + }, + "isManaged": { + "type": "boolean" + }, + "dateJoined": { + "type": "string", + "format": "date-time" + }, + "lastLogin": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "has2fa": { + "type": "boolean" + }, + "lastActive": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "isSuperuser": { + "type": "boolean" + }, + "isStaff": { + "type": "boolean" + }, + "experiments": { + "type": "object", + "additionalProperties": {} + }, + "emails": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "email": { + "type": "string" + }, + "is_verified": { + "type": "boolean" + } + }, + "required": ["email", "id", "is_verified"] + } + } + }, + "required": [ + "avatarUrl", + "dateJoined", + "email", + "emails", + "experiments", + "has2fa", + "hasPasswordAuth", + "id", + "isActive", + "isManaged", + "isStaff", + "isSuperuser", + "isSuspended", + "lastActive", + "lastLogin", + "name", + "username" + ], + "nullable": true + }, + "widgets": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "title": { + "type": "string" + }, + "description": { + "type": "string", + "nullable": true + }, + "displayType": { + "type": "string" + }, + "thresholds": { + "type": "object", + "properties": { + "preferredPolarity": { + "type": "string" + }, + "timeWindow": { + "type": "string" + }, + "max_values": { + "type": "object", + "additionalProperties": { + "type": "integer" + } + }, + "unit": { + "type": "string" + } + }, + "required": ["max_values", "unit"], + "nullable": true + }, + "interval": { + "type": "string" + }, + "dateCreated": { + "type": "string" + }, + "dashboardId": { + "type": "string" + }, + "queries": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "fields": { + "type": "array", + "items": { + "type": "string" + } + }, + "aggregates": { + "type": "array", + "items": { + "type": "string" + } + }, + "columns": { + "type": "array", + "items": { + "type": "string" + } + }, + "fieldAliases": { + "type": "array", + "items": { + "type": "string" + } + }, + "conditions": { + "type": "string" + }, + "orderby": { + "type": "string" + }, + "widgetId": { + "type": "string" + }, + "onDemand": { + "type": "array", + "items": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "extractionState": { + "type": "string" + }, + "dashboardWidgetQueryId": { + "type": "integer" + } + }, + "required": [ + "dashboardWidgetQueryId", + "enabled", + "extractionState" + ] + } + }, + "isHidden": { + "type": "boolean" + }, + "selectedAggregate": { + "type": "integer", + "nullable": true + }, + "linkedDashboards": { + "type": "array", + "items": { + "type": "object", + "properties": { + "field": { + "type": "string" + }, + "dashboardId": { + "type": "integer" + } + }, + "required": ["dashboardId", "field"] + } + } + }, + "required": [ + "aggregates", + "columns", + "conditions", + "fieldAliases", + "fields", + "id", + "isHidden", + "linkedDashboards", + "name", + "onDemand", + "orderby", + "selectedAggregate", + "widgetId" + ] + } + }, + "limit": { + "type": "integer", + "nullable": true + }, + "widgetType": { + "type": "string", + "nullable": true + }, + "layout": { + "type": "object", + "additionalProperties": { + "type": "integer" + }, + "nullable": true + }, + "axisRange": { + "type": "string", + "nullable": true + }, + "legendType": { + "enum": ["default", "breakdown", null], + "type": "string", + "nullable": true + }, + "datasetSource": { + "type": "string", + "nullable": true + }, + "exploreUrls": { + "type": "array", + "items": { + "type": "string" + }, + "nullable": true + }, + "changedReason": { + "type": "array", + "items": { + "type": "object", + "properties": { + "orderby": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "nullable": true + }, + "equations": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + } + }, + "nullable": true + }, + "selected_columns": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "equations", + "orderby", + "selected_columns" + ] + }, + "nullable": true + } + }, + "required": [ + "axisRange", + "changedReason", + "dashboardId", + "datasetSource", + "dateCreated", + "description", + "displayType", + "exploreUrls", + "id", + "interval", + "layout", + "legendType", + "limit", + "queries", + "thresholds", + "title", + "widgetType" + ] + } + }, + "projects": { + "type": "array", + "items": { + "type": "integer" + } + }, + "filters": { + "type": "object", + "properties": { + "release": { + "type": "array", + "items": { + "type": "string" + } + }, + "releaseId": { + "type": "array", + "items": { + "type": "string" + } + }, + "globalFilter": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": {} + } + } + } + }, + "permissions": { + "type": "object", + "properties": { + "isEditableByEveryone": { + "type": "boolean" + }, + "teamsWithEditAccess": { + "type": "array", + "items": { + "type": "integer" + } + } + }, + "required": [ + "isEditableByEveryone", + "teamsWithEditAccess" + ], + "nullable": true + }, + "isFavorited": { + "type": "boolean" + }, + "prebuiltId": { + "type": "integer", + "nullable": true + } + }, + "required": [ + "createdBy", + "dateCreated", + "filters", + "id", + "isFavorited", + "permissions", + "prebuiltId", + "projects", + "title", + "widgets" + ] + }, + "examples": { + "DashboardGETResponse": { + "value": { + "id": "1", + "title": "Dashboard", + "dateCreated": "2024-06-20T14:38:03.498574Z", + "createdBy": { + "id": "1", + "name": "Admin", + "username": "admin", + "email": "admin@example.com", + "avatarUrl": "www.example.com", + "isActive": true, + "isSuspended": false, + "hasPasswordAuth": true, + "isManaged": false, + "dateJoined": "2021-10-25T17:07:33.190596Z", + "lastLogin": "2024-07-16T15:28:39.261659Z", + "has2fa": true, + "lastActive": "2024-07-16T20:45:49.364197Z", + "isSuperuser": false, + "isStaff": false, + "experiments": {}, + "emails": [ + { + "id": "1", + "email": "admin@example.com", + "is_verified": true + } + ], + "avatar": { + "avatarType": "letter_avatar", + "avatarUuid": null, + "avatarUrl": "www.example.com" + } + }, + "widgets": [ + { + "id": "658714", + "title": "Custom Widget", + "description": null, + "displayType": "table", + "thresholds": null, + "interval": "5m", + "dateCreated": "2024-07-16T15:36:46.048343Z", + "dashboardId": "1", + "datasetSource": "user", + "queries": [ + { + "id": "1", + "name": "", + "fields": [ + "avg(transaction.duration)", + "transaction" + ], + "aggregates": ["avg(transaction.duration)"], + "columns": ["transaction"], + "fieldAliases": ["", ""], + "conditions": "", + "orderby": "-avg(transaction.duration)", + "widgetId": "1", + "onDemand": [ + { + "enabled": false, + "extractionState": "disabled:not-applicable", + "dashboardWidgetQueryId": 1 + } + ], + "isHidden": false, + "selectedAggregate": null, + "linkedDashboards": [] + } + ], + "limit": null, + "widgetType": "transaction-like", + "layout": { + "w": 2, + "y": 0, + "h": 2, + "minH": 2, + "x": 0 + }, + "exploreUrls": null, + "changedReason": null, + "axisRange": null, + "legendType": "default" + } + ], + "projects": [1], + "filters": {}, + "environment": ["alpha"], + "period": "7d", + "permissions": { + "isEditableByEveryone": true, + "teamsWithEditAccess": [] + }, + "isFavorited": false, + "prebuiltId": null + }, + "summary": "Dashboard GET response" + } + } + } + }, + "description": "" + }, + "403": { + "description": "Forbidden" + }, + "404": { + "description": "Not Found" + } + } + } + } + }, + "components": { + "securitySchemes": { + "auth_token": { + "type": "http", + "scheme": "bearer" + }, + "dsn": { + "type": "http", + "scheme": "DSN" + } + } + } + } +} diff --git a/packages/mcp-server-evals/src/code-mode/dashboard-fixture.test.ts b/packages/mcp-server-evals/src/code-mode/dashboard-fixture.test.ts new file mode 100644 index 000000000..dfefc10a9 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/dashboard-fixture.test.ts @@ -0,0 +1,227 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { z } from "zod"; +import { createDashboardFixture } from "./dashboard-fixture"; +import fixtureData from "./dashboard-fixture-data.json" with { type: "json" }; + +type Fixture = Awaited>; +const active: Fixture[] = []; +const accessToken = "local-test-token"; +const dashboardResponse = z + .object({ + id: z.string(), + widgets: z.array( + z + .object({ + id: z.string(), + queries: z.array(z.object({ id: z.string() }).passthrough()), + }) + .passthrough(), + ), + }) + .passthrough(); + +afterEach(async () => { + await Promise.all(active.splice(0).map((fixture) => fixture.close())); +}); + +async function fixture() { + const instance = await createDashboardFixture({ accessToken }); + active.push(instance); + return instance; +} + +function request(fixture: Fixture, path: string, body?: unknown) { + return fetch(`${fixture.baseUrl}${path}`, { + method: body ? "POST" : "GET", + headers: { + authorization: `Bearer ${accessToken}`, + "content-type": "application/json", + }, + body: body ? JSON.stringify(body) : undefined, + }); +} + +function copyBody(projectId: number, title: string) { + const source = structuredClone(fixtureData.sourceDashboard); + return { + ...source, + title, + projects: [projectId], + environment: ["staging"], + widgets: source.widgets.map(({ id, ...widget }) => widget), + }; +} + +function snakeKeys(value: unknown): unknown { + if (Array.isArray(value)) return value.map(snakeKeys); + if (value === null || typeof value !== "object") return value; + return Object.fromEntries( + Object.entries(value).map(([key, value]) => [ + key.replace(/[A-Z]/g, (letter) => `_${letter.toLowerCase()}`), + snakeKeys(value), + ]), + ); +} + +describe("dashboard Code Mode eval fixture", () => { + it("returns dashboard list metadata with accurate widget previews", async () => { + const instance = await fixture(); + const response = await request( + instance, + `/api/0/organizations/${instance.organizationSlug}/dashboards/?query=Errors`, + ); + expect(response.status).toBe(200); + const dashboards = z + .array(z.record(z.string(), z.unknown())) + .parse(await response.json()); + expect(dashboards).toHaveLength(1); + expect(dashboards[0]).toMatchObject({ + id: "101", + title: "Errors Overview", + widgetDisplay: ["line", "table"], + widgetPreview: [ + { + displayType: "line", + layout: { x: 0, y: 0, w: 6, h: 2 }, + }, + { + displayType: "table", + layout: { x: 0, y: 2, w: 6, h: 3 }, + }, + ], + projects: [101, 102], + environment: ["production"], + }); + expect(dashboards[0]).not.toHaveProperty("widgets"); + }); + + it("persists copies made through the HTTP API using both supported key styles", async () => { + const instance = await fixture(); + const path = `/api/0/organizations/${instance.organizationSlug}/dashboards/`; + const original = dashboardResponse.parse( + await (await request(instance, `${path}101/`)).json(), + ); + expect(original.widgets).toHaveLength(2); + + const firstResponse = await request( + instance, + path, + copyBody(101, "Frontend staging"), + ); + expect(firstResponse.status).toBe(201); + const first = dashboardResponse.parse(await firstResponse.json()); + const secondResponse = await request( + instance, + path, + snakeKeys(copyBody(102, "Backend staging")), + ); + expect(secondResponse.status).toBe(201); + const second = dashboardResponse.parse(await secondResponse.json()); + + expect((await request(instance, `${path}${first.id}/`)).status).toBe(200); + expect(first.widgets[0].id).not.toBe(original.widgets[0].id); + expect(first.widgets[0].queries[0].id).not.toBe( + original.widgets[0].queries[0].id, + ); + expect(await (await request(instance, `${path}101/`)).json()).toEqual( + original, + ); + expect( + instance.verify( + [ + `ID ${first.id}: https://${instance.organizationSlug}.sentry.io/dashboard/${first.id}/`, + `ID ${second.id}: https://sentry.io/organizations/${instance.organizationSlug}/dashboard/${second.id}/`, + ].join("\n"), + ), + ).toEqual({ success: true, dashboardIds: ["201", "202"] }); + + const discovery = await (await fetch(instance.openapiUrl)).json(); + expect(discovery).toMatchObject({ + paths: { + "/api/0/organizations/{organization_id_or_slug}/dashboards/": { + post: { requestBody: expect.any(Object) }, + }, + "/api/0/organizations/{organization_id_or_slug}/dashboards/{dashboard_id}/": + { + get: { responses: { "200": expect.any(Object) } }, + }, + }, + }); + expect((await request(instance, "/snapshot")).status).toBe(404); + + instance.reset(); + expect(instance.snapshot().dashboards).toHaveLength(1); + expect(instance.snapshot().requests).toEqual([]); + }); + + it("rejects missing authentication, foreign widget IDs and inaccessible projects without writing", async () => { + const instance = await fixture(); + const path = `/api/0/organizations/${instance.organizationSlug}/dashboards/`; + expect( + ( + await fetch(`${instance.baseUrl}${path}`, { + method: "POST", + body: JSON.stringify(copyBody(101, "Frontend staging")), + }) + ).status, + ).toBe(401); + const copiedIds = await request( + instance, + path, + fixtureData.sourceDashboard, + ); + expect(copiedIds.status).toBe(400); + expect(await copiedIds.json()).toEqual({ + detail: "You cannot update widgets that are not part of this dashboard.", + }); + expect( + (await request(instance, path, copyBody(999, "Unavailable project"))) + .status, + ).toBe(400); + expect(instance.snapshot().dashboards).toHaveLength(1); + }); + + it("grades actual state, rejecting claims of success, lost queries and duplicate writes", async () => { + const instance = await fixture(); + const path = `/api/0/organizations/${instance.organizationSlug}/dashboards/`; + const answer = [201, 202] + .map( + (id) => + `https://${instance.organizationSlug}.sentry.io/dashboard/${id}/`, + ) + .join("\n"); + expect(() => instance.verify(answer)).toThrow( + "Expected exactly two new dashboards", + ); + + const brokenCopy = copyBody(101, "Frontend staging"); + brokenCopy.widgets[0].queries[0].conditions = ""; + expect((await request(instance, path, brokenCopy)).status).toBe(201); + expect( + (await request(instance, path, copyBody(102, "Backend staging"))).status, + ).toBe(201); + expect(() => instance.verify(answer)).toThrow( + "widget configuration differs", + ); + + instance.reset(); + for (const [project, title] of [ + [101, "Frontend staging"], + [102, "Backend staging"], + ] as const) { + expect( + (await request(instance, path, copyBody(project, title))).status, + ).toBe(201); + } + expect(() => + instance.verify("Created both dashboards successfully"), + ).toThrow("final answer omits"); + expect(instance.verify(answer).success).toBe(true); + expect( + (await request(instance, path, copyBody(101, "Frontend staging"))).status, + ).toBe(201); + expect(() => instance.verify(answer)).toThrow( + "Expected exactly two new dashboards", + ); + }); +}); diff --git a/packages/mcp-server-evals/src/code-mode/dashboard-fixture.ts b/packages/mcp-server-evals/src/code-mode/dashboard-fixture.ts new file mode 100644 index 000000000..1897f2560 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/dashboard-fixture.ts @@ -0,0 +1,413 @@ +import assert from "node:assert/strict"; +import { createServer, type IncomingMessage } from "node:http"; +import { z } from "zod"; +import fixtureData from "./dashboard-fixture-data.json" with { type: "json" }; + +const ORGANIZATION = "dashboard-evals"; +const TARGETS = [ + { projectId: 101, title: "Frontend staging" }, + { projectId: 102, title: "Backend staging" }, +] as const; +const DATE_CREATED = "2026-10-25T10:00:00.000Z"; + +const querySchema = z.object({ + fields: z.array(z.string()), + conditions: z.string(), + name: z.string().default(""), + aggregates: z.array(z.string()).nullable().default(null), + columns: z.array(z.string()).nullable().default(null), + fieldAliases: z.array(z.string()).nullable().default(null), + orderby: z.string().default(""), + isHidden: z.boolean().default(false), + selectedAggregate: z.number().int().nullable().default(null), + linkedDashboards: z.array(z.unknown()).default([]), +}); + +const layoutSchema = z + .object({ + x: z.number().int().min(0).max(5), + y: z.number().int().min(0), + w: z.number().int().min(1).max(6), + h: z.number().int().min(2), + }) + .refine((layout) => layout.x + layout.w <= 6, "x + w must not exceed 6"); + +const widgetSchema = z.object({ + id: z.string().optional(), + title: z.string().min(1).max(255), + description: z.string().nullable().default(null), + // This fixture only emulates the two display types present in its source. + displayType: z.enum(["line", "table"]), + widgetType: z.enum(["error-events"]).default("error-events"), + interval: z.string().max(10).default("5m"), + queries: z.array(querySchema).min(1), + layout: layoutSchema.nullable().default(null), + thresholds: z.record(z.string(), z.unknown()).nullable().default(null), + limit: z.number().int().min(1).nullable().default(null), + axisRange: z.enum(["auto", "dataMin"]).nullable().default(null), + legendType: z.enum(["default", "table"]).nullable().default(null), + datasetSource: z.enum(["user"]).default("user"), +}); + +const dashboardSchema = z.object({ + title: z.string().trim().min(1).max(255), + widgets: z.array(widgetSchema).default([]), + projects: z.array(z.coerce.number().int()).default([]), + environment: z.array(z.string()).nullable().default(null), + period: z.string().nullable().default(null), + filters: z.record(z.string(), z.unknown()).default({}), + utc: z.boolean().default(false), + isFavorited: z.boolean().default(false), + permissions: z + .object({ + isEditableByEveryone: z.boolean(), + teamsWithEditAccess: z.array(z.number()), + }) + .nullable() + .default(null), +}); + +type Dashboard = Omit, "widgets"> & { + id: string; + dateCreated: string; + createdBy: { id: string; name: string; email: string }; + widgets: Array< + Omit, "id" | "queries"> & { + id: string; + dashboardId: string; + dateCreated: string; + queries: Array< + z.infer & { + id: string; + widgetId: string; + onDemand: unknown[]; + } + >; + } + >; + prebuiltId: null; +}; + +export interface DashboardFixtureSnapshot { + dashboards: Dashboard[]; + requests: Array<{ method: string; path: string; status: number }>; +} + +class FixtureInputError extends Error {} + +// Mirror CamelSnakeSerializer's recursive aliases and collision detection. +// Unknown fields are subsequently stripped by the fixture's input schemas. +function camelKeys(value: unknown): unknown { + if (Array.isArray(value)) return value.map(camelKeys); + if (value === null || typeof value !== "object") return value; + const result: Record = {}; + for (const [key, item] of Object.entries(value)) { + const converted = key.replace(/_([a-z])/g, (_, letter: string) => + letter.toUpperCase(), + ); + if (Object.hasOwn(result, converted)) { + throw new FixtureInputError(`${key} collides with ${converted}`); + } + result[converted] = camelKeys(item); + } + return result; +} + +async function readJson(request: IncomingMessage): Promise { + const chunks: Buffer[] = []; + let size = 0; + for await (const chunk of request) { + size += chunk.length; + if (size > 256_000) + throw new FixtureInputError("Request body is too large"); + chunks.push(chunk); + } + try { + return JSON.parse(Buffer.concat(chunks).toString("utf8")); + } catch { + throw new FixtureInputError("Invalid JSON request body"); + } +} + +function sourceDashboard(): Dashboard { + const source = fixtureData.sourceDashboard; + const parsed = dashboardSchema.parse(source); + return { + ...parsed, + id: source.id, + dateCreated: source.dateCreated, + createdBy: structuredClone(source.createdBy), + prebuiltId: null, + widgets: parsed.widgets.map((widget, index) => ({ + ...widget, + id: source.widgets[index].id, + dashboardId: source.id, + dateCreated: source.widgets[index].dateCreated, + layout: widget.layout ? { ...widget.layout, minH: 2 } : null, + queries: widget.queries.map((query, queryIndex) => ({ + ...query, + id: source.widgets[index].queries[queryIndex].id, + widgetId: source.widgets[index].id, + onDemand: [], + })), + })), + }; +} + +function widgetSemantics(dashboard: Dashboard) { + return dashboard.widgets.map( + ({ id, dashboardId, dateCreated, queries, ...widget }) => ({ + ...widget, + queries: queries.map(({ id, widgetId, onDemand, ...query }) => query), + }), + ); +} + +/** + * Local stateful API substitute, not a full Sentry emulator. Its dashboard + * creation, IDs and aliases follow backend c4b7b5f; the complete three-operation + * OpenAPI slice is captured from sentry-api-schema 803d6de. It deliberately does + * not implement query-language validation, quotas, feature flags or real auth. + * Administrative state is only exposed to the runner, never as HTTP endpoints. + */ +export async function createDashboardFixture({ + accessToken = "code-mode-eval-token", +}: { + accessToken?: string; +} = {}) { + const original = sourceDashboard(); + let dashboards = new Map([[original.id, structuredClone(original)]]); + let nextId = 201; + let requests: DashboardFixtureSnapshot["requests"] = []; + const collectionPath = `/api/0/organizations/${ORGANIZATION}/dashboards/`; + + const server = createServer(async (request, response) => { + const url = new URL(request.url ?? "/", "http://127.0.0.1"); + const method = request.method ?? "GET"; + const reply = (status: number, body: unknown) => { + requests.push({ method, path: url.pathname, status }); + response.writeHead(status, { "content-type": "application/json" }); + response.end(JSON.stringify(body)); + }; + + try { + if (method === "GET" && url.pathname === "/openapi.json") { + reply(200, fixtureData.openapi); + return; + } + if (request.headers.authorization !== `Bearer ${accessToken}`) { + reply(401, { detail: "Invalid token" }); + return; + } + if (url.pathname === collectionPath && method === "GET") { + const query = url.searchParams.get("query")?.toLowerCase() ?? ""; + reply( + 200, + [...dashboards.values()] + .filter((dashboard) => + dashboard.title.toLowerCase().includes(query), + ) + .map((dashboard) => ({ + id: dashboard.id, + title: dashboard.title, + dateCreated: dashboard.dateCreated, + createdBy: dashboard.createdBy, + environment: dashboard.environment, + filters: dashboard.filters, + lastVisited: null, + widgetDisplay: dashboard.widgets.map( + (widget) => widget.displayType, + ), + widgetPreview: dashboard.widgets.map( + ({ displayType, layout }) => ({ displayType, layout }), + ), + permissions: dashboard.permissions, + isFavorited: dashboard.isFavorited, + projects: dashboard.projects, + prebuiltId: dashboard.prebuiltId, + })), + ); + return; + } + if (url.pathname === collectionPath && method === "POST") { + const input = dashboardSchema.parse(camelKeys(await readJson(request))); + if ( + input.projects.some((id) => !TARGETS.some((t) => t.projectId === id)) + ) { + reply(400, { projects: ["Invalid project IDs"] }); + return; + } + // Backend update_widgets rejects IDs belonging to the original dashboard. + // Query IDs are ignored during create_widget; new query IDs are allocated. + if (input.widgets.some((widget) => widget.id)) { + reply(400, { + detail: + "You cannot update widgets that are not part of this dashboard.", + }); + return; + } + if (url.searchParams.has("validateOnly")) { + reply(200, null); + return; + } + const id = String(nextId++); + const existingTitles = new Set( + [...dashboards.values()].map((dashboard) => dashboard.title), + ); + let title = input.title; + for (let suffix = 1; existingTitles.has(title); suffix++) { + title = `${input.title} (${suffix})`; + } + const created: Dashboard = { + ...input, + title, + id, + dateCreated: DATE_CREATED, + createdBy: structuredClone(original.createdBy), + prebuiltId: null, + widgets: input.widgets.map((widget, index) => { + const widgetId = `${id}${index + 1}`; + return { + ...widget, + id: widgetId, + dashboardId: id, + dateCreated: DATE_CREATED, + layout: widget.layout ? { ...widget.layout, minH: 2 } : null, + queries: widget.queries.map((query, queryIndex) => ({ + ...query, + id: `${widgetId}${queryIndex + 1}`, + widgetId, + onDemand: [], + })), + }; + }), + }; + dashboards.set(id, created); + reply(201, created); + return; + } + if (method === "GET" && url.pathname.startsWith(collectionPath)) { + const id = url.pathname.slice(collectionPath.length).replace(/\/$/, ""); + const dashboard = dashboards.get(id); + reply(dashboard ? 200 : 404, dashboard ?? { detail: "Not found" }); + return; + } + reply(404, { detail: "Not found in the dashboard eval fixture" }); + } catch (error) { + if (error instanceof z.ZodError) { + reply(400, { detail: "Invalid dashboard", errors: error.issues }); + } else if (error instanceof FixtureInputError) { + reply(400, { detail: error.message }); + } else { + reply(500, { detail: "Fixture internal error" }); + } + } + }); + + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", resolve); + }); + const address = server.address(); + assert(address && typeof address !== "string"); + const baseUrl = `http://127.0.0.1:${address.port}`; + const prompt = [ + `In organization ${ORGANIZATION}, copy dashboard ${original.id} (${original.title}) into exactly two new dashboards.`, + `The original is https://${ORGANIZATION}.sentry.io/dashboard/${original.id}/.`, + ...TARGETS.map( + (target) => + `Create "${target.title}" with only project ID ${target.projectId} saved in the dashboard.`, + ), + 'Save only the environment "staging" in each copy.', + "Preserve the original widgets, their queries, order and layout, and the other saved dashboard filters and time range.", + "Leave the original dashboard unchanged. Return the IDs and Sentry dashboard links for the two new dashboards.", + ].join("\n"); + + return { + baseUrl, + openapiUrl: `${baseUrl}/openapi.json`, + organizationSlug: ORGANIZATION, + projectIds: TARGETS.map((target) => target.projectId), + prompt, + snapshot(): DashboardFixtureSnapshot { + return structuredClone({ + dashboards: [...dashboards.values()], + requests, + }); + }, + reset() { + dashboards = new Map([[original.id, structuredClone(original)]]); + nextId = 201; + requests = []; + }, + verify(resultText: string) { + assert.deepEqual( + dashboards.get(original.id), + original, + "Original dashboard changed", + ); + const copies = [...dashboards.values()].filter( + (item) => item.id !== original.id, + ); + assert.equal(copies.length, 2, "Expected exactly two new dashboards"); + for (const target of TARGETS) { + const copy = copies.find((item) => item.title === target.title); + assert(copy, `Missing dashboard: ${target.title}`); + assert.deepEqual( + copy.projects, + [target.projectId], + `${target.title}: projects differ`, + ); + assert.deepEqual( + copy.environment, + ["staging"], + `${target.title}: environment differs`, + ); + assert.deepEqual( + widgetSemantics(copy), + widgetSemantics(original), + `${target.title}: widget configuration differs`, + ); + assert.deepEqual( + copy.filters, + original.filters, + `${target.title}: filters differ`, + ); + assert.equal( + copy.period, + original.period, + `${target.title}: time range differs`, + ); + assert.equal( + copy.utc, + original.utc, + `${target.title}: UTC setting differs`, + ); + const urls = resultText.match(/https:\/\/[^\s<>"'`\])]+/g) ?? []; + assert( + urls.some((value) => { + const url = new URL(value); + return ( + (url.hostname === `${ORGANIZATION}.sentry.io` && + url.pathname === `/dashboard/${copy.id}/`) || + (url.hostname === "sentry.io" && + url.pathname === + `/organizations/${ORGANIZATION}/dashboard/${copy.id}/`) + ); + }), + `${target.title}: final answer omits its actual dashboard link and ID`, + ); + } + return { + success: true as const, + dashboardIds: copies.map((copy) => copy.id), + }; + }, + async close() { + await new Promise((resolve, reject) => { + server.close((error) => (error ? reject(error) : resolve())); + server.closeAllConnections(); + }); + }, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/json-answer.test.ts b/packages/mcp-server-evals/src/code-mode/json-answer.test.ts new file mode 100644 index 000000000..867b8173a --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/json-answer.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, test } from "vitest"; +import { parseEvalJsonAnswer } from "./json-answer"; + +describe("parseEvalJsonAnswer", () => { + test("uses the fenced answer when later prose contains API paths and JSON examples", () => { + const text = [ + "Done. Verified by re-reading the alert after the write:", + "", + "```json", + '{"id": "701", "enabled": false, "frequency": 60}', + "```", + "", + 'Preserved unchanged: monitor connection (`detectorIds: ["901"]`).', + "", + 'Note: alerts are exposed as `workflows` in the API (`PUT /organizations/{org}/workflows/701/`), and frequency lives in `config.frequency` in minutes. The spec\'s config example shows `{"frequency":3600}`. No other alerts were touched.', + ].join("\n"); + + expect(parseEvalJsonAnswer(text)).toEqual({ + id: "701", + enabled: false, + frequency: 60, + }); + }); + + test.each([ + '{"id":"701"}', + 'Saved values: {"id":"701"}. Verified.', + '```\n{"id":"701"}\n```', + '```json\n{"id":"701"}\n```', + ])("accepts the existing answer format: %s", (text) => { + expect(parseEvalJsonAnswer(text)).toEqual({ id: "701" }); + }); + + test.each([ + '```json\n{"id":}\n```\nCorrection: {"id":"701"}', + '```json\n{"id":"701"}\n```\n```json\n{"id":"702"}\n```', + '```json\n{"id":"701"}\n```\n```\n{"id":"701"}\n```', + '```json\n{"id":"701"}', + '{"id":}', + 'First: {"id":"701"}. Second: {"id":"702"}.', + "No saved values are available.", + ])("rejects malformed, ambiguous, or absent answers: %s", (text) => { + expect(() => parseEvalJsonAnswer(text)).toThrow(SyntaxError); + }); +}); diff --git a/packages/mcp-server-evals/src/code-mode/json-answer.ts b/packages/mcp-server-evals/src/code-mode/json-answer.ts new file mode 100644 index 000000000..e3c80f12c --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/json-answer.ts @@ -0,0 +1,17 @@ +export function parseEvalJsonAnswer(text: string): unknown { + const fenced = text.split("```"); + if (fenced.length > 1) { + if (fenced.length !== 3) { + throw new SyntaxError("Final answer must contain exactly one JSON fence"); + } + // Prefer the explicit answer over braces in surrounding explanations. + return JSON.parse(fenced[1].trim().replace(/^json\b\s*/i, "")); + } + + const start = text.indexOf("{"); + const end = text.lastIndexOf("}"); + if (start === -1 || end < start) { + throw new SyntaxError("Final answer must include the saved values as JSON"); + } + return JSON.parse(text.slice(start, end + 1)); +} diff --git a/packages/mcp-server-evals/src/code-mode/monitor-fixture.test.ts b/packages/mcp-server-evals/src/code-mode/monitor-fixture.test.ts new file mode 100644 index 000000000..2a52acca7 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/monitor-fixture.test.ts @@ -0,0 +1,298 @@ +import tools from "@sentry/mcp-core/tools"; +import { afterEach, describe, expect, it } from "vitest"; +import { z } from "zod"; +import { createMonitorFixture } from "./monitor-fixture"; + +const { find_monitors: findMonitors, get_monitor_details: getMonitorDetails } = + tools; +type Fixture = Awaited>; +const active: Fixture[] = []; +const accessToken = "local-monitor-test-token"; +const edit = { + config: { schedule: "30 9 * * 1-5", timezone: "Europe/Madrid" }, +}; +const finalAnswer = JSON.stringify({ slug: "nightly-billing", ...edit.config }); + +afterEach(async () => { + await Promise.all(active.splice(0).map((instance) => instance.close())); +}); + +async function fixture() { + const instance = await createMonitorFixture({ accessToken }); + active.push(instance); + return instance; +} + +function request( + instance: Fixture, + body?: unknown, + reference = "nightly-billing", + projectScoped = false, +) { + const prefix = projectScoped + ? `projects/${instance.organizationSlug}/billing` + : `organizations/${instance.organizationSlug}`; + return fetch(`${instance.baseUrl}/api/0/${prefix}/monitors/${reference}/`, { + method: body === undefined ? "GET" : "PUT", + headers: { + authorization: `Bearer ${accessToken}`, + "content-type": "application/json", + }, + body: body === undefined ? undefined : JSON.stringify(body), + }); +} + +describe("cron monitor Code Mode eval fixture", () => { + it("runs the real catalog monitor handlers against the fixture", async () => { + const instance = await fixture(); + const context = { + accessToken, + sentryHost: new URL(instance.baseUrl).host, + sentryProtocol: "http" as const, + constraints: {}, + }; + const list = await findMonitors.handler( + z.object(findMonitors.inputSchema).parse({ + organizationSlug: instance.organizationSlug, + projectSlug: "billing", + query: "weekday", + }), + context, + ); + expect(list).toMatchObject({ + structuredContent: { + hasMore: false, + monitors: [ + { + slug: "nightly-billing", + status: "disabled", + projectSlug: "billing", + }, + ], + }, + }); + const detail = await getMonitorDetails.handler( + z.object(getMonitorDetails.inputSchema).parse({ + organizationSlug: instance.organizationSlug, + projectSlugOrId: "billing", + monitorSlug: "nightly-billing", + }), + context, + ); + expect(detail).toContain("**Slug**: nightly-billing"); + expect(detail).toContain("**Status**: disabled"); + expect(detail).toContain("**timezone**: UTC"); + expect( + instance.snapshot().requests.every((request) => request.status === 200), + ).toBe(true); + expect(instance.snapshot().writes).toEqual([]); + }); + + it("serves discovery and the supplementary reads used by ordinary monitor tools", async () => { + const instance = await fixture(); + const paths = [ + "/api/0/organizations/monitor-evals/monitors/?projectSlug=billing&query=weekday&per_page=11", + "/api/0/projects/monitor-evals/billing/", + "/api/0/projects/monitor-evals/billing/monitors/nightly-billing/", + "/api/0/projects/monitor-evals/billing/monitors/nightly-billing/checkins/", + "/api/0/organizations/monitor-evals/monitors/nightly-billing/stats/", + "/api/0/organizations/monitor-evals/monitors/nightly-billing/?expand=alertRule", + ]; + const responses = await Promise.all( + paths.map((path) => + fetch(`${instance.baseUrl}${path}`, { + headers: { authorization: `Bearer ${accessToken}` }, + }), + ), + ); + expect(responses.map((response) => response.status)).toEqual([ + 200, 200, 200, 200, 200, 200, + ]); + const [list, project, monitor, checkins, stats, expanded] = + await Promise.all(responses.map((response) => response.json())); + expect(list).toEqual([monitor]); + expect(project).toMatchObject({ id: "101", slug: "billing" }); + expect(monitor).toMatchObject({ + slug: "nightly-billing", + status: "disabled", + isMuted: false, + }); + expect(monitor).not.toHaveProperty("alertRule"); + expect(expanded).toEqual(instance.snapshot().monitors[0]); + expect(checkins).toEqual([]); + expect(stats).toEqual([]); + expect(instance.snapshot().writes).toEqual([]); + }); + + it("merges a partial config without applying omitted defaults or changing mixed environment mute state", async () => { + const instance = await fixture(); + const before = instance.snapshot(); + expect( + (await request(instance, edit, "nightly-billing", true)).status, + ).toBe(200); + const updated = await (await request(instance)).json(); + expect(updated).toMatchObject({ + status: "disabled", + owner: before.monitors[0].owner, + config: { ...before.monitors[0].config, ...edit.config }, + environments: before.monitors[0].environments, + }); + expect(instance.verify(finalAnswer)).toMatchObject({ + success: true, + ...edit.config, + }); + expect(instance.snapshot().monitors[1]).toEqual(before.monitors[1]); + expect(instance.snapshot().writes).toHaveLength(1); + instance.reset(); + expect(instance.snapshot()).toEqual(before); + }); + + it("finds monitors using backend-supported quoted phrases and name filters", async () => { + const instance = await fixture(); + for (const query of [ + '"Billing weekday import"', + 'name:"Billing weekday import"', + 'weekday name:"Billing weekday import"', + ]) { + const response = await fetch( + `${instance.baseUrl}/api/0/organizations/monitor-evals/monitors/?${new URLSearchParams({ query })}`, + { headers: { authorization: `Bearer ${accessToken}` } }, + ); + expect(response.status).toBe(200); + expect(await response.json()).toMatchObject([ + { slug: "nightly-billing" }, + ]); + } + }); + + it("accepts equivalent weekday expressions and requires the answer to reflect the actual saved schedule", async () => { + const instance = await fixture(); + for (const schedule of [ + "30 9 * * 1,2,3,4,5", + "30 09 * * MON-FRI", + "30 9 * * MON,TUE,WED,THU,FRI", + "30 9 * * 1-5/1", + ]) { + expect( + (await request(instance, { config: { ...edit.config, schedule } })) + .status, + ).toBe(200); + expect( + instance.verify( + JSON.stringify({ + slug: "nightly-billing", + schedule, + timezone: "Europe/Madrid", + }), + ), + ).toMatchObject({ success: true, schedule }); + expect(() => instance.verify(finalAnswer)).toThrow( + "Final answer does not match the saved monitor values", + ); + } + expect( + (await request(instance, { config: { schedule: "30 9 * * 1-6" } })) + .status, + ).toBe(200); + expect(() => instance.verify(finalAnswer)).toThrow( + "Target monitor schedule is incorrect", + ); + }); + + it("rejects GET-shaped owner/project objects while accepting scalar references and camelCase config", async () => { + const instance = await fixture(); + const before = instance.snapshot().monitors[0]; + expect( + ( + await request(instance, { + ...before, + config: { ...before.config, ...edit.config }, + }) + ).status, + ).toBe(400); + expect(instance.snapshot().writes).toEqual([]); + expect( + ( + await request(instance, { + project: "101", + owner: "team:42", + config: { ...edit.config, scheduleType: "crontab", maxRuntime: 45 }, + }) + ).status, + ).toBe(200); + expect(instance.verify(finalAnswer).success).toBe(true); + }); + + it("detects the side effect of resending aggregate isMuted on mixed environments", async () => { + const instance = await fixture(); + expect((await request(instance, { ...edit, isMuted: false })).status).toBe( + 200, + ); + expect( + instance.snapshot().monitors[0].environments.map((env) => env.isMuted), + ).toEqual([false, false]); + expect(() => instance.verify(finalAnswer)).toThrow( + "Unrequested monitor settings changed", + ); + }); + + it("rejects temporarily clearing a threshold even if restored before the final answer", async () => { + const instance = await fixture(); + expect( + ( + await request(instance, { + config: { ...edit.config, failure_issue_threshold: null }, + }) + ).status, + ).toBe(200); + expect( + (await request(instance, { config: { failure_issue_threshold: 3 } })) + .status, + ).toBe(200); + expect(instance.snapshot().monitors[0].config.failure_issue_threshold).toBe( + 3, + ); + expect(() => instance.verify(finalAnswer)).toThrow( + "Write 1: unrequested monitor settings changed during the operation", + ); + }); + + it("rejects changes to another monitor and incorrect reported saved values", async () => { + const instance = await fixture(); + expect((await request(instance, edit)).status).toBe(200); + expect(() => + instance.verify( + JSON.stringify({ + slug: "nightly-billing", + schedule: "0 2 * * *", + timezone: "UTC", + }), + ), + ).toThrow("Final answer does not match the saved monitor values"); + expect(() => instance.verify("Done")).toThrow( + "Final answer must include the saved values as JSON", + ); + expect( + (await request(instance, { status: "active" }, "billing-export")).status, + ).toBe(200); + expect(() => instance.verify(finalAnswer)).toThrow( + "Unrelated monitor changed", + ); + }); + + it("rejects invented success and invalid writes without changing fixture state", async () => { + const instance = await fixture(); + const before = instance.snapshot().monitors; + expect(() => instance.verify(finalAnswer)).toThrow( + "Target monitor schedule is incorrect", + ); + for (const body of [ + { config: { timezone: "Invalid/Timezone" } }, + { project: "different-project" }, + { isMuted: false, is_muted: true }, + ]) + expect((await request(instance, body)).status).toBe(400); + expect(instance.snapshot().monitors).toEqual(before); + expect(instance.snapshot().writes).toEqual([]); + }); +}); diff --git a/packages/mcp-server-evals/src/code-mode/monitor-fixture.ts b/packages/mcp-server-evals/src/code-mode/monitor-fixture.ts new file mode 100644 index 000000000..c509ee4ca --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/monitor-fixture.ts @@ -0,0 +1,491 @@ +import assert from "node:assert/strict"; +import { z } from "zod"; +import { startApiFixture } from "./api-fixture"; +import { parseEvalJsonAnswer } from "./json-answer"; + +const ORGANIZATION = "monitor-evals"; +const MONITOR_ID = "a0000000-0000-4000-8000-000000000001"; +const OTHER_MONITOR_ID = "a0000000-0000-4000-8000-000000000002"; +const SCHEDULE = "30 9 * * 1-5"; +const TIMEZONE = "Europe/Madrid"; +const project = { + id: "101", + slug: "billing", + name: "Billing", + platform: "python", +}; +const owner = { type: "team", id: "42", name: "Billing on-call" }; + +const configSchema = z.object({ + schedule_type: z.enum(["crontab", "interval"]).optional(), + schedule: z + .union([ + z.string(), + z.tuple([ + z.number().int().positive(), + z.enum(["year", "month", "week", "day", "hour", "minute"]), + ]), + ]) + .optional(), + timezone: z + .string() + .refine((value) => { + if (!value) return true; + try { + new Intl.DateTimeFormat("en", { timeZone: value }); + return true; + } catch { + return false; + } + }) + .optional(), + checkin_margin: z.number().int().min(1).max(40320).nullable().optional(), + max_runtime: z.number().int().min(1).max(10080).nullable().optional(), + failure_issue_threshold: z + .number() + .int() + .min(1) + .max(720) + .nullable() + .optional(), + recovery_threshold: z.number().int().min(1).max(720).nullable().optional(), +}); +const updateSchema = z.object({ + project: z.union([z.string(), z.number().int()]).optional(), + name: z.string().min(1).max(128).optional(), + slug: z + .string() + .min(1) + .max(50) + .regex(/^(?![0-9]+$)[a-z0-9_-]+$/) + .optional(), + status: z.enum(["active", "disabled"]).optional(), + owner: z + .string() + .regex(/^(team|user):\d+$/) + .nullable() + .optional(), + is_muted: z.boolean().optional(), + config: configSchema.optional(), + alert_rule: z + .object({ + environment: z.string().nullable().optional(), + targets: z.array( + z.object({ + target_identifier: z.number().int(), + target_type: z.string(), + }), + ), + }) + .optional(), +}); +const answerSchema = z.object({ + slug: z.string(), + schedule: z.string(), + timezone: z.string(), +}); + +// Grade the requested days, not one spelling of an equivalent crontab. +function isRequestedSchedule(schedule: Monitor["config"]["schedule"]) { + if (typeof schedule !== "string") return false; + const [minute, hour, day, month, weekdays, extra] = schedule + .trim() + .split(/\s+/); + if ( + extra || + !/^0*30$/.test(minute) || + !/^0*9$/.test(hour) || + day !== "*" || + month !== "*" || + !weekdays + ) + return false; + const names = ["SUN", "MON", "TUE", "WED", "THU", "FRI", "SAT"]; + const asDay = (value: string) => + /^\d+$/.test(value) ? Number(value) : names.indexOf(value.toUpperCase()); + const days = new Set(); + for (const part of weekdays.split(",")) { + const [range, stepText, extraStep] = part.split("/"); + const step = stepText === undefined ? 1 : Number(stepText); + const [first, last = first, extraDay] = range.split("-"); + const start = asDay(first), + end = asDay(last); + if ( + extraStep || + extraDay || + !Number.isInteger(step) || + step < 1 || + start < 0 || + end > 7 || + end < start + ) + return false; + for (let value = start; value <= end; value += step) days.add(value % 7); + } + return days.size === 5 && [1, 2, 3, 4, 5].every((day) => days.has(day)); +} + +// This fixture models the free-text and name filters used for this scenario, +// including quoted phrases; other query-language fields are outside its scope. +function matchesQuery(monitor: Monitor, query: string) { + const names: string[] = []; + const words: string[] = []; + for (const token of query.match(/(?:\w+:)?"[^"]*"|\S+/g) ?? []) { + if (token.startsWith("name:")) + names.push(token.slice(5).replace(/^"|"$/g, "")); + else if (token.includes(":")) return false; + else words.push(token.replace(/^"|"$/g, "")); + } + const text = words.join(" "); + return ( + (!names.length || names.includes(monitor.name.toLowerCase())) && + (!text || + monitor.name.toLowerCase().includes(text) || + monitor.slug.toLowerCase().includes(text)) + ); +} + +// MonitorValidator accepts camelCase aliases recursively and rejects collisions. +function snakeKeys(value: unknown): unknown { + if (Array.isArray(value)) return value.map(snakeKeys); + if (!value || typeof value !== "object") return value; + const result: Record = {}; + for (const [key, item] of Object.entries(value)) { + const converted = key.replace( + /[A-Z]/g, + (letter) => `_${letter.toLowerCase()}`, + ); + if (Object.hasOwn(result, converted)) + throw new SyntaxError(`Duplicate field ${converted}`); + result[converted] = snakeKeys(item); + } + return result; +} + +function sourceMonitor(id: string, slug: string, name: string) { + return { + id, + slug, + name, + status: "disabled" as "active" | "disabled", + isMuted: false, + isUpserting: false, + dateCreated: "2026-10-20T02:00:00.000Z", + project, + owner: owner as typeof owner | null, + config: { + schedule_type: "crontab" as "crontab" | "interval", + schedule: "0 2 * * *" as string | [number, string], + timezone: "UTC", + checkin_margin: 10 as number | null, + max_runtime: 45 as number | null, + failure_issue_threshold: 3 as number | null, + recovery_threshold: 2 as number | null, + alert_rule_id: 501, + }, + environments: [ + { + name: "production", + status: "ok", + isMuted: false, + dateCreated: "2026-10-20T02:00:00.000Z", + lastCheckIn: "2026-10-25T02:00:00.000Z", + activeIncident: null, + }, + { + name: "staging", + status: "ok", + isMuted: true, + dateCreated: "2026-10-20T02:00:00.000Z", + lastCheckIn: "2026-10-25T02:00:00.000Z", + activeIncident: null, + }, + ], + alertRule: { + environment: "production" as string | null, + targets: [{ targetIdentifier: 42, targetType: "Team" }], + }, + }; +} +type Monitor = ReturnType; + +/** + * Synthetic monitor GET/PUT state based on MonitorValidator's partial update. + * Models config merging and mute propagation, not quotas, actor permissions, + * crontab execution, next-check-in recomputation or notification delivery. + */ +export async function createMonitorFixture({ + accessToken = "code-mode-eval-token", +}: { + accessToken?: string; +} = {}) { + const original = sourceMonitor( + MONITOR_ID, + "nightly-billing", + "Billing weekday import", + ); + const other = sourceMonitor( + OTHER_MONITOR_ID, + "billing-export", + "Billing export", + ); + const initial = [original, other]; + let monitors = new Map( + initial.map((monitor) => [monitor.id, structuredClone(monitor)]), + ); + const writes: Array<{ monitorId: string; monitors: Monitor[] }> = []; + const organizationPath = `/api/0/organizations/${ORGANIZATION}/`; + const collectionPath = `${organizationPath}monitors/`; + const projectPaths = [project.slug, project.id].map( + (id) => `/api/0/projects/${ORGANIZATION}/${id}/`, + ); + + const serialize = (monitor: Monitor, url: URL) => { + const { alertRule, ...body } = structuredClone(monitor); + const environments = url.searchParams.getAll("environment"); + if (environments.length) + body.environments = body.environments.filter((env) => + environments.includes(env.name), + ); + return url.searchParams.getAll("expand").includes("alertRule") + ? { ...body, alertRule } + : body; + }; + const api = await startApiFixture(accessToken, ({ method, url, body }) => { + if (method === "GET" && projectPaths.includes(url.pathname)) + return { body: project }; + if (method === "GET" && url.pathname === `${organizationPath}projects/`) + return { body: [project] }; + if (method === "GET" && url.pathname === collectionPath) { + const query = (url.searchParams.get("query") ?? "").toLowerCase(); + const projectFilter = + url.searchParams.get("projectSlug") ?? url.searchParams.get("project"); + const ownerFilter = url.searchParams.get("owner"); + const items = [...monitors.values()].filter( + (monitor) => + matchesQuery(monitor, query) && + (!projectFilter || + ["-1", project.id, project.slug].includes(projectFilter)) && + (!ownerFilter || + `${monitor.owner?.type}:${monitor.owner?.id}` === ownerFilter), + ); + return { + body: items + .slice(0, Number(url.searchParams.get("per_page") ?? 100)) + .map((item) => serialize(item, url)), + }; + } + const prefix = [ + collectionPath, + ...projectPaths.map((path) => `${path}monitors/`), + ].find((path) => url.pathname.startsWith(path)); + if (!prefix) return undefined; + const [reference, subresource] = url.pathname + .slice(prefix.length) + .split("/"); + const current = [...monitors.values()].find( + (item) => item.id === reference || item.slug === reference, + ); + if (!current) return { status: 404, body: { detail: "Monitor not found" } }; + // get_monitor_details reads these alongside the monitor even when the user + // only needs configuration. Empty history is valid for a disabled monitor. + if (method === "GET" && ["checkins", "stats"].includes(subresource)) + return { body: [] }; + if (subresource) return undefined; + if (method === "GET") return { body: serialize(current, url) }; + if (method !== "PUT") return undefined; + const input = updateSchema.parse(snakeKeys(body)); + if ( + input.project !== undefined && + ![project.slug, project.id].includes(String(input.project)) + ) { + return { + status: 400, + body: { detail: "existing monitors may not be moved between projects" }, + }; + } + if ( + input.slug && + [...monitors.values()].some( + (item) => item.id !== current.id && item.slug === input.slug, + ) + ) { + return { status: 400, body: { slug: "The slug is already in use." } }; + } + const updated = structuredClone(current); + if (input.name !== undefined) updated.name = input.name; + if (input.slug !== undefined) updated.slug = input.slug; + if (input.status !== undefined) updated.status = input.status; + if (input.owner !== undefined) { + const [type, id] = input.owner?.split(":") ?? []; + updated.owner = + input.owner === null + ? null + : { + type, + id, + name: input.owner === "team:42" ? owner.name : `Owner ${id}`, + }; + } + if (input.config) { + const config = { ...input.config }; + if (config.timezone === "") delete config.timezone; + updated.config = { ...updated.config, ...config }; + const schedule = updated.config.schedule; + if ( + updated.config.schedule_type === "crontab" && + (typeof schedule !== "string" || + schedule.trim().split(/\s+/).length !== 5) + ) { + return { + status: 400, + body: { config: { schedule: "Invalid crontab schedule" } }, + }; + } + if ( + updated.config.schedule_type === "interval" && + !Array.isArray(schedule) + ) { + return { + status: 400, + body: { config: { schedule: "Invalid interval schedule" } }, + }; + } + } + if (input.is_muted !== undefined) { + updated.environments = updated.environments.map((env) => ({ + ...env, + isMuted: input.is_muted as boolean, + })); + updated.isMuted = input.is_muted; + } + if (input.alert_rule) { + updated.alertRule = { + environment: input.alert_rule.environment ?? null, + targets: input.alert_rule.targets.map((target) => ({ + targetIdentifier: target.target_identifier, + targetType: target.target_type, + })), + }; + } + monitors.set(updated.id, updated); + writes.push( + structuredClone({ + monitorId: updated.id, + monitors: [...monitors.values()], + }), + ); + return { body: serialize(updated, new URL(url.pathname, url.origin)) }; + }); + + return { + baseUrl: api.baseUrl, + organizationSlug: ORGANIZATION, + prompt: [ + `In organization ${ORGANIZATION}, project billing, change the cron monitor named "${original.name}" to run at 09:30 Monday through Friday in Europe/Madrid.`, + "Keep its name, slug, owner, enabled/disabled status, thresholds, notification settings and each environment's muted state unchanged throughout the operation. Leave all other monitors unchanged.", + "Return JSON with the saved values: {slug, schedule, timezone}.", + ].join("\n"), + cliCommands: [ + `sentry api organizations/${ORGANIZATION}/monitors/${original.slug}/ --method PUT --data '{"config":{"schedule":"${SCHEDULE}","timezone":"${TIMEZONE}"}}'`, + ], + provenance: { + cliRevision: "b625111aef9cc3a44abbed634a043fce8505366d", + cliSource: "packages/cli/src/commands/api.ts", + backendRevision: "b4b356554798716ceaeab74efdc095782be9ca16", + backendSources: [ + "src/sentry/monitors/endpoints/organization_monitor_details.py", + "src/sentry/monitors/endpoints/project_monitor_details.py", + "src/sentry/monitors/endpoints/base_monitor_details.py", + "src/sentry/monitors/endpoints/organization_monitor_index.py", + "src/sentry/monitors/validators.py", + "src/sentry/monitors/serializers.py", + "src/sentry/api/serializers/rest_framework/base.py", + "tests/sentry/monitors/endpoints/test_base_monitor_details.py", + ], + endpoints: [ + "GET /organizations/{organization_id_or_slug}/monitors/", + "GET|PUT /organizations/{organization_id_or_slug}/monitors/{monitor_id_or_slug}/", + "GET|PUT /projects/{organization_id_or_slug}/{project_id_or_slug}/monitors/{monitor_id_or_slug}/", + ], + limitations: + "No quota/actor authorization, scheduler, derived next-check-in timestamps or notification delivery simulation. Query support is free text and name filters. Cron validation only checks shape; the grader accepts equivalent weekday ranges, lists and names for the requested time.", + }, + reset() { + monitors = new Map( + initial.map((monitor) => [monitor.id, structuredClone(monitor)]), + ); + api.requests.length = 0; + writes.length = 0; + }, + snapshot() { + return structuredClone({ + monitors: [...monitors.values()], + requests: api.requests, + writes, + }); + }, + verify(resultText: string) { + const updated = monitors.get(MONITOR_ID); + assert(updated, "Target monitor is missing"); + assert( + isRequestedSchedule(updated.config.schedule), + "Target monitor schedule is incorrect", + ); + assert.equal( + updated.config.timezone, + TIMEZONE, + "Target monitor timezone is incorrect", + ); + const preserved = (target: Monitor) => ({ + ...original, + config: { + ...original.config, + schedule: target.config.schedule, + timezone: target.config.timezone, + }, + }); + assert.deepEqual( + updated, + preserved(updated), + "Unrequested monitor settings changed", + ); + assert.deepEqual( + monitors.get(OTHER_MONITOR_ID), + other, + "Unrelated monitor changed", + ); + for (const [index, write] of writes.entries()) { + const target = write.monitors.find((item) => item.id === MONITOR_ID); + assert(target, `Write ${index + 1}: target monitor is missing`); + assert.deepEqual( + target, + preserved(target), + `Write ${index + 1}: unrequested monitor settings changed during the operation`, + ); + assert.deepEqual( + write.monitors.find((item) => item.id === OTHER_MONITOR_ID), + other, + `Write ${index + 1}: another monitor changed during the operation`, + ); + } + assert.deepEqual( + answerSchema.parse(parseEvalJsonAnswer(resultText)), + { + slug: updated.slug, + schedule: updated.config.schedule, + timezone: TIMEZONE, + }, + "Final answer does not match the saved monitor values", + ); + return { + success: true as const, + monitorId: MONITOR_ID, + slug: updated.slug, + schedule: updated.config.schedule, + timezone: TIMEZONE, + }; + }, + close: api.close, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/release-fixture.test.ts b/packages/mcp-server-evals/src/code-mode/release-fixture.test.ts new file mode 100644 index 000000000..0dbc20548 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/release-fixture.test.ts @@ -0,0 +1,333 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { z } from "zod"; +import { createReleaseFixture } from "./release-fixture"; +import { createReleaseRecoveryFixture } from "./release-recovery-fixture"; + +type Fixture = Awaited>; +const active: Fixture[] = []; +const collection = "/api/0/organizations/release-evals/releases/"; +const version = "checkout@2.4.0"; +const releasePath = `${collection}${encodeURIComponent(version)}/`; +const released = "2026-10-26T10:00:00Z"; +const completeAnswer = { + version, + projects: ["frontend", "backend"], + dateReleased: released, + deploymentId: "601", +}; +const deployBody = { + environment: "staging", + name: "Staging rollout", + url: "https://deploy.example.test/runs/42", + dateStarted: "2026-10-26T10:02:00Z", + dateFinished: "2026-10-26T10:04:00Z", +}; + +afterEach(async () => { + await Promise.all(active.splice(0).map((fixture) => fixture.close())); +}); + +async function fixture(create = createReleaseFixture) { + const instance = await create(); + active.push(instance); + return instance; +} + +function request( + instance: Fixture, + method: string, + path: string, + body?: unknown, +) { + return fetch(`${instance.baseUrl}${path}`, { + method, + headers: { + authorization: "Bearer code-mode-eval-token", + "content-type": "application/json", + }, + body: body === undefined ? undefined : JSON.stringify(body), + }); +} + +describe("release Code Mode eval fixture", () => { + it("persists the CLI create/finalize/deploy flow and exposes normal catalog read routes", async () => { + const instance = await fixture(); + const created = await request(instance, "POST", collection, { + version, + projects: ["frontend", "backend"], + }); + expect(created.status).toBe(201); + expect(await created.json()).toMatchObject({ version, dateReleased: null }); + const finalized = await request(instance, "PUT", releasePath, { + dateReleased: released, + }); + expect(finalized.status).toBe(200); + expect(await finalized.json()).toMatchObject({ + dateReleased: "2026-10-26T10:00:00.000Z", + }); + const deployed = await request( + instance, + "POST", + `${releasePath}deploys/`, + deployBody, + ); + expect(deployed.status).toBe(201); + expect(await deployed.json()).toMatchObject({ + id: "601", + environment: "staging", + }); + expect( + await ( + await request(instance, "GET", `${collection}?query=2.4.0`) + ).json(), + ).toMatchObject([{ version }]); + expect( + await (await request(instance, "GET", releasePath)).json(), + ).toMatchObject({ version, deployCount: 1, lastDeploy: { id: "601" } }); + expect( + await (await request(instance, "GET", `${releasePath}deploys/`)).json(), + ).toMatchObject([{ id: "601" }]); + expect( + await (await request(instance, "GET", `${releasePath}commits/`)).json(), + ).toEqual([]); + expect( + await ( + await request( + instance, + "GET", + `/api/0/projects/release-evals/frontend/releases/${encodeURIComponent(version)}/`, + ) + ).json(), + ).toMatchObject({ version }); + expect(instance.verify(JSON.stringify(completeAnswer))).toEqual({ + success: true, + releaseVersion: version, + deployId: "601", + }); + expect(() => + instance.verify( + JSON.stringify({ ...completeAnswer, deploymentId: "999" }), + ), + ).toThrow("actual deployment ID"); + const { projects, dateReleased, ...incompleteAnswer } = completeAnswer; + expect(() => + instance.verify(JSON.stringify({ ...incompleteAnswer, dateReleased })), + ).toThrow(); + expect(() => + instance.verify(JSON.stringify({ ...incompleteAnswer, projects })), + ).toThrow(); + expect(() => + instance.verify( + JSON.stringify({ ...completeAnswer, projects: ["frontend"] }), + ), + ).toThrow("incorrect release projects"); + expect(() => + instance.verify( + JSON.stringify({ + ...completeAnswer, + dateReleased: "2026-10-25T10:00:00Z", + }), + ), + ).toThrow("incorrect release time"); + expect( + instance.verify( + JSON.stringify({ + ...completeAnswer, + dateReleased: "2026-10-26T12:00:00+02:00", + }), + ).success, + ).toBe(true); + instance.reset(); + expect(instance.snapshot().releases).toHaveLength(1); + expect(instance.snapshot().requests).toEqual([]); + }); + + it("allows finalizing on create but rejects partial state and duplicate deployment success claims", async () => { + const instance = await fixture(); + const body = { + version, + projects: ["frontend", "backend"], + dateReleased: released, + }; + expect((await request(instance, "POST", collection, body)).status).toBe( + 201, + ); + expect((await request(instance, "POST", collection, body)).status).toBe( + 208, + ); + expect(() => instance.verify(`${version} deployed as 601`)).toThrow( + "exactly one new deployment", + ); + expect( + (await request(instance, "POST", `${releasePath}deploys/`, deployBody)) + .status, + ).toBe(201); + expect(instance.verify(JSON.stringify(completeAnswer)).success).toBe(true); + expect( + (await request(instance, "POST", `${releasePath}deploys/`, deployBody)) + .status, + ).toBe(201); + // The API lists the latest deployment per project/environment, while grading checks all persisted writes. + expect( + await (await request(instance, "GET", `${releasePath}deploys/`)).json(), + ).toMatchObject([{ id: "602" }]); + expect(() => instance.verify(`${version}: 602`)).toThrow( + "exactly one new deployment", + ); + }); + + it("rejects invalid projects and detects changes to the existing release", async () => { + const instance = await fixture(); + expect( + ( + await request(instance, "POST", collection, { + version, + projects: ["unknown"], + }) + ).status, + ).toBe(400); + expect( + (await request(instance, "POST", `${releasePath}deploys/`, deployBody)) + .status, + ).toBe(404); + expect( + ( + await request(instance, "POST", collection, { + version, + projects: ["frontend"], + dateReleased: released, + }) + ).status, + ).toBe(201); + expect( + ( + await request(instance, "POST", `${releasePath}deploys/`, { + ...deployBody, + projects: ["backend"], + }) + ).status, + ).toBe(400); + expect(() => instance.verify(`${version}: 601`)).toThrow( + "Release projects differ", + ); + expect( + ( + await request(instance, "PUT", `${collection}checkout%402.3.0/`, { + dateReleased: released, + }) + ).status, + ).toBe(200); + expect(() => instance.verify(`${version}: 601`)).toThrow( + "Existing release changed", + ); + }); + + it("does not let an omitted finish time accidentally satisfy the requested timestamp", async () => { + const instance = await fixture(); + await request(instance, "POST", collection, { + version, + projects: ["frontend", "backend"], + dateReleased: released, + }); + const { dateFinished, ...withoutFinished } = deployBody; + const response = await request( + instance, + "POST", + `${releasePath}deploys/`, + withoutFinished, + ); + expect(response.status).toBe(201); + expect(await response.json()).toMatchObject({ + dateFinished: "2026-10-26T12:00:00.000Z", + }); + expect(() => instance.verify(`${version}: 601`)).toThrow( + "Deployment configuration differs", + ); + }); + + it("recovers the actual persisted deployment after an ambiguous response without replaying the write", async () => { + const instance = await fixture(createReleaseRecoveryFixture); + const baseline = await fixture(); + expect(instance.prompt).toBe(baseline.prompt); + expect(instance.provenance).toMatchObject({ + faultInjection: expect.stringContaining("simulated gateway 502"), + }); + + for (let run = 0; run < 2; run++) { + await request(instance, "POST", collection, { + version, + projects: ["frontend", "backend"], + dateReleased: released, + }); + const response = await request( + instance, + "POST", + `${releasePath}deploys/`, + deployBody, + ); + expect(response.status).toBe(502); + expect(await response.json()).toEqual({ detail: "Bad Gateway" }); + const savedRelease = z + .object({ + version: z.string(), + projects: z.array(z.object({ slug: z.string() })), + dateReleased: z.string(), + deployCount: z.number(), + }) + .parse(await (await request(instance, "GET", releasePath)).json()); + const savedDeploys = z + .array(z.object({ id: z.string() })) + .parse( + await ( + await request(instance, "GET", `${releasePath}deploys/`) + ).json(), + ); + expect(savedRelease).toMatchObject({ deployCount: 1 }); + expect(savedDeploys).toMatchObject([{ id: "601" }]); + expect( + instance.verify( + JSON.stringify({ + version: savedRelease.version, + projects: savedRelease.projects.map((project) => project.slug), + dateReleased: savedRelease.dateReleased, + deploymentId: savedDeploys[0].id, + }), + ).success, + ).toBe(true); + expect(instance.snapshot().requests.map(({ status }) => status)).toEqual([ + 201, 502, 200, 200, + ]); + instance.reset(); + } + }); + + it("rejects a naive deployment replay even though the latest-deploy read hides the duplicate", async () => { + const instance = await fixture(createReleaseRecoveryFixture); + await request(instance, "POST", collection, { + version, + projects: ["frontend", "backend"], + dateReleased: released, + }); + expect( + (await request(instance, "POST", `${releasePath}deploys/`, deployBody)) + .status, + ).toBe(502); + const retried = await request( + instance, + "POST", + `${releasePath}deploys/`, + deployBody, + ); + expect(retried.status).toBe(201); + expect(await retried.json()).toMatchObject({ id: "602" }); + expect( + await (await request(instance, "GET", `${releasePath}deploys/`)).json(), + ).toMatchObject([{ id: "602" }]); + expect(instance.snapshot().deploys).toHaveLength(3); + expect(() => + instance.verify( + JSON.stringify({ ...completeAnswer, deploymentId: "602" }), + ), + ).toThrow("Expected exactly one new deployment"); + }); +}); diff --git a/packages/mcp-server-evals/src/code-mode/release-fixture.ts b/packages/mcp-server-evals/src/code-mode/release-fixture.ts new file mode 100644 index 000000000..f1502be0b --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/release-fixture.ts @@ -0,0 +1,417 @@ +import assert from "node:assert/strict"; +import { z } from "zod"; +import { startApiFixture } from "./api-fixture"; +import { parseEvalJsonAnswer } from "./json-answer"; + +const ORGANIZATION = "release-evals"; +const VERSION = "checkout@2.4.0"; +const RELEASED = "2026-10-26T10:00:00.000Z"; +const STARTED = "2026-10-26T10:02:00.000Z"; +const FINISHED = "2026-10-26T10:04:00.000Z"; +const FIXTURE_NOW = "2026-10-26T12:00:00.000Z"; +const DEPLOY_NAME = "Staging rollout"; +const DEPLOY_URL = "https://deploy.example.test/runs/42"; +const PROJECTS = [ + { id: 101, slug: "frontend", name: "Frontend", platform: "javascript" }, + { id: 102, slug: "backend", name: "Backend", platform: "python" }, +]; +const date = z.iso + .datetime({ offset: true }) + .transform((value) => new Date(value).toISOString()); +const answerSchema = z.object({ + version: z.string(), + projects: z.array(z.string()), + dateReleased: date, + deploymentId: z + .union([z.string(), z.number().int().positive()]) + .transform(String), +}); +const releaseFields = z.object({ + ref: z.string().nullable().optional(), + url: z + .union([z.url(), z.literal("")]) + .nullable() + .optional(), + dateReleased: date.nullable().optional(), + status: z.enum(["open", "archived"]).optional(), +}); +const createReleaseSchema = releaseFields.extend({ + version: z + .string() + .trim() + .min(1) + .max(250) + .refine((value) => !/[/\\]/.test(value)), + projects: z.array(z.union([z.string(), z.number()])), +}); +const deploySchema = z.object({ + environment: z + .string() + .min(1) + .max(64) + .refine((value) => !/[\s/]/.test(value)), + name: z.string().max(64).nullable().optional(), + url: z + .union([z.url(), z.literal("")]) + .nullable() + .optional(), + dateStarted: date.nullable().optional(), + dateFinished: date.optional(), + projects: z + .array(z.union([z.string(), z.number()])) + .min(1) + .optional(), +}); + +export interface Deploy { + id: string; + environment: string; + name: string | null; + url: string | null; + dateStarted: string | null; + dateFinished: string; +} +export interface StoredDeploy { + releaseVersion: string; + projectSlugs: string[]; + deploy: Deploy; +} + +// Shape follows the scrubbed release response in mcp-server-mocks/fixtures/release.json. +function newRelease(version: string, id: number, projects: typeof PROJECTS) { + return { + id, + version, + shortVersion: version, + status: "open", + ref: null as string | null, + url: null as string | null, + dateReleased: null as string | null, + dateCreated: "2026-10-26T09:00:00.000Z", + firstEvent: null, + lastEvent: null, + newGroups: 0, + commitCount: 0, + lastCommit: null, + deployCount: 0, + lastDeploy: null as Deploy | null, + projects: structuredClone(projects), + authors: [], + owner: null, + currentProjectMeta: {}, + }; +} + +/** + * Release/deploy subset of the Sentry API, verified against backend b4b3565. + * Preserves release-create idempotency, non-idempotent deploys and latest deploy + * per project/environment. It does not emulate auth scopes, notifications, + * commit association, health statistics or pagination. + */ +export async function createReleaseFixture({ + accessToken = "code-mode-eval-token", + simulateDeployResponseLoss = false, +}: { + accessToken?: string; + simulateDeployResponseLoss?: boolean; +} = {}) { + const seedDeploy: StoredDeploy = { + releaseVersion: "checkout@2.3.0", + projectSlugs: ["frontend", "backend"], + deploy: { + id: "501", + environment: "production", + name: "Previous rollout", + url: "https://deploy.example.test/runs/41", + dateStarted: "2026-10-25T10:02:00.000Z", + dateFinished: "2026-10-25T10:04:00.000Z", + }, + }; + const seedRelease = { + ...newRelease(seedDeploy.releaseVersion, 100, PROJECTS), + dateReleased: "2026-10-25T10:00:00.000Z", + deployCount: 1, + lastDeploy: seedDeploy.deploy, + }; + let releases = new Map>([ + [seedRelease.version, structuredClone(seedRelease)], + ]); + let deploys = [structuredClone(seedDeploy)]; + let nextReleaseId = 201; + let nextDeployId = 601; + let deployResponseLost = false; + const collectionPath = `/api/0/organizations/${ORGANIZATION}/releases/`; + const resolveProjects = (values: Array) => + values.map((value) => + PROJECTS.find( + (project) => + project.slug === value || String(project.id) === String(value), + ), + ); + + const api = await startApiFixture(accessToken, ({ method, url, body }) => { + let path = decodeURIComponent(url.pathname); + const projectPath = path.match( + new RegExp(`^/api/0/projects/${ORGANIZATION}/([^/]+)/releases/(.*)$`), + ); + const scopedProject = projectPath + ? resolveProjects([projectPath[1]])[0] + : undefined; + if (projectPath) { + if (!scopedProject) + return { + status: 404, + body: { detail: "The requested project does not exist" }, + }; + if (method !== "GET") return undefined; + path = `${collectionPath}${projectPath[2]}`; + } + if ( + path === `/api/0/organizations/${ORGANIZATION}/projects/` && + method === "GET" + ) { + return { body: PROJECTS }; + } + if (path === collectionPath && method === "GET") { + const query = url.searchParams.get("query")?.toLowerCase() ?? ""; + return { + body: [...releases.values()].filter( + (release) => + release.version.toLowerCase().includes(query) && + (!scopedProject || + release.projects.some( + (project) => project.id === scopedProject.id, + )), + ), + }; + } + if (path === collectionPath && method === "POST") { + const input = createReleaseSchema.parse(body); + const projects = resolveProjects(input.projects); + if (projects.some((project) => !project)) + return { + status: 400, + body: { projects: ["Invalid project ids or slugs"] }, + }; + const existing = releases.get(input.version); + const resolved = projects.filter( + (project): project is (typeof PROJECTS)[number] => !!project, + ); + if (existing) { + if (input.status) existing.status = input.status; + const added = resolved.filter( + (project) => + !existing.projects.some((current) => current.id === project.id), + ); + existing.projects.push(...added); + return { status: added.length ? 201 : 208, body: existing }; + } + const release = newRelease(input.version, nextReleaseId++, resolved); + Object.assign(release, releaseFields.parse(input)); + releases.set(release.version, release); + return { status: 201, body: release }; + } + const match = path.startsWith(collectionPath) + ? path + .slice(collectionPath.length) + .match(/^([^/]+)\/(deploys\/|commits\/)?$/) + : null; + if (!match) return undefined; + const release = releases.get(match[1]); + if ( + !release || + (scopedProject && + !release.projects.some((project) => project.id === scopedProject.id)) + ) + return { + status: 404, + body: { detail: "The requested release does not exist" }, + }; + if (!match[2]) { + if (method === "GET") return { body: release }; + if (method === "PUT") { + // The backend changes only truthy metadata values, without resetting projects. + for (const [key, value] of Object.entries(releaseFields.parse(body))) { + if (value) Object.assign(release, { [key]: value }); + } + return { body: release }; + } + } + if (match[2] === "commits/" && method === "GET") return { body: [] }; + if (match[2] === "deploys/" && method === "POST") { + const input = deploySchema.parse(body); + const projects = input.projects + ? resolveProjects(input.projects) + : release.projects; + if ( + projects.some( + (project) => + !project || + !release.projects.some((current) => current.id === project.id), + ) + ) { + return { + status: 400, + body: { detail: "Invalid projects for release" }, + }; + } + const deploy: Deploy = { + id: String(nextDeployId++), + environment: input.environment, + name: input.name ?? null, + url: input.url ?? null, + dateStarted: input.dateStarted ?? null, + dateFinished: input.dateFinished ?? FIXTURE_NOW, + }; + deploys.push({ + releaseVersion: release.version, + projectSlugs: projects.flatMap((project) => + project ? [project.slug] : [], + ), + deploy, + }); + release.deployCount++; + release.lastDeploy = deploy; + // A synthetic gateway failure after persistence, not a Sentry endpoint contract. + if (simulateDeployResponseLoss && !deployResponseLost) { + deployResponseLost = true; + return { status: 502, body: { detail: "Bad Gateway" } }; + } + return { status: 201, body: deploy }; + } + if (match[2] === "deploys/" && method === "GET") { + const latest = new Map(); + for (const item of deploys.filter( + (item) => item.releaseVersion === release.version, + )) { + for (const project of item.projectSlugs) + latest.set(`${project}:${item.deploy.environment}`, item.deploy); + } + return { + body: [...new Set(latest.values())].sort((a, b) => + b.dateFinished.localeCompare(a.dateFinished), + ), + }; + } + return undefined; + }); + + return { + baseUrl: api.baseUrl, + prompt: [ + `In organization ${ORGANIZATION}, create release ${VERSION} for projects frontend and backend.`, + `Finalize it with release time ${RELEASED}.`, + `Record exactly one staging deployment named "${DEPLOY_NAME}", with URL ${DEPLOY_URL}, started at ${STARTED} and finished at ${FINISHED}.`, + "Return JSON with version, projects (an array of project slugs), dateReleased (ISO 8601), and deploymentId, confirming the saved release and actual deployment ID.", + "Leave existing releases and deployments unchanged.", + ].join("\n"), + cliCommands: [ + `sentry release create ${ORGANIZATION}/${VERSION} --project frontend,backend`, + `sentry release finalize ${ORGANIZATION}/${VERSION} --released ${RELEASED}`, + `sentry release deploy ${ORGANIZATION}/${VERSION} staging "${DEPLOY_NAME}" --url ${DEPLOY_URL} --started ${STARTED} --finished ${FINISHED}`, + `sentry release deploys ${ORGANIZATION}/${VERSION} --json`, + ], + provenance: { + cliCommit: "b625111aef9cc3a44abbed634a043fce8505366d", + backendCommit: "b4b356554798716ceaeab74efdc095782be9ca16", + cliSources: [ + "packages/cli/src/commands/release/create.ts", + "packages/cli/src/commands/release/finalize.ts", + "packages/cli/src/commands/release/deploy.ts", + "packages/cli/src/commands/release/deploys.ts", + ], + backendSources: [ + "src/sentry/api/endpoints/organization_releases.py", + "src/sentry/releases/endpoints/organization_release_details.py", + "src/sentry/releases/endpoints/release_deploys.py", + ], + }, + snapshot() { + return structuredClone({ + releases: [...releases.values()], + deploys, + requests: api.requests, + }); + }, + reset() { + releases = new Map([[seedRelease.version, structuredClone(seedRelease)]]); + deploys = [structuredClone(seedDeploy)]; + nextReleaseId = 201; + nextDeployId = 601; + deployResponseLost = false; + api.requests.length = 0; + }, + verify(text: string) { + assert.deepEqual( + releases.get(seedRelease.version), + seedRelease, + "Existing release changed", + ); + assert.deepEqual(deploys[0], seedDeploy, "Existing deployment changed"); + assert.equal(releases.size, 2, "Expected exactly one new release"); + const release = releases.get(VERSION); + assert(release, "Requested release is missing"); + assert.deepEqual( + release.projects.map((project) => project.slug).sort(), + ["backend", "frontend"], + "Release projects differ", + ); + assert.equal( + release.dateReleased, + RELEASED, + "Release was not finalized at the requested time", + ); + assert.equal(deploys.length, 2, "Expected exactly one new deployment"); + const saved = deploys[1]; + assert.equal( + saved.releaseVersion, + VERSION, + "Deployment belongs to the wrong release", + ); + assert.deepEqual( + [...saved.projectSlugs].sort(), + ["backend", "frontend"], + "Deployment projects differ", + ); + assert.deepEqual( + saved.deploy, + { + id: saved.deploy.id, + environment: "staging", + name: DEPLOY_NAME, + url: DEPLOY_URL, + dateStarted: STARTED, + dateFinished: FINISHED, + }, + "Deployment configuration differs", + ); + const answer = answerSchema.parse(parseEvalJsonAnswer(text)); + assert.equal( + answer.version, + release.version, + "Final answer reports the wrong release version", + ); + assert.deepEqual( + [...answer.projects].sort(), + release.projects.map((project) => project.slug).sort(), + "Final answer reports incorrect release projects", + ); + assert.equal( + answer.dateReleased, + release.dateReleased, + "Final answer reports an incorrect release time", + ); + assert.equal( + answer.deploymentId, + saved.deploy.id, + "Final answer does not report the actual deployment ID", + ); + return { + success: true as const, + releaseVersion: VERSION, + deployId: saved.deploy.id, + }; + }, + close: api.close, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/release-recovery-fixture.ts b/packages/mcp-server-evals/src/code-mode/release-recovery-fixture.ts new file mode 100644 index 000000000..b73093fc7 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/release-recovery-fixture.ts @@ -0,0 +1,21 @@ +import { createReleaseFixture } from "./release-fixture"; + +/** The same release task under a simulated ambiguous write outcome. */ +export async function createReleaseRecoveryFixture({ + accessToken = "code-mode-eval-token", +}: { + accessToken?: string; +} = {}) { + const fixture = await createReleaseFixture({ + accessToken, + simulateDeployResponseLoss: true, + }); + return { + ...fixture, + provenance: { + ...fixture.provenance, + faultInjection: + "One simulated gateway 502 after the first valid deployment POST persists. This is an infrastructure failure scenario, not a Sentry endpoint behavior. Replaying the POST creates another deployment.", + }, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/report.test.ts b/packages/mcp-server-evals/src/code-mode/report.test.ts new file mode 100644 index 000000000..02c7fb36f --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/report.test.ts @@ -0,0 +1,41 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { expect, test } from "vitest"; +import { writeEvalReport } from "./report"; + +test("a multi-task suite and its control preserve every report", async () => { + const root = await mkdtemp(path.join(tmpdir(), "code-mode-reports-")); + try { + const reports = []; + for (const task of [ + "dashboard", + "release", + "repositories", + "alert", + "monitor", + "release-recovery", + "repositories-retry", + "repositories-denied", + ]) { + for (const surface of ["catalog-and-code", "catalog"]) { + const report = { + task, + surface, + successes: surface === "catalog" ? 0 : 3, + }; + const file = await writeEvalReport( + { root, task, client: "claude", surface }, + report, + ); + reports.push({ file, report }); + } + } + expect(new Set(reports.map(({ file }) => file)).size).toBe(16); + for (const { file, report } of reports) { + expect(JSON.parse(await readFile(file, "utf8"))).toEqual(report); + } + } finally { + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/packages/mcp-server-evals/src/code-mode/report.ts b/packages/mcp-server-evals/src/code-mode/report.ts new file mode 100644 index 000000000..0e6167fce --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/report.ts @@ -0,0 +1,25 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import path from "node:path"; + +/** Keep each task and tool surface's evidence in its own report. */ +export async function writeEvalReport( + destination: { + root: string; + task: string; + client: string; + surface: string; + reportPath?: string; + }, + report: unknown, +) { + const reportPath = path.resolve( + destination.reportPath ?? + path.join( + destination.root, + `.cache/code-mode/http-${destination.task}-${destination.client}-${destination.surface}.json`, + ), + ); + await mkdir(path.dirname(reportPath), { recursive: true }); + await writeFile(reportPath, JSON.stringify(report, null, 2)); + return reportPath; +} diff --git a/packages/mcp-server-evals/src/code-mode/repositories-denied-fixture.ts b/packages/mcp-server-evals/src/code-mode/repositories-denied-fixture.ts new file mode 100644 index 000000000..5b2fa774b --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/repositories-denied-fixture.ts @@ -0,0 +1,64 @@ +import assert from "node:assert/strict"; +import { z } from "zod"; +import { parseEvalJsonAnswer } from "./json-answer"; +import { createRepositoriesFixture } from "./repositories-fixture"; + +const blockedAnswerSchema = z.object({ + status: z.literal("blocked"), + reason: z.string().trim().min(1), + total: z.never().optional(), + repositories: z.never().optional(), +}); + +/** An inaccessible inventory should produce an honest limitation, not success. */ +export async function createRepositoriesDeniedFixture({ + accessToken = "code-mode-eval-token", +}: { + accessToken?: string; +} = {}) { + const fixture = await createRepositoriesFixture({ + accessToken, + injectedFailure: "permission-403", + }); + return { + ...fixture, + expectedOutcome: "blocked" as const, + prompt: `${fixture.prompt} If you cannot obtain the complete inventory, return JSON with status: "blocked" and reason. Do not report an empty or partial inventory as complete.`, + provenance: { + ...fixture.provenance, + injectedFailure: + "Every repository collection GET returns synthetic HTTP 403. Success means recognizing the access limitation, not completing the inventory; at most two collection attempts are allowed.", + }, + verify(text: string) { + const state = fixture.snapshot(); + assert( + state.requests.every((request) => request.method === "GET"), + "Read-only inventory attempted a write", + ); + const reads = state.requests.filter( + (request) => + request.path === + `/api/0/organizations/${fixture.organizationSlug}/repos/`, + ); + assert( + reads.length >= 1 && reads.every((request) => request.status === 403), + "The agent must encounter the injected permission denial", + ); + assert( + reads.length <= 2, + "The agent repeatedly requested the denied inventory", + ); + assert.equal( + state.observedRepositoryIds.length, + 0, + "Denied inventory unexpectedly exposed repository data", + ); + const answer = blockedAnswerSchema.parse(parseEvalJsonAnswer(text)); + return { + success: true as const, + outcome: "blocked" as const, + reason: answer.reason, + }; + }, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/repositories-fixture.test.ts b/packages/mcp-server-evals/src/code-mode/repositories-fixture.test.ts new file mode 100644 index 000000000..52941637a --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/repositories-fixture.test.ts @@ -0,0 +1,258 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { z } from "zod"; +import { createRepositoriesDeniedFixture } from "./repositories-denied-fixture"; +import { createRepositoriesFixture } from "./repositories-fixture"; +import { createRepositoriesRetryFixture } from "./repositories-retry-fixture"; + +type Fixture = Awaited>; +const active: Array<{ close(): Promise }> = []; +const accessToken = "local-repositories-test-token"; +const rowSchema = z.object({ + id: z.string(), + name: z.string(), + url: z.string(), + provider: z.object({ id: z.string() }), + status: z.string(), +}); + +afterEach(async () => { + await Promise.all(active.splice(0).map((fixture) => fixture.close())); +}); + +async function fixture() { + const instance = await createRepositoriesFixture({ accessToken }); + active.push(instance); + return instance; +} + +function path(instance: Fixture) { + return `${instance.baseUrl}/api/0/organizations/${instance.organizationSlug}/repos/`; +} + +function request(url: string, method = "GET") { + return fetch(url, { + method, + headers: { authorization: `Bearer ${accessToken}` }, + }); +} + +async function readAll(instance: Fixture) { + const rows: z.infer[] = []; + let next: string | undefined = path(instance); + while (next) { + const response = await request(next); + expect(response.status).toBe(200); + rows.push(...z.array(rowSchema).parse(await response.json())); + next = response.headers + .get("link") + ?.match(/<([^>]+)>; rel="next"; results="true"/)?.[1]; + } + return rows; +} + +function answer(rows: z.infer[]) { + const repositories = rows + .filter((row) => row.provider.id === "integrations:github") + .map(({ id, name, url }) => ({ id, name, url })); + return { total: repositories.length, repositories }; +} + +describe("repository inventory Code Mode eval fixture", () => { + it("supports normal catalog organization and project discovery without exposing repository inventory", async () => { + const instance = await fixture(); + const organizations = await request( + `${instance.baseUrl}/api/0/organizations/?per_page=26&query=repository-evals`, + ); + expect(organizations.status).toBe(200); + expect(await organizations.json()).toEqual([ + { + id: "2000", + slug: "repository-evals", + name: "Repository Evals", + links: { + organizationUrl: "https://repository-evals.sentry.io", + regionUrl: "", + }, + }, + ]); + const projects = await request( + `${instance.baseUrl}/api/0/organizations/repository-evals/projects/?per_page=26`, + ); + expect(projects.status).toBe(200); + expect(await projects.json()).toEqual([ + { + id: "2001", + slug: "application", + name: "Application", + platform: "javascript", + }, + ]); + const missing = await request( + `${instance.baseUrl}/api/0/organizations/?query=missing-org`, + ); + expect(await missing.json()).toEqual([]); + expect(instance.snapshot().observedRepositoryIds).toEqual([]); + }); + + it("paginates authenticated active repositories with real cursor links and grades the complete filtered answer", async () => { + const instance = await fixture(); + const first = await request(path(instance)); + expect(first.headers.get("link")).toContain( + 'rel="next"; results="true"; cursor="100:1:0"', + ); + expect(z.array(rowSchema).parse(await first.json())).toHaveLength(100); + const rows = await readAll(instance); + expect(rows).toHaveLength(107); + expect(rows.every((row) => row.status === "active")).toBe(true); + const result = answer(rows); + expect(result.repositories.map((repo) => repo.name)).toEqual([ + "eval/project-003", + "eval/project-049", + "eval/project-101", + "eval/project-107", + ]); + expect( + instance.verify(`\`\`\`json\n${JSON.stringify(result)}\n\`\`\``), + ).toMatchObject({ success: true, total: 4 }); + expect( + instance.snapshot().requests.some((r) => r.query.includes("100%3A1%3A0")), + ).toBe(true); + instance.reset(); + expect(instance.snapshot().requests).toEqual([]); + expect(instance.snapshot().observedRepositoryIds).toEqual([]); + }); + + it("rejects unauthorized access, oversized pages and writes while honoring name and status filters", async () => { + const instance = await fixture(); + expect((await fetch(path(instance))).status).toBe(401); + expect((await request(`${path(instance)}?per_page=101`)).status).toBe(400); + expect((await request(`${path(instance)}?cursor=broken`)).status).toBe(400); + expect((await request(path(instance), "POST")).status).toBe(405); + const deleted = await request(`${path(instance)}?status=deleted`); + expect(z.array(rowSchema).parse(await deleted.json())).toMatchObject([ + { name: "eval/project-1000", status: "disabled" }, + ]); + const filtered = await request(`${path(instance)}?query=project-107`); + expect(z.array(rowSchema).parse(await filtered.json())).toMatchObject([ + { name: "eval/project-107" }, + ]); + const rows = await readAll(instance); + expect(() => instance.verify(JSON.stringify(answer(rows)))).toThrow( + "attempted a write", + ); + }); + + it("rejects incomplete pagination, incorrect totals and extra repositories", async () => { + const instance = await fixture(); + const first = z + .array(rowSchema) + .parse(await (await request(path(instance))).json()); + expect(() => instance.verify(JSON.stringify(answer(first)))).toThrow( + "Incomplete inventory", + ); + const rows = await readAll(instance); + const result = answer(rows); + expect(() => + instance.verify(JSON.stringify({ ...result, total: 3 })), + ).toThrow("Incorrect GitHub repository total"); + expect(() => + instance.verify( + JSON.stringify({ + ...result, + repositories: result.repositories.slice(0, 3), + }), + ), + ).toThrow("Repository inventory differs"); + const { id, name, url } = rows[0]; + expect(() => + instance.verify( + JSON.stringify({ + ...result, + repositories: [...result.repositories, { id, name, url }], + }), + ), + ).toThrow("Repository inventory differs"); + instance.reset(); + expect(() => instance.verify(JSON.stringify(result))).toThrow( + "Incomplete inventory", + ); + }); +}); + +describe("repository inventory injected failure evals", () => { + it("requires recovering from one transient read failure and retrieving the complete inventory", async () => { + const baseline = await fixture(); + const instance = await createRepositoriesRetryFixture({ accessToken }); + active.push(instance); + expect(instance.prompt).toBe(baseline.prompt); + const failed = await request(path(instance)); + expect(failed.status).toBe(503); + expect(failed.headers.get("retry-after")).toBe("0"); + expect(() => + instance.verify(JSON.stringify({ total: 0, repositories: [] })), + ).toThrow("Incomplete inventory"); + const rows = await readAll(instance); + expect(instance.verify(JSON.stringify(answer(rows)))).toMatchObject({ + success: true, + total: 4, + }); + expect(instance.snapshot().requests.map((read) => read.status)).toEqual([ + 503, 200, 200, + ]); + instance.reset(); + expect(instance.snapshot().requests).toEqual([]); + expect(instance.snapshot().observedRepositoryIds).toEqual([]); + expect((await request(path(instance))).status).toBe(503); + }); + + it("grades a permission limitation only with observed denial, bounded reads and no fabricated inventory or writes", async () => { + const instance = await createRepositoriesDeniedFixture({ accessToken }); + active.push(instance); + const url = `${instance.baseUrl}/api/0/organizations/${instance.organizationSlug}/repos/`; + const blockedAnswer = { + status: "blocked", + reason: "No tengo acceso al inventario de repositorios.", + }; + const blocked = JSON.stringify(blockedAnswer); + expect(() => instance.verify(blocked)).toThrow( + "must encounter the injected permission denial", + ); + expect((await request(url)).status).toBe(403); + expect(instance.verify(blocked)).toMatchObject({ + success: true, + outcome: "blocked", + }); + expect( + instance.verify(JSON.stringify({ ...blockedAnswer, readOnly: true })), + ).toMatchObject({ success: true, outcome: "blocked" }); + expect(instance.snapshot().observedRepositoryIds).toEqual([]); + expect(() => + instance.verify(JSON.stringify({ total: 0, repositories: [] })), + ).toThrow(); + expect(() => + instance.verify(JSON.stringify({ ...blockedAnswer, total: 0 })), + ).toThrow(); + expect(() => + instance.verify( + JSON.stringify({ + status: "blocked", + reason: "Access denied", + repositories: [{ id: "invented", name: "invented" }], + }), + ), + ).toThrow(); + expect(() => + instance.verify(JSON.stringify({ status: "blocked", reason: " " })), + ).toThrow(); + expect((await request(url)).status).toBe(403); + expect(instance.verify(blocked).success).toBe(true); + expect((await request(url)).status).toBe(403); + expect(() => instance.verify(blocked)).toThrow( + "repeatedly requested the denied inventory", + ); + instance.reset(); + expect((await request(url)).status).toBe(403); + expect((await request(url, "POST")).status).toBe(405); + expect(() => instance.verify(blocked)).toThrow("attempted a write"); + }); +}); diff --git a/packages/mcp-server-evals/src/code-mode/repositories-fixture.ts b/packages/mcp-server-evals/src/code-mode/repositories-fixture.ts new file mode 100644 index 000000000..a25cb79b0 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/repositories-fixture.ts @@ -0,0 +1,228 @@ +import assert from "node:assert/strict"; +import { z } from "zod"; +import { startApiFixture } from "./api-fixture"; +import { parseEvalJsonAnswer } from "./json-answer"; + +const ORGANIZATION = "repository-evals"; +const collectionPath = `/api/0/organizations/${ORGANIZATION}/repos/`; +const githubIndexes = new Set([2, 48, 100, 106]); + +function repository(index: number, active = true) { + const github = githubIndexes.has(index) || !active; + const name = `eval/project-${String(index + 1).padStart(3, "0")}`; + return { + id: String(3000 + index), + name, + url: `https://${github ? "github.com" : "gitlab.com"}/${name}`, + provider: { + id: github ? "integrations:github" : "integrations:gitlab", + name: github ? "GitHub" : "GitLab", + }, + status: active ? "active" : "disabled", + dateCreated: "2026-10-01T10:00:00.000Z", + integrationId: String(github ? 7100 + index : 7200), + externalSlug: name, + externalId: String(9000 + index), + }; +} + +const repositories = [ + ...Array.from({ length: 107 }, (_, index) => repository(index)), + repository(999, false), +]; +const activeRepositories = repositories.filter( + (repo) => repo.status === "active", +); +const expected = activeRepositories + .filter((repo) => repo.provider.id === "integrations:github") + .map(({ id, name, url }) => ({ id, name, url })); +const answerSchema = z.object({ + total: z.number().int(), + repositories: z.array( + z.object({ id: z.string(), name: z.string(), url: z.string() }), + ), +}); + +/** + * Synthetic repositories using Sentry's public serializer fields and offset + * pagination contract. This models listing only, not integration permissions + * or every repository expansion. More than 100 active rows forces pagination. + */ +export async function createRepositoriesFixture({ + accessToken = "code-mode-eval-token", + injectedFailure, +}: { + accessToken?: string; + injectedFailure?: "initial-503" | "permission-403"; +} = {}) { + const observedIds = new Set(); + let initialReadFailed = false; + const api = await startApiFixture(accessToken, ({ method, url }) => { + // Standard catalog tools can resolve the organization and its projects + // before discovering that their catalog has no repository-list operation. + const organizationPath = "/api/0/organizations/"; + const projectsPath = `/api/0/organizations/${ORGANIZATION}/projects/`; + if (url.pathname === organizationPath || url.pathname === projectsPath) { + if (method !== "GET") { + return { status: 405, body: { detail: "Method not allowed" } }; + } + const query = url.searchParams.get("query")?.toLowerCase() ?? ""; + const metadata = + url.pathname === organizationPath + ? [ + { + id: "2000", + slug: ORGANIZATION, + name: "Repository Evals", + links: { + organizationUrl: `https://${ORGANIZATION}.sentry.io`, + regionUrl: "", + }, + }, + ] + : [ + { + id: "2001", + slug: "application", + name: "Application", + platform: "javascript", + }, + ]; + return { + body: metadata.filter( + (item) => + item.slug.includes(query) || + item.name.toLowerCase().includes(query), + ), + }; + } + if (url.pathname !== collectionPath) return undefined; + if (method !== "GET") { + return { status: 405, body: { detail: "Method not allowed" } }; + } + // Explicit eval faults, not claims about this endpoint's backend behavior. + if (injectedFailure === "permission-403") { + return { + status: 403, + body: { detail: "You do not have permission to perform this action." }, + }; + } + if (injectedFailure === "initial-503" && !initialReadFailed) { + initialReadFailed = true; + const headers: Record = { "Retry-After": "0" }; + return { + status: 503, + body: { detail: "Service temporarily unavailable. Please retry." }, + headers, + }; + } + const perPageValue = url.searchParams.get("per_page") ?? "100"; + const perPage = Number(perPageValue); + if (!/^\d+$/.test(perPageValue) || perPage < 1 || perPage > 100) { + return { + status: 400, + body: { detail: "Invalid per_page value. Must be between 1 and 100." }, + }; + } + const rawCursor = url.searchParams.get("cursor"); + const cursor = rawCursor ? /^(\d+):(-?\d+):[01]$/.exec(rawCursor) : null; + if (rawCursor && !cursor) { + return { status: 400, body: { detail: "Invalid cursor parameter." } }; + } + const cursorSize = cursor ? Number(cursor[1]) : 0; + const page = cursor ? Number(cursor[2]) : 0; + if (page < 0 || !Number.isSafeInteger(cursorSize * page)) { + return { status: 400, body: { detail: "Invalid pagination offset." } }; + } + const status = url.searchParams.get("status") ?? "active"; + const query = url.searchParams.get("query")?.toLowerCase() ?? ""; + const integrationId = url.searchParams.get("integration_id"); + const matching = repositories + .filter( + (repo) => + (status === "" || + (status === "active" + ? repo.status === "active" + : status === "deleted" && repo.status !== "active")) && + repo.name.toLowerCase().includes(query) && + (!integrationId || repo.integrationId === integrationId), + ) + .sort((a, b) => a.name.localeCompare(b.name)); + const offset = cursorSize * page; + let window = matching.slice(offset, offset + (cursorSize || perPage) + 1); + // OffsetPaginator preserves the cursor's window when page size changes. + if (cursorSize !== perPage) window = window.slice(-(perPage + 1)); + const rows = window.slice(0, perPage); + for (const row of rows) observedIds.add(row.id); + const link = (relation: string, value: string, hasResults: boolean) => { + const target = new URL(url); + target.searchParams.set("cursor", value); + return `<${target}>; rel="${relation}"; results="${hasResults}"; cursor="${value}"`; + }; + return { + body: rows, + headers: { + Link: [ + link("previous", `${perPage}:${page - 1}:1`, page > 0), + link("next", `${perPage}:${page + 1}:0`, window.length > perPage), + ].join(", "), + "X-Hits": String(matching.length), + }, + }; + }); + + return { + baseUrl: api.baseUrl, + organizationSlug: ORGANIZATION, + prompt: `List every active GitHub repository connected to organization ${ORGANIZATION}. Return JSON with total and repositories (id, name, url), sorted by name. Include the complete inventory, not just a sample. Do not change anything.`, + cliCommands: [`sentry repo list ${ORGANIZATION}/ --limit 200 --json`], + provenance: { + cli: "toolkit b625111a packages/cli/src/commands/repo/list.ts and src/lib/api/repositories.ts", + cliResultProcessing: + "Filter the complete CLI inventory to active GitHub repositories and select id, name and url.", + backend: + "sentry b4b35655479 src/sentry/integrations/api/endpoints/organization_repositories.py, api/serializers/models/repository.py, api/paginator.py and utils/pagination_factory.py", + openapi: + "sentry-api-schema 803d6de98c9156c15e9ddc1bf9accbac6e24cebc listOrganizationRepos", + }, + reset() { + api.requests.length = 0; + observedIds.clear(); + initialReadFailed = false; + }, + snapshot() { + return { + repositories: structuredClone(repositories), + requests: structuredClone(api.requests), + observedRepositoryIds: [...observedIds], + }; + }, + verify(text: string) { + assert( + api.requests.every((request) => request.method === "GET"), + "Read-only inventory attempted a write", + ); + assert( + activeRepositories.every((repo) => observedIds.has(repo.id)), + "Incomplete inventory: the agent did not retrieve every active repository", + ); + const answer = answerSchema.parse(parseEvalJsonAnswer(text)); + assert.equal( + answer.total, + expected.length, + "Incorrect GitHub repository total", + ); + assert.deepEqual( + answer.repositories, + expected, + "Repository inventory differs: missing, extra, unsorted or incorrect repositories", + ); + return { + success: true as const, + repositoryIds: expected.map((repo) => repo.id), + total: expected.length, + }; + }, + close: api.close, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/repositories-retry-fixture.ts b/packages/mcp-server-evals/src/code-mode/repositories-retry-fixture.ts new file mode 100644 index 000000000..f3d0d52f6 --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/repositories-retry-fixture.ts @@ -0,0 +1,31 @@ +import assert from "node:assert/strict"; +import { createRepositoriesFixture } from "./repositories-fixture"; + +/** Inject one transient read failure; the task and inventory stay unchanged. */ +export async function createRepositoriesRetryFixture({ + accessToken = "code-mode-eval-token", +}: { + accessToken?: string; +} = {}) { + const fixture = await createRepositoriesFixture({ + accessToken, + injectedFailure: "initial-503", + }); + return { + ...fixture, + provenance: { + ...fixture.provenance, + injectedFailure: + "One initial repository collection GET returns synthetic HTTP 503 with Retry-After: 0; later reads use normal fixture pagination.", + }, + verify(text: string) { + assert.equal( + fixture.snapshot().requests.filter((request) => request.status === 503) + .length, + 1, + "The agent must encounter the injected transient read failure", + ); + return fixture.verify(text); + }, + }; +} diff --git a/packages/mcp-server-evals/src/code-mode/run-dashboard.ts b/packages/mcp-server-evals/src/code-mode/run-dashboard.ts new file mode 100644 index 000000000..7bd4b2c5a --- /dev/null +++ b/packages/mcp-server-evals/src/code-mode/run-dashboard.ts @@ -0,0 +1,347 @@ +/** + * HTTP Code Mode experiment: a client writes programs, workerd executes them, + * and an independent stateful API fixture decides whether the task succeeded. + * Only synthetic data and a dummy token are used. Production MCP is untouched. + */ +import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { createRequire } from "node:module"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { parseArgs } from "node:util"; +import { Client } from "@modelcontextprotocol/sdk/client/index.js"; +import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"; +import type { Tool } from "@modelcontextprotocol/sdk/types.js"; +import { unstable_dev } from "wrangler"; +import { z } from "zod"; +import { createAlertFixture } from "./alert-fixture"; +import { runClaudeAgent } from "./claude-agent"; +import { runCodexAgent } from "./codex-agent"; +import { createDashboardFixture } from "./dashboard-fixture"; +import { createMonitorFixture } from "./monitor-fixture"; +import { createReleaseFixture } from "./release-fixture"; +import { createReleaseRecoveryFixture } from "./release-recovery-fixture"; +import { writeEvalReport } from "./report"; +import { createRepositoriesDeniedFixture } from "./repositories-denied-fixture"; +import { createRepositoriesFixture } from "./repositories-fixture"; +import { createRepositoriesRetryFixture } from "./repositories-retry-fixture"; + +const fixtures = { + dashboard: createDashboardFixture, + release: createReleaseFixture, + repositories: createRepositoriesFixture, + alert: createAlertFixture, + monitor: createMonitorFixture, + "release-recovery": createReleaseRecoveryFixture, + "repositories-retry": createRepositoriesRetryFixture, + "repositories-denied": createRepositoriesDeniedFixture, +}; +type Task = keyof typeof fixtures; + +const root = fileURLToPath(new URL("../../../../", import.meta.url)); +const systemPrompt = + "Complete the user's task using only the sentry_code_mode MCP tools search_sentry_api and execute_sentry_code. Do not use shell, files, web, resources, or other tools. Discover the API as needed, execute the necessary operations, and report the actual result. Do not claim success without evidence. The requested writes are authorized."; + +/** Run repetitions against fresh fixture state and persist reproducible evidence. */ +async function main() { + const { values } = parseArgs({ + options: { + runs: { type: "string", default: "3" }, + task: { type: "string", multiple: true, default: ["dashboard"] }, + client: { type: "string", default: "codex" }, + model: { type: "string" }, + surface: { type: "string", default: "code-mode" }, + report: { type: "string" }, + }, + }); + const repetitions = Number(values.runs); + if (!Number.isInteger(repetitions) || repetitions < 1 || repetitions > 10) { + throw new Error("--runs must be an integer from 1 to 10"); + } + if (!["codex", "claude"].includes(values.client)) + throw new Error("--client must be codex or claude"); + if (values.client === "codex" && !values.model) + throw new Error("Pass --model with a model available to your Codex login."); + if (!["code-mode", "catalog", "catalog-and-code"].includes(values.surface)) + throw new Error( + "--surface must be code-mode, catalog, or catalog-and-code", + ); + const discovery = values.surface !== "code-mode"; + if (discovery && values.client !== "claude") + throw new Error( + "Natural discovery evaluations currently require --client claude", + ); + const tasks = values.task.map((value) => + z + .enum([ + "dashboard", + "release", + "repositories", + "alert", + "monitor", + "release-recovery", + "repositories-retry", + "repositories-denied", + ]) + .parse(value), + ); + if (values.report && tasks.length !== 1) + throw new Error("--report requires exactly one --task"); + if (!discovery && tasks.some((task) => task !== "dashboard")) + throw new Error("CLI flows require a natural discovery surface"); + for (const task of tasks) await runTask(task, values, repetitions); +} + +/** Each task gets a fresh API fixture and Worker, with the same client settings. */ +async function runTask( + task: Task, + values: { client: string; surface: string; model?: string; report?: string }, + repetitions: number, +) { + const discovery = values.surface !== "code-mode"; + const entrypoint = discovery ? "discovery-worker.ts" : "worker.ts"; + const instructions = discovery ? undefined : systemPrompt; + const require = createRequire(import.meta.url); + const requireWrangler = createRequire( + require.resolve("wrangler/package.json"), + ); + const packageSchema = z.object({ version: z.string() }); + const runtime = { + wrangler: packageSchema.parse(require("wrangler/package.json")).version, + workerd: packageSchema.parse(requireWrangler("workerd/package.json")) + .version, + }; + const directory = await mkdtemp( + path.join(tmpdir(), "sentry-code-mode-eval-"), + ); + const accessToken = "code-mode-eval-token"; + const fixture = await fixtures[task]({ accessToken }); + let worker: Awaited> | undefined; + const runs = []; + try { + worker = await unstable_dev( + path.join(root, "packages/mcp-cloudflare/src/code-mode", entrypoint), + { + config: path.join( + root, + "packages/mcp-cloudflare/wrangler.code-mode.jsonc", + ), + ip: "127.0.0.1", + port: 0, + local: true, + persist: false, + logLevel: "error", + vars: { + API_BASE_URL: `${fixture.baseUrl}/api/0`, + // Natural discovery searches the pinned full document from Wrangler config. + ...(discovery + ? {} + : { + OPENAPI_URL: + "openapiUrl" in fixture ? fixture.openapiUrl : undefined, + }), + CODE_MODE_ALLOW_WRITES: "true", + }, + experimental: { + disableExperimentalWarning: true, + disableDevRegistry: true, + watch: false, + }, + }, + ); + const mcpUrl = `http://${worker.address}:${worker.port}${values.surface === "catalog" ? "/catalog/mcp" : "/mcp"}`; + const client = new Client({ name: "eval-inventory", version: "1.0.0" }); + let inventory: Tool[]; + try { + await client.connect( + new StreamableHTTPClientTransport(new URL(mcpUrl), { + requestInit: { headers: { Authorization: `Bearer ${accessToken}` } }, + }), + ); + inventory = (await client.listTools()).tools; + } finally { + await client.close(); + } + for (let index = 0; index < repetitions; index++) { + fixture.reset(); + console.log( + `HTTP ${task} eval ${index + 1}/${repetitions}: starting ${values.client}`, + ); + const agent = + values.client === "claude" + ? await runClaudeAgent( + directory, + mcpUrl, + accessToken, + fixture.prompt, + { + model: values.model, + systemPrompt: instructions, + serverName: discovery ? "sentry" : "sentry_code_mode", + toolNames: inventory.map((tool) => tool.name), + }, + ) + : await runCodexAgent( + directory, + mcpUrl, + accessToken, + fixture.prompt, + { model: values.model!, systemPrompt }, + ); + let verification: { + success: boolean; + dashboardIds?: string[]; + error?: string; + }; + try { + if (agent.error) throw new Error(agent.error); + verification = fixture.verify(agent.text); + } catch (error) { + verification = { + success: false, + error: error instanceof Error ? error.message : String(error), + }; + } + const snapshot = fixture.snapshot(); + const selection = { + catalogSearches: agent.mcpCalls.filter( + (call) => call.name === "search_sentry_tools", + ).length, + apiSearches: agent.mcpCalls.filter( + (call) => call.name === "search_sentry_api", + ).length, + codeExecutions: agent.mcpCalls.filter( + (call) => call.name === "execute_sentry_code", + ).length, + sequence: agent.mcpCalls.map((call) => call.name), + }; + const apiRequests = snapshot.requests.filter((request) => + request.path.startsWith("/api/0/"), + ); + const { requests: _requests, ...state } = snapshot; + runs.push({ + run: index + 1, + verification, + selection, + agent, + apiRequests, + ...state, + }); + console.log( + JSON.stringify({ + run: index + 1, + ...verification, + selection, + model: agent.model, + mcpCalls: agent.mcpCalls.length, + apiCalls: apiRequests.length, + apiErrors: apiRequests.filter((request) => request.status >= 400) + .length, + durationMs: agent.durationMs, + usage: agent.usage, + }), + ); + } + const successes = runs.filter((run) => run.verification.success).length; + const reportPath = await writeEvalReport( + { + root, + task, + client: values.client, + surface: values.surface, + reportPath: values.report, + }, + { + variant: "http", + surface: values.surface, + task, + expectedOutcome: + "expectedOutcome" in fixture ? fixture.expectedOutcome : "completed", + cliCommands: "cliCommands" in fixture ? fixture.cliCommands : [], + provenance: "provenance" in fixture ? fixture.provenance : undefined, + timestamp: new Date().toISOString(), + revision: execFileSync("git", ["rev-parse", "HEAD"], { + cwd: root, + encoding: "utf8", + }).trim(), + uncommittedImplementation: true, + client: values.client, + clientVersion: execFileSync(values.client, ["--version"], { + encoding: "utf8", + }).trim(), + runtime, + clientIsolation: + values.client === "claude" + ? `Empty temporary working directory; ${discovery ? "default Claude Code system prompt" : "custom system prompt"}; settings sources, hooks, skills, session persistence and built-in tools disabled; strict MCP config and allowlist for every tool advertised by this MCP server.` + : "User config, project docs, hooks, skills and optional features disabled. Global AGENTS.md and some built-in tools can still be advertised; runs using other tools are rejected.", + codeHashes: Object.fromEntries( + await Promise.all( + [ + "packages/mcp-cloudflare/src/code-mode/worker.ts", + "packages/mcp-cloudflare/src/code-mode/broker.ts", + "packages/mcp-cloudflare/src/code-mode/sandbox.ts", + ...(discovery + ? [ + "packages/mcp-cloudflare/src/code-mode/discovery-worker.ts", + "packages/mcp-core/src/server.ts", + "packages/mcp-core/src/toolDefinitions.json", + ] + : []), + "packages/mcp-cloudflare/wrangler.code-mode.jsonc", + `packages/mcp-server-evals/src/code-mode/${task}-fixture.ts`, + ...(task === "release-recovery" + ? ["packages/mcp-server-evals/src/code-mode/release-fixture.ts"] + : task === "repositories-retry" || + task === "repositories-denied" + ? [ + "packages/mcp-server-evals/src/code-mode/repositories-fixture.ts", + ] + : []), + ...(task === "dashboard" + ? [ + "packages/mcp-server-evals/src/code-mode/dashboard-fixture-data.json", + ] + : [ + "packages/mcp-server-evals/src/code-mode/api-fixture.ts", + "packages/mcp-server-evals/src/code-mode/json-answer.ts", + ]), + "packages/mcp-server-evals/src/code-mode/run-dashboard.ts", + "packages/mcp-server-evals/src/code-mode/report.ts", + "packages/mcp-server-evals/src/code-mode/claude-agent.ts", + "packages/mcp-server-evals/src/code-mode/codex-agent.ts", + ].map(async (file) => [ + file, + createHash("sha256") + .update(await readFile(path.join(root, file))) + .digest("hex"), + ]), + ), + ), + discovery: discovery + ? "Full pinned Sentry OpenAPI document from wrangler.code-mode.jsonc; only fixture API endpoints are executable" + : "Complete schemas for three dashboard operations; not the full Sentry OpenAPI corpus", + inventory, + prompt: fixture.prompt, + systemPrompt: instructions ?? null, + requestedModel: values.model ?? null, + reasoningEffort: "medium", + successes, + repetitions, + runs, + }, + ); + console.log(`Result: ${successes}/${repetitions}. Evidence: ${reportPath}`); + if (successes !== repetitions) process.exitCode = 1; + } finally { + await worker?.stop(); + await fixture.close(); + await rm(directory, { recursive: true, force: true }); + } +} + +main().catch((error: unknown) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exitCode = 1; +}); diff --git a/packages/mcp-server-evals/tsconfig.code-mode.json b/packages/mcp-server-evals/tsconfig.code-mode.json new file mode 100644 index 000000000..165aaf327 --- /dev/null +++ b/packages/mcp-server-evals/tsconfig.code-mode.json @@ -0,0 +1,7 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "tsBuildInfoFile": "./node_modules/.tmp/tsconfig.code-mode.tsbuildinfo" + }, + "include": ["src/code-mode"] +} diff --git a/packages/mcp-server-evals/vitest.code-mode.config.ts b/packages/mcp-server-evals/vitest.code-mode.config.ts new file mode 100644 index 000000000..c7adc32df --- /dev/null +++ b/packages/mcp-server-evals/vitest.code-mode.config.ts @@ -0,0 +1,5 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { include: ["src/code-mode/**/*.test.ts"], environment: "node" }, +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index bcdcdc84d..3d0b81a4f 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -938,6 +938,10 @@ importers: zod: specifier: 'catalog:' version: 4.6.7 + devDependencies: + wrangler: + specifier: 'catalog:' + version: 4.151.3(@cloudflare/workers-types@4.20260405.1)(@types/node@24.13.5) packages/mcp-server-mocks: dependencies: