Repository navigation
test(evals): a real-model evaluation of Skilled OpenAPI, with a dry run that runs in CI - #1103
Merged
frontegg-david merged 4 commits intoNov 4, 2026
Merged
Conversation
…d OpenAPI The protocol e2e tests check what the server answers. Nothing checked what a model does with those answers: whether it finds the skill, loads it, runs the action or a workflow, and tells the user the right thing. apps/evals/skilled-openapi runs five billing tasks against the e2e app's server in its production settings, gives the model the server's tools, descriptions and initialize instructions, and runs a tool-use loop until the model answers. Each task is judged on the requests the mock billing API recorded and on the final answer, never on what the model says it did. A JSON report and a text summary give success, tool calls by name and in order, skill reloads, actions run before any load, tokens and latency, per task and in total. The eval calls the Claude API (claude-sonnet-5-5 by default), so it has its own eval target and never runs in CI. Its dry run plays a fixed script in place of the model, needs no API key, and runs in the project's test target, so the harness can't break unnoticed. The mock billing API can now hold seeded invoices, answer its reports without delay, and record each request's body and status; the e2e tests keep their behavior.
…feat/skilled-model-evals
frontegg-david
marked this pull request as ready for review
November 4, 2026 22:01
Contributor
Author
|
@coderabbitai review |
Contributor
|
Warning Review limit reachedYour organization has reached its usage spending cap. Adjust your spending cap in the billing tab. Next included review available in 22 minutes. View limit details
|
Contributor
Action not completed
|
alexmercerpo
approved these changes
Nov 4, 2026
…ted and the ANTHROPIC_API_KEY secret is set The job reports without failing the workflow: the summary goes on its job page and the JSON report is uploaded as an artifact. A run without the secret (a fork, or a repository that hasn't set it) or with a change that doesn't affect the eval skips with a one-line note.
Contributor
Performance Test ResultsStatus: ✅ All tests passed Summary
Total: 101 tests across 21 projects 📊 View full report in workflow run Generated at: 2026-11-04T22:28:00.759Z |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Stacks on #1102.
The protocol e2e tests check what a Skilled OpenAPI server answers. Nothing measured what a model does with those
answers: whether it finds the skill, loads it, runs the action or a workflow, and tells the user the right thing. This
PR adds a model evaluation that runs that whole loop with a real model.
It lives in a new Nx project,
evals-skilled-openapi(apps/evals/skilled-openapi). Itsevaltarget calls theClaude API, so it never runs in CI. Its
testtarget, which CI runs, holds the unit tests and a dry run that plays afixed script in place of the model, so the harness can't break unnoticed.
What a run does
copying them: a bundle signed at start,
unprotectedOps: 'deny', and the mock billing API over HTTPS.initializeinstructions and the date. It runs every tool call the model makes over MCP and returns the results,until the model answers or reaches the turn limit (10 by default).
it did.
The tasks
invoice-statusrefund-within-limitrefund-over-limitexport-revenue-reportaccount-plan-by-searchThe last task needs a skill the model has to search for: no line of the catalog in
search_skill's descriptionmentions the account plan. The dry run checks that this stays true.
Metrics, per task and in total
The model
claude-sonnet-5-5by default;SKILLED_EVAL_MODELpicks another model, andSKILLED_EVAL_EFFORTsetsoutput_config.effort.@anthropic-ai/sdk, already a root dev dependency, so the lockfile doesn't change.Each request is cached up to its last block, so the tools and system prompt every task shares are read from the
cache.
send the prompt or the tool results and get the reply. Another provider is one more adapter.
wrong one. A refusal shows as
stopped: refusal.ANTHROPIC_API_KEYonly. It isn't printed, logged or written to the report, and the serverthe harness starts gets it blanked. The report's API requests leave out the
Authorizationheader.Running the real eval
export ANTHROPIC_API_KEY=... npx nx run evals-skilled-openapi:evalA run sends about 15 to 30 requests, and at most 50 (five tasks times the turn limit). Each request carries about
2,500 tokens of tools and system prompt; the measured tool definitions are 6,600 characters. A run comes to about
60,000 to 120,000 input tokens, most of them cache reads, and 5,000 to 20,000 output tokens. With Claude Sonnet 5.5
that is well under one US dollar per run. The summary's
Tokensline gives the real figures.Changes to the e2e app's mock billing API
All of them are optional, so the e2e tests keep their behavior:
ProductionEnvironmenttakes these options and passes them to the mock.The project's eslint config allows its imports of the e2e app's helpers, as
libs/adaptersdoes formcp-from-openapi.User-facing change
plugin, and the plugin's README points to it.
Tests
Added, in
evals-skilled-openapi'stesttarget (7 suites, 57 tests):tasks.spec.ts(23): each task's checks pass and fail on recorded requests and answers, including the month acrossa year.
metrics.spec.ts(7): counts, call order, acting before loading, reloads byskillIdandskillIds, totals.run-task.spec.ts(7): the loop, parallel calls sent back together, failed calls, the turn limit, a failed modelrequest, and the scripted model.
anthropic-model.spec.ts(5): the request (tools, system prompt, caching, effort), the append-only history, andreplies cut off at
max_tokens.report.spec.ts(3): the summary and the files written.cli.spec.ts(8): options, their errors, and the missing-key error, raised before any server starts.dry-run.spec.ts(4): the whole--dry-runcommand against the real server passes 5 of 5 and writes a report withno credential in it; a careless script fails its check; an unknown tool and a failed workflow reach the model as
error results; the catalog leaves the account plan to a search.
Run:
npx nx test evals-skilled-openapi --coverage: 7 suites, 57 tests passed; coverage 97.3% statements, 94.0%branches, 98.1% functions, 98.0% lines.
npx nx run evals-skilled-openapi:eval:dry-run: 5 of 5 tasks passed.jest.e2e.config.ts), after the mock changes: 8 suites, 81 tests passed.npx nx run evals-skilled-openapi:typecheck,npx nx run demo-e2e-skilled-openapi:typecheck, andtscon the e2eapp's
tsconfig.e2e.json: no errors.npx nx run evals-skilled-openapi:lint, prettier,yarn install --immutable: clean.The real eval was not run: it needs an API key and costs money.
In CI
A new job, "Skilled OpenAPI eval" in the On Push workflow, runs the real eval next to the E2E tests:
evals-skilled-openapi(nx show projects --affected), and theANTHROPIC_API_KEYrepository secret is available (a pull request from a fork never gets it);skilled-openapi-eval-reportartifact;Each run is one eval run: about 15–30 requests. Set the secret with
gh secret set ANTHROPIC_API_KEY --repo agentfront/frontmcp.