feat: add support for benchmarks to experiments

.changeset/sparkly-waves-post.md

@@ -0,0 +1,8 @@
+---
+'@primer/agent-eval': minor
+---
+
+Add support for running benchmark experiments with the `benchmark` option.
+
+Calls to `getExperiment` and `listExperiments` now require `benchmarksDirectory`,
+including when loading only scenario-backed experiments.

AGENTS.md

@@ -31,6 +31,16 @@
   `SKILL.md` concise, put detailed guidance in references, and use npm/npx in
   skill examples.
 
+## Evidence for decisions
+
+- When a decision, recommendation, or review finding relies on a resource,
+  include the specific evidence and explain how it supports the conclusion.
+- Cite repository files with paths and line numbers. For external resources,
+  link to the relevant section or comment and quote the supporting passage
+  when line numbers are unavailable.
+- Read the cited resource before relying on it. Distinguish what it explicitly
+  states from your inference, and state uncertainty or conflicting evidence.
+
 ## Code
 
 - Parse, don't validate. Use the type system to gurantee correctness.
@@ -123,7 +133,7 @@ When adding an experiment, follow these rules:
   it, when appropriate
 - New experiments are added at `./experiments` as a new `*.ts` file
 - Provide an appropriate name for the experiment as its file name
-- Export an `experiment` named export created with `defineConfig`
+- Use a default export created with `defineConfig`
 - Provide an appropriate name and description for what we are testing
 - Select appropriate models and scenarios based on the experiment
 - For treatments, make sure to include the fewest number possible to meet the

docs/experiments.md

@@ -2,9 +2,9 @@
 
 Experiments are used to determine how different setups (or treatments) affect the performance of an agent on a given task. By default, they live in an `experiments` folder in your project.
 
-Each experiment defines a set of scenarios that are used to evaluate the
-performance of each treatment. An experiment also defines a set of models to
-use.
+Each experiment defines either a set of scenarios or a benchmark whose
+capabilities and scenarios are used to evaluate each treatment. An experiment
+also defines a set of models to use.
 
 As an example, you may want to design an experiment comparing the performance of
 your MCP server to a skills based approach. With this in mind, you create two
@@ -55,3 +55,37 @@ export default defineConfig({
 You can interact with experiments using the `experiments` subcommand of the `agent-eval` CLI. This sub-command gives you access to run experiments, create run plans to use for sharding, or merge the results of a plan.
 
 Use `agent-eval experiments --help` to see the available commands and options.
+
+## Reports
+
+Experiment runs print a summary grouped by treatment, scenario, and model
+variant. When the experiment selects a `benchmark` instead of `scenarios`, the
+report adds a `Capability` column and capability totals between each treatment
+and its scenarios:
+
+```text
+treatment
+  capability
+    scenario
+      model / reasoning effort
+```
+
+Treatment totals include all completed capabilities. Scenarios and model
+variants are grouped separately within each capability, even when a scenario
+appears in more than one capability. Reports keep runner results separate and
+show a `Runner` column when SDK trials are present.
+
+Each level includes run counts, check summaries, and implementation-agent usage
+totals. Partial or sharded runs show only the groups with completed results.
+Scenario-backed experiments retain their existing report format.
+
+On the website, experiment pages show overall treatment summaries, then
+capability summaries with their scenario results. Runner, model, and reasoning
+effort groups remain separate. Website resource usage is averaged per trial.
+Scenario output links open the matching capability and scenario on the run
+detail page, which also supports filtering by capability.
+
+Benchmark-backed result bundles preserve capability metadata in `output.json`
+and a `capabilityId` on each trial. Older experiment bundles without this
+metadata remain readable, but their results stay grouped by scenario because
+capability membership cannot be inferred reliably.

experiments/benchmark.ts

@@ -0,0 +1,16 @@
+import {defineConfig} from '@primer/agent-eval/experiment'
+
+export default defineConfig({
+  name: 'benchmark',
+  description: 'An end-to-end test for running experiments against benchmarks',
+  models: ['gpt-5.6-luna'],
+  benchmark: 'noop',
+  treatments: [
+    {
+      name: 'noop',
+      async setup() {
+        console.log('local setup')
+      },
+    },
+  ],
+})

packages/agent-eval/src/__snapshots__/package.test.ts.snap

@@ -11,6 +11,7 @@ exports[`@primer/agent-eval exports 1`] = `
   "listExperiments",
   "ExperimentOutputFileSchema",
   "ExperimentTrialOutputSchema",
+  "parseExperimentTrialOutput",
   "getScenario",
   "listScenarios",
   "addCheckResults",

packages/agent-eval/src/benchmark/output.ts

@@ -399,10 +399,11 @@ async function listBenchmarkOutputFiles({
 export {
   BenchmarkOutputFileSchema,
   BenchmarkTrialOutputSchema,
+  CapabilityOutputSchema,
   parseBenchmarkTrialOutput,
   createBenchmarkOutput,
   listBenchmarkOutputFiles,
   mergeBenchmarkOutputFiles,
   writeBenchmarkOutput,
 }
-export type {BenchmarkOutput, BenchmarkTrialOutput}
+export type {BenchmarkOutput, BenchmarkTrialOutput, CapabilityOutput}

packages/agent-eval/src/cli/commands/experiment.ts

@@ -14,6 +14,7 @@ import {logger} from '../../logger'
 import {createPlanFromManifest, selectPlanTrials, runPlan} from '../../plan'
 import {parseShard} from '../../shard'
 import {
+  benchmarksOption,
   copilotConcurrencyOption,
   containerConcurrencyOption,
   experimentsOption,
@@ -82,6 +83,7 @@ const experimentCommand = defineCommand({
             description: 'Create an experiment plan',
           },
           args: {
+            benchmarks: benchmarksOption,
             experiments: experimentsOption,
             name: {
               type: 'positional',
@@ -98,25 +100,35 @@ const experimentCommand = defineCommand({
           async run({args}) {
             logger.info('Planning experiment: %s', args.name)
 
+            const benchmarksDirectory = path.resolve(args.benchmarks)
             const experimentsDirectory = path.resolve(args.experiments)
             const scenariosDirectory = path.resolve(args.scenarios)
             const outputPath = path.resolve(args['output-path'])
 
             logger.debug({
+              benchmarksDirectory,
               experimentsDirectory,
               scenariosDirectory,
               outputPath,
             })
 
             const experiment = await getExperiment({
+              benchmarksDirectory,
               experimentsDirectory,
               scenariosDirectory,
               name: args.name,
             })
-            const plan = createExperimentPlan({experiment})
-            const manifest = createExperimentPlanManifest({experiment, plan})
+            const plan = createExperimentPlan({
+              experiment,
+            })
+            const manifest = createExperimentPlanManifest({
+              experiment,
+              plan,
+            })
 
-            await host.fs.mkdir(path.dirname(outputPath), {recursive: true})
+            await host.fs.mkdir(path.dirname(outputPath), {
+              recursive: true,
+            })
             await host.fs.writeFile(outputPath, JSON.stringify(manifest, null, 2), 'utf-8')
 
             logger.info('Wrote experiment plan to: %s', path.relative(process.cwd(), outputPath))
@@ -128,6 +140,7 @@ const experimentCommand = defineCommand({
             description: 'Run an experiment plan from a manifest',
           },
           args: {
+            benchmarks: benchmarksOption,
             experiments: experimentsOption,
             'copilot-concurrency': copilotConcurrencyOption,
             'container-concurrency': containerConcurrencyOption,
@@ -143,6 +156,7 @@ const experimentCommand = defineCommand({
             token: githubCopilotTokenOption,
           },
           async run({args}) {
+            const benchmarksDirectory = path.resolve(args.benchmarks)
             const experimentsDirectory = path.resolve(args.experiments)
             const copilotConcurrency = getConcurrencyValue(args['copilot-concurrency'], 'copilot-concurrency')
             const containerConcurrency = getConcurrencyValue(args['container-concurrency'], 'container-concurrency')
@@ -156,6 +170,7 @@ const experimentCommand = defineCommand({
 
             logger.debug({
               artifactsDirectory,
+              benchmarksDirectory,
               experimentsDirectory,
               copilotConcurrency,
               containerConcurrency,
@@ -172,6 +187,7 @@ const experimentCommand = defineCommand({
 
             const contents = await host.fs.readFile(planPath, 'utf-8')
             const manifest = await parseExperimentPlanManifest({
+              benchmarksDirectory,
               experimentsDirectory,
               contents,
               scenariosDirectory,
@@ -210,6 +226,7 @@ const experimentCommand = defineCommand({
         description: 'Run an experiment',
       },
       args: {
+        benchmarks: benchmarksOption,
         experiments: experimentsOption,
         'copilot-concurrency': copilotConcurrencyOption,
         'container-concurrency': containerConcurrencyOption,
@@ -225,6 +242,7 @@ const experimentCommand = defineCommand({
       async run({args}) {
         logger.info('Running experiment: %s', args.name)
 
+        const benchmarksDirectory = path.resolve(args.benchmarks)
         const experimentsDirectory = path.resolve(args.experiments)
         const copilotConcurrency = getConcurrencyValue(args['copilot-concurrency'], 'copilot-concurrency')
         const containerConcurrency = getConcurrencyValue(args['container-concurrency'], 'container-concurrency')
@@ -236,6 +254,7 @@ const experimentCommand = defineCommand({
 
         logger.debug({
           artifactsDirectory,
+          benchmarksDirectory,
           experimentsDirectory,
           copilotConcurrency,
           containerConcurrency,
@@ -245,6 +264,7 @@ const experimentCommand = defineCommand({
         })
 
         const experiment = await getExperiment({
+          benchmarksDirectory,
           experimentsDirectory,
           scenariosDirectory,
           name: args.name,

packages/agent-eval/src/experiment/config.ts

@@ -4,7 +4,35 @@ import {ModelVariantConfigSchema} from '../model'
 import {ControlTreatment, TreatmentConfigSchema, TreatmentSetupSchema} from '../treatment'
 import {InlineScenarioConfigSchema} from '../scenario/config'
 
-const ExperimentConfigSchema = z.strictObject({
+const BenchmarkExperimentConfig = z.strictObject({
+  name: z.string(),
+  description: z.string(),
+  models: z.array(ModelVariantConfigSchema),
+  runners: CopilotRunnerConfigSchema,
+  benchmark: z.string().check(z.minLength(1)),
+  setup: z.optional(TreatmentSetupSchema),
+  treatments: z.array(TreatmentConfigSchema).check(
+    z.refine(
+      treatments => {
+        const names = new Set<string>([ControlTreatment.name, 'Benchmark'])
+        for (const treatment of treatments) {
+          if (names.has(treatment.name)) {
+            return false
+          }
+          names.add(treatment.name)
+        }
+        return true
+      },
+      {
+        message: 'Treatment names must be unique and cannot use the reserved names "Control" or "Benchmark"',
+      },
+    ),
+  ),
+})
+
+type BenchmarkExperimentConfig = z.infer<typeof BenchmarkExperimentConfig>
+
+const ScenarioExperimentConfig = z.strictObject({
   name: z.string(),
   description: z.string(),
   models: z.array(ModelVariantConfigSchema),
@@ -30,6 +58,10 @@ const ExperimentConfigSchema = z.strictObject({
   ),
 })
 
+type ScenarioExperimentConfig = z.infer<typeof ScenarioExperimentConfig>
+
+const ExperimentConfigSchema = z.union([ScenarioExperimentConfig, BenchmarkExperimentConfig])
+
 type ExperimentConfig = z.infer<typeof ExperimentConfigSchema>
 
 function defineConfig(config: z.input<typeof ExperimentConfigSchema>): ExperimentConfig {

packages/agent-eval/src/experiment/experiment.ts

@@ -2,8 +2,23 @@ import type {ExperimentConfig} from './config'
 import type {ModelVariant} from '../model'
 import type {Scenario} from '../scenario/scenario'
 import type {Treatment, TreatmentSetup} from '../treatment'
+import type {Benchmark} from '../benchmark/benchmark'
 
-type Experiment = {
+type BenchmarkExperiment = {
+  type: 'benchmark'
+  id: string
+  filepath: string
+  name: ExperimentConfig['name']
+  description: ExperimentConfig['description']
+  models: Array<ModelVariant>
+  runners: ExperimentConfig['runners']
+  benchmark: Benchmark
+  setup?: TreatmentSetup
+  treatments: Array<Treatment>
+}
+
+type ScenarioExperiment = {
+  type: 'scenario'
   id: string
   filepath: string
   name: ExperimentConfig['name']
@@ -15,4 +30,6 @@ type Experiment = {
   treatments: Array<Treatment>
 }
 
-export type {Experiment}
+type Experiment = ScenarioExperiment | BenchmarkExperiment
+
+export type {Experiment, ScenarioExperiment, BenchmarkExperiment}

packages/agent-eval/src/experiment/get.ts

@@ -3,21 +3,24 @@ import type {Experiment} from './experiment'
 import {listExperiments} from './list'
 
 type GetExperimentOptions = {
+  benchmarksDirectory: string
   experimentsDirectory: string
   host?: Host
   scenariosDirectory: string
   name: string
 }
 
 async function getExperiment({
+  benchmarksDirectory,
   experimentsDirectory,
   host = DefaultHost,
   name,
   scenariosDirectory,
 }: GetExperimentOptions): Promise<Experiment> {
   const experiments = await listExperiments({
-    host,
+    benchmarksDirectory,
     experimentsDirectory,
+    host,
     scenariosDirectory,
   })
   const experiment = experiments.find(candidate => {