feat: add support for benchmarks to experiments by joshblack · Pull Request #373 · primer/agent-eval

joshblack · 2026-10-09T19:43:22Z

Closes #329

This PR adds benchmark-backed experiments so we can compare custom treatments against control and benchmark conditions using the same capability groups.

Capability prerequisites run for every treatment, after experiment setup and before treatment setup. Benchmark-level setup runs only for Benchmark. Created and restored plans preserve the same setup order and treatment isolation.

Changelog

New

Add the benchmark experiment option with automatic Control and Benchmark treatments.
Add capability-aware experiment results and reporting.
Add regression coverage for setup order and treatment isolation in created and restored plans.

Changed

Require benchmarksDirectory in calls to getExperiment and listExperiments, including scenario-backed experiments.
Update capability and setup guidance for benchmark-backed experiments.
Use default exports in experiment authoring guidance and examples.
Require specific source evidence for resource-based agent decisions and recommendations.

Removed

Preserve benchmark experiments alongside duplicate trial ID checks and legacy runner defaults. Update CLI and website callers for benchmark-backed experiments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c99bf7e6-b5a0-46f7-b599-0787499df607

Copilot

🟡 Changes recommended

Benchmark experiment plans incorrectly restrict shared capability setup to the Benchmark treatment, breaking control and custom treatment prerequisites.

4 open findings

What changed in this PR

Adds benchmark-backed experiments across planning, result persistence, CLI/website reporting, and consumer guidance.

Changes:

Adds benchmark-based experiment configuration and execution.
Preserves capability metadata through plans and result bundles.
Adds capability-aware CLI and website reporting.

File	Description
`.changeset/sparkly-waves-post.md`	Adds the package release note.
`docs/experiments.md`	Documents benchmark-backed experiments and reports.
`experiments/benchmark.ts`	Adds an end-to-end benchmark experiment.
`packages/agent-eval/src/__snapshots__/package.test.ts.snap`	Updates package exports snapshot.
`packages/agent-eval/src/benchmark/output.ts`	Exports capability output types/schema.
`packages/agent-eval/src/cli/commands/experiment.ts`	Adds benchmark-directory CLI wiring.
`packages/agent-eval/src/experiment/config.ts`	Accepts benchmark experiment configurations.
`packages/agent-eval/src/experiment/experiment.ts`	Adds scenario/benchmark experiment variants.
`packages/agent-eval/src/experiment/get.ts`	Passes benchmark directory when loading.
`packages/agent-eval/src/experiment/list.ts`	Loads benchmark-backed experiments.
`packages/agent-eval/src/experiment/output.test.ts`	Tests capability-aware output bundles.
`packages/agent-eval/src/experiment/output.ts`	Persists and validates capability membership.
`packages/agent-eval/src/experiment/plan.test.ts`	Tests benchmark experiment planning.
`packages/agent-eval/src/experiment/plan.ts`	Creates and restores benchmark experiment trials.
`packages/agent-eval/src/experiment/report.test.ts`	Tests hierarchical experiment reports.
`packages/agent-eval/src/experiment/report.ts`	Adds capability-level report summaries.
`packages/agent-eval/src/index.ts`	Exports the experiment trial parser.
`packages/agent-eval/src/treatment.ts`	Adds setup composition support.
`skills/agent-eval/SKILL.md`	Links experiment reporting guidance.
`skills/agent-eval/references/cli.md`	Documents experiment benchmark-directory support.
`skills/agent-eval/references/experiments.md`	Documents benchmark experiment behavior.
`skills/agent-eval/references/trials-and-results.md`	Documents capability-aware bundles.
`website/src/app/components/ExperimentResults.tsx`	Renders capability and runner summaries.
`website/src/app/components/RunDetailsView.tsx`	Uses capability-specific scenario anchors.
`website/src/app/components/ScenarioAnchors.test.tsx`	Tests capability links and targets.
`website/src/experiment-results.test.ts`	Tests capability/runner grouping.
`website/src/experiment-results.ts`	Produces capability-aware summaries.
`website/src/experiments.ts`	Loads and presents benchmark experiments.
`website/src/result-files.test.ts`	Tests capability bundle parsing.
`website/src/result-files.ts`	Parses experiment capability membership.
`website/src/run-details.ts`	Adds capabilities to trial details.
`website/src/runs.test.ts`	Updates output fixtures.
`website/src/scenario-anchor.ts`	Generates capability-specific anchors.
`website/src/test-fixtures.ts`	Adds experiment capability metadata.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+| Required fields     | `name`, `description`, `models`, `treatments`, and either `scenarios` or `benchmark` |
+| Optional fields     | `setup`, `runners`                                                                   |
+| CLI identifier      | Filename without extension, not display `name`                                       |
+| Scenario references | Folder IDs; alternatively `{path, name?}`                                            |
+| Benchmark reference | Benchmark filename ID; selects its capabilities and scenarios                        |


… trial setup for all treatments' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Restored benchmark plans currently omit or reorder capability setup for multiple treatments.

3 open findings

2 resolved since last review

🧠 Review effort: Balanced

@@ -0,0 +1,16 @@
+import {defineConfig} from '@primer/agent-eval/experiment'
+
+export default defineConfig({


Update experiment guidance to use default exports and require source evidence for resource-based decisions. Clarify capability setup scopes and the benchmarksDirectory migration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b79d13c1-4b88-4e32-9faa-d425f1e2b55d

Copilot

🔵 Needs a closer look

The changeset omits breaking type migrations, and plan parsing introduces an unrelated compatibility break.

2 open findings

1 resolved since last review

Correct documentation for shared capability and benchmark setup

Previously missed (2)

In code that hasn't changed since last review

Avoid strict parsing to preserve manifest compatibility

packages/agent-eval/src/experiment/plan.ts:82

Changing this parser from z.object to z.strictObject makes previously accepted experiment plan manifests fail whenever they contain an extra top-level field, even though this feature does not require stricter parsing and the equivalent benchmark parser remains permissive (packages/agent-eval/src/benchmark/plan.ts:58). Keep the existing parsing behavior to avoid an unrelated CLI compatibility break.

Document migration for breaking Experiment type changes

.changeset/sparkly-waves-post.md:8

The migration note covers the new loader argument, but it omits two other breaking public type changes: Experiment is now a discriminated union whose scenarios field requires narrowing, and ExperimentOutput now requires a capabilities map (packages/agent-eval/src/experiment/experiment.ts:7-35, packages/agent-eval/src/experiment/output.ts:171-176). Please state how existing TypeScript consumers should update; the changeset guidance requires migration instructions for breaking changes.

🧠 Review effort: Balanced

joshblack and others added 6 commits October 9, 2026 12:52

feat: add support for benchmarks to experiments

38620da

refactor: inline getExperimentTreatments

cdb1f45

feat: add support for benchmark experiments to result

8bb7586

feat(website): add support for visualizing benchmark experiment results

9362f53

chore: add changeset

68438bb

Copilot AI balanced review requested due to automatic review settings October 9, 2026 19:43

Copilot started reviewing on behalf of joshblack October 9, 2026 19:43 View session

Copilot AI reviewed Oct 9, 2026

View reviewed changes

Potential fix for pull request finding 'Compose capability setup into…

e24311b

… trial setup for all treatments' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 12, 2026 16:10

Copilot started reviewing on behalf of joshblack October 12, 2026 16:11 View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread experiments/benchmark.ts

@@ -0,0 +1,16 @@

import {defineConfig} from '@primer/agent-eval/experiment'

export default defineConfig({

Copilot AI balanced review requested due to automatic review settings October 12, 2026 16:23

Copilot started reviewing on behalf of joshblack October 12, 2026 16:24 View session

joshblack merged commit a5d1340 into main Oct 12, 2026
9 checks passed

joshblack deleted the feat/add-benchmark-to-experiments branch October 12, 2026 16:28

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat: add support for benchmarks to experiments - #373

feat: add support for benchmarks to experiments#373
joshblack merged 8 commits into
mainfrom
feat/add-benchmark-to-experiments

joshblack commented Oct 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,16 @@
		import {defineConfig} from '@primer/agent-eval/experiment'

		export default defineConfig({

Conversation

joshblack commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog

New

Changed

Removed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshblack commented Oct 9, 2026 •

edited

Loading