Skip to content

docs(benchmark): remove the withdrawn earlier benchmark #231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dgesteves merged 1 commit into from
Oct 22, 2026
Merged

docs(benchmark): remove the withdrawn earlier benchmark #231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 6 additions & 56 deletions bench/agents/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,59 +1,9 @@
# Agent benchmark
# Agent benchmark v1

Does an agent write UI that fits the design system more often with onsystem than without it? This runs Claude Code on the same UI tasks twice, once as it ships and once with the [Claude Code plugin](https://onsystem.vercel.app/docs/plugin), and scores what it wrote with `onsystem check`.
An earlier, smaller benchmark, replaced by [bench/v2](../v2).

The runs below were made on 2026-10-08, before the rename, with `@dgesteves/design-system-mcp` 0.1 and its `design-system` plugin.
Its files stay here because bench/v2 builds on them: v2's task prompts end with the same instruction as [run.mjs](run.mjs), its chatbot tasks reuse [tasks.json](tasks.json), and its cost estimate ([config/usage-model.json](../v2/config/usage-model.json)) is measured from these runs.

A larger benchmark, across three providers, five repositories and five conditions, and scored with `tsc`, onsystem and a blind-labelled sample as well, is set up in [bench/v2](../v2).

## Setup

- **Project:** [vercel/ai-chatbot](https://github.com/vercel/ai-chatbot) at `c2f8235`, a real shadcn/ui app with 23 components and 52 tokens. Dependencies are not installed.
- **Tasks:** ten components a chat product needs ([tasks.json](tasks.json)), for example a usage banner with a dismiss button, a delete-all-chats confirmation dialog, a settings card with a green "Connected" label, and a message toolbar of icon buttons. Each prompt ends with the same instruction: use the project's existing UI components and design tokens, create only that file, and do not run commands.
- **Conditions:**
- `base`: `claude -p` with no plugin.
- `plugin`: the same, plus `--plugin-dir` with the plugin (MCP server, skill and hook) and the MCP tools allowed.
- Both use `--permission-mode acceptEdits` and a fresh copy of the project per run.
- **Models:** Claude Haiku 4.5 and Claude Opus 5, one run per task, condition and model, so 40 runs in all.
- **Scoring:** `onsystem check` on every file the run created or changed, against a baseline of the untouched project, so only findings the run introduced count. A task is clean when it has no errors.

## Results

| Model | Condition | Clean tasks | Errors | Cost | Avg. time | Avg. turns |
| ---------------- | --------- | ----------: | -----: | ----: | --------: | ---------: |
| Claude Haiku 4.5 | base | 6 / 10 | 13 | $1.31 | 46 s | 13.8 |
| Claude Haiku 4.5 | plugin | **10 / 10** | **0** | $1.17 | 42 s | 16.0 |
| Claude Opus 5 | base | 8 / 10 | 5 | $7.74 | 108 s | 15.8 |
| Claude Opus 5 | plugin | **10 / 10** | **0** | $8.31 | 141 s | 21.1 |

No run produced a warning.

What the base runs got wrong:

| Task | Haiku 4.5 | Opus 5 |
| ---------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| rate-limit-alert | 9 × `no-hardcoded-color` (Tailwind's red palette) | none |
| settings-card | 2 × `no-hardcoded-color` | 4 × `no-hardcoded-color` (`emerald` for "Connected") |
| profile-card | none | 1 × `no-hardcoded-color` (an `oklch()` literal for the avatar circle) |
| share-dialog | 1 × `prefer-design-system-component` (native `<label>`; the project has `Label`) | none |
| usage-banner | 1 × `icon-button-accessible-name` | none |

With the plugin, the agents called the design-system tools 5.6 (Haiku) and 7.3 (Opus) times per task on average: `search_components`, `get_component`, `get_tokens` and `check_ui`. The lint-on-edit hook never had to report an error back. Looking the design system up before writing was enough, so the hook acted as a safety net that was never needed in these runs. A [separate run](https://onsystem.vercel.app/docs/plugin), where Claude was asked to write hardcoded values on purpose, shows the hook catching them and Claude fixing them.

## Reading it honestly

- **Frontier models already do well here.** Opus 5 reads the codebase and copies its patterns. Its misses were colors the prompt named ("green", "colored circle") that the project has no token for. Smaller, faster models miss more, and they gained the most: on Haiku the plugin also cost less and finished sooner.
- **The sample is small.** It's one project and ten tasks, with one run per cell. Treat the numbers as a direction, not a rate.
- **The score is the tool's own.** It only counts what `check` can see: hardcoded values, invented components, props and variants, native elements, and icon buttons without names. It does not judge whether the UI is good.

The files each run produced are in [output/](output/), and the per-run results, with every finding, cost and turn count, are in [results/](results/).

## Reproduce

```sh
git clone https://github.com/vercel/ai-chatbot && git -C ai-chatbot checkout c2f8235
pnpm build # in this repository
node bench/agents/run.mjs ./ai-chatbot . ./bench-out --concurrency 5 --model haiku
```

Each run calls Claude Code headless and is billed to your account: about $0.06 to $0.25 per run with Haiku 4.5 and $0.45 to $1.30 with Opus 5.
- [run.mjs](run.mjs): the harness, which runs Claude Code headless on each task with and without the plugin and scores the files with `onsystem check`.
- [tasks.json](tasks.json): the tasks.
- [results/](results/) and [output/](output/): the record of each run and the files it wrote.
21 changes: 4 additions & 17 deletions bench/v2/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,7 @@ Does onsystem make coding agents write UI that fits a project's design system, a

**Results (2026-10-12, onsystem 0.5.0):** 1,200 runs, 6 models from 3 providers, 5 real repositories. With onsystem's skill, MCP server and hook together, design-system errors per run, as onsystem 0.5.0 counts them, fell from 0.85 to 0.00, with no measurable change in `tsc` pass rate (93% → 92%, Δ 0 [−4, +3]) and the same cost per run. The MCP server and the hook each do most of the work on their own; the gain is largest for small models. That count is onsystem's own, of the errors its MCP server and hook show the agent; [Threats to validity](#threats-to-validity) says what it cannot show: dub's colors in 0.5.0, chart tokens used for status colors, and a blind sample too small to confirm it. Full tables with confidence intervals: [results/standard/results.md](results/standard/results.md); per-run data in `results/standard/runs/` and `runs.csv`; transcripts on the [bench-v2-2026-10-12 release](https://github.com/dgesteves/onsystem/releases/tag/bench-v2-2026-10-12). A stratified 60-file sample was labelled blind by an independent AI reviewer ([LABELLER.md](results/standard/labels/LABELLER.md)).

## What changes from v1

[v1](../agents) was one repository, Claude only, 40 runs, and onsystem scored its own output. It showed a direction, not a result.

| | v1 (2026-10-08) | v2 |
| ------------ | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Agent | Claude Code headless, with the user's own settings loaded | One neutral agent loop over each provider's API, nothing else loaded |
| Models | Claude Haiku 4.5, Claude Opus 5 | Small and mid tiers from Anthropic, OpenAI and Google; frontier tier optional |
| Repositories | 1 (vercel/chatbot) | 5 from `corpus/`, pinned, installed with scripts off |
| Tasks | 10 | 20 (4 per repository), fixed prompts |
| Conditions | base, plugin | none, skill, mcp, hook, all |
| Repeats | 1 | 1 to 3 per cell |
| Scoring | `onsystem check` | `tsc` in each project's own setup, `onsystem check`, blind labels on a stratified sample, and @shadcn/lint (added after the standard run) |
| Statistics | counts | Wilson intervals for rates, cluster bootstrap for means and paired differences |
An earlier, smaller benchmark, [v1](../agents), was replaced by this one.

## Method

Expand All @@ -41,7 +28,7 @@ Each arm adds shipped onsystem files to the same loop, unchanged:

| Arm | What the agent gets on top of the file tools |
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `none` | Nothing. The prompt still asks it to use the project's components and tokens, as v1's did. |
| `none` | Nothing. The prompt still asks it to use the project's components and tokens. |
| `skill` | The Claude Code plugin's `SKILL.md`, the way Agent Skills work: its name and description in the system prompt and a `load_skill` tool that returns the whole file. The skill names the MCP tools, which this arm does not have; its other instructions (tokens, variants, native elements, icon buttons) still apply. |
| `mcp` | The onsystem MCP server (this checkout's `dist/cli.js`), started over stdio in the run's copy of the repository. Its five tools, with their own descriptions and schemas, as `mcp__onsystem__<tool>`, and its instructions in the system prompt, as MCP clients do. |
| `hook` | The plugin's hook script run after every write or edit, with the PostToolUse input Claude Code gives it. On exit 2 its stderr goes back to the model with the tool result; on exit 0 its `additionalContext` is added as a note. Its message points at `get_component` and `get_tokens`, which this arm does not have. |
Expand Down Expand Up @@ -123,7 +110,7 @@ Rates get 95% [Wilson score intervals](https://en.wikipedia.org/wiki/Binomial_pr

- **Not a product test.** The loop is not Claude Code, Codex or Gemini CLI. Real clients add their own prompts and tools, which can change how much onsystem helps.
- **One reasoning level.** `medium` means different things to different providers; other levels may change the picture.
- **The prompt asks for the design system.** Every arm is told to use the project's components and tokens. A prompt that does not ask would likely show larger effects; this one is the conservative case, and the same as v1.
- **The prompt asks for the design system.** Every arm is told to use the project's components and tokens. A prompt that does not ask would likely show larger effects; this one is the conservative case.
- **The scorers' blind spots.** `tsc` sees invented APIs, not hardcoded values. onsystem sees both but is the tool under test; its precision on the corpus is in `corpus/labels.json`. @shadcn/lint, added after the standard run, sees hardcoded values and restyled components, not invented APIs, and reads less in some repositories (its own section, Independent scorer: @shadcn/lint, says where). No tool judges whether the UI does what was asked; only the blind labels do, and only on a sample.
- **onsystem counts what it feeds back.** The MCP server's `check_ui` and the hook give the agent the same errors that `onsystem check` counts afterwards, so the arms with either end close to zero partly by construction. No run added a suppression comment or changed a file other than its own.
- **dub, as of onsystem 0.5.0.** 0.5.0 found no color tokens in dub, so the color rules stayed quiet there in every arm. dub keeps its tokens in a shared Tailwind config package, which onsystem reads since [#107](https://github.com/dgesteves/onsystem/pull/107), merged after 0.6.0, and re-scoring these outputs with it raises dub's count in every arm. A rerun on the released version is planned.
Expand Down Expand Up @@ -166,7 +153,7 @@ node bench/v2/run.mjs score --with shadcn-lint --out bench/v2/results/standard
| **`standard`** | **1,200** | `small` plus mid models, 5 arms, 20 tasks, 1 repeat | **$45** | **$214** | $17 / $17 / $11 |
| `large` | 2,100 | small and mid models with 3 repeats, plus frontier models with 1 | $256 | $1,178 | $79 / $136 / $41 |

**How the estimate is made.** Per run, the number of requests, the context a run adds by reading and writing, and the output tokens come from v1's 40 transcripts ([config/usage-model.json](config/usage-model.json)): Claude Haiku 4.5's runs for small models, Claude Opus 5's for mid and frontier models, v1's base condition for `none` and the larger of its two conditions for the other arms. v1's own fixed prompt (Claude Code's, about 34,000 tokens per request) is replaced by this harness's (about 700 tokens, computed from its text; the skill adds about 170, plus 570 once loaded; the MCP server about 1,570). _Expected_ assumes each request reads the earlier context from the prompt cache and writes only what is new. _Pessimistic_ assumes no cache hits at all, 50% more requests and context, and 50% more output. As a check, the same formula given v1's setup gives $0.17 per Haiku 4.5 run where v1 measured $0.13: it errs high. Token counts use 4 characters per token, and other providers' tokenizers differ by tens of percent. The canary measures real cost per run, and `plan --usage-from <canary output>` re-estimates from it.
**How the estimate is made.** Per run, the number of requests, the context a run adds by reading and writing, and the output tokens come from v1's transcripts ([config/usage-model.json](config/usage-model.json)): Claude Haiku 4.5's runs for small models, Claude Opus 5's for mid and frontier models, v1's base condition for `none` and the larger of its two conditions for the other arms. v1's own fixed prompt (Claude Code's, about 34,000 tokens per request) is replaced by this harness's (about 700 tokens, computed from its text; the skill adds about 170, plus 570 once loaded; the MCP server about 1,570). _Expected_ assumes each request reads the earlier context from the prompt cache and writes only what is new. _Pessimistic_ assumes no cache hits at all, 50% more requests and context, and 50% more output. As a check, the same formula given v1's setup gives $0.17 per Haiku 4.5 run where v1 measured $0.13: it errs high. Token counts use 4 characters per token, and other providers' tokenizers differ by tens of percent. The canary measures real cost per run, and `plan --usage-from <canary output>` re-estimates from it.

**What each size can conclude.** Half-widths of the 95% Wilson interval for one cell (one model, one arm): ±17 to ±20 points with 20 runs, ±10 to ±12 with 60, ±6 to ±7 with 180 (for rates between 80% and 50%). For differences between an arm and `none`, a simulation with this harness's own interval code (20 tasks whose difficulty varies a lot, as v1's did; the arm removes a share of each task's failures; 300 simulated benchmarks per point) gives the smallest effect found 80% of the time:

Expand Down
Loading
Loading