How good is your Mac at running LLM coding agents? A reproducible, single-command benchmark for local coding models on Apple Silicon. Each model gets one real coding task, one tool (a shell), and a hard 15-minute clock to produce a working Flask app. Pass/fail is verified by an actual HTTP request, not a vibe check.
uv sync
uv run benchmark run --set recommendedThat's it. The harness spins up each model in turn, runs it against the
task in an isolated workspace, watches for infinite loops, boots the app
the model generated, curls it, and writes a Markdown scoreboard to
reports/results_<job_id>.md. Past reports are checked into this repo
(reports/*.md) so you can see exactly what you'd be signing up for.
Most "local LLM leaderboards" measure perplexity on a frozen test set, or pass rates on toy one-line puzzles. That doesn't tell you whether a model can actually build a thing on your hardware in a reasonable amount of time. This project does the dumb-but-honest version of that:
- One real, end-to-end coding task per run (currently: a Flask web app).
- Two agent harnesses — a minimal ReAct loop (
litellm+ abashtool) and the real Aider CLI — so you can compare how a model behaves under different scaffolding. - Hardware-aware metrics: CPU, memory, token counts, and optionally
GPU utilisation, GPU power (mW), die temp, and fan RPM from Apple's
powermetrics. See if your M-series chip is throttling before you publish a "Qwen is slow" take. - Loop detection on both backends, so a stuck model gets killed in seconds instead of eating the full 15-minute timeout.
- Deterministic file layout: one job ID per run, all artefacts
collated under it, cleanup is a single
rm -rf. - No network sandbox tricks — generated code runs on your box. Only
run this in the workspaces the harness creates (it already keeps them
outside the repo tree so
uv initcan't climb back in).
If you want to contribute a task, a model, or a different agent harness, see Contributing below — we genuinely want PRs.
# 1. Install
uv sync
# 2. See what Ollama models you already have (vs. the catalog)
uv run benchmark list
# 3. Pull the recommended benchmark set (or a subset)
uv run prefetch --set recommended --dry-run # preview downloads
uv run prefetch --set recommended # actually pull
# Optional: focused Qwen coding comparison batch
uv run prefetch --set qwen-coding --dry-run
uv run benchmark run --set qwen-coding
uv run benchmark run --set qwen-coding --rounds 5 --order balanced
# 4. Smallest possible sanity run (~30s)
uv run benchmark run --set poc
# 5. Full recommended set — 10 local models
uv run benchmark run --set recommended
# 6. Open the auto-generated report
open reports/results_$(ls -1 reports/ | tail -1)Each completed run writes reports/results_<job_id>.md automatically.
Open a finished result directory and run ./run.sh to serve the
model-generated app on http://localhost:8181 for manual inspection.
Sample headline row from a recent --set recommended run:
| Model | Pass | Wall time | Notes |
|---|---|---|---|
qwen3.5:35b-a3b-coding-mxfp8 |
✅ | 12.5 s | Real rhyme-group generator. |
gemma4:e4b |
✅ | 16.8 s | Structured tuples, clean HTML. |
gemma4:e2b |
✅ | 13.0 s | Tiny, static-list, fastest. |
qwen3.6:latest |
❌ | 180.3 s | Infinite planning loop, wrote nothing. |
gemma4:26b |
❌ | 70.5 s | Analysis paralysis, then loop-guard fired. |
Full detailed example: reports/results_20260417_073034.md. A cross-run
meta-analysis across 10 jobs from one day is in
claude_report.md.
Stable findings so far:
- Smaller models finish more often than larger ones on this task — the 2B–4B Gemmas ship static-list apps in 12–30 s; base 26B–35B models routinely burn the full time budget "planning" rhyme dictionaries they never write to disk.
- Run-to-run variance is large. One sample per model is not a
leaderboard. Plan for
n ≥ 3. - Aider's SEARCH/REPLACE diff format is a real failure surface for local models — a model that fails under Aider may pass under the ReAct harness and vice versa.
- One job per invocation. Each
benchmark rungets a single job ID (YYYYMMDD.HHMMSS); every per-model run is collated under it. Results live inresults/<job_id>/<run-dir>/, workspaces in~/.limerick-benchmark/workspaces/<job_id>/<run-dir>/with a symlink back. Single-run jobs still use the bare model slug; repeated-round jobs prefix run index / round / position so repeated model IDs do not collide. Cleanup isrm -rf results/<job_id>plus the matching workspace tree. - Serial runs, no GPU contention. One model at a time so the numbers are comparable.
- Workspace prep. Workspaces are pre-seeded with task data files
(for the limerick task,
tasks/limericks.txtgets copied in).- ReAct (default): the workspace is not pre-initialised —
running
uv initand installing Flask is part of what the model is being graded on. - Aider (
--agent aider): the workspace is pre-bootstrapped as auvproject with Flask installed, because Aider can't run setup commands. The task prompt has an environment note that reflects whichever mode you picked.
- ReAct (default): the workspace is not pre-initialised —
running
- One of two agent backends drives the run.
- ReAct —
litellm+ a singlebashtool (60s per-command timeout). Loop guards abort on 5 identical commands in a row, repeateduv init, repeated whole-file rewrites, or 5 malformed tool calls. - Aider — the real Aider CLI in headless mode, plus a harness
wrapper that watches for low log-line uniqueness, repeating log
cycles, per-file edit caps, and workspace-hash stagnation (kills
the run if the file tree hasn't changed for 300 s by default —
override with
--aider-stagnation-timeout).
- ReAct —
- Hard 15-minute clock on the whole run (
--timeoutto change). - Metrics sampled every 5 seconds. CPU / memory / tokens always;
GPU utilisation / power / die temp / fan RPM if you pass
--enable-hardware-metrics(needssudoforpowermetrics). - Automated evaluator. Requires canonical entry point
app.py, runsuv run python app.py, expects HTTP 200 on port 8181, checks the response body for a 5-line limerick shape and either a refresh<meta>tag or asetInterval(call. Writes arun.shinto the result dir so you can re-boot the server manually. - Auto-report. At the end of the job, a Markdown summary lands in
reports/results_<job_id>.md.
See tasks/limerick.md for the full prompt.
Build a Python web app using Flask and
uvthat shows a limerick in the browser, rotates to a new one every 5 seconds, and listens on port 8181.
Tasks are just Markdown files in tasks/. Adding a new one is as simple
as dropping a new .md in there and (if needed) a corresponding data
seed and an evaluator branch. PRs welcome — see Contributing.
The first two checks are automated gates: if the server doesn't start, there is nothing to evaluate.
| Check | Type | Points |
|---|---|---|
| Server starts without error | Automated gate | — |
GET / returns HTTP 200 |
Automated gate | — |
| Page displays a recognisable 5-line limerick | Human | 0–40 |
Auto-refreshes every ~5 s (meta refresh or setInterval) |
Human | 0–30 |
| Code quality / approach (uv usage, structure, etc.) | Human | 0–30 |
Reference: Claude Opus 4.7 = 100 points. All other scores are relative to that.
- macOS with Apple Silicon (developed on M5 Max, 64 GB). Tested on macOS Sequoia.
- Ollama installed and running (
ollama serve). - Python 3.11+ and
uv. ANTHROPIC_API_KEYonly if you want to run the Anthropic reference models (--set reference).sudoaccess only if you pass--enable-hardware-metrics.
Nothing else. No Docker, no cloud, no sign-up.
| Flag | Default | Notes |
|---|---|---|
--set {poc,v1,recommended,qwen-coding,local,reference} |
— | Named model set. qwen-coding is the focused comparison batch: gemma4:e4b, qwen3.5:35b-a3b-coding-mxfp8, and qwen3.6:35b-a3b-coding-mxfp8; local = whatever is already in your Ollama store; reference = Anthropic cloud. Mutually exclusive with --model. |
--model MODEL_ID […] |
— | One or more explicit model IDs. Unknown IDs are treated as Ollama models. |
--task NAME |
limerick |
Task file name (without .md) in tasks/. |
--rounds N |
1 |
Repeat the full model list N times in one job. |
--order {balanced,random,fixed} |
balanced |
Per-round ordering. balanced rotates start position each round; random shuffles each round; fixed repeats the same order. |
--seed INT |
— | Seed for reproducible --order random runs. |
--timeout SECONDS |
900 | Per-model hard limit. |
--agent {react,aider} |
react |
Agent backend. |
--aider-stagnation-timeout SECONDS |
300 | Abort an Aider run if the workspace tree stays unchanged this long. |
--skip-missing |
off | Skip Ollama models that aren't pulled, instead of aborting the run plan. |
--enable-hardware-metrics |
off | Collect GPU / thermal / fan metrics via powermetrics (prompts for sudo). |
uv run benchmark list # local models vs. catalog
uv run benchmark report --job-id 20260417.083818
uv run benchmark report --job-id <id> --output reports/mine.md| Flag | Default | Notes |
|---|---|---|
--set {poc,v1,recommended,qwen-coding,all} |
— | Named model set. |
--model MODEL_ID […] |
— | Specific model IDs to pull. |
--dry-run |
off | Show the plan without downloading. |
--yes, -y |
off | Skip the confirmation prompt. |
results/
└── 20260417.193159/ ← one job ID per invocation
├── job.json ← what was requested
├── gemma4_e2b/
│ ├── workspace -> ~/.limerick-benchmark/workspaces/… ← generated code
│ ├── run.sh ← boot the generated server manually
│ ├── summary.json ← tokens, timing, evaluator result, failure class
│ ├── metrics.csv ← 5-second samples throughout the run
│ └── trace.jsonl ← full agent message history
└── qwen3.5_9b/
└── …
Workspaces live outside the repo tree at
~/.limerick-benchmark/workspaces/ so uv init inside them can't walk
up and auto-register as a member of this project's pyproject.toml.
Keep it that way.
metrics.csv columns:
| Column | Source | Notes |
|---|---|---|
timestamp |
wall clock | ISO 8601 |
elapsed_s |
benchmark | seconds since run start |
cpu_percent |
psutil | all cores averaged |
memory_percent |
psutil | system RAM |
gpu_utilization_percent |
powermetrics | Apple GPU busy % |
gpu_power_mw |
powermetrics | GPU power draw in mW |
die_temp_c |
powermetrics | CPU/GPU die temperature |
fan_rpm |
powermetrics SMC | first fan, if present |
tokens_in |
LiteLLM | cumulative prompt tokens |
tokens_out |
LiteLLM | cumulative completion tokens |
api_calls |
benchmark | cumulative model API calls |
summary.json now carries the higher-signal facts that feed repeated-run
analysis:
- Lifecycle + timing:
started_at,agent_finished_at,eval_started_at,eval_finished_at,finished_at,wall_seconds,agent_seconds,eval_seconds,first_edit_seconds,startup_seconds - Trace-derived behavior:
trace_line_count,assistant_answer_count,verification_marker_count,self_correction_marker_count,format_fixation_marker_count,duplicate_output_attempt_count - Workspace/code artifacts:
workspace_file_count,files_created_count,files_modified_count,app_py_sha256,app_py_bytes,app_py_loc,uses_render_template_string,uses_inline_html,route_count,dependency_count
Repeated-run Markdown reports aggregate those raw facts into model-level variance and stability summaries:
- wall-time spread (
stddev,p90) alongside median/fastest/slowest - distinct
app.pyhash counts and distinct solution-shape counts - median timing breakdowns for agent/edit/eval/startup phases
- per-position pass rate and median wall time so round-order effects are visible
|
tool_calls| benchmark | cumulative bash tool invocations |
GPU/thermal/fan columns are populated only when
--enable-hardware-metrics is on. To skip repeated sudo prompts, add
to /etc/sudoers (via visudo):
yourusername ALL=(ALL) NOPASSWD: /usr/bin/powermetrics
Both backends abort early if they're stuck, so a bad run costs seconds, not the full 15-minute budget:
- ReAct — aborts on 5 consecutive identical commands, 5 consecutive
redundant
uv initattempts, 3+ rewrites of the same file in a row, or 5 malformed / unknown tool calls.summary.jsonrecords which guard tripped. - Aider — aborts on low log-line uniqueness over a rolling window,
any detectable repeating log cycle, any single file edited more than
AIDER_MAX_EDITS_PER_FILEtimes, or the workspace tree hash not changing for 300 s by default. Override with--aider-stagnation-timeout. Aider stdout is prefixed with[N/total:model-id:agent]so multi-model runs stay grep-friendly.
A stuck run is recorded with finish_reason: stuck_loop and the HTTP
evaluation is skipped.
| Model ID | Role |
|---|---|
claude-opus-4-7 |
Reference — 100 pts |
claude-sonnet-4-6 |
Reference — better |
claude-haiku-4-5-20251001 |
Reference — good |
Chosen to answer specific questions: does size help? does a coding fine-tune beat a larger general model? does Apple MLX runtime differ from GGUF? does Qwen 3.6 beat 3.5 at the same weight class?
| # | Model ID | Size | Family | Question |
|---|---|---|---|---|
| 1 | qwen3.5:9b |
6.6 GB | Qwen 3.5 | Small/fast Qwen baseline |
| 2 | gemma4:e2b |
7.2 GB | Gemma 4 | Small/fast Gemma baseline |
| 3 | gemma4:e4b |
9.6 GB | Gemma 4 | One size up from e2b |
| 4 | gemma4:e2b-mlx-bf16 |
10 GB | Gemma 4 MLX | MLX vs. GGUF runtime |
| 5 | qwen3.5:27b-coding-mxfp8 |
31 GB | Qwen 3.5 Coder | Coding fine-tune vs. bigger general? |
| 6 | qwen3.5:35b-a3b |
24 GB | Qwen 3.5 | Large MoE baseline |
| 7 | gemma4:26b |
18 GB | Gemma 4 | Quality jump from small Gemmas? |
| 8 | qwen3.5:35b-a3b-coding-mxfp8 |
38 GB | Qwen 3.5 Coder | Coding fine-tune at 35B |
| 9 | qwen3.6:latest |
24 GB | Qwen 3.6 | Same size as #6 — does 3.6 matter? |
| 10 | qwen3.6:35b-a3b-q8_0 |
39 GB | Qwen 3.6 | Gen 3.6 at higher precision |
Full catalog with every variant and exclusion notes: models.yaml.
PRs are very welcome. The low-friction ways to contribute:
- Run the benchmark on your hardware and open a PR with the result Markdown. Different Apple Silicon generations (M1/M2/M3/M4/M5 and their Pro/Max/Ultra variants) produce different numbers; real data points are the most useful thing this repo can collect.
- Propose a new task. Drop a new
tasks/<name>.mdand a corresponding evaluator branch. Good candidates: "build a CLI that parses a CSV," "implement a job queue with Redis," "generate an interactive data viz." Tasks should be small enough to finish in 15 minutes but have a verifiable artefact. - Add a model family. Edit
models.yaml— mark anything that won't fit in 64 GB unified memory or that's NVIDIA-only (FP4, etc.) withexclude: trueand a reason. - Add a third agent backend. Current backends live in
benchmark/agent.py(_run_reactand_run_aider). A Codex / Aider architect mode / your-own-agent plug-in would be a great PR. - Harden the evaluator. See the "Recommendations" section of
claude_report.mdfor a concrete list of known gaps (per-job harness pinning,n ≥ 3sampling, partial credit forbody_missing_limerick, …).
Agent-facing repo conventions live in AGENTS.md.
CLAUDE.md and GEMINI.md are symlinks to it — edit AGENTS.md
directly so Claude Code, Codex, Gemini, and Aider all read the same
guidance. Run the test suite before submitting:
uv run python -m unittest discover tests- The ReAct loop executes arbitrary shell commands from model output.
The harness runs it only inside workspaces under
~/.limerick-benchmark/workspaces/(outside the repo). Do not run the benchmark in the repo root, and do not point it at a workspace you don't want a model writing to. - The Aider backend runs a subprocess without the bash-tool shim, but still operates on the workspace directory, so the same rule applies.
- Generated workspaces and
results/artefacts are gitignored on purpose (with a.gitkeeppassthrough inresults/). Published score writeups (results_YYYYMMDD_HHMMSS.md,claude_report.md, etc.) are intentionally tracked.
Built on top of Ollama, Aider, LiteLLM, uv, and a lot of Apple Silicon heat.