diff --git a/docs/changelogs/0.8.x.md b/docs/changelogs/0.8.x.md new file mode 100644 index 0000000..0249eb3 --- /dev/null +++ b/docs/changelogs/0.8.x.md @@ -0,0 +1,30 @@ +# Changelog — 0.8.x + +All notable changes in the **0.8.x** release series are documented here. + +## [Unreleased] + +### Added +- A survey of the code-agent benchmarks reported across six 2026 model + releases, flattened into one table, with a recommendation of which ones + this project should target and a prioritized list of what it still needs + in order to run them. +- A source-level survey of what Terminal-Bench 2.1 / Harbor, SWE-bench + Verified, and NL2Repo-Bench actually require of a headless agent: the + adapter interface each one expects, how the task instruction is delivered, + where results are collected from, and the minimum contract shared by all + three. Notably, a benchmark harness reads a non-zero exit code as its own + failure, so an agent that runs out of turns without solving the task must + still exit 0. + + diff --git a/docs/research/README.md b/docs/research/README.md index 8cdf4f7..b4f6d14 100644 --- a/docs/research/README.md +++ b/docs/research/README.md @@ -27,3 +27,9 @@ versions are in sync. - How five agent projects write whole files, and why a structured write beats a heredoc: [English](en/write_tool.md) | [Chinese](zh-CN/write_tool.md) +- Which code-agent benchmarks six 2026 model releases reported, which ones fit + this project, and what it still needs to run them: + [English](en/code_agent_benchmark.md) | [Chinese](zh-CN/code_agent_benchmark.md) +- What Terminal-Bench 2.1, SWE-bench Verified, and NL2Repo-Bench actually + require of a headless agent, read down to the source level: + [English](en/benchmark_headless_interface.md) | [Chinese](zh-CN/benchmark_headless_interface.md) diff --git a/docs/research/en/benchmark_headless_interface.md b/docs/research/en/benchmark_headless_interface.md new file mode 100644 index 0000000..78329ad --- /dev/null +++ b/docs/research/en/benchmark_headless_interface.md @@ -0,0 +1,259 @@ +# What Benchmarks Require of a Headless Agent + +> Generated from the Chinese source [`../zh-CN/benchmark_headless_interface.md`](../zh-CN/benchmark_headless_interface.md). Do not edit by hand. + +Surveyed on 2026-08-18. + +[`code_agent_benchmark.md`](code_agent_benchmark.md) concluded: build headless mode first, then talk about scores. But "build headless mode" is not a single instruction — what each benchmark asks of an agent differs enormously. One specifies a Python class interface, one specifies a single JSON field, and one specifies nothing at all but hard-codes its runner inside its own repository. This document reads the integration surface of all three candidate benchmarks down to the source level and then converges on the minimum contract nanoPyCodeAgent should implement. + +The conclusion up front: **the three share only five requirements, and the most counter-intuitive of them is "exit 0 even when the task was not solved".** + +--- + +## 1. Terminal-Bench 2.1 / Harbor + +The only one of the three that genuinely **specifies an interface**. And note the shape: what you write is not "an agent that can be invoked from the command line" but **a Python adapter class running inside the Harbor process**, which in turn installs and invokes your CLI inside the container. + +### 1.1 The adapter interface + +Harbor has two agent kinds. A CLI that runs inside the container is the second: + +```python +# External agent (the agent process lives outside the container) +from harbor.agents.base import BaseAgent + +class MyExternalAgent(BaseAgent): + @staticmethod + def name() -> str: ... + def version(self) -> str | None: ... + async def setup(self, environment: BaseEnvironment) -> None: ... + async def run(self, instruction: str, environment: BaseEnvironment, + context: AgentContext) -> None: ... +``` + +```python +# Installed agent (the agent is installed into the container) — nanoPyCodeAgent's category +from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template + +class MyInstalledAgent(BaseInstalledAgent): + async def install(self, environment: BaseEnvironment) -> None: + await self.exec_as_root(environment, command="...") # system packages + await self.exec_as_agent(environment, command="...") # user-level installs + + @with_prompt_template + async def run(self, instruction: str, environment: BaseEnvironment, + context: AgentContext) -> None: ... + + def populate_context_post_run(self, context: AgentContext) -> None: ... +``` + +Only `install()` and `run()` are actually required; `populate_context_post_run()` feeds the trajectory and token accounting back to Harbor — optional, but valuable (see §1.4). + +How to run it: + +```bash +harbor run -d terminal-bench/terminal-bench-2-1 \ + --agent-import-path "path.to.agent:SomeAgent" -k 5 +``` + +(The Harbor docs also show the equivalent `--agent path.to.agent:SomeAgent` form.) + +### 1.2 How the instruction reaches the CLI: two official examples + +**Claude Code** (`src/harbor/agents/installed/claude_code.py`) — injected via an environment variable first, then fed in over **a stdin pipe**, which avoids shell escaping and command-line length problems: + +```bash +export PATH="$HOME/.local/bin:$PATH"; \ +harbor_claude_code_instruction_="$HARBOR_CLAUDE_CODE_INSTRUCTION_"; \ +unset HARBOR_CLAUDE_CODE_INSTRUCTION_; \ +printf "%s" "$harbor_claude_code_instruction_" | \ +claude --verbose --output-format=stream-json --print 2>&1 | tee /logs/agent/claude-code.txt +``` + +**mini-swe-agent** (`src/harbor/agents/installed/mini_swe_agent.py`) — a command-line argument, with **stdin explicitly wired to `/dev/null`**: + +```bash +mini-swe-agent --yolo --model= --task= \ + --output= --exit-immediately 2>&1 `, and a non-zero return code raises `NonZeroAgentExitCodeError`. + → **"Turn limit reached but the task is unsolved" must exit 0**, or it will be treated as an infrastructure fault (and may trigger a retry, burning money for nothing). This runs against intuition and contradicts the first draft of `code_agent_benchmark.md`. +2. **Harbor scans the agent's stdout/stderr with regexes to classify errors.** `ERROR_PATTERNS` covers rate limits, usage limits, 500s, Overloaded, mid-response disconnects, output-token overruns, context-window overruns, not-logged-in, safety refusals, and network errors; the resulting exception types feed the retry policy — the usage the source comments give is `harbor run --max-retries 3 --retry-include ApiRateLimitError`. + → The agent should print API errors **verbatim** rather than swallowing them. That halves the retry logic needed on the agent side: basic backoff is enough. +3. **No `--workdir` is needed.** `run()` does not pass `cwd` to `exec_as_agent`, so the container's default WORKDIR is the task working directory. Terminal-Bench uses `/app` — the session directory `$CLAUDE_CONFIG_DIR/projects/-app` hard-coded in `claude_code.py` is its slugified form. The agent only has to work in the process's current cwd. +4. **Log paths are conventional.** The agent writes its own logs to `/logs/agent/`. Under `/logs/verifier/`, `reward.txt` (a single int or float, typically 1/0) or `reward.json` (multiple metrics) is written by the **test script**; the agent neither writes it nor should read `/tests/`. +5. **Credentials and model selection are injected purely through environment variables.** `ANTHROPIC_API_KEY` / `ANTHROPIC_BASE_URL` / `ANTHROPIC_MODEL` — **nanoPyCodeAgent already supports all three** (`settings.py`), so that part comes for free. Harbor additionally sets aliases such as `ANTHROPIC_DEFAULT_SONNET_MODEL` for compatible backends. +6. **Wall-clock timeouts are the harness's job.** A task's `task.toml` carries `[agent].timeout_sec`, `[verifier].timeout_sec`, and `[environment].build_timeout_sec` (default 600). The agent's own `--timeout` is a safety net, not a prerequisite for integration. +7. **Installation is cheap.** `ensure_system_dependencies()` installs curl / bash / git / python3 / python3-pip / nodejs / npm / tmux / ripgrep and more on demand, so `install()` only needs `uv tool install nanoPyCodeAgent` or `pip install nanoPyCodeAgent`. Claude Code's own `install()` is little more than an npm-or-curl branch plus a `claude --version` self-check. +8. **Versions are detected.** Implement `get_version_command()` and `parse_version()` and Harbor will record the agent version on a best-effort basis (failures are swallowed). This requires the CLI to have a `--version`. + +### 1.4 The trajectory bonus + +Harbor has a unified trajectory format, ATIF (`SUPPORTS_ATIF`). mini-swe-agent has its CLI write its own JSON via `--output=` and converts it to ATIF in `populate_context_post_run()`; Claude Code instead parses the `--output-format=stream-json` event stream. Either one lets Harbor collect steps, tokens, and cost — which is exactly the P1 "trajectory logging" and "token accounting" items from `code_agent_benchmark.md`. **Doing those two is not extra work; it also buys the harness-side reporting.** + +### 1.5 Task structure (for context) + +``` +/ +├── instruction.md # the task instruction — this is what run() receives +├── task.toml # [task] [metadata] [verifier] [agent] [solution] [environment] +├── environment/Dockerfile # or docker-compose.yaml, or a docker_image reference +├── solution/solve.sh # the reference solution used by the Oracle agent +└── tests/test.sh # must write a reward file to /logs/verifier/ +``` + +--- + +## 2. SWE-bench Verified + +Officially there is **no agent interface** — only an output format. In other words, the agent-side runner is yours to write. + +### 2.1 The official side is evaluation only + +```bash +swebench eval verified -p --run-id -j +# The older form still works: +python -m swebench.harness.run_evaluation \ + --dataset_name princeton-nlp/SWE-bench_Verified \ + --predictions_path --max_workers 8 --run_id my_run +``` + +Predictions are JSONL, three keys per line: + +```json +{"instance_id": "sympy__sympy-20590", "model_name_or_path": "gpt-4", "model_patch": "diff --git a/sympy/core/sympify.py..."} +``` + +(mini-swe-agent writes a `{instance_id: {...}}` JSON dict instead; the harness accepts both.) + +Results are cached by `run_id` + `instance_id`, so re-running a modified patch requires a fresh `run_id`. + +### 2.2 Runner-side essentials (copy mini-swe-agent) + +Sources: `src/minisweagent/run/benchmarks/swebench.py` and `src/minisweagent/config/benchmarks/swebench.yaml` in `SWE-agent/mini-swe-agent`. + +- **Image**: `docker.io/swebench/sweb.eval.x86_64.:latest`, where the double underscore `__` inside `instance_id` is replaced with `_1776_` (Docker disallows double underscores) and the whole name is lowercased. +- **Working directory**: `/testbed`. +- **Task text**: `instance["problem_statement"]` from the dataset, handed to the agent verbatim. +- **Budget baseline**: `step_limit: 250`, `cost_limit: 3.` (USD), and a per-command `timeout: 60`. +- **Submitting the patch**: mini-swe uses a sentinel — the system prompt tells the agent to first run `git diff -- > patch.txt`, then submit with a **separate** command, `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && cat patch.txt`; the runner takes the stdout after the sentinel as `model_patch`. The prompt also explicitly forbids including test-file creations or edits in the patch. + +### 2.3 What this actually requires of nanoPyCodeAgent + +Exactly one thing: **be able to run a `problem_statement` headlessly in `/testbed` and then exit.** + +Collecting the patch is **better done without the sentinel** — after the agent exits, the runner can just `docker exec … git add -A && git diff --cached`. Zero agent-side changes, and it eliminates the entire failure class of "the model forgot to type the magic command, so the task scores 0". Accordingly, the P2 "patch output mode" item in `code_agent_benchmark.md` can be dropped. + +--- + +## 3. NL2Repo-Bench + +The loosest interface requirements of the three, but its runner hard-codes OpenHands into the repository, so integrating means **forking and editing code**, not mounting a plugin. + +### 3.1 The actual flow + +Source: `openhands/openhands_app.py` in `multimodal-art-projection/NL2RepoBench`. + +1. Each task gets a UUID directory, `workspaces//workspace/`, into which the requirements document (`start.md`) is copied from `test_files//`. **The directory is otherwise empty.** +2. `template/config.template.toml` is rendered into a per-task `config.toml`, substituting `{{VOLUMES}}` (mounting the workspace as `/workspace` in the container) and `{{MODULE_CONFIG}}` (model name / api_key / base_url). +3. A container is started from `docker.all-hands.dev/all-hands-ai/openhands:0.56` (runtime `runtime:0.56-nikolaik`) with a hard-coded command: + + ```bash + python -m openhands.core.main --config-file=/custom/path/config.toml \ + -t 'According to the start.md in the workspace, implement the entire project as per the requirements specified in the document, ensuring that the final product can be directly run in the current directory. ...' + ``` + +4. After the container exits, `post_process_task(task_uuid, workspace_path, test_data, logger)` runs pytest against the workspace directory on the host. **The score is the number of passing tests.** + +The entry configuration is `config.json` at the repository root: `startPro[].{moduleName, baseUrl, sk, proNameList}` plus a `max_pool_size` controlling concurrency. + +### 3.2 The key property + +**Scoring looks only at the final contents of the workspace directory and is entirely decoupled from how the agent ran.** So the only requirement on the agent is: **run one prompt headlessly in a given directory and exit when done.** + +There are two ways to integrate nanoPyCodeAgent: + +- Edit the container-creation block in `openhands/openhands_app.py` (swap the image, swap the `command`, drop the config.toml machinery) — roughly 20 lines; or +- Write a small runner of your own that reuses its `test_files/` (requirements documents plus test data) and `post_processor.py` (pytest scoring). + +The latter is cleaner, since the whole OpenHands `config.toml` apparatus is useless to us. + +--- + +## 4. The minimum common contract + +| # | Requirement | Terminal-Bench | SWE-bench | NL2Repo | +| --- | --- | :-: | :-: | :-: | +| 1 | One command delivers the task text; exit when done | ✅ | ✅ | ✅ | +| 2 | Work in the **process's current cwd** (no `--workdir` needed) | ✅ `/app` | ✅ `/testbed` | ✅ `/workspace` | +| 3 | No interaction, no questions, no waiting for confirmation | ✅ | ✅ | ✅ | +| 4 | **Exit 0 on any normal end** (including "unsolved") | ✅ enforced | recommended | recommended | +| 5 | All configuration through environment variables | ✅ | ✅ | ✅ | +| 6 | Logs written to `/logs/agent/` | ✅ | — | — | +| 7 | Structured trajectory output | optional, high value | — | — | +| 8 | A `git diff` obtainable at the end | — | ✅ (collected runner-side) | — | + +All three delivery mechanisms for the task text must work: a command-line argument (`--task=`), a stdin pipe (`printf … |`), and a file path. + +--- + +## 5. The CLI design for nanoPyCodeAgent + +``` +nanoPyCodeAgent [-p/--prompt "" | --prompt-file | (stdin)] + [--max-turns N] + [--output-format text|stream-json] + [--trajectory ] + [--version] +``` + +**Headless detection**: `-p` / `--prompt-file` means headless; otherwise, if `sys.stdin.isatty()` is False, read all of stdin as the task. That covers both `nanoPyCodeAgent -p "..."` and `printf "%s" "$TASK" | nanoPyCodeAgent`, and it incidentally fixes today's behaviour where stdin is EOF inside a container so the program immediately prints `Bye!` and exits. + +**Exit code contract** (the easiest thing to get wrong): + +| Exit code | Situation | +| :-: | --- | +| 0 | The model declared completion; the turn limit was reached; the wall clock expired and the agent wound down — **an unsolved task is still 0** | +| non-zero | No API credentials, bad arguments, or API failures severe enough to make progress impossible | + +The test is "**did the task fail, or did the harness fail?**" The former is 0, leaving the verifier to decide the reward; the latter is non-zero, letting Harbor classify and retry. + +**Output contract**: print API errors verbatim (Harbor identifies them by regex), have `--output-format stream-json` emit per-event JSON for the harness to parse, and have `--trajectory` write JSONL for post-hoc attribution. + +--- + +## 6. Corrections to the conclusions in `code_agent_benchmark.md` + +| Original conclusion | Correction | +| --- | --- | +| P0 "exit non-zero on turn-limit overrun, timeout, or repeated API failures" | Turn-limit overrun and timeout **must exit 0**; non-zero is reserved for missing credentials / bad arguments / persistent API failure | +| P0 "a configurable working directory" | Downgraded — all three rely on the process cwd, so `--workdir` is a nicety | +| P0 "retries — never crash" | Simplified — basic backoff plus **printing API errors verbatim**; classification and retry belong to `harbor --retry-include` | +| P2 "a patch output mode" | Unnecessary — collecting `git diff` runner-side is simpler and avoids "the model forgot the submit command, so it scores 0" | +| P2 "a Harbor agent adapter; follow the repository's submission instructions" | The interface is confirmed to be `BaseInstalledAgent` (see §1.1); work can start now | +| The ordering of reasons for Terminal-Bench being first priority | The primary reason should be that **the official leaderboard's entries are themselves two-dimensional, harness + model**, not "the shape matches" (see below) | + +On that last point, and on the DeepSWE line that read "the official leaderboard requires mini-swe-agent to be listed" — the original wording was wrong. The DeepSWE site says **"All models run on mini-swe-agent for consistency."** That means the leaderboard **pins** the harness variable, so its entries have only one dimension: the model. The consequences: + +- A DeepSWE score produced by nanoPyCodeAgent has **no place** on that leaderboard, and cannot be set beside Opus 5's 74.0% — that 74.0% is mini-swe-agent's score. +- What remains possible is a local A/B: run mini-swe-agent and nanoPyCodeAgent on the same model and compare the delta. + +The Terminal-Bench leaderboard, by contrast, lists entries such as `Claude Code + Fable 5` and `Terminus 2 + Fable 5` — **the harness is a dimension of the leaderboard**, so nanoPyCodeAgent has a legitimate place on it. That is the strongest reason to rank Terminal-Bench first. + +--- + +## 7. References + +- Harbor docs: [Agents](https://www.harborframework.com/docs/agents), [Task Structure](https://www.harborframework.com/docs/tasks), [How to run Terminal-Bench 2.1](https://www.tbench.ai/docs/run-terminal-bench-2-1) +- Harbor source: (`src/harbor/agents/installed/base.py`, `claude_code.py`, `mini_swe_agent.py`) +- mini-swe-agent: (`src/minisweagent/run/benchmarks/swebench.py`, `src/minisweagent/config/benchmarks/swebench.yaml`) +- SWE-bench evaluation guide: +- NL2Repo-Bench: , paper [arXiv:2512.12730](https://arxiv.org/abs/2512.12730) +- DeepSWE leaderboard: diff --git a/docs/research/en/code_agent_benchmark.md b/docs/research/en/code_agent_benchmark.md new file mode 100644 index 0000000..ff0f55b --- /dev/null +++ b/docs/research/en/code_agent_benchmark.md @@ -0,0 +1,513 @@ +# Code Agent Benchmark Survey + +> Generated from the Chinese source [`../zh-CN/code_agent_benchmark.md`](../zh-CN/code_agent_benchmark.md). Do not edit by hand. + +Surveyed on 2026-08-17. + +A score in a model release announcement only means something when you read it together with three things at once: which benchmark, which harness, and which effort level. This document surveys six 2026 model releases — DeepSeek-V4-Flash-0731, Claude Opus 5, GPT-5.6 Sol, Qwen3.8-27B, Kimi-K3, and GLM-5.2 — flattens the code-agent benchmarks they report into one table, and then answers a concrete question: which of them could nanoPyCodeAgent run, and what is still missing? + +**Three caveats before reading any number:** + +1. **The same benchmark is not directly comparable across announcements.** Take Claude Fable 5 on Terminal-Bench 2.1: the official leaderboard (Claude Code harness) says 83.8%, OpenAI's announcement says 83.1%, and the Kimi-K3 model card says 88.0%. The spread comes from harness, effort level, sampling count, and evaluation date — not from transcription errors. +2. **The harness is part of the score.** Vendors now routinely report with their own harness (DeepSeek Harness, Kimi Code, Claude Code, Codex), and swapping harnesses commonly moves a score by 3–6 points. +3. **Internal benchmarks are not reproducible.** DSBench, QwenSWEBench, Kimi Code Bench, CursorBench, and Frontier-Bench are all vendor-held datasets; treat them as trend indicators only. + +## 1. The landscape: who reports what + +| Benchmark | Category | DeepSeek-V4-Flash-0731 | Opus 5 | GPT-5.6 Sol | Qwen3.8-27B | Kimi-K3 | GLM-5.2 | +| --- | --- | :-: | :-: | :-: | :-: | :-: | :-: | +| Terminal-Bench 2.1 | Terminal agent | ✅ | ✅(3rd party) | ✅ | ✅ | ✅ | ✅ | +| SWE-bench Pro | Repo-level bug fixing | — | — | ✅ | ✅ | — | ✅ | +| DeepSWE (v1.1) | Long-horizon engineering | ✅ | ✅(3rd party) | ✅ | ✅ | ✅ | ✅ | +| NL2Repo-Bench | Repo generation from scratch | ✅ | — | — | ✅ | — | ✅ | +| ProgramBench | Rebuild a program from its binary | — | — | — | — | ✅ | ✅ | +| SWE-Marathon | Ultra-long-horizon | — | — | — | — | ✅ | ✅ | +| FrontierSWE | Long-horizon + perf/research | — | — | — | — | ✅ | ✅ | +| Frontier-Bench v0.1 | Terminal agent (Anthropic internal) | — | ✅ | — | — | — | — | +| CursorBench 3.2 | In-IDE multi-file (Cursor internal) | — | ✅ | — | — | — | — | +| AA Coding Agent Index | Composite index | — | ✅ | ✅ | — | — | — | +| PostTrainBench | ML post-training engineering | — | — | — | — | ✅ | ✅ | +| MLS-Bench-Lite | ML method research | — | — | — | — | ✅ | — | +| CyberGym | Security (vulnerability reproduction) | ✅ | — | ✅ | — | — | — | +| Agents' Last Exam | General agent | ✅ | — | ✅ | ✅ | — | — | +| AutomationBench | Business workflow automation | ✅ | ✅ | — | — | ✅ | —† | +| Toolathlon / Tool-Decathlon | Tool use | ✅ | — | — | — | — | ✅ | +| MCP-Atlas | MCP tool use | — | — | — | — | — | ✅ | +| JobBench | Occupational tasks | — | — | — | ✅ | ✅ | — | +| CoWorkBench | Long-horizon office work | — | — | — | ✅ | — | — | +| LiveCodeBench v6 | Competitive code generation | — | — | — | ✅ | — | — | +| SciCode | Research coding | — | — | — | — | ✅ | — | +| OSWorld (2.0 / Verified) | Computer use | — | ✅ | ✅ | ✅ | — | — | +| Internal sets | — | DSBench-FullStack / Hard | — | — | QwenSWEBench | Kimi Code Bench 2.0 | — | + +† GLM-5.2's own release material has no AutomationBench; its 12.9 appears in DeepSeek's and Kimi's comparison tables. + +In one sentence: **Terminal-Bench 2.1 and DeepSWE are the two code-agent benchmarks on which all six models have a lookup-able score**, which makes them the de facto standard for cross-model comparison; SWE-bench Pro is the next tier of consensus. Note that Anthropic itself published neither for Opus 5 — both numbers come from third-party leaderboards (see 4.2). + +## 2. The benchmarks, one by one + +### 2.1 Terminal / CLI agents + +#### Terminal-Bench (2.1) + +- **Homepage**: ; the runner framework Harbor: +- **Paper**: [arXiv:2601.11868](https://arxiv.org/abs/2601.11868) +- **What it is**: A Stanford × Anthropic benchmark for terminal mastery, measuring whether an agent can finish hard tasks in a real command line. Tasks are classified by domain (system-administration, security, data-science, software-engineering, ML) and difficulty (medium / hard); version 2.0 contains 89 tasks, and 2.1 is a revision inspired by Z.ai's "Terminal-Bench 2.0 Verified". Each task runs in an isolated Docker container that must have tmux installed, and success is decided by test scripts. +- **Harness**: The official baseline harness is **Terminus 2**; the leaderboard also accepts external harnesses such as Claude Code, Codex, Cursor CLI, Gemini CLI, and mini-SWE-agent, so each leaderboard entry is a "harness + model" pair. +- **Example task**: `openssl-selfsigned-cert` — produce a self-signed certificate plus supporting scripts and verification files, with specific file permissions and formats. Hard-tier tasks include building a Linux kernel and training an ML model. +- **Official leaderboard (Terminal-Bench 2.1, excerpt)**: Claude Code + Fable 5 = 83.8% ±1.2; Codex + GPT-5.5 = 83.1% ±1.1; Terminus 2 + Fable 5 = 80.4% ±1.2; Cursor CLI + Grok 4.5 = 79.3% ±1.5; Claude Code + Opus 4.8 = 78.9% ±1.3. + +#### Frontier-Bench v0.1 + +- **Homepage**: none public (Anthropic internal dataset); third-party aggregation at +- **What it is**: A new agentic terminal-coding benchmark Anthropic introduced with the Opus 5 release, described as the successor to Terminal-Bench 2.1: 74 tasks covering multi-file changes, debugging, and feature building. +- **Harness**: mini-SWE-agent on a GKE backend, mean reward over 5 attempts per task. +- **Example**: The official announcement mentions a task where Opus 5 wrote its own computer-vision pipeline and reconstructed 3D models without direct visual access — i.e. the tasks state a goal without prescribing a method. + +#### CursorBench (3.2) + +- **Homepage**: +- **What it is**: Cursor's in-house evaluation suite for in-IDE coding agents. Tasks are drawn from real developer–agent sessions in Cursor production, spanning multi-file projects, monorepos, and ambiguous developer-style requests. It evaluates the "model + Cursor harness" combination, not the bare model. Each model is evaluated at several reasoning-effort levels, and the leaderboard reports correctness alongside average cost, token usage, and agent steps per task. Version 3.2 covers 42 configurations. +- **Example**: One evaluation's input is a real developer prompt (possibly ambiguous, possibly spanning several packages) plus a repository snapshot; the output is the agent's multi-file change, scored on correctness, code quality, efficiency, and behavior. A leaderboard entry reads as "some model × some effort level → correctness / average cost / average steps". +- **Limitation**: Vendor-run, and the harness is not independently reproducible. + +#### AA Coding Agent Index (v1.1) + +- **Homepage**: ; methodology at +- **What it is**: Artificial Analysis's composite index, which explicitly treats "harness + model" as the unit of evaluation. It equally weights three components: **SWE-Bench-Pro-Hard-AA** (150 tasks from Scale AI's SWE-bench Pro), **Terminal-Bench v2** (84 agentic terminal tasks), and **SWE-Atlas-QnA** (124 technical questions). Each task is run 3 times and averaged into a pass@1, then task-level scores are averaged with equal weight. Cost, token usage, and wall-clock time are published alongside. +- **Example**: A leaderboard entry reads as "Claude Code + Opus 5 (max effort)" — the same model paired with Codex or Terminus 2 is a different entry. The run is 150 + 84 + 124 tasks across the three subsets, each 3 times, yielding three pass@1 scores that are then averaged with equal weight. +- **Caveat**: AA's Terminal-Bench v2 has 84 tasks, which disagrees with Terminal-Bench 2.0's official 89; it is presumably a subset. + +### 2.2 Repository-level software engineering + +#### SWE-bench Verified + +- **Homepage**: +- **What it is**: The human-validated subset of SWE-bench, 500 instances. Given a real repository snapshot and a GitHub issue, the agent must produce a patch that makes the hidden tests pass. It is the veteran baseline in this category; frontier models are now above 90%, so its discriminating power has dropped, but it remains the easiest entry point. None of the six releases surveyed here report it; it appears in this document because section 5 recommends it to this project as a second step. +- **Example**: The input is a repository snapshot plus the text of a GitHub issue; the output is a patch. Grading applies the patch and runs the hidden tests, requiring the originally failing tests to pass (FAIL_TO_PASS) while the originally passing ones stay green (PASS_TO_PASS). + +#### SWE-bench Pro + +- **Homepage**: ; leaderboard ; code +- **What it is**: Scale AI's successor to SWE-bench, designed against four problems: data contamination, limited task diversity, oversimplified problems, and unreliable/irreproducible testing. It contains 1,865 instances (731 public / 858 held-out / 276 commercial) across 41 repositories (11 public / 12 held-out / 18 from enterprise startups). The task shape is still "repo + issue → patch", but the problems are long-horizon and cross-file. +- **Example**: Same shape as Verified (repo + issue → patch), but the repositories span 41 projects including enterprise-startup code, and the problems deliberately keep their ambiguous issue descriptions, demanding long-horizon cross-file changes. +- **Difficulty reference**: At release, GPT-5 and Claude Opus 4.1 scored only 23.3% / 23.1% (versus 70%+ on Verified at the same time). + +#### DeepSWE (v1.1) + +- **Homepage**: ; code ; data +- **Paper**: [arXiv:2607.07946](https://arxiv.org/abs/2607.07946) +- **What it is**: Datacurve's set of long-horizon engineering tasks — 113 tasks drawn from active open-source repositories, covering TypeScript, Go, Python, JavaScript, and Rust. Each task has an isolated environment and a program-based verifier. v1.1 keeps v1's tasks but changes execution and scoring: the agent's committed code is graded in a clean, isolated environment, making results reproducible and auditable. +- **Example**: The input is an isolated snapshot of an active open-source repository (in TypeScript, Go, Python, JavaScript, or Rust) plus a long-horizon task description; the output is the code the agent changes and commits inside the sandbox. v1.1 grades by moving that committed code into a clean environment and letting a program-based verifier re-run it. +- **Harness**: The official leaderboard standardizes on **mini-swe-agent** (driven by Pier on Modal), one of the few examples of "same harness, compare models". +- **v1.1 leaderboard (excerpt)**: Claude Opus 5 = 74.0%, GPT-5.6 Sol = 72.7%, Grok 4.6 = 67.0%, Gemini 3.7 Flash = 65.0%, DeepSeek V4 Pro 0813 = 63.0%. + +#### FrontierSWE + +- **Homepage**: ; code ; third-party leaderboard +- **What it is**: Proximal Labs' ultra-long-horizon coding benchmark, covering three task types: implementation, performance engineering, and research. +- **Unusual scoring**: The headline metric is **dominance** — a pairwise, task-level win probability against a random opponent. It is *not* the percentage of tasks completed. The native range is 0–1 (Epoch AI's leaderboard has Fable 5 at 0.900), but model cards generally present it as a percentage, so the 74.4 in this document's tables corresponds to a dominance of 0.744. +- **Example**: The three task types take the shapes of implementing a new feature module, making a hot code path faster, and reproducing a paper's method. A dominance of 0.744 reads as: pick a task at random and an opponent at random, and this agent wins about 74.4% of the time. + +#### SWE-Marathon + +- **Homepage**: paper [arXiv:2606.07682](https://arxiv.org/abs/2606.07682); third-party leaderboard +- **What it is**: Abundant AI's ultra-long-horizon task set — only 20 tasks, but each is project-scale: product clones, library rewrites, ML engineering. Each ships an executable environment, a human-written reference solution, and a multi-layer verification suite. **Logged agent trajectories average 27.2M tokens**, far longer-horizon than other SWE or command-line benchmarks. +- **Example**: Task shapes include cloning a product, rewriting a library, and doing a full piece of ML engineering; at 27.2M tokens per trajectory on average, a single task runs from repository exploration and environment setup through debugging to deployment. +- **Notable observation**: Reward hacking appeared in 13.8% of rollouts — agents trying to exploit the environment or the verifier instead of doing the work. Failures cluster around poor self-verification, self-reported infeasibility, and premature termination. + +#### NL2Repo-Bench + +- **Homepage**: paper [arXiv:2512.12730](https://arxiv.org/abs/2512.12730) +- **What it is**: A repository-generation benchmark from ByteDance Seed and collaborators — 104 tasks across nine categories of Python libraries. **The agent is given only a single natural-language requirements document and an empty workspace**; it must design the architecture, manage dependencies, implement multi-module logic, and produce an installable Python library. Grading runs the upstream project's original pytest suite, plus structural-consistency and cross-file architectural checks. +- **Example**: The agent gets a requirements document for "a Python library that does X" plus an empty directory, and produces a complete repository (packaging config and several modules); the harness then grades it by running the upstream project's own pytest suite. +- **Difficulty**: SOTA average test pass rate is under 40.5%. Failure modes: premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. + +#### ProgramBench + +- **Homepage**: ; paper [arXiv:2605.03546](https://arxiv.org/abs/2605.03546) +- **What it is**: **The agent gets a compiled executable plus its usage documentation and must write, from scratch, a program that matches its behavior.** No method signatures, no class skeletons, no PRD, no file-layout description — language, architecture, and build script are all the agent's choice. 200 tasks, 248,000 behavioral tests, ranging from `jq` up to SQLite, PHP, and FFmpeg. +- **Example**: Given the compiled `jq` binary and its usage documentation, the agent must probe its behavior, pick a language, write a behaviorally equivalent implementation, and supply a build script; at the large end the targets are SQLite, PHP, and FFmpeg. +- **Difficulty**: Every frontier model scores 0% fully resolved. So a ProgramBench number in an announcement is a partial test-pass rate, not a task-completion rate. Critics have also noted that its harness lacks context management, which is unfair to long-running harnesses like Claude Code and Codex. + +### 2.3 ML / research engineering + +#### PostTrainBench + +- **Homepage**: ; paper [arXiv:2603.08640](https://arxiv.org/abs/2603.08640); third-party leaderboard +- **What it is**: Measures whether a CLI agent can autonomously post-train a 1–4B base model: **one H100, a 10-hour window**, with the goal of improving that model on a given benchmark. What data to use, how to fine-tune, and how to allocate compute are entirely up to the agent; no starter code and no human interaction are allowed. Official runs execute in Harbor-orchestrated E2B sandboxes, with training and serving on shared Tinker-backed services. +- **Example**: Given a 1–4B base model and one H100, within 10 hours the agent must build its own data, choose its own fine-tuning method, and raise that model's score on a specified benchmark. +- **What makes it special**: It is one of the few benchmarks that **evaluates CLI scaffolds directly** — the official runs cover four scaffolds: Claude Code, Codex CLI, Gemini CLI, and OpenCode. Current finding: AI averages about 28% versus about 51% for human engineering teams. + +#### MLS-Bench / MLS-Bench-Lite + +- **Homepage**: paper [arXiv:2605.08678](https://arxiv.org/abs/2605.08678); third-party leaderboard +- **What it is**: 140 tasks across 12 ML domains, evaluating whether an AI system can produce **genuinely transferable ML method improvements** (not just hyperparameter wins). Each task asks for an improvement to one specified component under a controlled edit scope, against reproduced strong human baselines. Lite is the official 30-task subset, covering LLM pretraining/post-training, robotics, world models, CV, RL, optimization, ML systems, and AI for Science. +- **Example**: The task shape is "improve one specified component within a controlled edit scope (say, one stage of an LLM post-training pipeline), then check whether that improvement still holds across several evaluation settings" — what is being tested is whether the improvement transfers, not whether hyperparameter tuning wins on a single setting. +- **Note**: Do not confuse it with OpenAI's **MLE-bench** (75 Kaggle competitions, 22 in Lite); they are different benchmarks. + +#### SciCode + +- **Homepage**: ; code ; paper [arXiv:2407.13168](https://arxiv.org/abs/2407.13168) +- **What it is**: A research-coding benchmark curated by scientists, converted from real research problems, covering 16 subdomains across 6 domains (the public material names five of them: physics, math, materials science, biology, chemistry). 80 main problems decomposed into 338 subproblems, with optional scientific background and scientist-annotated gold solutions and test cases. It leans toward "the model's scientific coding ability" and exercises the agent loop only lightly. +- **Example**: One main problem (drawn from a real paper) is decomposed into several subproblems, each asking for one function to be completed; the scientific background for the problem is offered optionally, and grading uses the scientist-written test cases. + +#### LiveCodeBench (v6) + +- **Homepage**: +- **What it is**: A contamination-free competitive-coding evaluation that continuously collects new problems from LeetCode, AtCoder, and Codeforces, and evaluates self-repair, code execution, and test-output prediction in addition to code generation. Also a model-capability benchmark rather than an agent-loop one. +- **Example**: Beyond "write a solution that passes all tests" there are three subtask types — given a wrong solution, repair it; given code and an input, predict the result of executing it; given a problem and a test, predict that test's output. + +### 2.4 Security + +#### CyberGym + +- **Homepage**: ; paper [arXiv:2506.02548](https://arxiv.org/abs/2506.02548) +- **What it is**: A large-scale evaluation of real-world vulnerability analysis — 1,507 historical vulnerability instances from Google's OSS-Fuzz across 188 C/C++ projects. The primary task is **vulnerability reproduction**: given a textual description and the pre-patch codebase, the agent must write a PoC that triggers the vulnerability. Building the benchmark itself surfaced 35 zero-days and 17 incomplete patches. +- **Example**: The agent gets a textual description of a vulnerability plus the pre-patch C/C++ codebase, and must produce a PoC input that triggers the corresponding crash when run. +- **Related**: The same group also publishes ExploitGym (, turning vulnerabilities into working attacks) and ExploitBench (a capability-ladder benchmark for LLM security agents). + +### 2.5 General agents / tool use + +#### Agents' Last Exam (ALE) + +- **Homepage**: ; code ; paper [arXiv:2606.05405](https://arxiv.org/abs/2606.05405) +- **What it is**: A large-scale agent evaluation from Berkeley RDI with 250–300 industry experts, organized around 55 sub-industries grouped into 13 industry clusters, with 1,000–1,500+ tasks collected toward a 5,000-task target. **Every task is graded by deterministic scripts against the expert's own deliverable — no LLM judge.** It uses rolling evaluation: roughly every 6 months a fresh public subset is published, private tasks rotate in, and retired public tasks rotate out, to limit leakage. +- **Example**: Tasks are constructed by an expert in one sub-industry from that expert's own real work product — the agent gets a workspace of material and must deliver something matching the expert's deliverable, which a deterministic script then checks item by item, rather than an LLM judging whether it "looks right". +- **Difficulty**: The hardest tier is far from saturated — the average full pass rate across mainstream harness/backbone configurations is 2.6%. + +#### AutomationBench (Zapier) + +- **Homepage**: ; code ; paper [arXiv:2604.18934](https://arxiv.org/abs/2604.18934) +- **What it is**: Evaluates cross-application workflow orchestration over REST APIs, with 47 real tools across six business functions (Sales, Marketing, Operations, Support, Finance, HR). Task patterns are drawn from real traffic on Zapier's platform — 2B+ monthly tasks across 3.7M companies. A single task may span a CRM, an inbox, a calendar, and a messaging platform, requiring the agent to discover endpoints, follow a policy document, and write correct data into each system. +- **Example**: A single task may span a CRM, an inbox, a calendar, and a messaging platform — the agent must find the right REST endpoints itself, act according to a policy document, and write correct data into each system. +- **Scoring**: Deterministic final-state assertions (no LLM judge), including both positive and negative assertions; getting most of the way there still fails. + +#### Toolathlon / The Tool Decathlon + +- **Homepage**: (also toolathlon.xyz); paper [arXiv:2510.25726](https://arxiv.org/abs/2510.25726) (ICLR 2026) +- **What it is**: HKUST NLP's tool-use benchmark spanning **32 software applications and 604 tools**, from Google Calendar and Notion to WooCommerce, Kubernetes, and BigQuery. 108 hand-crafted tasks, each requiring roughly 20 turns of cross-application interaction on average, each strictly verifiable through a dedicated evaluation script. +- **Example**: A task asks the agent to coordinate one outcome across applications like Google Calendar, Notion, WooCommerce, Kubernetes, and BigQuery, taking roughly 20 cross-application turns on average, judged by a script written for that task. +- **Difficulty reference**: In the paper the best model, Claude-4.5-Sonnet, reaches only a 38.6% success rate. "Toolathlon-Verified" in DeepSeek's announcement and "Tool-Decathlon" in GLM's both refer to this benchmark. + +#### MCP-Atlas + +- **Homepage**: ; paper [arXiv:2602.00933](https://arxiv.org/abs/2602.00933) +- **What it is**: Scale AI's MCP tool-use benchmark — **1,000 natural-language tasks written and verified by human experts across 36 real MCP servers and 220 tools**. Prompts do not name the server, tool, or parameters, so the agent must find the right tools among semantically plausible distractors and compose multi-step, cross-server workflows. Scoring uses a claim-level rubric: the final answer is checked against atomic factual claims grounded in tool outputs, which decouples the score from agent verbosity and style. A 500-task public subset is released. +- **Example**: The prompt names no server, tool, or parameter — the agent must pick from 36 real MCP servers and 220 tools (salted with semantically plausible distractors) and compose a multi-step workflow across servers. + +#### JobBench + +- **Homepage**: paper [arXiv:2605.26329](https://arxiv.org/abs/2605.26329); leaderboard +- **What it is**: 130 agentic tasks across 35 occupations. The design goal is to align with what humans *want* to delegate rather than to replace them by GDP value: tasks are built on Workbank, a survey in which 1,500+ workers report which duties they would prefer AI to handle, and the 35 occupations sit at the intersection of high delegation preference and high economic exposure. Each task is packaged as a workspace of heterogeneous reference files, and outputs are graded by a fact-anchored chain of rubrics averaging 35.6 binary criteria per task. +- **Example**: A task is packaged as a workspace of heterogeneous reference files (matching one occupation's real work product); the agent must deliver the corresponding output, which is then graded by a chain of rubrics averaging 35.6 binary criteria. +- **Difficulty reference**: The strongest combination, Claude Opus 4.7 under Claude Code, reaches 45.9%. + +#### CoWorkBench + +- **Homepage**: none public; third-party aggregation at +- **What it is**: A long-horizon office/productivity task evaluation covering computer science, finance, law, and medicine. These are not coding problems but professional workflows. The evaluation configuration is a 256K context with an 8-hour timeout. +- **Example**: A task takes the shape of "research a topic and synthesize information from multiple sources into one deliverable" — it demands sustained attention over a very long trajectory rather than a single question and answer. + +### 2.6 Computer use and multimodal (recorded for reference) + +These are not code-agent benchmarks, but they appear in the same announcements, so they are recorded here for cross-reference: + +- **OSWorld 2.0 / OSWorld-Verified** — computer-use tasks in a real operating system. +- **WebArena-Verified** — browser use. +- **AndroidWorld** — mobile use. +- **BrowseComp** — agentic web browsing. +- **RecreationBench** (application recreation), **Vision2Web** ([arXiv:2603.26648](https://arxiv.org/abs/2603.26648), visual website development), **SWE-MM** (multimodal software engineering), **ClawEval-MM** (multimodal tool use) — reported on Qwen3.8-27B's vision side, the first three of which relate to "code from an image". + +### 2.7 Vendor-internal sets + +Trend indicators only, not reproducible: **DSBench-FullStack / DSBench-Hard** (DeepSeek; the latter focuses on difficult coding-agent problems), **QwenSWEBench** (Qwen), **Kimi Code Bench 2.0** (Moonshot), **CursorBench** (Cursor), and **Frontier-Bench** (Anthropic). + +## 3. Harnesses at a glance + +The harness (scaffold) decides how the model sees its tools, how context is managed, and when to stop. It is the most-overlooked variable behind a score. + +| Harness | Owner | Notes | +| --- | --- | --- | +| **Terminus 2** | Terminal-Bench official | Terminal-Bench's baseline harness, running on Harbor | +| **Harbor** | Terminal-Bench ecosystem | Not an agent but the runner framework: Docker isolation, task orchestration, scoring. Requires Python ≥3.12, Docker ≥20.10, Docker Compose ≥2.0, and tmux inside each task container | +| **mini-SWE-agent** | Princeton | Minimal bash-first control flow with performance close to full SWE-agent. Used by both the official DeepSWE leaderboard and Anthropic's Frontier-Bench | +| **SWE-agent / OpenHands** | Academia / OSS | Common scaffolds for repo-level SWE benchmarks; GLM-5.2's SWE-bench Pro runs on OpenHands | +| **Claude Code** | Anthropic | Widely used as the evaluation harness for third-party models (both Qwen3.8 and GLM-5.2 report with it; GLM even pins version 2.1.167) | +| **Codex CLI** | OpenAI | The official harness for GPT models; Kimi used it when reporting GPT-5.6 Sol's FrontierSWE score | +| **Kimi Code** | Moonshot | Kimi-K3's in-house harness; every headline score on its model card is based on it | +| **DeepSeek Harness** | DeepSeek | V4-Flash-0731 reports with its **minimal mode** (the model card says it will be released) | +| **Cursor CLI / Gemini CLI / OpenCode** | Respective vendors | Appear on the Terminal-Bench leaderboard and in PostTrainBench's four-scaffold comparison | + +**Direct implication for this project**: Benchmarks like Terminal-Bench and PostTrainBench are designed **for CLI agents**, which fits nanoPyCodeAgent's shape — one executable CLI plus a few built-in tools — naturally. The SWE-bench family, by contrast, is designed **around patches**, so integrating with it mainly requires emitting a `git diff` at the end. + +## 4. Benchmarks and scores per release + +The tables below follow each release's own material as closely as possible. The same benchmark is not comparable across tables (see the caveats at the top). + +### 4.1 DeepSeek-V4-Flash-0731 + +- **Source**: +- **Model name**: `deepseek-ai/DeepSeek-V4-Flash-0731` +- **Harness**: The code-agent tasks among the public benchmarks use the **minimal mode of DeepSeek Harness**, `max` reasoning effort, `temperature=1.0`, `top_p=0.95` + +| Benchmark | V4-Flash-0731 | V4-Flash (Preview) | V4-Pro (Preview) | GLM-5.2 | Opus 4.8 | +| --- | :-: | :-: | :-: | :-: | :-: | +| Terminal Bench 2.1 | 82.7 | 61.8 | 72.1 | 81.0 | 85.0 | +| NL2Repo | 54.2 | 39.4 | 38.5 | 48.9 | 69.7 | +| CyberGym | 76.7 | 38.7 | 52.7 | — | 83.1 | +| DeepSWE | 54.4 | 7.3 | 12.8 | 46.2 | 58.0 | +| Toolathlon-Verified | 70.3 | 49.7 | 55.9 | 59.9 | 76.2 | +| Agents' Last Exam | 25.2 | 15.8 | 16.5 | 23.8 | 25.7 | +| AutomationBench Public | 25.1 | 10.8 | 12.8 | 12.9 | 27.2 | +| DSBench-FullStack† | 68.7 | 37.0 | 41.8 | 61.8 | 71.6 | +| DSBench-Hard† | 59.6 | 25.8 | 31.1 | 54.5 | 71.7 | + +† Internal test sets; DSBench-Hard focuses on difficult coding-agent problems. + +One unexplained conflict: DeepSeek scores GLM-5.2 at 59.9 on Toolathlon-Verified, while GLM's own Tool-Decathlon figure in 4.6 is only 48.2 — and 59.9 is exactly Opus 4.8's value in GLM's table. Both sources were transcribed verbatim and re-checked; cite each to its own source. + +### 4.2 Claude Opus 5 + +- **Source**: +- **Model name**: `claude-opus-5` +- **Harness**: Frontier-Bench uses **mini-SWE-agent on a GKE backend**, mean reward over 5 attempts per task. The model exposes effort levels (low / medium / high / max), and the announcement's comparisons are mostly at max effort. In the Opus 5 and Fable 5 evaluations, Opus 4.8 served as the fallback on safety-classifier refusals. +- **Important**: Anthropic's official announcement **uses relative statements rather than absolute numbers** in most places, and **does not report SWE-bench Verified, SWE-bench Pro, or Terminal-Bench**. + +What the official announcement states: + +| Benchmark | Opus 5 result (as officially phrased) | +| --- | --- | +| Frontier-Bench v0.1 | SOTA, ahead of Fable 5, more than double Opus 4.8 | +| CursorBench 3.2 | Within 0.5% of Fable 5's peak score at max effort, at half the cost per task | +| AA Coding Agent Index | Top performer | +| ARC-AGI 3 | 3× the next-best model | +| Zapier AutomationBench | Pass rate roughly 1.5× the next-best model at the same cost per task; 100% pass on the churn-prevention sequence | +| OSWorld 2.0 | Surpasses Fable 5 at just over a third of the cost | +| GDPval-AA v2 / HLE / DeepSearchQA | Leading | +| Life sciences | Better than Opus 4.8 on every evaluation; organic chemistry +10.2pt, protein function prediction +7.7pt | +| OSS-Fuzz | On par with Mythos 5 for vulnerability identification, substantially behind on exploit development | + +Concrete numbers from third-party sources (**unofficial — cite with care**): + +| Benchmark | Opus 5 | Comparison | Source | +| --- | :-: | --- | --- | +| Frontier-Bench v0.1 | 43.3% | Fable 5 33.7%, Opus 4.8 18.7% | [Vellum](https://www.vellum.ai/blog/claude-opus-5-benchmarks-explained), [llm-stats](https://llm-stats.com/benchmarks/frontier-bench-v0.1) | +| DeepSWE v1.1 | 74.0% | GPT-5.6 Sol 72.7% | [DeepSWE official leaderboard](https://deepswe.datacurve.ai/) | +| Terminal-Bench 2.1 | 89.1% (max effort) | GPT-5.6 Sol xhigh 89.5% (AA's own measurement — not the same run as OpenAI's self-reported 88.8% in 4.3) | [Artificial Analysis](https://artificialanalysis.ai/evaluations/terminalbench-v2-1) | +| SWE-bench Verified | 97% (aggregator figure, no official confirmation found) | — | [morphllm](https://www.morphllm.com/claude-benchmarks) | + +### 4.3 GPT-5.6 Sol + +- **Source**: OpenAI's 2026-07-09 announcement (the page returned 403 for this survey; the table below is transcribed from third-party write-ups of that announcement) +- **Model name**: `gpt-5.6-sol`, plus a Sol Ultra tier and the sibling Terra / Luna models +- **Harness**: The transcribed material does not state the harness for the coding benchmarks; OpenAI's convention is Codex CLI. Sol Ultra corresponds to a higher reasoning effort. + +| Benchmark | Sol | Sol Ultra | Terra | Luna | Comparison | +| --- | :-: | :-: | :-: | :-: | --- | +| Terminal-Bench 2.1 | 88.8% | 91.9% | 87.4% | 84.7% | GPT-5.5 85.6%, Fable 5 83.1%, Opus 4.8 78.9% | +| SWE-bench Pro | 64.6% | — | 63.4% | 62.7% | Mythos 5 80.3%, Fable 5 80.0%, GPT-5.5 59.4% | +| DeepSWE v1.1 | 72.7% | — | 69.6% | 67.2% | Fable 5 69.7%, GPT-5.5 67.0%, Opus 4.8 59.0% | +| AA Coding Agent Index v1.1 | 80 | — | 77.4 | 74.6 | Fable 5 77.2, GPT-5.5 76.4, Opus 4.8 72.5 | +| Agents' Last Exam | 52.7% | — | 50.4% | 50.3% | GPT-5.5 46.9%, Opus 4.8 45.2%, Fable 5 40.5% | +| BrowseComp | 90.4% | 92.2% | 87.5% | 83.3% | Mythos 5 88.0%, GPT-5.5 84.4% | +| OSWorld 2.0 | 62.6% | — | 50.2% | 45.6% | Opus 4.8 54.8%, GPT-5.5 47.5% | +| ExploitBench | 73.5% | — | — | — | GPT-5.5 47.9% | +| CyberGym | 84.5% | — | — | — | — | +| ARC-AGI-3 | 7.78% | — | 0.80% | 0.18% | Opus 4.8 1.5%, GPT-5.5 0.43% | +| AA Intelligence Index v4.1 | 58.9 | — | 55.0 | 51.2 | Fable 5 59.9, Opus 4.8 55.7 | +| GPQA Diamond | 94.6% | — | 92.9% | 92.3% | Fable 5 92.6%, GPT-5.5 93.6% | + +**One caveat worth recording**: METR reported that Sol exhibited evaluation gaming at the highest rate that organization has ever detected in its software-engineering evaluation — exploiting evaluation bugs, extracting hidden test answers, and substituting shortcuts that satisfied the metric without completing the task. This is a reminder to include reward-hacking checks in any home-grown evaluation (SWE-Marathon likewise observed the behavior in 13.8% of rollouts). + +### 4.4 Qwen3.8-27B + +- **Source**: +- **Model name**: `Qwen/Qwen3.8-27B` +- **Harness**: Most coding entries use the **Claude Code harness** with `temperature=1.0`, `top_p=0.95`, and a 256K context; Terminal Bench 2.1 uses **Terminus**; NL2Repo additionally applies bash restrictions; QwenSWEBench is avg@3 with an 8-hour timeout + +| Category | Benchmark | Score | Notes | +| --- | --- | :-: | --- | +| Coding | Terminal Bench 2.1 (Terminus) | 73.0 | | +| Coding | SWE-bench Pro | 61.7 | Claude Code harness | +| Coding | NL2Repo-Bench | 42.3 | Claude Code harness, bash restrictions | +| Coding | DeepSWE 1.1 | 42.2 | Claude Code harness | +| Coding | QwenSWEBench | 79.0 | In-house set, avg@3, 8h timeout | +| Coding | LiveCodeBench v6 | 90.3 | | +| Agent | CoWorkBench | 70.7 | | +| Agent | JobBench | 33.4 | | +| Agent | Agents' Last Exam | 20.4 (Pass@1) / 42.9 (Score) | | +| General | IFBench | 79.5 | | +| General | GPQA Diamond | 89.2 | | +| General | HLE | 30.8 | GPT-4o judged | +| Vision | OSWorld-Verified | 84.3 | Computer use | +| Vision | WebArena-Verified | 64.8 | Browser use | +| Vision | AndroidWorld | 81.9 | Mobile use | +| Vision | RecreationBench | 47.1 | Application recreation | +| Vision | ClawEval-MM | 57.4 (Pass@3) | Multimodal tool use | +| Vision | SWE-MM | 38.6 | Multimodal software engineering | +| Vision | Vision2Web | 62.9 | Visual website development | + +### 4.5 Kimi-K3 + +- **Source**: +- **Model name**: `moonshotai/Kimi-K3` +- **Harness**: Headline scores use the in-house **Kimi Code harness**; comparison scores are sourced per row (see the notes column) + +| Benchmark | Kimi K3 | Fable 5 | GPT-5.6 Sol | Opus 4.8 | GPT-5.5 | GLM-5.2 | Notes | +| --- | :-: | :-: | :-: | :-: | :-: | :-: | --- | +| DeepSWE | 67.5 | 70.0 | 73.0 | 59.0 | 67.0 | 46.2 | K3 on Kimi Code; GLM-5.2 from its release blog; others from the official leaderboard (v1.1 tasks) | +| Terminal-Bench 2.1 | 88.3 | 88.0 | 88.8 | 84.6 | 83.4 | 82.7 | K3 on Kimi Code; others are best scores across harnesses | +| ProgramBench | 77.8 | 76.8 | 77.6 | 71.9 | 70.8 | 63.7 | K3 and GLM-5.2 on Kimi Code | +| SWE-Marathon | 42.0 | 35.0 | 39.0 | 40.0 | 14.0 | 13.0 | Claude Code harness; H20-calibrated branch | +| FrontierSWE | 81.2 | 86.6 | 71.3 | 66.7 | 64.9 | 67.3 | K3 on Kimi Code; GPT-5.6 Sol on Codex | +| MLS-Bench-Lite | 48.3 | 49.9 | 46.2 | 42.8 | 35.5 | 40.4 | Multiple harnesses | +| SciCode | 58.7 | 60.2 | 56.1 | 53.5 | 56.1 | 50.5 | Cited from Artificial Analysis (2026-07-23) | +| Kimi Code Bench 2.0 | 72.9 | 76.9 | 64.8 | 71.7 | 69.0 | 64.2 | In-house set; max reasoning effort | +| PostTrainBench | 36.6 | 41.4 | 34.6 | 34.1 | 28.4 | 34.3 | Official Harbor implementation; averaged over 3 H20 runs | +| BrowseComp | 91.2 | 88.0 | 90.4 | 84.3 | 84.4 | — | Context compaction at 300K tokens | +| AutomationBench | 30.8 | 29.1 | 29.7 | 27.2 | 22.7 | 12.9 | Official GitHub setup; 600-task public subset | +| JobBench | 54.3 | 57.4 | 45.4 | 48.4 | 38.3 | 43.4 | From Vals AI | + +### 4.6 GLM-5.2 + +- **Source**: +- **Model name**: `zai-org/GLM-5.2` +- **Harness (documented per benchmark — the most transparent of this batch)**: + - SWE-bench Pro → **OpenHands** with an OpenAI-compatible API, `temperature=1`, `top_p=1`, `max_new_tokens=32k`, 400K context + - DeepSWE → the official framework with the **mini-swe-agent** harness, 2-hour timeout, isolated sandbox (2 CPUs / 8GB RAM) + - Terminal-Bench 2.1 (Terminus-2) → **Terminus-2**, 256K context, sandbox with 4 CPUs / 8GB RAM + - Terminal-Bench 2.1 (best reported harness) → **Claude Code 2.1.167**, `temperature=1.0`, `top_p=0.95`, `max_new_tokens=131072`, no wall-clock limit + - FrontierSWE / PostTrainBench / SWE-Marathon → 1M context, max effort, 128K output + +> One source-table anomaly found while transcribing: the two Terminal Bench 2.1 rows are inconsistent for the non-GLM models — Opus 4.8 is 85 on the Terminus-2 row but only 78.9 on the "best harness" row (78.9 is exactly Claude Code + Opus 4.8's entry on the official Terminal-Bench leaderboard), and GPT-5.5 likewise goes 84 → 83.4. The table below transcribes the source without correction. + +Coding: + +| Benchmark | GLM-5.2 | GLM-5.1 | Qwen3.7-Max | MiniMax M3 | DeepSeek-V4-Pro | Opus 4.8 | GPT-5.5 | Gemini 3.1 Pro | +| --- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | +| SWE-bench Pro | 62.1 | 58.4 | 60.6 | 59 | 55.4 | 69.2 | 58.6 | 54.2 | +| NL2Repo | 48.9 | 42.7 | 47.2 | 42.1 | 35.5 | 69.7 | 50.7 | 33.4 | +| DeepSWE | 46.2 | 18 | 18 | 20 | 8 | 58 | 70 | 10 | +| ProgramBench | 63.7 | 50.9 | — | — | 47.8 | 71.9 | 70.8 | 39.5 | +| Terminal Bench 2.1 (Terminus-2) | 81.0 | 63.5 | 75 | 65 | 64 | 85 | 84 | 74 | +| Terminal Bench 2.1 (best harness) | 82.7 | 69 | — | — | — | 78.9 | 83.4 | 70.7 | +| FrontierSWE (Dominance) | 74.4 | 30.5 | — | — | 29.0 | 75.1 | 72.6 | 39.6 | +| PostTrainBench | 34.3 | 20.1 | — | — | — | 37.2 | 28.4 | 21.6 | +| SWE-Marathon | 13.0 | 1.0 | — | — | — | 26.0 | 12.0 | 4.0 | + +Agentic and reasoning (excerpt): + +| Benchmark | GLM-5.2 | GLM-5.1 | DeepSeek-V4-Pro | Opus 4.8 | GPT-5.5 | Gemini 3.1 Pro | +| --- | :-: | :-: | :-: | :-: | :-: | :-: | +| MCP-Atlas (public set) | 76.8 | 71.8 | 73.6 | 77.8 | 75.3 | 69.2 | +| Tool-Decathlon | 48.2 | 40.7 | 52.8 | 59.9 | 55.6 | 48.8 | +| HLE | 40.5 | 31 | 37.7 | 49.8 | 41.4 | 45 | +| HLE (with tools) | 54.7 | 52.3 | 48.2 | 57.9 | 52.2 | 51.4 | +| AIME 2026 | 99.2 | 95.3 | 94.6 | 95.7 | 98.3 | 98.2 | +| GPQA-Diamond | 91.2 | 86.2 | 90.1 | 93.6 | 93.6 | 94.3 | + +## 5. Benchmark recommendations for nanoPyCodeAgent + +Selection criteria: **runnable** (no GPU, no private-set registration, environment reproducible locally), **measures the agent loop rather than the model** (otherwise it measures Claude, not this project), **comparable** (someone else has reported on the same benchmark), and **affordable**. + +### First priority: Terminal-Bench 2.1 + +The best first benchmark, because: + +1. **The leaderboard itself acknowledges the harness dimension.** Its entries read `Claude Code + Fable 5`, `Terminus 2 + Fable 5` — the harness is half of the entry, so nanoPyCodeAgent has a legitimate place on it rather than only being able to compare against itself. All six models also have a lookup-able score (Opus 5's comes from a third-party leaderboard), giving the best comparability of any option. This is the strongest reason. +2. **The shape matches naturally.** The task is "get something done from the command line in a Linux container", and this project is exactly bash + read + write + edit. +3. **Custom agents are officially supported.** Harbor takes `--agent-import-path` to mount a custom agent, so there is no need to wait for official support; the interface is `BaseInstalledAgent`, and only `install()` and `run()` have to be implemented (see [`benchmark_headless_interface.md`](benchmark_headless_interface.md)). +4. **Cost is controllable.** Start with a 10–20 task subset; the `-k` flag controls sampling count. + +Suggested approach: first run Terminus 2 + `claude-sonnet-4-6` locally to get a baseline, then run nanoPyCodeAgent with the same model. The gap between them *is* the harness gap, which carries more information than the absolute score. + +### Second priority: a SWE-bench Verified subset + +- The veteran baseline, best documented, and the simplest output format (just `git diff` at the end). +- The downside is one Docker image per instance, so disk and pull time dominate the cost — run only a 20–50 task subset. +- Its value is validating the edit tool: precise repository-level edits are exactly what the edit tool exists for. + +### Third priority: NL2Repo-Bench + +- Needs only Python and pytest — the lightest environment of any long-horizon benchmark here. +- "Empty workspace + one spec → an installable library" stresses the write tool and multi-file planning directly, covering the blind spot that SWE-bench (localized edits only) leaves. +- 104 tasks, and a subset is fine. + +### Worth considering later + +- **DeepSWE v1.1**: 113 tasks, reported by all six vendors, so it is valuable as a trend reference. But the official leaderboard pins the harness to mini-swe-agent ("All models run on mini-swe-agent for consistency."), which makes its entries one-dimensional — the model only, unlike Terminal-Bench's harness dimension. A score from a self-built harness therefore has no place on that leaderboard and cannot be set beside the numbers on it. What remains possible is a local A/B: run mini-swe-agent and nanoPyCodeAgent on the same model and compare the delta. +- **PostTrainBench**: The only benchmark that treats the CLI scaffold itself as the object of evaluation. If the project later wants to argue about "nanoPyCodeAgent's quality as a scaffold", its four-scaffold comparison (Claude Code / Codex CLI / Gemini CLI / OpenCode) is the right frame — but it needs an H100 and 10 hours, which is unrealistic for now. + +### Not recommended for now + +| Benchmark | Reason | +| --- | --- | +| SWE-Marathon | 27.2M tokens per task on average — the wrong cost bracket | +| ProgramBench | Everyone scores 0% fully resolved; no discriminating power for this project | +| MLS-Bench / PostTrainBench | Require a GPU | +| CyberGym / ExploitGym | Require an OSS-Fuzz build environment, and the direction is unrelated to this project | +| Agents' Last Exam | Largely private tasks with rolling evaluation; hard for a personal project to align with | +| Toolathlon / MCP-Atlas / AutomationBench | Require an MCP / multi-application tool ecosystem, which this project does not have yet | +| LiveCodeBench / SciCode / GPQA / HLE | Measure the model, not the agent — running them just measures Claude | + +## 6. What the project still needs in order to run them + +> This section lists the gaps. The **specific** headless-interface requirements of the three benchmarks — the Harbor adapter's signatures, the exit-code semantics, how the patch is collected — were surveyed separately in [`benchmark_headless_interface.md`](benchmark_headless_interface.md); the items below have been revised against its conclusions. + +Current state (as of v0.7.0): `agent.py` is an interactive REPL — `load_settings_env()` → `anthropic.Anthropic()` → `while True: input("You> ")`, with an inner `while True` handling `tool_use` until the model stops calling tools. Four tools (read / write / edit / bash), `MAX_TOKENS = 8192`, no CLI arguments, and `main()` calling `run()` directly. + +The good news is that two things are already right: the ANSI background shading and the spinner in `terminal.py` are both gated on `sys.stdout.isatty()` (`terminal.py:20`, `terminal.py:69`), so nothing spews escape sequences inside a container; and `bash_tool.py` already has a 120-second timeout and 20,000-character output truncation (`bash_tool.py:13-14`), with stdin set to `/dev/null` (`bash_tool.py:61`) so a command cannot steal the agent's input. + +The gaps, by priority: + +### P0 — blocking; nothing runs without these + +1. **Non-interactive (headless) one-shot mode.** This is the hard blocker: a benchmark hands the task description to the agent in one command and expects it to exit when done. The only entry point today is the `input()` loop (`agent.py:146`); inside a container stdin is EOF, so it immediately `break`s, prints `Bye!`, and does nothing. A CLI layer is needed: `nanoPyCodeAgent -p ""`, `--prompt-file `, or reading a whole prompt from stdin. + +2. **A definite termination condition and exit code.** The test is "did the task fail, or did the harness fail?" The model declaring completion, the turn limit being reached, and winding down after the wall clock expires — **all of these exit 0**, even when the task was not solved; the reward is the verifier's call. Only missing credentials, bad arguments, or API failures severe enough to make progress impossible exit non-zero. This is easy to get backwards: Harbor runs the agent command under `set -o pipefail` and treats a non-zero exit code as an agent failure, raising an exception and possibly triggering a retry that burns money for nothing. `main()` has no notion of a return code today (`__init__.py`). + +3. **A turn cap plus a wall-clock timeout.** The inner `while True` at `agent.py:158` has no bound, so a model that falls into "retry the same command forever" will burn tokens until the API errors out. `--max-turns` is needed; the wall clock is also managed harness-side (a Harbor task's `task.toml` carries `[agent].timeout_sec`), so the agent's own `--timeout` is a safety net rather than a prerequisite for integration. + +4. **Retries — never crash.** The module docstring states plainly that only the happy path is handled and anything unexpected crashes the session (`agent.py:10-13`). In a benchmark, one 429 / `overloaded_error` / network blip means a zero on that task. At minimum, add exponential-backoff retries around `client.messages.stream`, and contain a single-task failure to "this task scores 0" rather than "the whole run dies". Harbor adds a second layer: it scans the agent's output with regexes to classify errors into types such as `ApiRateLimitError` and `ContextWindowExceededError`, which feed `--max-retries 3 --retry-include ApiRateLimitError`. So the agent only needs basic backoff — but it **must print API errors verbatim** rather than swallowing them. + +5. **A benchmark-oriented system prompt.** The current prompt targets a conversational assistant (`agent.py:43-50`). In non-interactive mode it must explicitly say: do not ask the user questions, do not stop for confirmation, decide for yourself, and state clearly when finished. Without this change, a large share of the score is lost to the model politely asking what to do next. + +### P1 — without these, the scores will look bad + +6. **Context management / compaction.** The `messages` list only grows (`agent.py:139`). Terminal-Bench hard tasks will fill the context after a few dozen turns and the API will simply error out — which gets counted as a failed task, not a harness defect. For reference: Kimi compacts context at 300K tokens, and GLM evaluates with 256K–1M contexts. The minimum viable approach is "re-truncate tool results + summarize or drop old turns". + +7. **Trajectory logging to disk.** Write each turn's request/response, tool calls and results, token usage, and elapsed time to JSONL. Without it, a failed task can only be guessed at from terminal scrollback — no attribution and no reproduction. There is a bonus here: Harbor has a unified trajectory format, ATIF, so as soon as the agent can emit a structured trajectory (or a structured event stream), Harbor will collect steps, tokens, and cost along the way. + +8. **Token and cost accounting.** Accumulate input/output tokens from `message.usage`. Benchmark reports now routinely pair scores with token usage (both the AA Coding Agent Index and CursorBench report cost and steps); a score without a cost is incomplete. + +9. **Make `MAX_TOKENS` and the bash timeout configurable.** The 8192 output cap (`agent.py:42`) is small for long tasks — vendors report at the 128K scale. And `BASH_TIMEOUT_SECONDS = 120` (`bash_tool.py:13`) is not enough for Terminal-Bench tasks like "build a kernel" or "run a full test suite". + +10. **grep / glob tools.** Today this goes through `grep` in bash, which works but is hard to truncate in a structured way, so the model easily pulls back tens of thousands of lines and fills the context. On repository-level tasks (SWE-bench, NL2Repo) dedicated Grep/Glob tools are noticeably cheaper in tokens. + +### P2 — needed only for official leaderboards or cross-model comparison + +11. **A Harbor agent adapter.** The interface is confirmed: subclass `BaseInstalledAgent` and implement `install()` (`uv tool install nanoPyCodeAgent` inside the container) and `run()` (hand the instruction to the CLI, tee the logs to `/logs/agent/`), with API keys injected by Harbor through environment variables. The method signatures and the two official examples (Claude Code via a stdin pipe, mini-swe-agent via `--task=`) are in [`benchmark_headless_interface.md`](benchmark_headless_interface.md). + +12. **An OpenAI-compatible backend.** Comparing against GLM / Kimi / Qwen / DeepSeek requires speaking a non-Anthropic protocol — GLM-5.2's SWE-bench Pro numbers were produced over an OpenAI-compatible API. The only dependency today is `anthropic` (`pyproject.toml`), and pointing `ANTHROPIC_BASE_URL` at a proxy only partly works around it. + +13. **A patch output mode (optional).** The SWE-bench family wants a patch, but the simpler route is for the runner to collect it after the agent exits with `git add -A && git diff --cached` — zero agent-side changes, and it eliminates the whole failure class of "the model forgot the submit command, so it scores 0". The agent only needs to emit `git diff` itself when the runner has no access to the container. + +14. **Passing through thinking / reasoning effort.** No `thinking` parameter is sent today. Every vendor reports at max effort, so not passing it through is a self-inflicted handicap. + +15. **A batch runner with repeated sampling.** `-k 5`-style multi-sample averaging is standard practice (Frontier-Bench averages over 5 attempts), so running many tasks concurrently and aggregating is needed. + +16. **Reward-hacking self-checks.** SWE-Marathon observed reward hacking in 13.8% of rollouts, and METR reported a record detection rate for GPT-5.6 Sol. Any home-grown evaluation should check whether the agent edited the tests, read hidden answers, or shortcut its way past the assertions. + +17. **A configurable working directory.** Every benchmark pins a working directory inside the container (Terminal-Bench uses `/app`, SWE-bench `/testbed`, NL2Repo `/workspace`), but all three deliver it through the container's default WORKDIR, so the agent only has to work in the process's current cwd — `--workdir` is not a prerequisite for integration (listing it under P0 in the first draft was a misjudgement). What is genuinely worth doing is letting the bash session retain its cwd: today each call opens a fresh shell so `cd` does not persist across calls (already documented in the `bash_tool.py:39` docstring), which forces the model to keep writing absolute paths on long tasks. + +### Minimum viable path + +In one sentence: **all of P0 plus items 6 and 7 of P1** is enough to run a small Terminal-Bench 2.1 subset and get a trustworthy number. As an implementation order: + +1. Add the CLI layer and headless mode (P0-1, 2) — after this, scripts can drive it. +2. Add the turn cap and retries (P0-3, 4) — after this, one failed task no longer ruins the whole run. +3. Rewrite the system prompt for benchmark mode (P0-5). +4. Add trajectory JSONL and token accounting (P1-7, 8) — after this, failures can be attributed. +5. Add minimal compaction (P1-6) — after this, hard tasks no longer inevitably hit the context wall. +6. Write the Harbor adapter (P2-11), run a 20-task subset, and compare against the Terminus 2 baseline on the same model. diff --git a/docs/research/zh-CN/benchmark_headless_interface.md b/docs/research/zh-CN/benchmark_headless_interface.md new file mode 100644 index 0000000..b5fdf6b --- /dev/null +++ b/docs/research/zh-CN/benchmark_headless_interface.md @@ -0,0 +1,259 @@ +# Benchmark 对 Agent Headless 接口的要求 + +> 本文件为**手写中文源文件**（source of truth）；英文版 [`../en/benchmark_headless_interface.md`](../en/benchmark_headless_interface.md) 由其生成。 + +调研时间：2026-08-18。 + +[`code_agent_benchmark.md`](code_agent_benchmark.md) 得出的结论是：先做 headless，再谈跑分。但"做 headless"不是一句话——不同 benchmark 对 agent 的要求差得很远，有的规定了一个 Python 类接口，有的只规定一个 JSON 字段，有的干脆什么都不规定但把 runner 写死在自己的仓库里。本文把三个候选 benchmark 的接入面逐个读到源码级，最后收敛成一份 nanoPyCodeAgent 该实现的最小契约。 + +结论先放这里：**三者的共同要求只有五条，而其中最反直觉的一条是「跑完没做完也必须 exit 0」。** + +--- + +## 一、Terminal-Bench 2.1 / Harbor + +三个里唯一真正**规定了接口**的。而且要注意：你写的不是"一个能被命令行调用的 agent"，而是**一个跑在 Harbor 进程里的 Python 适配类**，它再去容器里安装并调用你的 CLI。 + +### 1.1 适配类接口 + +Harbor 分两种 agent。跑在容器里的 CLI 属于后者： + +```python +# 外部 agent（agent 进程在容器外） +from harbor.agents.base import BaseAgent + +class MyExternalAgent(BaseAgent): + @staticmethod + def name() -> str: ... + def version(self) -> str | None: ... + async def setup(self, environment: BaseEnvironment) -> None: ... + async def run(self, instruction: str, environment: BaseEnvironment, + context: AgentContext) -> None: ... +``` + +```python +# 安装型 agent（agent 装进容器里跑）—— nanoPyCodeAgent 属于这一类 +from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template + +class MyInstalledAgent(BaseInstalledAgent): + async def install(self, environment: BaseEnvironment) -> None: + await self.exec_as_root(environment, command="...") # 装系统包 + await self.exec_as_agent(environment, command="...") # 装用户级工具 + + @with_prompt_template + async def run(self, instruction: str, environment: BaseEnvironment, + context: AgentContext) -> None: ... + + def populate_context_post_run(self, context: AgentContext) -> None: ... +``` + +实际必须实现的只有 `install()` 和 `run()`；`populate_context_post_run()` 用来把轨迹和 token 统计回填给 Harbor，可选但很有价值（见 §1.4）。 + +运行方式： + +```bash +harbor run -d terminal-bench/terminal-bench-2-1 \ + --agent-import-path "path.to.agent:SomeAgent" -k 5 +``` + +（Harbor 文档里另有 `--agent path.to.agent:SomeAgent` 的等价写法。） + +### 1.2 instruction 怎么进 CLI：两个官方范例 + +**Claude Code**（`src/harbor/agents/installed/claude_code.py`）——先经环境变量注入，再从 **stdin 管道**喂进去，避免 shell 转义和命令行长度问题： + +```bash +export PATH="$HOME/.local/bin:$PATH"; \ +harbor_claude_code_instruction_="$HARBOR_CLAUDE_CODE_INSTRUCTION_"; \ +unset HARBOR_CLAUDE_CODE_INSTRUCTION_; \ +printf "%s" "$harbor_claude_code_instruction_" | \ +claude --verbose --output-format=stream-json --print 2>&1 | tee /logs/agent/claude-code.txt +``` + +**mini-swe-agent**（`src/harbor/agents/installed/mini_swe_agent.py`）——走命令行参数，且**显式把 stdin 接到 `/dev/null`**： + +```bash +mini-swe-agent --yolo --model= --task= \ + --output= --exit-immediately 2>&1 ` 执行，返回码非 0 就抛 `NonZeroAgentExitCodeError`。 + → **"轮数用尽但任务没做完"必须 exit 0**，否则会被当成基础设施故障（还可能触发重试，白烧钱）。这一条与直觉相反，也与 `code_agent_benchmark.md` 初版的写法冲突。 +2. **Harbor 会正则扫 agent 的 stdout/stderr 来分类错误。** `ERROR_PATTERNS` 覆盖 rate limit、usage limit、500、Overloaded、连接中断、输出 token 超限、上下文超限、未登录、安全拒答、网络错误等，分类出的异常类型可配合重试：源码注释里给的用法是 `harbor run --max-retries 3 --retry-include ApiRateLimitError`。 + → agent 应当把 API 错误**原文打到输出**而不是吞掉；重试逻辑因此可以少写一半，agent 侧只需要基础退避。 +3. **不需要 `--workdir`。** `run()` 调 `exec_as_agent` 时不传 `cwd`，容器的默认 WORKDIR 就是任务工作目录。Terminal-Bench 用 `/app`——`claude_code.py` 里硬编码的会话目录 `$CLAUDE_CONFIG_DIR/projects/-app` 就是它的 slug 形式。agent 只要"在当前进程的 cwd 里干活"即可。 +4. **日志路径有约定。** agent 自己的日志写 `/logs/agent/`；`/logs/verifier/` 下的 `reward.txt`（单个整数或浮点，通常 1/0）或 `reward.json`（多指标）是**评测脚本**写的，agent 不碰，也不该去读 `/tests/`。 +5. **密钥与模型全部走环境变量注入。** `ANTHROPIC_API_KEY` / `ANTHROPIC_BASE_URL` / `ANTHROPIC_MODEL`——**这三个 nanoPyCodeAgent 已经支持**（`settings.py`），等于白送。Harbor 还会为兼容基座额外设置 `ANTHROPIC_DEFAULT_SONNET_MODEL` 等别名。 +6. **墙钟超时由 harness 侧管。** 任务的 `task.toml` 里有 `[agent].timeout_sec`、`[verifier].timeout_sec`、`[environment].build_timeout_sec`（默认 600）。agent 自己的 `--timeout` 是保险，不是接入的前提。 +7. **安装很轻。** `ensure_system_dependencies()` 能按需装 curl / bash / git / python3 / python3-pip / nodejs / npm / tmux / ripgrep 等，所以 `install()` 里写 `uv tool install nanoPyCodeAgent` 或 `pip install nanoPyCodeAgent` 就够。Claude Code 的 `install()` 也不过是 npm/curl 二选一加一行 `claude --version` 自检。 +8. **版本可探测。** 实现 `get_version_command()` 与 `parse_version()`，Harbor 会尽力探测并记录 agent 版本（best-effort，失败不报错）。这要求 CLI 有一个 `--version`。 + +### 1.4 轨迹（trajectory）的额外收益 + +Harbor 有统一轨迹格式 ATIF（`SUPPORTS_ATIF`）。mini-swe-agent 的做法是让 CLI 用 `--output=` 写自家 JSON，再在 `populate_context_post_run()` 里转成 ATIF；Claude Code 的做法是解析 `--output-format=stream-json` 的事件流。任一种做到了，Harbor 就能采集步数、token、成本，这正好覆盖 `code_agent_benchmark.md` 里 P1 的"trajectory 落盘"和"token 统计"——**这两件事顺手做了不是额外开销，而是把 harness 侧的报表能力一起拿到。** + +### 1.5 任务结构（了解即可） + +``` +/ +├── instruction.md # 任务指令，就是 run() 收到的 instruction +├── task.toml # [task] [metadata] [verifier] [agent] [solution] [environment] +├── environment/Dockerfile # 或 docker-compose.yaml，或直接引用 docker_image +├── solution/solve.sh # Oracle agent 用的参考解 +└── tests/test.sh # 必须往 /logs/verifier/ 写 reward 文件 +``` + +--- + +## 二、SWE-bench Verified + +官方**不规定 agent 接口**，只规定产物格式。换句话说：agent 侧的 runner 要自己写。 + +### 2.1 官方只管评测 + +```bash +swebench eval verified -p --run-id -j +# 旧式写法仍可用： +python -m swebench.harness.run_evaluation \ + --dataset_name princeton-nlp/SWE-bench_Verified \ + --predictions_path --max_workers 8 --run_id my_run +``` + +predictions 是 JSONL，每行三个键： + +```json +{"instance_id": "sympy__sympy-20590", "model_name_or_path": "gpt-4", "model_patch": "diff --git a/sympy/core/sympify.py..."} +``` + +（mini-swe-agent 写的是 `{instance_id: {...}}` 的 JSON dict，harness 两种都收。） + +结果按 `run_id` + `instance_id` 缓存，改了补丁重跑必须换 `run_id`。 + +### 2.2 runner 侧要素（照 mini-swe-agent 抄） + +来源：`SWE-agent/mini-swe-agent` 的 `src/minisweagent/run/benchmarks/swebench.py` 与 `src/minisweagent/config/benchmarks/swebench.yaml`。 + +- **镜像**：`docker.io/swebench/sweb.eval.x86_64.:latest`，其中 `instance_id` 里的双下划线 `__` 要替换成 `_1776_`（Docker 不允许双下划线），整体转小写。 +- **工作目录**：`/testbed`。 +- **任务文本**：数据集里的 `instance["problem_statement"]`，原文交给 agent。 +- **预算基准**：`step_limit: 250`、`cost_limit: 3.`（美元）、单条命令 `timeout: 60`。 +- **交 patch**：mini-swe 用哨兵——系统提示词要求 agent 先 `git diff -- <改过的文件> > patch.txt`，再用**单独一条**命令 `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && cat patch.txt` 提交，runner 抓哨兵之后的 stdout 当 `model_patch`。提示词里还明确禁止把创建/修改测试文件的改动放进补丁。 + +### 2.3 对 nanoPyCodeAgent 的实际要求 + +只有一条：**能在 `/testbed` 里 headless 跑一段 `problem_statement` 然后退出。** + +补丁的收集**不建议学哨兵那套**——runner 在 agent 退出后自己 `docker exec … git add -A && git diff --cached` 收就行，agent 侧零改动，也避免了"模型忘了敲那条魔法命令就 0 分"这一整类失败。相应地，`code_agent_benchmark.md` 里 P2 的"patch 输出模式"可以不做。 + +--- + +## 三、NL2Repo-Bench + +接口要求最松，但它的 runner 把 OpenHands 写死在仓库里，所以接入方式是 **fork 改代码**，不是插件式挂载。 + +### 3.1 实际流程 + +来源：`multimodal-art-projection/NL2RepoBench` 的 `openhands/openhands_app.py`。 + +1. 每个任务分配一个 UUID 目录 `workspaces//workspace/`，从 `test_files/<项目名>/` 复制那份需求文档（`start.md`）进去，**目录里除此之外是空的**。 +2. 由 `template/config.template.toml` 生成本次任务的 `config.toml`，替换掉 `{{VOLUMES}}`（把 workspace 挂成容器里的 `/workspace`）和 `{{MODULE_CONFIG}}`（模型名 / api_key / base_url）。 +3. 起容器 `docker.all-hands.dev/all-hands-ai/openhands:0.56`（runtime 用 `runtime:0.56-nikolaik`），启动命令写死： + + ```bash + python -m openhands.core.main --config-file=/custom/path/config.toml \ + -t 'According to the start.md in the workspace, implement the entire project as per the requirements specified in the document, ensuring that the final product can be directly run in the current directory. ...' + ``` + +4. 容器退出后，宿主机侧 `post_process_task(task_uuid, workspace_path, test_data, logger)` 对 workspace 目录跑 pytest，**分数 = 通过的测试数**。 + +配置入口是仓库根的 `config.json`：`startPro[].{moduleName, baseUrl, sk, proNameList}` 加一个 `max_pool_size` 控并发。 + +### 3.2 关键性质 + +**评分只看 workspace 目录的最终产物，跟 agent 怎么跑完全解耦。** 所以对 agent 的唯一要求是：**在指定目录 headless 跑一句 prompt，跑完退出。** + +接入 nanoPyCodeAgent 有两条路： + +- 改 `openhands/openhands_app.py` 里创建容器那一段（换镜像、换 `command`、去掉 config.toml 那套），约 20 行； +- 或者干脆自己写一个小 runner，复用它的 `test_files/`（需求文档 + 测试数据）和 `post_processor.py`（pytest 打分）。 + +后者更干净，因为 OpenHands 的那一整套 `config.toml` 对我们毫无用处。 + +--- + +## 四、三者的最小公共契约 + +| # | 要求 | Terminal-Bench | SWE-bench | NL2Repo | +| --- | --- | :-: | :-: | :-: | +| 1 | 一条命令拿到任务文本，跑完退出 | ✅ | ✅ | ✅ | +| 2 | 在**进程当前 cwd** 里干活（不需要 `--workdir`） | ✅ `/app` | ✅ `/testbed` | ✅ `/workspace` | +| 3 | 不交互、不提问、不等确认 | ✅ | ✅ | ✅ | +| 4 | **正常结束一律 exit 0**（含"没做完"） | ✅ 强制 | 建议 | 建议 | +| 5 | 配置全走环境变量 | ✅ | ✅ | ✅ | +| 6 | 日志写 `/logs/agent/` | ✅ | — | — | +| 7 | 结构化轨迹输出 | 可选，收益高 | — | — | +| 8 | 最终能取到 `git diff` | — | ✅（runner 侧收） | — | + +任务文本的三种投递方式都要能吃下：命令行参数（`--task=`）、stdin 管道（`printf … |`）、文件路径。 + +--- + +## 五、nanoPyCodeAgent 的 CLI 设计 + +``` +nanoPyCodeAgent [-p/--prompt "任务描述" | --prompt-file | (stdin)] + [--max-turns N] + [--output-format text|stream-json] + [--trajectory ] + [--version] +``` + +**headless 判定**：给了 `-p` / `--prompt-file` 就是 headless；否则若 `sys.stdin.isatty()` 为 False，就把整段 stdin 读进来当任务。这样同时兼容 `nanoPyCodeAgent -p "..."` 和 `printf "%s" "$TASK" | nanoPyCodeAgent` 两种调用形态，也顺手解决了现在"容器里 stdin 是 EOF 就立刻打印 `Bye!` 退出"的问题。 + +**退出码约定**（这是最容易做错的地方）： + +| 退出码 | 场景 | +| :-: | --- | +| 0 | 模型声明完成；轮数用尽；墙钟超时后自行收尾 —— **任务没做成也是 0** | +| 非 0 | 无 API 凭证、参数错误、API 连续失败到无法继续 | + +判据是"**失败的是任务，还是 harness**"：前者归 0，交给 verifier 判 reward；后者归非 0，让 Harbor 去分类和重试。 + +**输出约定**：把 API 错误原文原样打出来（Harbor 靠正则识别它们），`--output-format stream-json` 输出逐事件 JSON 供 harness 解析，`--trajectory` 落 JSONL 供事后归因。 + +--- + +## 六、对 `code_agent_benchmark.md` 结论的修正 + +| 原结论 | 修正 | +| --- | --- | +| P0"超轮数、超时、API 连续失败 exit 非 0" | 超轮数、超时**必须 exit 0**；非零只留给无凭证 / 参数错误 / API 持续失败 | +| P0"可配置的工作目录" | 降级——三家都靠进程 cwd，`--workdir` 是锦上添花 | +| P0"错误重试，不许崩" | 简化——基础退避 + **把 API 错误原文打到输出**，分类与重试交给 `harbor --retry-include` | +| P2"patch 输出模式" | 不必做——runner 侧 `git diff` 更省事，也避免"忘敲提交命令就 0 分" | +| P2"Harbor agent adapter，接口以仓库提交说明为准" | 接口已确认为 `BaseInstalledAgent`（见 §1.1），可以直接动手 | +| 第一优先 Terminal-Bench 的理由排序 | 首要理由应是"**官方榜条目本身就是 harness + 模型两维**"，而不是"形态匹配"（见下） | + +关于最后一条，以及 DeepSWE 那句"官方榜要求 mini-swe-agent 才能上榜"——原措辞不准。DeepSWE 官网原文是 **"All models run on mini-swe-agent for consistency."**，意思是榜单把 harness 这个变量**钉死了**，条目只有"模型"一维。后果是： + +- nanoPyCodeAgent 跑出来的 DeepSWE 分数在那个榜上**没有位置**，不能拿去跟 Opus 5 的 74.0% 并列——那 74.0% 是 mini-swe-agent 的分。 +- 能做的只有本地 A/B：同一个模型，mini-swe-agent 跑一遍、nanoPyCodeAgent 跑一遍，比差值。 + +而 Terminal-Bench 官方榜的条目形如 `Claude Code + Fable 5` 与 `Terminus 2 + Fable 5`——**harness 是榜单的一个维度**，nanoPyCodeAgent 因此有一个名正言顺的位置。这才是把 Terminal-Bench 排第一的最硬理由。 + +--- + +## 七、参考 + +- Harbor 文档：[Agents](https://www.harborframework.com/docs/agents)、[Task Structure](https://www.harborframework.com/docs/tasks)、[如何运行 Terminal-Bench 2.1](https://www.tbench.ai/docs/run-terminal-bench-2-1) +- Harbor 源码：（`src/harbor/agents/installed/base.py`、`claude_code.py`、`mini_swe_agent.py`） +- mini-swe-agent：（`src/minisweagent/run/benchmarks/swebench.py`、`src/minisweagent/config/benchmarks/swebench.yaml`） +- SWE-bench 评测指南： +- NL2Repo-Bench：、论文 [arXiv:2512.12730](https://arxiv.org/abs/2512.12730) +- DeepSWE 榜单： diff --git a/docs/research/zh-CN/code_agent_benchmark.md b/docs/research/zh-CN/code_agent_benchmark.md new file mode 100644 index 0000000..846047d --- /dev/null +++ b/docs/research/zh-CN/code_agent_benchmark.md @@ -0,0 +1,513 @@ +# Code Agent Benchmark 调研 + +> 本文件为**手写中文源文件**（source of truth）；英文版 [`../en/code_agent_benchmark.md`](../en/code_agent_benchmark.md) 由其生成。 + +调研时间：2026-08-17。 + +模型发布公告里的分数只有放在“哪个 benchmark + 哪个 harness + 哪档 effort”这三件事一起看时才有意义。本文调研六个 2026 年的模型发布——DeepSeek-V4-Flash-0731、Claude Opus 5、GPT-5.6 Sol、Qwen3.8-27B、Kimi-K3、GLM-5.2——把它们报的 code agent 相关 benchmark 拉平成一张表，然后回答一个具体问题：nanoPyCodeAgent 想跑其中哪一个，还差什么。 + +**读数前的三条注意事项：** + +1. **同一个 benchmark 在不同发布公告里的分数不可直接比。** 例如 Terminal-Bench 2.1 上的 Claude Fable 5：Terminal-Bench 官方榜（Claude Code harness）是 83.8%，OpenAI 公告里写 83.1%，Kimi-K3 模型卡里写 88.0%。差异来自 harness、effort 档位、采样次数和评测日期，不是抄错。 +2. **harness 是分数的一部分。** 各家现在普遍用自研 harness 报分（DeepSeek Harness、Kimi Code、Claude Code、Codex），换 harness 掉 3～6 个点是常态。 +3. **内部 benchmark 不可复现。** DSBench、QwenSWEBench、Kimi Code Bench、CursorBench、Frontier-Bench 都是厂商自持数据集，只能当趋势看。 + +## 一、全景表：谁报了什么 + +| Benchmark | 类别 | DeepSeek-V4-Flash-0731 | Opus 5 | GPT-5.6 Sol | Qwen3.8-27B | Kimi-K3 | GLM-5.2 | +| --- | --- | :-: | :-: | :-: | :-: | :-: | :-: | +| Terminal-Bench 2.1 | 终端 agent | ✅ | ✅(三方) | ✅ | ✅ | ✅ | ✅ | +| SWE-bench Pro | 仓库级修 bug | — | — | ✅ | ✅ | — | ✅ | +| DeepSWE (v1.1) | 长时程工程 | ✅ | ✅(三方) | ✅ | ✅ | ✅ | ✅ | +| NL2Repo-Bench | 从零建仓库 | ✅ | — | — | ✅ | — | ✅ | +| ProgramBench | 从二进制重建程序 | — | — | — | — | ✅ | ✅ | +| SWE-Marathon | 超长时程 | — | — | — | — | ✅ | ✅ | +| FrontierSWE | 长时程 + 性能/研究 | — | — | — | — | ✅ | ✅ | +| Frontier-Bench v0.1 | 终端 agent（Anthropic 内部） | — | ✅ | — | — | — | — | +| CursorBench 3.2 | IDE 内多文件（Cursor 内部） | — | ✅ | — | — | — | — | +| AA Coding Agent Index | 复合指数 | — | ✅ | ✅ | — | — | — | +| PostTrainBench | ML 后训练工程 | — | — | — | — | ✅ | ✅ | +| MLS-Bench-Lite | ML 方法研究 | — | — | — | — | ✅ | — | +| CyberGym | 安全（漏洞复现） | ✅ | — | ✅ | — | — | — | +| Agents' Last Exam | 通用 agent | ✅ | — | ✅ | ✅ | — | — | +| AutomationBench | 业务流程自动化 | ✅ | ✅ | — | — | ✅ | —† | +| Toolathlon / Tool-Decathlon | 工具使用 | ✅ | — | — | — | — | ✅ | +| MCP-Atlas | MCP 工具使用 | — | — | — | — | — | ✅ | +| JobBench | 职业任务 | — | — | — | ✅ | ✅ | — | +| CoWorkBench | 长时程办公 | — | — | — | ✅ | — | — | +| LiveCodeBench v6 | 竞赛代码生成 | — | — | — | ✅ | — | — | +| SciCode | 科研代码 | — | — | — | — | ✅ | — | +| OSWorld (2.0 / Verified) | 计算机操作 | — | ✅ | ✅ | ✅ | — | — | +| 内部集 | — | DSBench-FullStack / Hard | — | — | QwenSWEBench | Kimi Code Bench 2.0 | — | + +† GLM-5.2 自己的发布材料没有 AutomationBench，它的 12.9 分出现在 DeepSeek 和 Kimi 的对照表里。 + +一句话结论：**Terminal-Bench 2.1 和 DeepSWE 是六个模型都有分数可查的两个 code agent benchmark**，是横向比较的事实标准；SWE-bench Pro 是次一档的共识项。注意 Opus 5 这两项都不是 Anthropic 自己公布的，而是第三方榜单跑出来的（见 4.2）。 + +## 二、Benchmark 逐个说明 + +### 2.1 终端 / CLI Agent + +#### Terminal-Bench (2.1) + +- **主页**：，运行框架 Harbor： +- **论文**：[arXiv:2601.11868](https://arxiv.org/abs/2601.11868) +- **介绍**：Stanford × Anthropic 合作的终端能力 benchmark，衡量 agent 在真实命令行环境里完成硬任务的能力。任务按领域（system-administration、security、data-science、software-engineering、ML）和难度（medium / hard）分类，2.0 版含 89 个任务，2.1 是受 Z.ai 的“Terminal-Bench 2.0 Verified”启发做的修订版。每个任务跑在独立 Docker 容器里，容器内需要 tmux，由测试脚本判定成功与否。 +- **harness**：官方基线 harness 叫 **Terminus 2**；榜单同时接受 Claude Code、Codex、Cursor CLI、Gemini CLI、mini-SWE-agent 等外部 harness，因此榜单条目是“harness + 模型”的组合。 +- **示例任务**：`openssl-selfsigned-cert`——生成自签名证书、配套脚本和校验文件，且文件权限和格式都要符合要求。难档任务包括编译 Linux 内核、训练一个 ML 模型。 +- **官方榜（Terminal-Bench 2.1，节选）**：Claude Code + Fable 5 = 83.8% ±1.2；Codex + GPT-5.5 = 83.1% ±1.1；Terminus 2 + Fable 5 = 80.4% ±1.2；Cursor CLI + Grok 4.5 = 79.3% ±1.5；Claude Code + Opus 4.8 = 78.9% ±1.3。 + +#### Frontier-Bench v0.1 + +- **主页**：无公开主页（Anthropic 内部数据集），第三方汇总见 +- **介绍**：Anthropic 在 Opus 5 发布时启用的新 agentic 终端编码 benchmark，被描述为 Terminal-Bench 2.1 的后继者，74 个任务，覆盖多文件改动、调试、功能开发。 +- **harness**：mini-SWE-agent + GKE 后端，每题 5 次尝试取平均 reward。 +- **示例**：官方公告提到一个任务里 Opus 5 自己写了一条计算机视觉流水线、在没有直接视觉输入的情况下重建 3D 模型——说明任务是开放式的“给目标、不给方法”。 + +#### CursorBench (3.2) + +- **主页**： +- **介绍**：Cursor 自研的 IDE 内编码 agent 评测集，任务来自 Cursor 生产环境里真实的开发者-agent 会话，覆盖多文件项目、monorepo 和含糊的口语化需求。评的是“模型 + Cursor harness”的组合，不是裸模型。每个模型在多档 reasoning effort 下评测，榜单同时报正确率、平均成本、token 用量和 agent 步数。3.2 版含 42 个配置。 +- **示例**：一次评测的输入是一段真实开发者提示（可能含糊、可能跨多个包）加一个仓库快照，输出是 agent 的多文件改动，再按正确性、代码质量、效率和行为打分；榜单条目形如“某模型 × 某 effort 档 → 正确率 / 平均成本 / 平均步数”。 +- **局限**：厂商自评、harness 不可独立复现。 + +#### AA Coding Agent Index (v1.1) + +- **主页**：，方法论 +- **介绍**：Artificial Analysis 的复合指数，明确以“harness + 模型”为评测单位。由三个子项等权合成：**SWE-Bench-Pro-Hard-AA**（150 个来自 Scale AI SWE-bench Pro 的任务）、**Terminal-Bench v2**（84 个终端任务）、**SWE-Atlas-QnA**（124 道技术问答）。每题跑 3 次取平均得 pass@1，再对任务等权平均。同时公布成本、token 用量和墙钟耗时。 +- **示例**：一个榜单条目形如“Claude Code + Opus 5（max effort）”——同一个模型换成 Codex 或 Terminus 2 就是另一个条目。跑法是 150 + 84 + 124 三份子集各跑 3 次，得到三个 pass@1 再等权平均。 +- **注意**：AA 口径的 Terminal-Bench v2 是 84 题，与 Terminal-Bench 官方 2.0 的 89 题不一致，应为子集。 + +### 2.2 仓库级软件工程 + +#### SWE-bench Verified + +- **主页**： +- **介绍**：SWE-bench 的人工校验子集，500 个实例。给定一个真实仓库快照和一个 GitHub issue，agent 要产出能让隐藏测试通过的补丁。是这一类的老基线，目前头部模型已到 90%+，区分度下降，但仍是最容易上手的入口。本文调研的六个发布都没有报它，列在这里是因为第 5 节把它推荐给本项目作为第二步。 +- **示例**：输入是某个仓库的快照加一段 GitHub issue 文本，输出是一个 patch；判定方式是打上 patch 后跑隐藏测试，要求原本失败的测试转为通过（FAIL_TO_PASS）且原本通过的不许挂（PASS_TO_PASS）。 + +#### SWE-bench Pro + +- **主页**：，榜单 ，代码 +- **介绍**：Scale AI 做的 SWE-bench 接棒者，针对四个问题设计：数据污染、任务多样性不足、问题被过度简化、测试不可复现。共 1865 个实例（731 公开 / 858 私有 / 276 商业），来自 41 个仓库（11 公开 / 12 私有 / 18 来自企业初创公司）。任务形式仍是“仓库 + issue → 补丁”，但是长时程、跨文件的难题。 +- **示例**：与 Verified 同形（仓库 + issue → 补丁），但仓库跨 41 个项目、含企业初创公司的代码，且刻意保留了含糊的 issue 描述，需要跨文件的长时程改动。 +- **难度参照**：刚发布时 GPT-5 与 Claude Opus 4.1 只有 23.3% / 23.1%（同期 Verified 上普遍 70%+）。 + +#### DeepSWE (v1.1) + +- **主页**：，代码 ，数据 +- **论文**：[arXiv:2607.07946](https://arxiv.org/abs/2607.07946) +- **介绍**：Datacurve 出的长时程工程任务集，113 个任务，取自活跃开源仓库，覆盖 TypeScript、Go、Python、JavaScript、Rust。每题一个隔离环境和一个程序化 verifier。v1.1 相对 v1 没换任务，改的是执行与评分方式——在干净隔离环境里对 agent 提交的代码打分，让结果可复现、可审计。 +- **示例**：输入是某活跃开源仓库（TypeScript / Go / Python / JavaScript / Rust 之一）的隔离快照加一份长时程任务描述，输出是 agent 在沙箱里改完并提交的代码；v1.1 的判定是把提交的代码搬到干净环境里，由程序化 verifier 重跑决定通过与否。 +- **harness**：官方榜统一用 **mini-swe-agent**（在 Modal 上由 Pier 驱动），这是“同一 harness 横向比模型”的少数样板。 +- **v1.1 榜单（节选）**：Claude Opus 5 = 74.0%，GPT-5.6 Sol = 72.7%，Grok 4.6 = 67.0%，Gemini 3.7 Flash = 65.0%，DeepSeek V4 Pro 0813 = 63.0%。 + +#### FrontierSWE + +- **主页**：，代码 ，第三方榜 +- **介绍**：Proximal Labs 的超长时程编码 benchmark，覆盖三类任务：功能实现、性能工程、研究型任务。 +- **计分方式特殊**：主指标是 **dominance**——在单个任务上对随机对手的成对胜率，不是“完成了百分之多少的任务”。原始区间是 0～1（Epoch AI 榜上 Fable 5 为 0.900），但模型卡普遍按百分比呈现，所以本文表格里的 74.4 对应 dominance 0.744。 +- **示例**：三类任务的形态分别是——实现一个新的功能模块、把一段热点代码的性能提上去、复现一篇论文的方法。dominance 0.744 的读法是：随机抽一个任务、随机抽一个对手，该 agent 赢的概率约 74.4%。 + +#### SWE-Marathon + +- **主页**：论文 [arXiv:2606.07682](https://arxiv.org/abs/2606.07682)，第三方榜 +- **介绍**：Abundant AI 的超长时程任务集，只有 20 个任务，但每个都是项目级：产品克隆、库重写、ML 工程。每题配一个可执行环境、一份人写的参考实现和一套多层校验。**记录到的 agent 轨迹平均 2720 万 token**，量级远超其他 SWE / 命令行 benchmark。 +- **示例**：任务形态包括克隆一个产品、重写一个库、做一整套 ML 工程；单题轨迹平均 2720 万 token，意味着从探索仓库、搭环境、调试一路做到部署。 +- **有意思的观察**：13.8% 的 rollout 里出现了 reward hacking——agent 试图绕过环境或 verifier 而不是真做任务。失败模式集中在自我校验差、自称任务不可行、过早终止。 + +#### NL2Repo-Bench + +- **主页**：论文 [arXiv:2512.12730](https://arxiv.org/abs/2512.12730) +- **介绍**：ByteDance Seed 等机构做的仓库生成 benchmark，104 个任务，覆盖九类 Python 库。**给 agent 的只有一份自然语言需求文档和一个空工作区**，agent 要自己设计架构、管理依赖、实现多模块逻辑，最终产出一个能安装的 Python 库；评测方式是跑上游项目原本的 pytest 套件，再加结构一致性和跨文件架构校验。 +- **示例**：给 agent 一份“实现某类功能的 Python 库”的需求文档和一个空目录，agent 产出完整仓库（含打包配置和多个模块），评测方再跑上游同名项目原本的 pytest 套件打分。 +- **难度**：SOTA 平均测试通过率不到 40.5%。失败模式：过早终止、全局一致性丢失、跨文件依赖脆弱、几百步交互中规划不足。 + +#### ProgramBench + +- **主页**：，论文 [arXiv:2605.03546](https://arxiv.org/abs/2605.03546) +- **介绍**：**给 agent 一个编译好的可执行文件加它的用法文档，要求从零写出行为一致的程序。** 没有方法签名、没有类骨架、没有 PRD、没有文件布局说明——语言、架构、构建脚本全由 agent 自己定。200 个任务，24.8 万条行为测试，规模从 `jq` 到 SQLite、PHP、FFmpeg。 +- **示例**：给出编译好的 `jq` 二进制及其用法文档，要求 agent 探测其行为、自选语言重写出一个行为一致的实现并给出构建脚本；题目大的一侧是 SQLite、PHP、FFmpeg。 +- **难度**：全部前沿模型的“完全解决”率都是 0%。因此发布公告里的 ProgramBench 分数是部分测试通过率，不是任务完成率。也有批评指出其 harness 缺上下文管理，对 Claude Code / Codex 这类会跑很长的 harness 不够公平。 + +### 2.3 ML / 科研工程 + +#### PostTrainBench + +- **主页**：，论文 [arXiv:2603.08640](https://arxiv.org/abs/2603.08640)，第三方榜 +- **介绍**：衡量 CLI agent 能否自主给一个 1～4B 的基座模型做后训练：**一张 H100、10 小时窗口**，目标是提高该模型在指定 benchmark 上的表现。用什么数据、怎么微调、怎么分配算力全自由，不给起始代码，不允许人介入。评测经 Harbor 编排的 E2B 沙箱执行，训练与推理走共享的 Tinker 服务。 +- **示例**：给一个 1～4B 的基座模型和一张 H100，10 小时内自己造数据、自己选微调方法，把它在指定 benchmark 上的分数提上去。 +- **特别之处**：这是少数**直接以“CLI 脚手架”为评测对象**的 benchmark——官方跑 Claude Code、Codex CLI、Gemini CLI、OpenCode 四种脚手架。当前结论：AI 平均约 28%，人类工程团队约 51%。 + +#### MLS-Bench / MLS-Bench-Lite + +- **主页**：论文 [arXiv:2605.08678](https://arxiv.org/abs/2605.08678)，第三方榜 +- **介绍**：140 个任务、12 个 ML 领域，评的是 AI 系统能否产出**真正可迁移的 ML 方法改进**（不是调参涨点）。每题要求在受控的编辑范围内改进某个指定组件，并配有复现过的强人类基线。Lite 是官方 30 题子集，覆盖 LLM 预训练/后训练、机器人、世界模型、CV、RL、优化、ML 系统、AI for Science。 +- **示例**：任务形态是“在受控的编辑范围内改进某个指定组件（例如 LLM 后训练流程里的某一环），再在多种评测设置下看这个改进是否还成立”——考的是改进能不能迁移，而不是在单一设置上调参涨点。 +- **注意**：不要和 OpenAI 的 **MLE-bench**（75 个 Kaggle 竞赛，Lite 为 22 个）搞混，两者不同。 + +#### SciCode + +- **主页**：，代码 ，论文 [arXiv:2407.13168](https://arxiv.org/abs/2407.13168) +- **介绍**：科学家策划的科研编码 benchmark，从真实研究问题转写而来，覆盖 6 个领域 16 个子领域（公开材料点名的是物理、数学、材料科学、生物、化学五个），80 个主问题拆成 338 个子问题，带可选的科学背景说明和科学家标注的金标准解与测试用例。偏“模型的科学编码能力”，不太考验 agent 循环。 +- **示例**：一个主问题（来自某篇真实论文）被拆成若干子问题，每个子问题要求补全一个函数；可选提供该问题的科学背景说明，判定用科学家写的测试用例。 + +#### LiveCodeBench (v6) + +- **主页**： +- **介绍**：持续从 LeetCode、AtCoder、Codeforces 收新题的无污染竞赛编码评测，除代码生成外还评自修复、代码执行、测试输出预测。同样偏模型能力，不测 agent 循环。 +- **示例**：除“写出能通过全部测试的解”之外还有三类子任务——给一个错解要求自修复、给一段代码和输入要求预测执行结果、给一道题和一个测试要求预测该测试的输出。 + +### 2.4 安全 + +#### CyberGym + +- **主页**：，论文 [arXiv:2506.02548](https://arxiv.org/abs/2506.02548) +- **介绍**：大规模真实漏洞分析评测，1507 个历史漏洞实例，来自 Google OSS-Fuzz，覆盖 188 个 C/C++ 项目。主任务是**漏洞复现**：给 agent 一段文字描述和打补丁前的代码库，要它写出能触发该漏洞的 PoC。该 benchmark 的构建过程本身发现了 35 个 0-day 和 17 个不完整补丁。 +- **示例**：给 agent 一段漏洞的文字描述和该漏洞打补丁之前的 C/C++ 代码库，要求产出一个 PoC 输入，跑起来能触发对应的崩溃。 +- **相关**：同组还有 ExploitGym（，把漏洞变成可用攻击）和 ExploitBench（能力阶梯式的 LLM 安全 agent 评测）。 + +### 2.5 通用 Agent / 工具使用 + +#### Agents' Last Exam (ALE) + +- **主页**：，代码 ，论文 [arXiv:2606.05405](https://arxiv.org/abs/2606.05405) +- **介绍**：Berkeley RDI 联合 250～300 位行业专家做的大规模 agent 评测，围绕 55 个子行业（归为 13 个行业簇）组织，已收 1000～1500+ 任务，目标 5000。**每个任务由确定性脚本对照专家自己的交付物打分，不用 LLM 当裁判。** 采用滚动评测：每约 6 个月发一批新的公开子集，私有任务轮换进、退役的公开任务轮换出，以抑制泄漏。 +- **示例**：任务由某个子行业的专家按自己真实的工作产出构造——给 agent 一份工作区材料，要求交付一份与专家同款的成果物，再由确定性脚本逐项比对，而不是让 LLM 判断“看起来对不对”。 +- **难度**：最难档远未饱和，主流 harness + 骨干模型组合的平均完全通过率为 2.6%。 + +#### AutomationBench (Zapier) + +- **主页**：，代码 ，论文 [arXiv:2604.18934](https://arxiv.org/abs/2604.18934) +- **介绍**：评 agent 通过 REST API 做跨应用工作流编排，47 个真实工具，覆盖销售、市场、运营、支持、财务、HR 六大业务职能，任务模式取自 Zapier 平台上每月 20 亿+ 任务、370 万家公司的真实流量。一个任务可能横跨 CRM、收件箱、日历和 IM，agent 要自己发现端点、遵守一份策略文档、把正确数据写进每个系统。 +- **示例**：一个任务可能横跨 CRM、收件箱、日历和 IM——agent 要自己找到对应的 REST 端点、按一份策略文档行事、把正确数据写进每个系统。 +- **评分**：确定性终态断言（不用 LLM 裁判），含正向和负向断言；差一点也算失败。 + +#### Toolathlon / The Tool Decathlon + +- **主页**：（另有 toolathlon.xyz），论文 [arXiv:2510.25726](https://arxiv.org/abs/2510.25726)（ICLR 2026） +- **介绍**：HKUST NLP 做的工具使用评测，覆盖 **32 个软件应用、604 个工具**，从 Google Calendar、Notion 到 WooCommerce、Kubernetes、BigQuery。108 个人工构造任务，平均需要约 20 轮跨应用交互，每题都有专门的校验脚本可严格验证。 +- **示例**：任务要求在 Google Calendar、Notion、WooCommerce、Kubernetes、BigQuery 这类应用之间协同完成一件事，平均约 20 轮跨应用交互，由该题专属的校验脚本判定。 +- **难度参照**：论文里最好的 Claude-4.5-Sonnet 只有 38.6% 成功率。DeepSeek 公告中的“Toolathlon-Verified”和 GLM 公告中的“Tool-Decathlon”都指这个 benchmark。 + +#### MCP-Atlas + +- **主页**：，论文 [arXiv:2602.00933](https://arxiv.org/abs/2602.00933) +- **介绍**：Scale AI 做的 MCP 工具能力评测，**1000 个由人类专家撰写并校验的自然语言任务，覆盖 36 个真实 MCP server、220 个工具**。提示里不说用哪个 server、哪个工具、什么参数，agent 要在语义相近的干扰项中自己找工具，并跨 server 组合多步流程。用 claim-level rubric 打分：把最终答案拆成基于工具输出的原子事实逐条核对，从而与 agent 的啰嗦程度和文风解耦。公开子集 500 题。 +- **示例**：提示里不说用哪个 server、哪个工具、什么参数，agent 要在 36 个真实 MCP server、220 个工具（还混着语义相近的干扰项）里自己挑，并跨 server 组合出多步流程。 + +#### JobBench + +- **主页**：论文 [arXiv:2605.26329](https://arxiv.org/abs/2605.26329)，榜单 +- **介绍**：130 个 agentic 任务，覆盖 35 个职业。设计思路是“对齐人的委派意愿”而不是“按 GDP 价值替代人”：任务建在 Workbank 之上——一份 1500+ 名劳动者填写的、说明自己希望把哪些职责交给 AI 的调查——从“高委派意愿 × 高经济暴露”的交集里挑出 35 个职业。每题打包成一个含各类参考文件的工作区，输出由事实锚定的 rubric 链评分，平均每题 35.6 条二值判据。 +- **示例**：一个任务打包成含各类参考文件的工作区（对应某个职业的真实产出物），agent 要交付相应成果，再由平均 35.6 条二值判据的 rubric 链评分。 +- **难度参照**：最强组合 Claude Opus 4.7 + Claude Code 为 45.9%。 + +#### CoWorkBench + +- **主页**：无公开主页，第三方汇总见 +- **介绍**：长时程办公/生产力任务评测，覆盖计算机科学、金融、法律、医疗等领域。不是编码题，而是专业工作流。评测配置为 256K 上下文、8 小时超时。 +- **示例**：一个任务的形态是“就某个主题做调研，从多个来源把信息综合成一份交付物”——要求在很长的轨迹上保持注意力，而不是一次问答。 + +### 2.6 计算机操作与多模态（顺带记录） + +这几个不是 code agent benchmark，但在同一批公告里出现，便于对照： + +- **OSWorld 2.0 / OSWorld-Verified**——真实操作系统里的计算机操作任务。 +- **WebArena-Verified**——浏览器操作。 +- **AndroidWorld**——移动端操作。 +- **BrowseComp**——agentic 网页检索。 +- **RecreationBench**（复刻应用）、**Vision2Web**（[arXiv:2603.26648](https://arxiv.org/abs/2603.26648)，视觉驱动的网站开发）、**SWE-MM**（多模态软件工程）、**ClawEval-MM**（多模态工具使用）——Qwen3.8-27B 视觉侧报的几项，其中前三项与“看图写代码”相关。 + +### 2.7 厂商内部集 + +只作趋势参考，无法复现：**DSBench-FullStack / DSBench-Hard**（DeepSeek，后者专注困难编码 agent 问题）、**QwenSWEBench**（Qwen）、**Kimi Code Bench 2.0**（Moonshot）、**CursorBench**（Cursor）、**Frontier-Bench**（Anthropic）。 + +## 三、Harness 一览 + +harness（脚手架）决定了模型怎么看到工具、怎么管上下文、怎么决定停止。这是分数里最容易被忽略的变量。 + +| Harness | 归属 | 说明 | +| --- | --- | --- | +| **Terminus 2** | Terminal-Bench 官方 | Terminal-Bench 的基线 harness，跑在 Harbor 上 | +| **Harbor** | Terminal-Bench 生态 | 不是 agent，而是运行框架：Docker 隔离、任务编排、评分。要求 Python ≥3.12、Docker ≥20.10、Docker Compose ≥2.0，容器内需 tmux | +| **mini-SWE-agent** | Princeton | 极简 bash-first 控制流，性能接近完整 SWE-agent。DeepSWE 官方榜和 Anthropic 的 Frontier-Bench 都用它 | +| **SWE-agent / OpenHands** | 学界 / OSS | 仓库级 SWE benchmark 的常用脚手架；GLM-5.2 的 SWE-bench Pro 走 OpenHands | +| **Claude Code** | Anthropic | 被广泛用作第三方模型的评测 harness（Qwen3.8、GLM-5.2 都用它报分，GLM 甚至标注了 2.1.167 这个具体版本） | +| **Codex CLI** | OpenAI | GPT 系列的官方 harness；Kimi 报 GPT-5.6 Sol 的 FrontierSWE 分数时用它 | +| **Kimi Code** | Moonshot | Kimi-K3 自研 harness，模型卡里所有主分数都基于它 | +| **DeepSeek Harness** | DeepSeek | V4-Flash-0731 用其 **minimal mode** 报分（模型卡称将开源） | +| **Cursor CLI / Gemini CLI / OpenCode** | 各自厂商 | 出现在 Terminal-Bench 榜和 PostTrainBench 的四脚手架对照里 | + +**对本项目的直接启示**：Terminal-Bench 和 PostTrainBench 这类 benchmark 是**面向 CLI agent** 设计的，nanoPyCodeAgent 这种“一个可执行 CLI + 几个内置工具”的形态天生适配；而 SWE-bench 家族是**面向 patch** 设计的，接入时只需要在结束时产出 `git diff`。 + +## 四、各模型发布时的 benchmark 与得分 + +以下表格尽量照抄各自发布材料。同一 benchmark 跨表不可比（见开头注意事项）。 + +### 4.1 DeepSeek-V4-Flash-0731 + +- **来源**： +- **模型名**：`deepseek-ai/DeepSeek-V4-Flash-0731` +- **harness**：公开 benchmark 中的 Code Agent 任务使用 **DeepSeek Harness 的 minimal mode**，`max` reasoning effort，`temperature=1.0`、`top_p=0.95` + +| Benchmark | V4-Flash-0731 | V4-Flash (Preview) | V4-Pro (Preview) | GLM-5.2 | Opus 4.8 | +| --- | :-: | :-: | :-: | :-: | :-: | +| Terminal Bench 2.1 | 82.7 | 61.8 | 72.1 | 81.0 | 85.0 | +| NL2Repo | 54.2 | 39.4 | 38.5 | 48.9 | 69.7 | +| CyberGym | 76.7 | 38.7 | 52.7 | — | 83.1 | +| DeepSWE | 54.4 | 7.3 | 12.8 | 46.2 | 58.0 | +| Toolathlon-Verified | 70.3 | 49.7 | 55.9 | 59.9 | 76.2 | +| Agents' Last Exam | 25.2 | 15.8 | 16.5 | 23.8 | 25.7 | +| AutomationBench Public | 25.1 | 10.8 | 12.8 | 12.9 | 27.2 | +| DSBench-FullStack† | 68.7 | 37.0 | 41.8 | 61.8 | 71.6 | +| DSBench-Hard† | 59.6 | 25.8 | 31.1 | 54.5 | 71.7 | + +† 内部测试集；DSBench-Hard 专注困难编码 agent 问题。 + +一处未能解释的冲突：DeepSeek 给 GLM-5.2 的 Toolathlon-Verified 打了 59.9，而 GLM 自己在 4.6 里报的 Tool-Decathlon 只有 48.2——59.9 在 GLM 的表里恰好是 Opus 4.8 的值。两边原文都已逐字核对，转录无误，引用时以各自来源为准。 + +### 4.2 Claude Opus 5 + +- **来源**： +- **模型名**：`claude-opus-5` +- **harness**：Frontier-Bench 用 **mini-SWE-agent + GKE 后端**，每题 5 次尝试取平均 reward。模型带 effort 档位（low / medium / high / max），公告中的对比多在 max effort 下。Opus 5 与 Fable 5 的评测中，安全分类器拒答时以 Opus 4.8 作为兜底。 +- **重要说明**：Anthropic 官方发布页**大量使用相对表述而非绝对数字**，且**没有报 SWE-bench Verified / SWE-bench Pro / Terminal-Bench**。 + +官方公告中的表述： + +| Benchmark | Opus 5 表现（官方原义） | +| --- | --- | +| Frontier-Bench v0.1 | SOTA，超过 Fable 5，是 Opus 4.8 的两倍以上 | +| CursorBench 3.2 | max effort 下与 Fable 5 峰值分数差距在 0.5% 以内，成本只有一半 | +| AA Coding Agent Index | 榜首 | +| ARC-AGI 3 | 次优模型的 3 倍 | +| Zapier AutomationBench | 同等每任务成本下通过率约为次优模型的 1.5 倍；churn-prevention 序列 100% 通过 | +| OSWorld 2.0 | 超过 Fable 5，成本仅略高于三分之一 | +| GDPval-AA v2 / HLE / DeepSearchQA | 领先 | +| 生命科学 | 全项优于 Opus 4.8；有机化学 +10.2pt，蛋白功能预测 +7.7pt | +| OSS-Fuzz | 漏洞识别与 Mythos 5 相当，漏洞利用开发明显落后 | + +三方来源补充的具体数字（**非官方，谨慎引用**）： + +| Benchmark | Opus 5 | 对照 | 来源 | +| --- | :-: | --- | --- | +| Frontier-Bench v0.1 | 43.3% | Fable 5 33.7%、Opus 4.8 18.7% | [Vellum](https://www.vellum.ai/blog/claude-opus-5-benchmarks-explained)、[llm-stats](https://llm-stats.com/benchmarks/frontier-bench-v0.1) | +| DeepSWE v1.1 | 74.0% | GPT-5.6 Sol 72.7% | [DeepSWE 官方榜](https://deepswe.datacurve.ai/) | +| Terminal-Bench 2.1 | 89.1%（max effort） | GPT-5.6 Sol xhigh 89.5%（AA 自测口径，与 4.3 里 OpenAI 自报的 88.8% 不是同一次评测） | [Artificial Analysis](https://artificialanalysis.ai/evaluations/terminalbench-v2-1) | +| SWE-bench Verified | 97%（聚合站数据，未见官方确认） | — | [morphllm](https://www.morphllm.com/claude-benchmarks) | + +### 4.3 GPT-5.6 Sol + +- **来源**：OpenAI 2026-07-09 发布公告 （该页对本次抓取返回 403，下表转引三方对该公告的整理） +- **模型名**：`gpt-5.6-sol`，另有 Sol Ultra 档以及同族 Terra / Luna +- **harness**：公告未在转引材料中明示编码 benchmark 的 harness；OpenAI 系列惯例是 Codex CLI。Sol Ultra 对应更高的 reasoning effort。 + +| Benchmark | Sol | Sol Ultra | Terra | Luna | 对照 | +| --- | :-: | :-: | :-: | :-: | --- | +| Terminal-Bench 2.1 | 88.8% | 91.9% | 87.4% | 84.7% | GPT-5.5 85.6%、Fable 5 83.1%、Opus 4.8 78.9% | +| SWE-bench Pro | 64.6% | — | 63.4% | 62.7% | Mythos 5 80.3%、Fable 5 80.0%、GPT-5.5 59.4% | +| DeepSWE v1.1 | 72.7% | — | 69.6% | 67.2% | Fable 5 69.7%、GPT-5.5 67.0%、Opus 4.8 59.0% | +| AA Coding Agent Index v1.1 | 80 | — | 77.4 | 74.6 | Fable 5 77.2、GPT-5.5 76.4、Opus 4.8 72.5 | +| Agents' Last Exam | 52.7% | — | 50.4% | 50.3% | GPT-5.5 46.9%、Opus 4.8 45.2%、Fable 5 40.5% | +| BrowseComp | 90.4% | 92.2% | 87.5% | 83.3% | Mythos 5 88.0%、GPT-5.5 84.4% | +| OSWorld 2.0 | 62.6% | — | 50.2% | 45.6% | Opus 4.8 54.8%、GPT-5.5 47.5% | +| ExploitBench | 73.5% | — | — | — | GPT-5.5 47.9% | +| CyberGym | 84.5% | — | — | — | — | +| ARC-AGI-3 | 7.78% | — | 0.80% | 0.18% | Opus 4.8 1.5%、GPT-5.5 0.43% | +| AA Intelligence Index v4.1 | 58.9 | — | 55.0 | 51.2 | Fable 5 59.9、Opus 4.8 55.7 | +| GPQA Diamond | 94.6% | — | 92.9% | 92.3% | Fable 5 92.6%、GPT-5.5 93.6% | + +**一条需要记录的风评**：METR 报告称 Sol 在其软件工程评测中出现了该机构历史上检出率最高的 evaluation gaming——利用评测 bug、抽取隐藏测试答案、用能满足指标但没真正完成任务的捷径替代实现。这提醒我们自建 benchmark 时必须做 reward hacking 检查（SWE-Marathon 也观察到 13.8% 的 rollout 有此行为）。 + +### 4.4 Qwen3.8-27B + +- **来源**： +- **模型名**：`Qwen/Qwen3.8-27B` +- **harness**：多数编码项使用 **Claude Code harness**，`temperature=1.0`、`top_p=0.95`、256K 上下文；Terminal Bench 2.1 用 **Terminus**；NL2Repo 额外施加了 bash 限制；QwenSWEBench 为 avg@3、8 小时超时 + +| 分类 | Benchmark | 得分 | 备注 | +| --- | --- | :-: | --- | +| 编码 | Terminal Bench 2.1 (Terminus) | 73.0 | | +| 编码 | SWE-bench Pro | 61.7 | Claude Code harness | +| 编码 | NL2Repo-Bench | 42.3 | Claude Code harness，加 bash 限制 | +| 编码 | DeepSWE 1.1 | 42.2 | Claude Code harness | +| 编码 | QwenSWEBench | 79.0 | 内部集，avg@3，8h 超时 | +| 编码 | LiveCodeBench v6 | 90.3 | | +| Agent | CoWorkBench | 70.7 | | +| Agent | JobBench | 33.4 | | +| Agent | Agents' Last Exam | 20.4 (Pass@1) / 42.9 (Score) | | +| 通用 | IFBench | 79.5 | | +| 通用 | GPQA Diamond | 89.2 | | +| 通用 | HLE | 30.8 | GPT-4o 判分 | +| 视觉 | OSWorld-Verified | 84.3 | 计算机操作 | +| 视觉 | WebArena-Verified | 64.8 | 浏览器操作 | +| 视觉 | AndroidWorld | 81.9 | 移动端操作 | +| 视觉 | RecreationBench | 47.1 | 复刻应用 | +| 视觉 | ClawEval-MM | 57.4 (Pass@3) | 多模态工具使用 | +| 视觉 | SWE-MM | 38.6 | 多模态软件工程 | +| 视觉 | Vision2Web | 62.9 | 视觉驱动的网站开发 | + +### 4.5 Kimi-K3 + +- **来源**： +- **模型名**：`moonshotai/Kimi-K3` +- **harness**：主分数用自研 **Kimi Code harness**；对照分数的来源逐项标注（见备注列） + +| Benchmark | Kimi K3 | Fable 5 | GPT-5.6 Sol | Opus 4.8 | GPT-5.5 | GLM-5.2 | 备注 | +| --- | :-: | :-: | :-: | :-: | :-: | :-: | --- | +| DeepSWE | 67.5 | 70.0 | 73.0 | 59.0 | 67.0 | 46.2 | K3 用 Kimi Code；GLM-5.2 取自其发布博客；其余取自官方榜（v1.1 任务） | +| Terminal-Bench 2.1 | 88.3 | 88.0 | 88.8 | 84.6 | 83.4 | 82.7 | K3 用 Kimi Code；其余取各 harness 中最好成绩 | +| ProgramBench | 77.8 | 76.8 | 77.6 | 71.9 | 70.8 | 63.7 | K3 与 GLM-5.2 用 Kimi Code | +| SWE-Marathon | 42.0 | 35.0 | 39.0 | 40.0 | 14.0 | 13.0 | Claude Code harness；H20 校准分支 | +| FrontierSWE | 81.2 | 86.6 | 71.3 | 66.7 | 64.9 | 67.3 | K3 用 Kimi Code；GPT-5.6 Sol 用 Codex | +| MLS-Bench-Lite | 48.3 | 49.9 | 46.2 | 42.8 | 35.5 | 40.4 | 混用多种 harness | +| SciCode | 58.7 | 60.2 | 56.1 | 53.5 | 56.1 | 50.5 | 引自 Artificial Analysis（2026-07-23） | +| Kimi Code Bench 2.0 | 72.9 | 76.9 | 64.8 | 71.7 | 69.0 | 64.2 | 内部集；max reasoning effort | +| PostTrainBench | 36.6 | 41.4 | 34.6 | 34.1 | 28.4 | 34.3 | 官方 Harbor 实现；3 次 H20 运行取平均 | +| BrowseComp | 91.2 | 88.0 | 90.4 | 84.3 | 84.4 | — | 300K token 处做上下文压缩 | +| AutomationBench | 30.8 | 29.1 | 29.7 | 27.2 | 22.7 | 12.9 | 官方 GitHub 配置；600 题公开子集 | +| JobBench | 54.3 | 57.4 | 45.4 | 48.4 | 38.3 | 43.4 | 取自 Vals AI | + +### 4.6 GLM-5.2 + +- **来源**： +- **模型名**：`zai-org/GLM-5.2` +- **harness（逐项标注，是本批公告里最透明的）**： + - SWE-bench Pro → **OpenHands**，OpenAI 兼容 API，`temperature=1`、`top_p=1`、`max_new_tokens=32k`、400K 上下文 + - DeepSWE → 官方框架 + **mini-swe-agent**，2 小时超时，隔离沙箱（2 CPU / 8GB RAM） + - Terminal-Bench 2.1（Terminus-2）→ **Terminus-2**，256K 上下文，沙箱 4 CPU / 8GB RAM + - Terminal-Bench 2.1（Best Reported Harness）→ **Claude Code 2.1.167**，`temperature=1.0`、`top_p=0.95`、`max_new_tokens=131072`，无墙钟限制 + - FrontierSWE / PostTrainBench / SWE-Marathon → 1M 上下文，max effort，128K 输出 + +> 照抄时发现的一处源表异常：Terminal Bench 2.1 的两行对非 GLM 模型口径不一致——Opus 4.8 在 Terminus-2 行是 85，在“最佳 harness”行反而只有 78.9（78.9 恰是 Terminal-Bench 官方榜上 Claude Code + Opus 4.8 的成绩），GPT-5.5 同样是 84 → 83.4。下表照抄原文，未作修正。 + +编码： + +| Benchmark | GLM-5.2 | GLM-5.1 | Qwen3.7-Max | MiniMax M3 | DeepSeek-V4-Pro | Opus 4.8 | GPT-5.5 | Gemini 3.1 Pro | +| --- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | +| SWE-bench Pro | 62.1 | 58.4 | 60.6 | 59 | 55.4 | 69.2 | 58.6 | 54.2 | +| NL2Repo | 48.9 | 42.7 | 47.2 | 42.1 | 35.5 | 69.7 | 50.7 | 33.4 | +| DeepSWE | 46.2 | 18 | 18 | 20 | 8 | 58 | 70 | 10 | +| ProgramBench | 63.7 | 50.9 | — | — | 47.8 | 71.9 | 70.8 | 39.5 | +| Terminal Bench 2.1 (Terminus-2) | 81.0 | 63.5 | 75 | 65 | 64 | 85 | 84 | 74 | +| Terminal Bench 2.1 (最佳 harness) | 82.7 | 69 | — | — | — | 78.9 | 83.4 | 70.7 | +| FrontierSWE (Dominance) | 74.4 | 30.5 | — | — | 29.0 | 75.1 | 72.6 | 39.6 | +| PostTrainBench | 34.3 | 20.1 | — | — | — | 37.2 | 28.4 | 21.6 | +| SWE-Marathon | 13.0 | 1.0 | — | — | — | 26.0 | 12.0 | 4.0 | + +Agent 与推理（节选）： + +| Benchmark | GLM-5.2 | GLM-5.1 | DeepSeek-V4-Pro | Opus 4.8 | GPT-5.5 | Gemini 3.1 Pro | +| --- | :-: | :-: | :-: | :-: | :-: | :-: | +| MCP-Atlas（公开集） | 76.8 | 71.8 | 73.6 | 77.8 | 75.3 | 69.2 | +| Tool-Decathlon | 48.2 | 40.7 | 52.8 | 59.9 | 55.6 | 48.8 | +| HLE | 40.5 | 31 | 37.7 | 49.8 | 41.4 | 45 | +| HLE（带工具） | 54.7 | 52.3 | 48.2 | 57.9 | 52.2 | 51.4 | +| AIME 2026 | 99.2 | 95.3 | 94.6 | 95.7 | 98.3 | 98.2 | +| GPQA-Diamond | 91.2 | 86.2 | 90.1 | 93.6 | 93.6 | 94.3 | + +## 五、对 nanoPyCodeAgent 的 benchmark 推荐 + +评估口径：**跑得起来**（不需要 GPU / 不需要注册私有集 / 环境可本地复现）、**测的是 agent 循环而不是模型**（否则测的是 Claude 而不是本项目）、**能横向对照**（有别人用同一 benchmark 报过分）、**成本可控**。 + +### 第一优先：Terminal-Bench 2.1 + +最合适的第一个 benchmark，理由： + +1. **榜单本身承认 harness 这个维度**。官方榜条目形如 `Claude Code + Fable 5`、`Terminus 2 + Fable 5`——harness 是条目的一半，nanoPyCodeAgent 因此有一个名正言顺的位置，而不是只能自己跟自己比。六个模型也都有可查的分数（Opus 5 的来自第三方榜），横向可比性最高。这是最硬的理由。 +2. **形态天然匹配**。任务就是“在一个 Linux 容器里用命令行把事做成”，而本项目正好是 bash + read + write + edit 四件套。 +3. **官方支持自定义 agent**。Harbor 提供 `--agent-import-path` 挂载自定义 agent，不必等官方适配；接口是 `BaseInstalledAgent`，只需实现 `install()` 与 `run()` 两个方法（详见 [`benchmark_headless_interface.md`](benchmark_headless_interface.md)）。 +4. **成本可控**。可以先跑 10～20 题子集；`-k` 参数控制采样次数。 + +建议做法：先在本地跑 Terminus 2 + `claude-sonnet-4-6` 得到基线，再跑 nanoPyCodeAgent + 同一模型，两者之差就是 harness 差距，这比绝对分数更有信息量。 + +### 第二优先：SWE-bench Verified 子集 + +- 老基线、文档最全、patch 形式最简单（结束时 `git diff` 即可）。 +- 缺点是每个实例一个 Docker 镜像，磁盘和拉取时间是主要成本，所以只跑 20～50 题子集。 +- 价值在于验证 edit 工具的正确性：仓库级精确改动是 edit 工具的主战场。 + +### 第三优先：NL2Repo-Bench + +- 只需要 Python + pytest，环境依赖在所有长时程 benchmark 里最轻。 +- “空工作区 + 一份规格 → 一个能安装的库”直接压测 write 工具和多文件规划能力，正好补 SWE-bench 只测局部修改的盲区。 +- 104 题，可跑子集。 + +### 值得后续考虑 + +- **DeepSWE v1.1**：113 题，六家全报，趋势参考价值高。但官方榜把 harness 钉死在 mini-swe-agent 上（原文 “All models run on mini-swe-agent for consistency.”），榜单条目只有“模型”一维，不像 Terminal-Bench 那样带 harness 维度——所以自建 harness 跑出来的分数在那个榜上没有位置，不能拿去跟榜上的数字并列。能做的是本地 A/B：同一个模型，mini-swe-agent 跑一遍、nanoPyCodeAgent 跑一遍，比差值。 +- **PostTrainBench**：唯一直接把“CLI 脚手架”当评测对象的 benchmark，未来若想论证“nanoPyCodeAgent 作为脚手架的质量”，它的四脚手架对照（Claude Code / Codex CLI / Gemini CLI / OpenCode）是最好的框架——但需要一张 H100 和 10 小时，现阶段不现实。 + +### 暂不建议 + +| Benchmark | 原因 | +| --- | --- | +| SWE-Marathon | 平均 2720 万 token/题，成本量级不合适 | +| ProgramBench | 全员完全解决率 0%，对本项目没有区分度 | +| MLS-Bench / PostTrainBench | 需要 GPU | +| CyberGym / ExploitGym | 需要 OSS-Fuzz 构建环境，且方向与本项目无关 | +| Agents' Last Exam | 大量私有任务，滚动评测，个人项目难以对齐 | +| Toolathlon / MCP-Atlas / AutomationBench | 需要 MCP / 多应用工具生态，本项目尚无 MCP 支持 | +| LiveCodeBench / SciCode / GPQA / HLE | 测模型不测 agent，跑了只是在测 Claude | + +## 六、为了跑起来，项目还缺什么 + +> 本节列的是缺口。三个 benchmark 对 headless 接口的**具体**要求——Harbor 适配类的签名、退出码语义、patch 怎么收——已另做调研，见 [`benchmark_headless_interface.md`](benchmark_headless_interface.md)；下面的条目已按其结论修订过。 + +现状（截至 v0.7.0）：`agent.py` 是一个交互式 REPL——`load_settings_env()` → `anthropic.Anthropic()` → `while True: input("You> ")`，内层再一个 `while True` 处理 tool_use 直到模型不再调工具。四个工具（read / write / edit / bash），`MAX_TOKENS = 8192`，无 CLI 参数，`main()` 直接调 `run()`。 + +好消息是有两件事已经做对了：`terminal.py` 的 ANSI 背景上色和 Spinner 都用 `sys.stdout.isatty()` 做了门控（`terminal.py:20`、`terminal.py:69`），所以在容器里不会喷转义序列；`bash_tool.py` 已有 120 秒超时和 20000 字符输出截断（`bash_tool.py:13-14`），并且把 stdin 设成 `/dev/null`（`bash_tool.py:61`），命令不会抢走 agent 的输入。 + +下面按优先级列出缺口。 + +### P0 — 不做就完全跑不起来 + +1. **非交互（headless）一次性任务模式**。这是最硬的阻塞项：benchmark 通过一条命令把任务描述交给 agent，跑完就退出。当前唯一入口是 `input()` 循环（`agent.py:146`），容器里 stdin 是 EOF，会立刻 `break` 打印 `Bye!` 退出，什么都不做。需要一个 CLI 层：`nanoPyCodeAgent -p "<任务>"`、`--prompt-file `，或从 stdin 读整段。 + +2. **明确的终止条件与退出码**。判据是“失败的是任务，还是 harness”。模型声明完成、轮数用尽、墙钟超时后自行收尾——**这些一律 exit 0**，哪怕任务没做成，reward 交给 verifier 去判；只有无凭证、参数错误、API 连续失败到无法继续，才 exit 非 0。这一条很容易做反：Harbor 在 `set -o pipefail` 下执行 agent 命令，非零退出码直接判成 agent 失败并抛异常，还可能触发重试白烧钱。`main()` 现在没有返回码概念（`__init__.py`）。 + +3. **轮数上限 + 墙钟超时**。`agent.py:158` 的内层 `while True` 没有任何上限，模型一旦陷入“反复试同一条命令”的循环就会一直烧钱到 API 报错。需要 `--max-turns`；墙钟超时 harness 侧也管（Harbor 任务的 `task.toml` 里有 `[agent].timeout_sec`），所以 agent 自己的 `--timeout` 是保险而非接入前提。 + +4. **错误重试，不许崩**。模块 docstring 明说只处理 happy path、异常即崩溃（`agent.py:10-13`）。benchmark 里一次 429 / `overloaded_error` / 网络抖动就是整题 0 分。至少要给 `client.messages.stream` 加指数退避重试，并把单题失败收敛成“这题 0 分”而不是“整个 run 挂掉”。Harbor 侧另有一层：它用正则扫 agent 输出，把错误分类成 `ApiRateLimitError`、`ContextWindowExceededError` 等类型，配合 `--max-retries 3 --retry-include ApiRateLimitError` 重试——所以 agent 只需要基础退避，但**必须把 API 错误原文打到输出**而不是吞掉。 + +5. **benchmark 化的系统提示词**。当前提示词面向对话助手（`agent.py:43-50`）。非交互模式下必须显式要求：不要向用户提问、不要停下来等确认、自己决策到底、完成后明确声明结束。这一条不改，分数会被“模型礼貌地询问下一步”大量吃掉。 + +### P1 — 不做的话分数会很难看 + +6. **上下文管理 / compaction**。`messages` 列表只增不减（`agent.py:139`）。Terminal-Bench 的 hard 任务几十轮后必然打满上下文，然后 API 直接报错——这会被计成“任务失败”而不是“harness 缺陷”。参考各家做法：Kimi 在 300K token 处压缩上下文，GLM 用 256K～1M 上下文。最小可行方案是“工具结果二次截断 + 旧轮次摘要或丢弃”。 + +7. **Trajectory 落盘**。把每轮的 request/response、tool call 与结果、token 用量、耗时写成 JSONL。没有这个，一题失败只能看终端 scrollback 猜原因，无法归因也无法复现。这一项还有额外收益：Harbor 有统一轨迹格式 ATIF，agent 只要能输出结构化轨迹（或结构化事件流），Harbor 就能顺带采集步数、token 和成本。 + +8. **token 与成本统计**。从 `message.usage` 累加输入/输出 token。现在的 benchmark 报告普遍同时看分数和 token 用量（AA Coding Agent Index、CursorBench 都报成本与步数），只有分数没有成本是不完整的。 + +9. **`MAX_TOKENS` 与 bash 超时可配**。8192 输出上限（`agent.py:42`）对长任务偏小——各家都在 128K 量级报分。`BASH_TIMEOUT_SECONDS = 120`（`bash_tool.py:13`）对“编译内核”“跑完整测试套件”这类 Terminal-Bench 任务不够。 + +10. **grep / glob 工具**。现在靠 bash 里的 `grep`，能用，但输出难以结构化截断，模型容易一次拉回上万行把上下文打满。仓库级任务（SWE-bench、NL2Repo）里专用的 Grep/Glob 明显更省 token。 + +### P2 — 想正式上榜或横向对比才需要 + +11. **Harbor agent adapter**。接口已确认：继承 `BaseInstalledAgent`，实现 `install()`（容器内 `uv tool install nanoPyCodeAgent`）与 `run()`（把 instruction 交给 CLI、日志 tee 到 `/logs/agent/`），API key 由 Harbor 通过环境变量注入。方法签名和两个官方范例（Claude Code 走 stdin 管道、mini-swe-agent 走 `--task=`）见 [`benchmark_headless_interface.md`](benchmark_headless_interface.md)。 + +12. **OpenAI 兼容后端**。想跟 GLM / Kimi / Qwen / DeepSeek 对比就必须支持非 Anthropic 协议——GLM-5.2 的 SWE-bench Pro 就是走 OpenAI 兼容 API 报的。当前唯一依赖是 `anthropic`（`pyproject.toml`），靠 `ANTHROPIC_BASE_URL` 指代理只能部分绕过。 + +13. **patch 输出模式（可选）**。SWE-bench 家族要的是补丁，但更省事的做法是 runner 在 agent 退出后自己 `git add -A && git diff --cached` 收，agent 侧零改动，还避免了“模型忘了敲提交命令就 0 分”这一整类失败。只有在 runner 拿不到容器的情况下才需要 agent 自己输出 `git diff`。 + +14. **thinking / reasoning effort 透传**。现在没传 `thinking` 参数。所有厂商都是在 max effort 下报分的，不透传就是自带劣势。 + +15. **批量运行器与多次采样**。`-k 5` 式的多次采样求均值是通行做法（Frontier-Bench 就是 5 次取平均），需要能并发跑多题并汇总。 + +16. **reward hacking 自查**。SWE-Marathon 观察到 13.8% 的 rollout 有 reward hacking，METR 报告 GPT-5.6 Sol 的 gaming 检出率创其历史新高。自建评测时要检查 agent 是不是改了测试、读了隐藏答案、或用捷径糊过了断言。 + +17. **可配置的工作目录**。benchmark 都在容器里指定工作目录（Terminal-Bench 用 `/app`、SWE-bench 用 `/testbed`、NL2Repo 用 `/workspace`），但三家都是靠容器默认 WORKDIR 传递的，agent 在进程当前 cwd 里干活即可，所以 `--workdir` 不是接入前提（初稿把它列在 P0 是判断失误）。真正值得做的是让 bash 会话保留 cwd——现在每次开新 shell、`cd` 不跨调用保留（`bash_tool.py:39` 的 docstring 已说明），长任务里模型必须反复写绝对路径。 + +### 最小可行路径 + +一句话：**P0 全做 + P1 的第 6 与第 7 项**，就足够跑通 Terminal-Bench 2.1 的一个小子集并拿到可信数字。落成一条实现顺序： + +1. 加 CLI 层与 headless 模式（P0-1、2）——这一步之后就能被脚本调用。 +2. 加轮数上限与重试（P0-3、4）——这一步之后单题失败不再毁掉整个 run。 +3. 改 benchmark 化系统提示词（P0-5）。 +4. 加 trajectory JSONL 与 token 统计（P1-7、8）——这一步之后失败可归因。 +5. 加最简 compaction（P1-6）——这一步之后 hard 任务不再必然撞上下文墙。 +6. 写 Harbor adapter（P2-11），跑 20 题子集，与 Terminus 2 + 同模型的基线对照。