Add swebench_bench.py: gold-check, pre-registered selection, post-hoc scoring, paired comparison - #236
Open
t-timms wants to merge 2 commits into
Open
Add swebench_bench.py: gold-check, pre-registered selection, post-hoc scoring, paired comparison#236t-timms wants to merge 2 commits into
t-timms wants to merge 2 commits into
Conversation
…post-hoc scoring, paired comparison Automates four steps of a SWE-bench evaluation, each learned on a real run where skipping it gave a wrong or unverifiable number: - gold-check: run the official harness with the GOLD patch on every candidate task. In the currently published images 8 of 23 SWE-bench Lite dev tasks do not resolve even with it (all 5 pvlib: NumPy 2 removed np.Inf; pyvista: no libGL.so.1; 2 pydicom: test-infra error), so a score on them says nothing about the agent. The failing test output is classified into a one-line reason. - selection: the first N gold-valid tasks of a seeded random order (prefix-stable, independent of input order), fixed before any agent runs, so the task set cannot be tuned to a result. - score: score the final patches (last prediction per instance) afterwards with the official harness, independent of any in-loop verify tool; empty patches never reach the harness; Wilson 95% intervals (clamped to [0, 1]), agent minutes and solved per hour from the runner's metrics. - compare: paired per-task comparison of two arms over several draws with an exact two-sided sign test on the tasks where they differ, next to pooled Wilson intervals (descriptive only: draws of the same tasks are not independent samples). Also: scripts/validate_driver.py's default smoke set replaced pvlib__pvlib-python-1606, whose gold patch no longer resolves in the current image, with pydicom__pydicom-1256 (gold-verified); docs/adding_a_driver.md updated; docs/benchmark_scoring.md; CHANGELOG. Tests: tests/test_swebench_bench.py (31), with the real 23-instance dev list and gold-check result and the per-task outcomes of six scored draws as fixtures; harness runs are faked.
🤖 Godspeed Review
SecurityNo secrets detected in changed files. |
`swebench_bench.py score` treated every task not in `resolved_ids` as unresolved. A task the harness never scored (an image pull refused by Docker Hub's anonymous rate limit, a container that did not start) landed in `error_ids` / `infra_failure_ids` / `incomplete_ids` and was silently counted as a failure, deflating that arm's score with no sign in the output. `score` now reads those lists (and any task missing from a report that accounts for every task), exits 3 without writing `--out`, and saves the details to `<out>.incomplete.json`. A resumable driver therefore re-scores instead of treating the draw as finished. `--allow-harness-errors` writes the score anyway, with `complete: false` and `harness_problems` recorded. An `ambiguous_failure` is recorded but still counts as unresolved. `compare` warns on draws scored that way.
🤖 Godspeed Review
SecurityNo secrets detected in changed files. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Four steps of a SWE-bench evaluation that I did by hand on a real run (Qwen3.8-27B vs KAT-REAP50 on a 16 GB GPU), each of which gave a wrong or unverifiable number when skipped or done loosely (details in
docs/benchmark_scoring.md):np.Inf; pyvista: missinglibGL.so.1; 2 pydicom: test-infrastructureAttributeError). A score on them measures nothing.scripts/validate_driver.py's default smoke set contained one of them (Make the agent-in-loop verify oracle work with swebench 5.x #231 noted this).Changes
scripts/swebench_bench.pywith subcommandsgold-check,score,compare(pure, tested logic for the Wilson interval, exact sign test, pre-registered selection, failure classification and report parsing; harness calls behindswebench5.x flags, interpreter fromGODSPEED_SWEBENCH_PYTHON, datasetSWE-bench/SWE-bench_Lite).error_ids,infra_failure_ids,incomplete_ids, or missing from a report that accounts for every task, e.g. a rate-limited image pull) is not counted as unresolved.scoreexits 3, saves the details to<out>.incomplete.jsonand does not write<out>, so a resumable driver re-scores.--allow-harness-errorswrites it anyway withcomplete: falseand the problems recorded.ambiguous_failureis recorded but still counts as unresolved.scripts/validate_driver.py: default smoke set swapspvlib__pvlib-python-1606forpydicom__pydicom-1256(gold-verified);docs/adding_a_driver.mdupdated.docs/benchmark_scoring.md, CHANGELOG.Verification
tests/test_swebench_bench.py: 36 tests (5 cover the fail-closed scoring: fatal vs ambiguous categories, an old-schema report, exit 3 with no score file,--allow-harness-errors, ambiguous-only). Fixtures are real: the 23 dev instances with the gold-check result, the per-task outcomes of six scored draws, and the tasks selected on 2026-09-24. They pin the reported intervals (53-98, 41-93, 22-78, 60-91, 31-69), the sign test (4 vs 0 -> 0.125), and reproduce the previously chosen 8-task set from seed 20260924.gold-check8/8 valid;scoreon the six draws gave 7/8, 6/8, 6/8 and 4/8, 4/8, 4/8, identical to the earlier scoring;comparegave 4 strictly more / 4 ties / 0 behind, p = 0.125, pooled 19/24 (59.5-90.8%) vs 12/24 (31.4-68.6%).ruff check .andruff format --check .clean.Not verified
ratelimit-remaining: 0, 100 per 3600 s), scoring one task whose image was not local made the harness reporterror_ids: [sympy__sympy-17139]with an emptyunresolved_ids. The old code would have counted that as an unresolved task; the new code exited 3 and wrote no score file (score.incomplete.jsononly). The unit tests fake the subprocess. Across 80 harness reports from the earlier run, none haderror,infra_failureorincompleteentries (two had oneambiguous_failureeach), so the earlier headline numbers are unaffected.gold-check(plain,--sample 3 --seed 20260925 --split test) was run end to end on the Lite test split (3/3 gold-valid), including thedatasetsload; the larger 40-task selection was driven by a separate chunked wrapper, not by this subcommand.