Make the agent-in-loop verify oracle work with swebench 5.x - #231
Merged
Merged
Conversation
experiments/swebench_lite/verify_patch.py failed on a fresh Linux box with swebench 5.0.2 installed, in three independent ways (each reproduced): - It ran the harness with the hard-coded interpreter /home/swebench_venv/bin/python3, which does not exist. Now the interpreter is $GODSPEED_SWEBENCH_PYTHON if set, else the legacy venv when it exists (WSL from Windows keeps working unchanged), else the current interpreter. - It always passed `--cache_level instance`; swebench 5.x removed that flag, so argparse exits with "unrecognized arguments". The flag is now only passed when `run_evaluation --help` lists it (probed once per interpreter). - The one-row dataset came only from benchmarks/swebench_lite_test.jsonl, which is not in the repo, so an empty dataset was written and every verification reported "harness failed". The row is now looked up in that file if present, otherwise in the HuggingFace SWE-bench Lite splits (requested split first). A stale empty dataset file is regenerated, and an instance that is found nowhere returns a clear message instead of invoking the harness. verify_patch() gains a `split` argument (default "dev", unchanged), and SWEBenchVerifyTool passes its own split through. Tests: tests/test_swebench_verify_patch.py (harness and datasets faked).
🤖 Godspeed Review
SecurityNo secrets detected in changed files. |
…e field Running the real harness (gold patches, swebench 5.0.2) with princeton-nlp/SWE-bench_Lite fails in make_test_spec with KeyError: 'image'. That older snapshot lacks the image, eval_script, log_parser, eval_type and difficulty columns that swebench 5.x reads; SWE-bench/SWE-bench_Lite has them. verify_patch built its one-row dataset from the old name, so it would have hit the same error. LITE_DATASET now points at SWE-bench/SWE-bench_Lite.
🤖 Godspeed Review
SecurityNo secrets detected in changed files. |
🤖 Godspeed Review
SecurityNo secrets detected in changed files. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
experiments/swebench_lite/verify_patch.py(the oracle behindswebench_verify_patch, used by--agent-in-loopandscripts/validate_driver.py) failed on a fresh Linux box withswebench5.0.2. Four independent failures, each reproduced:/home/swebench_venv/bin/python3ls /home/swebench_venv-> No such file$GODSPEED_SWEBENCH_PYTHON, else the legacy venv if it exists (WSL-from-Windows unchanged), else the current interpreter--cache_level instanceerror: unrecognized arguments: --cache_level instancerun_evaluation --helplists it (probed once per interpreter)benchmarks/swebench_lite_test.jsonl, which is not in the repoprinceton-nlp/SWE-bench_Litelacks the columns swebench 5.x readsKeyError: 'image'inmake_test_spec;SWE-bench/SWE-bench_Litehasimage,eval_script,log_parserLITE_DATASET = "SWE-bench/SWE-bench_Lite"verify_patch()gainssplit="dev"(default unchanged) andSWEBenchVerifyToolpasses its own split.Verification
tests/test_swebench_verify_patch.py: interpreter selection, command composition with/without the flag, help probing, dataset lookup (local, HF, missing), andverify_patch()end to end with the harness subprocess anddatasetsfaked. 29 passed together with the existingtest_swebench_verify_tool.py.ruff check .andruff format --check .(CI flags): clean.SWE-bench/SWE-bench_Litedev): 8 instances submitted, 8 completed, 0 errors; 6 resolved, 2 unresolved (see below).Related finding (not fixed here)
Two of those 8 dev instances cannot be resolved even by the gold patch in the currently published images, so they are invalid tasks today:
pvlib__pvlib-python-1606:AttributeError: np.Inf was removed in the NumPy 2.0 releasewhile importingpvlib(image has NumPy 2.x). This instance is inscripts/validate_driver.py's default smoke set (DEFAULT_SMOKE_INSTANCES), so that smoke run cannot pass on it.pyvista__pyvista-4315:ImportError: libGL.so.1: cannot open shared object file.Anyone benchmarking should gold-check an instance set first.
Not verified