Conversation
Two problems seen in a real SWE-bench Lite evaluation of local models: - The agent ran `pip install -e .` and `pip install "setuptools<81"` and, with Godspeed's own venv first on PATH, those installs landed in Godspeed's venv and leaked between tasks. - The agent's shell shares the host filesystem, and one model searched it for hidden tests and gold data (`find / -name "*1359*"`, greps of a failing test's name over /home, greps of the swebench package for FAIL_TO_PASS). Changes: - shell tool: on POSIX, $GODSPEED_SHELL_WRAPPER replaces /bin/bash (called as WRAPPER -c COMMAND). - scripts/agent_shell_isolate.sh: runs one command in a private mount + PID namespace. $HOME and /mnt are empty tmpfs, /tmp is a per-task private directory (persists across that task's commands), the workspace and task venv stay visible and writable, uv Pythons are visible read-only, /etc/resolv.conf is kept (on WSL2 it points into the masked /mnt/wsl), all capabilities are dropped, and it fails closed if namespaces cannot be created. - experiments/swebench_lite/isolation.py + run.py flags --isolate-agent-shell and --task-python VERSION: per-task throwaway venv (uv) first on PATH, wrapper env applied for the task and restored afterwards; preflight exits 2 if isolation is requested but unavailable. - scripts/audit_agent_commands.py: flags agent commands that search or read outside the workspace (wide-search, outside-path, hunt-term) in a run log. - docs/benchmark_hygiene.md, CHANGELOG. Tests: tests/test_tools/test_shell_wrapper.py, tests/test_benchmark_isolation.py (environment handling, preflight, and integration tests of the real script that skip where unprivileged namespaces are unavailable, incl. a regression test for the venv being visible when it lives under /tmp), tests/test_audit_agent_commands.py (built from real commands an agent ran).
🤖 Godspeed Review
SecurityNo secrets detected in changed files. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ), | ||
| ): | ||
| raise RuntimeError("boom") | ||
| assert "AGENT_PRIV" not in os.environ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two problems seen in a real SWE-bench Lite evaluation of local models (details in
docs/benchmark_hygiene.md):pip install -e .andpip install "setuptools<81"; with Godspeed's venv first onPATHthose landed in Godspeed's own venv (a task's editable install was still there for the next task).find / -name "test_datetime_list_inner_format",grep -rn <that name> /home/..., greps of theswebenchpackage forFAIL_TO_PASS,find / -name "gold_results*"andfind / -name "*1359*"(the instance number). Gold patches, the dataset cache and other runs' predictions live on that filesystem. In that unsandboxed run 21 of 376 shell commands left the workspace, versus 1 of 165 for another model under the same harness.Changes
$GODSPEED_SHELL_WRAPPERreplaces/bin/bash(called asWRAPPER -c COMMAND). Read once and cached; ignored on Windows.scripts/agent_shell_isolate.sh: one command per call in a private mount + PID namespace.$HOMEand/mntare empty tmpfs;/tmpis a per-task private directory that persists across that task's commands; the workspace and task venv stay visible and writable; uv-managed Pythons are visible read-only (an agent hadpip installed into the shared base interpreter);/etc/resolv.confis kept (on WSL2 it points into the masked/mnt/wsl, and without it the agent silently loses DNS); all capabilities are dropped; fails closed if namespaces cannot be created.experiments/swebench_lite/isolation.py+run.py --isolate-agent-shell/--task-python VERSION: per-task throwaway venv viauv(first onPATH,pytestpreinstalled), wrapper env applied for the task and restored afterwards, preflight exits 2 if isolation is requested but unavailable.scripts/audit_agent_commands.py: flags agent commands that search or read outside the workspace (wide-search,outside-path,hunt-term;--allow,--term,--fail-on-flag).docs/benchmark_hygiene.md(usage, requirements, and what this does not do), CHANGELOG.Verification
test_shell_wrapper.py,test_benchmark_isolation.py(13; environment handling, preflight, and integration tests of the real script: home empty and planted "gold" unreachable, workspace edits reach the host, private/tmppersists across commands, capabilities dropped and mounts cannot be undone, fails closed with a brokenunshare, usage error, venv visible at its own path under/tmp),test_audit_agent_commands.py(built from real commands an agent ran)./tmphad been replaced and silently skipped binding it; the test fails on that version and passes on the fix (checked).task_isolation(python="3.9", isolate_shell=True)+ the realShellTool: agent sees Python 3.9.25 from its own venv,$HOMEshows only the uv directory,/mntonly the resolver file, afind / -name "*1359*"hunt finds nothing,pip install sixworks over the network and lands only in the task venv, edits reach the host, environment restored afterwards.ruff check ./ruff format --check .clean;run.py --isolate-agent-shellwith nounshareon PATH exits 2 with a clear message.Not verified / limits
kernel.apparmor_restrict_unprivileged_userns=1, macOS, Windows), so CI may not exercise them; I ran them on WSL2 (kernel 6.18).test_shell*,test_benchmark_isolation,test_audit_agent_commands,test_source_encoding) pass (558 tests) and CI runs the rest.