[devin-ai-integration]

Summary

Test-only PR. It adds 4 new hermetic test files for the benchmark plumbing that runs without VMs or desktops. No changes to src/ or existing tests.

Module	Before	After
agentlab.benchmarks.osworld_axtree_preprocessing	0%	99%
agentlab.benchmarks.osworld	0%	99%
agentlab.benchmarks.setup_benchmark	0%	100%
agentlab.benchmarks.abstract_env	56%	100%

194 tests (193 pass, 1 strict xfail): test_osworld_axtree_preprocessing.py (83), test_osworld_unit.py (67), test_setup_benchmark.py (29), test_abstract_env.py (15). Each file runs in about 1s. The tests pass serially, with -n 4, and under pytest-randomly seeds 1–3.

How the tests stay hermetic:

• osworld: monkeypatch.setattr(osworld, "DesktopEnv", MagicMock()) works when desktop_env is absent (the module sets it to None). That lets the tests run the real OsworldGym.__init__/reset/step/close, the obs conversion and the action conversion. time.sleep is patched. OsworldBenchmark reads task JSON files written to tmp_path.
• The accessibility-tree XML is built by hand for both the ubuntu and windows namespaces. draw_bounding_boxes gets PNG bytes of a small gradient image (or a solid-colour image for the "single colour region" skip path).
• trim_accessibility_tree: a fake tiktoken module goes into sys.modules, so there's no BPE download.
• setup_benchmark: os.system / _ensure_repo are patched, so nothing is cloned. MINIWOB_URL is registered with monkeypatch before setup_miniwob writes os.environ directly, so the variable doesn't leak between tests.

Uncovered lines:

• osworld.py 31-32: the from desktop_env ... imports, which only run when desktop_env is installed.
• osworld_axtree_preprocessing.py 197: bottom_right < coords can't happen, because the positive-size check just before it already rejects those nodes.

Bugs found (xfail strict)

• tag_screenshot(..., platform="windows") doesn't pass platform on to draw_bounding_boxes. Windows trees are then read with the ubuntu component namespace and nothing gets tagged (test_windows_platform_tags_nodes).

Other things the tests pin (current behavior, not marked as bugs)

• convert_agentlab_action_to_computer_13("not a call") returns {"action_type": "NONE", "parameters": {}} instead of rejecting the action.
• AbstractBenchmark.get_version is annotated -> int but returns "1".

Run

uv run pytest tests/benchmarks/test_osworld_axtree_preprocessing.py tests/benchmarks/test_osworld_unit.py tests/benchmarks/test_setup_benchmark.py tests/benchmarks/test_abstract_env.py -p no:cacheprovider -q [-n 4]

Devin-Org: engineering

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/0272da97199448aba69dac1e9fa07eca
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/0272da97199448aba69dac1e9fa07eca?variant=devin
Requested by: @danigrim

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.