[devin-ai-integration]

Summary

Adds hermetic unit tests (5 new files, tests only — no src/ or existing-test changes) covering study orchestration and reproducibility bookkeeping without launching experiments.

• Study is built from a real bgym.DEFAULT_BENCHMARKS["miniwob_tiny_test"]() + GenericAgentArgs(CheatMiniWoBLLMArgs); run_experiments, prepare_backends, inspect_results, and find_incomplete are patched. ParallelStudies uses fake ProcessPoolExecutor/Pool/Manager (no real processes).
• Git-info tests use throwaway git.Repo.init(tmp_path) repos with monkeypatched GIT_AUTHOR_*/GIT_COMMITTER_*; journal tests write to tmp_path only.
• view_dep_graph runs plotting at import time, so tests patch bgym.DEFAULT_BENCHMARKS["visualwebarena"] with a fake benchmark, force matplotlib.use("Agg"), and stub plt.show.

Coverage (target modules, new tests only)

Module	Before (whole suite on main)	After	Goal
experiments.study	31%	99%	≥70%
experiments.reproducibility_util	37%	100%	≥80%
experiments.args	81%	97%	≥95%
experiments.view_dep_graph	0%	99%	≥70%
experiments.multi_server	78%	100%	≥95%

Remaining misses: args.py 301-315 is the if __name__ == "__main__" block; study.py 345-348 is the no-progress retry stop, unreachable because of the last_error_count bug below; view_dep_graph.py 290 is a defensive has_node guard.

Tests

167 tests: 162 passed, 5 xfailed (strict). Each file < 1s–4s; passes serially, with -n 4, and in random order (pytest-randomly).

xfail'd source bugs (strict=True)

1. study.Study.run: last_error_count is initialized to None and never updated, so "Last trial did not reduce the number of errors" never fires.
2. study.SequentialStudies.run: calls self.make_dir() without exp_root, so the parent study dir is created under RESULTS_DIR instead of the given exp_root.
3. reproducibility_util._get_git_info: repo.index.diff(repo.head.commit) is reversed — a staged new file is reported as D instead of A.
4. reproducibility_util.append_to_journal: when creating a new journal, headers come from info.keys() only, so result columns (avg_reward, std_err, …) are dropped.
5. multi_server.WebArenaInstanceVars.from_env_vars(prefix=...): the custom prefix isn't stored on the returned instance (it keeps prefix="WA_").

Run

uv run pytest tests/experiments/test_study_unit.py tests/experiments/test_reproducibility_util_unit.py tests/experiments/test_args_unit.py tests/experiments/test_view_dep_graph.py tests/experiments/test_multi_server_unit.py -p no:cacheprovider -q   # add -n 4 for xdist

Devin-Org: engineering

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/5559d53f195643d6843f249e797f1c88
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/5559d53f195643d6843f249e797f1c88?variant=devin
Requested by: @danigrim

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

🔍 Devin Review: 3 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)