[giladsamuels-cognition]

Summary

Before this PR, tests/experiments/test_study.py only had skipped tests, so make_study dispatch, Study.make_exp_args_list and the Study.run relaunch loop had no real coverage. This PR adds tests for them that don't need a MiniWoB server or a real LLM. They use CheatMiniWoBLLMArgs on miniwob_tiny_test (4 envs). No production code changes.

• make_study dispatch: one agent returns a Study. Several agents on miniwob return a single Study. Several agents on a webarena*-named benchmark (a dataclasses.replace copy of the tiny benchmark) return a SequentialStudies with one sub-study per agent. parallel_servers=2 returns a ParallelStudies.
• Cross-product: 2 agents × 4 envs gives 8 ExpArgs in agent-major order, with order 0..7 and logging levels passed through. Re-running make_exp_args_list after changing agents rebuilds the list. More than one agent on a webarena benchmark raises ValueError.
• Study.find_incomplete: checks the (n_incomplete, n_error) counts against the existing tests/data/test_study fixture.
• Study.run loop: set_reproducibility_info, _run, get_results and find_incomplete are patched at class level so the study still pickles through the real save(exp_root=tmp_path). The tests check that:
  • the loop stops early when nothing is left incomplete;
  • n_relaunch caps the number of trials;
  • it stops after the first trial when more than 30% of experiments error, and keeps going when fewer do;
  • relaunch_errors is passed to find_incomplete as include_errors, and strict_reproducibility reaches both set_reproducibility_info and _run.

Agent flags are deep-copied because set_benchmark changes them in place. This keeps the shared FLAGS_GPT_4o from being modified.

Note for reviewers (not changed here): in Study.run, last_error_count is never reassigned after it is set to None. So the "Last trial did not reduce the number of errors" check never fires. The tests don't depend on that branch.

pytest tests/experiments/test_study.py → 14 passed, 1 skipped.

Link to Devin session: https://app.devin.ai/sessions/3583e45b1ae5405ebd59c438e57d417b
Open in Devin Desktop: https://app.devin.ai/desktop/session/3583e45b1ae5405ebd59c438e57d417b?variant=devin
Requested by: @giladsamuels-cognition

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.