[devin-ai-integration]

Summary

Adds hermetic unit tests for result loading, aggregation and reporting in agentlab.analyze.inspect_results and agentlab.analyze.archive_studies. Test-only: two new files, no src/ changes.

• tests/analyze/test_inspect_results_unit.py: 92 tests (85 pass + 7 strict xfail)
• tests/analyze/test_archive_studies.py: 13 tests (12 pass + 1 strict xfail)

Fixtures are built two ways. Some are hand-made DataFrames that follow the load_result_df column conventions. Others are synthetic experiment dirs in tmp_path: a pickled ExpArgs with a test-local FakeAgentArgs, plus summary_info.json, experiment.log and step_*.pkl.gz files. Staleness is controlled with os.utime. An autouse fixture swaps loop.EXP_RESULT_CACHE for an empty dict so no state leaks between tests. Stats tests assert exact values (for example the binary std err (\sqrt{p(1-p)/n}) and the ddof=1 sample std err). The __main__ block of archive_studies runs through runpy with exp_utils.RESULTS_DIR monkeypatched to tmp_path.

Coverage (target modules, measured with these files only)

Module	Before (suite on main)	After
agentlab.analyze.inspect_results	31%	92%
agentlab.analyze.archive_studies	0%	95%

Uncovered lines that remain:

• inspect_results 650-660, 672-693, 711-718, 721, 735-736, 745: code after the undefined ERR_CLASS_MAP lookup. It can't run until the bug below is fixed.
• archive_studies 118-122: the move-to-archive branch. The script hardcodes archive_dir = None, so this branch is dead.

Source bugs found (tests marked xfail(strict=True))

1. map_err_key drops the last character when the message has no === logs === marker, because find() returns -1 and the slice becomes err_msg[:-1]. Example: "Some error" comes back as "Some erro".
2. categorize_error refers to ERR_CLASS_MAP, is_critical_server_error and is_minor_server_error, which are never defined or imported. Any row with an error raises NameError, and so do error_report_detailed, print_errors_chronologically and report_different_errors.
3. report_different_errors asserts that .sum() returns a DataFrame, but .sum() always returns a Series, so it raises AssertionError even when there are no errors (there is already a # TODO: fix this bug comment).
4. archive_studies.search_for_reasons_to_archive raises ZeroDivisionError on an empty summary (n_total == 0) when it should flag the study with "Empty summary DataFrame" / "No tasks".
5. get_study_summary notices a stale summary and rebuilds it, but get_exp_result hands back cached ExpResult objects with the old summary_info. The rebuilt summary and the CSV written to disk therefore keep the old rewards. Found by Devin Review.

Also observed, not marked as a bug: set_index_from_variables adds env.task_name to the index twice if index_white_list matches env.*.

Run

uv run pytest tests/analyze/test_inspect_results_unit.py tests/analyze/test_archive_studies.py -p no:cacheprovider -q        # 97 passed, 8 xfailed
uv run pytest tests/analyze/test_inspect_results_unit.py tests/analyze/test_archive_studies.py -p no:cacheprovider -q -n 4   # same
uv run --with pytest-cov pytest tests/analyze/test_inspect_results_unit.py tests/analyze/test_archive_studies.py -p no:cacheprovider -q --cov=agentlab.analyze.inspect_results --cov=agentlab.analyze.archive_studies --cov-report=term-missing

Devin-Org: engineering

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/f3bd4e2463ad4873993623c7bd85b3ae
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/f3bd4e2463ad4873993623c7bd85b3ae?variant=devin
Requested by: @danigrim

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Devin Review found 1 potential issue.