[devin-ai-integration]

Summary

Closes the coverage gap flagged by the code scan: the headline benchmark numbers (get_std_err, get_sample_std_err, summarize) and error bucketing (map_err_key, error_report) in src/agentlab/analyze/inspect_results.py had no direct tests, so a regression would silently misreport results.

New tests/analyze/test_inspect_results_stats.py (21 tests) builds small hand-made dataframes and asserts:

• get_std_err on binary rewards matches the Bernoulli formula, drops NaNs, and gives 0 for an all-success frame; non-binary rewards delegate to get_sample_std_err.
• get_sample_std_err matches std(ddof=1)/sqrt(n) and normalizes the singleton NaN to 0.
• summarize on a mixed success/error frame: avg_reward, std_err, avg_steps, n_completed (errored runs count as completed, unfinished ones do not), n_err, cum_cost, and cum_effective_cost replacing cum_cost; plus None when nothing completed, the no-cum_reward branch, and AssertionError when an errored run carries non-zero reward.
• map_err_key strips the === logs === tail and normalizes token counts and task names so equivalent failures collapse; error_report buckets them, lists tasks sorted by name, and honours max_stack_trace (get_exp_result is monkeypatched with lightweight fakes).

One behavior fix was needed to make bucketing correct: map_err_key sliced on err_msg.find("=== logs ===") unconditionally, so a message with no logs marker lost its last character (find returns -1).

-err_msg = err_msg[: err_msg.find("=== logs ===")].rstrip()
+logs_idx = err_msg.find("=== logs ===")
+if logs_idx >= 0:
+    err_msg = err_msg[:logs_idx].rstrip()

pytest tests/analyze passes (24 tests); black --check clean.

Devin-Org: engineering

Link to Devin session: https://app.devin.ai/sessions/2ac2d4e40de941eda7cea079da93e133
Open in Devin Desktop: https://app.devin.ai/desktop/session/2ac2d4e40de941eda7cea079da93e133?variant=devin

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring