Skip to content

Write conversation logs from the in-loop benchmark runner - #237

Open
t-timms wants to merge 1 commit into
mainfrom
fix/in-loop-conversation-logging
Open

t-timms wants to merge 1 commit into
mainfrom
fix/in-loop-conversation-logging

Conversation

@t-timms

@t-timms t-timms commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Why

experiments/swebench_lite/run_in_loop.py never created a ConversationLogger, so log_conversations: true (which the CLI honours) did nothing for benchmark runs. After a full night of agent-in-loop evaluation (dozens of agent sessions) there was no training/ directory at all. Benchmark runs generate exactly the trajectories any later tuning needs (tool calls, thinking, outcomes, all scored by the official harness), and they were being discarded.

Changes

  • The runner creates a ConversationLogger when settings.log_conversations is on, passes it to Conversation, writes the terminal session_end record (exit reason and code from EXIT_REASON_TO_CODE, iterations, tool calls, duration, cost) and closes the file on every exit path (normal, timeout, exception).
  • CHANGELOG entry.

Verification

  • tests/test_run_in_loop_wiring.py gains two tests: with log_conversations: true one <session>.conversation.jsonl exists, starts with the system record and ends with a session_end record (exit_reason: stopped, exit_code: 0); with it off no file is written. The enabled-case test fails without this change and passes with it (checked).
  • ruff check . / ruff format --check . clean.

Not verified

  • Not run against a live model; the stub agent loop produces only the system and session_end records, so the per-turn records rely on the existing Conversation logging that the CLI path already exercises.
  • The full suite was not re-run on this branch (CI runs it).

experiments/swebench_lite/run_in_loop.py never created a ConversationLogger, so with
log_conversations: true (which the CLI honours) every benchmark trajectory was discarded: after
a full evaluation night ~/.godspeed/training did not exist for those runs. The runner now writes
<global_dir>/training/<session>.conversation.jsonl, with the terminal session_end record
(exit reason and code from EXIT_REASON_TO_CODE), and closes the file on every exit path.

Tests: tests/test_run_in_loop_wiring.py gains a test that the log and its session_end record
exist when enabled and are absent when disabled; the enabled test fails without this change.
@github-actions

Copy link
Copy Markdown

🤖 Godspeed Review

LLM review skipped (no API key configured).

Security

No secrets detected in changed files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant