feat: add a headless CLI for running one task and exiting - #21
Merged
Conversation
Record why the next step is a headless entry point: the benchmark surveys both conclude that nothing can be measured until the agent can be driven programmatically end to end.
Fix two broken links, add the release-date placeholder to the version heading, and write up the first feature the benchmark surveys point at: a headless CLI with harness-compatible exit codes.
The only way in was an input() loop, so a container — where stdin is at EOF — made the agent print "Bye!" and exit having done nothing. Nothing about the harness could be measured, because nothing could drive it end to end. A task can now arrive three ways: -p/--prompt, --prompt-file, or piped on stdin (taken as the task whenever stdin is not a terminal). With no task and a terminal attached, the interactive session opens as before. The run happens in the process's working directory, and --max-turns caps how many model replies an unattended run may spend. Exit codes answer one question: did the harness fail, or did the task? Everything that is merely a bad outcome — the model gave up, the turn budget ran out, the work is half done — exits 0 and leaves the verdict to whatever scores the result, because a benchmark reads a non-zero code as a broken agent and may pay to retry the trial. Only a run that could not happen exits non-zero: 1 for missing credentials or an API that kept refusing, 2 for a misused command line. API errors are printed verbatim, since a harness classifies a failed run by pattern-matching that text. A headless run also gets its own system prompt. With no user at the other end, a clarifying question or a pause for approval ends the run with the task untouched, which scores exactly like a wrong answer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
-p/--prompt,--prompt-file, or piped stdin, while preserving the existing interactive REPL for a terminal with no task.--max-turnsbudget, and--versionsupport.Error handling
Validation
.venv/bin/pytest -q— 129 passedhello.py, then exited 0