Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,26 @@ uv tool install nanoPyCodeAgent # or: pipx install nanoPyCodeAgent
nanoPyCodeAgent
```

#### Run one task and exit

Give it a task and it works through it on its own, with no prompt and nothing
to confirm — the shape a script or a benchmark harness needs:

```bash
nanoPyCodeAgent -p "add a --version flag and run the tests"
nanoPyCodeAgent --prompt-file task.md
printf "%s" "$TASK" | nanoPyCodeAgent
```

The task is carried out in the current directory. `--max-turns N` caps how
many model replies one run may spend (50 by default).

A run like this exits `0` whenever the agent actually ran — including when it
gave up or ran out of turns with the task unfinished, which is for whatever
checks the result to judge. A non-zero exit means the run could not happen at
all: `1` for missing credentials or an API that kept refusing, `2` for a
misused command line.

#### Run a branch or tagged version

Run an unreleased branch or a specific release tag straight from GitHub:
Expand Down
18 changes: 18 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,24 @@ uv tool install nanoPyCodeAgent # 或: pipx install nanoPyCodeAgent
nanoPyCodeAgent
```

#### 一次性任务(非交互)

给它一个任务,它会自己做完退出——不给提示符,也不会停下来等确认,这正是脚本或
benchmark harness 需要的形态:

```bash
nanoPyCodeAgent -p "add a --version flag and run the tests"
nanoPyCodeAgent --prompt-file task.md
printf "%s" "$TASK" | nanoPyCodeAgent
```

任务在当前目录下执行。`--max-turns N` 限制一次运行最多花费多少轮模型回复(默认
50 轮)。

只要 agent 真的跑起来了,退出码就是 `0`——包括它放弃了、或者轮数用尽而任务没做
完,那该由检查结果的一方去判定。非零退出码表示这次运行根本没能进行:`1` 是缺少
凭据或 API 持续失败,`2` 是命令行用错了。

#### 运行某个分支或标签版本

直接从 GitHub 运行未发布的分支,或某个具体的发布标签:
Expand Down
5 changes: 5 additions & 0 deletions docs/changelogs/0.8.x.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ All notable changes in the **0.8.x** release series are documented here.
three. Notably, a benchmark harness reads a non-zero exit code as its own
failure, so an agent that runs out of turns without solving the task must
still exit 0.
- A headless CLI for running one task and exiting via `-p` / `--prompt`,
`--prompt-file`, or piped stdin. Headless runs use a dedicated
non-interactive system prompt, support a configurable `--max-turns` budget
and `--version`, keep the run itself on stdout, return benchmark-safe exit
codes, and preserve API and transport error text on stderr.

<!--
When cutting a release, copy the relevant items from [Unreleased] into a new
Expand Down
52 changes: 52 additions & 0 deletions docs/dev_notes/en/0.8.x.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Development Notes — 0.8.x

> Generated from the Chinese source [`../zh-CN/0.8.x.md`](../zh-CN/0.8.x.md). Do not edit by hand.

## 0.8.0 - YYYY.MM.DD

We have now built several basic tools — bash, write, read, and edit — roughly matching Pi's basic toolset. The next step is to learn from practice which capabilities genuinely need improving and which new ones would strengthen the harness. To do that, we need to run public benchmarks. The goal is to use benchmarks reported in mainstream model releases so that, with the model held constant, we can compare different harnesses and identify concrete optimization targets.

I first surveyed the landscape in [code_agent_benchmark](../../research/en/code_agent_benchmark.md). The survey showed that, regardless of which benchmark we choose, the first requirement is a non-interactive way to invoke the code agent. Only then can a program run the agent end to end on a task and evaluate the result. Based on the first benchmarks recommended by that survey, I then documented a practical non-interactive interface in [benchmark_headless_interface](../../research/en/benchmark_headless_interface.md).

The first feature is therefore a **headless CLI**, the first step in the survey's minimum viable path. It is the only hard blocker: the first three requirements shared by all three benchmarks — accept task text in one command and run to completion, work in the process's current directory, and never prompt, ask questions, or wait for confirmation — all depend on it. The only existing entry point is the `input("You> ")` loop in `agent.py`; inside a container stdin is EOF, so the process immediately prints `Bye!` and exits without doing anything. Until this works, none of the later harness improvements can be verified programmatically, so there is no basis for deciding what to optimize.

### Headless CLI contract

**Command-line shape**:

```text
nanoPyCodeAgent [-p/--prompt "<task>" | --prompt-file <path> | (piped stdin)]
[--max-turns N]
[--version]
```

`--version` is not decorative: Harbor uses an adapter's `get_version_command()` / `parse_version()` to probe and record the agent version on a best-effort basis. `_package_version()` already existed; it only needed to be exposed through the CLI.

**Headless detection**: passing `-p` or `--prompt-file` selects headless mode. When neither is present, the entire stdin stream becomes the task if `sys.stdin.isatty()` is false; only a tty with no supplied task starts the existing REPL. All three input forms are required. Harbor's two official examples already use two of them: Claude Code uses a stdin pipe (`printf … | claude --print`) to avoid shell quoting and command-length limits, while mini-swe-agent passes `--task=<shlex.quote(instruction)>` and explicitly connects stdin to `/dev/null`. This also fixes the behavior where EOF on stdin inside a container immediately printed `Bye!` and exited.

**Exit codes**:

| Exit code | Scenario |
| :-: | --- |
| 0 | The model declares completion; the turn budget is exhausted; the task is not completed |
| Non-zero | Missing API credentials; invalid arguments; API or transport errors |

"Exit 0 even when unfinished" is counterintuitive. Harbor runs the agent under `set -o pipefail`; a non-zero code immediately raises `NonZeroAgentExitCodeError`, marks the entire trial as an agent failure, and may trigger a retry that only wastes money. Whether the task succeeded belongs to the verifier reading the reward file under `/logs/verifier/`, not to the agent's process exit code.

This also exposes an existing bug: without API credentials, `run()` printed a message and returned, while `main()` had no concept of a return code, so the process ultimately exited 0. To the harness, that meant "the run completed but did not solve the task." An entire batch could quietly score zero with no indication that configuration was the problem. `main()` therefore has to return an integer that the generated console script can use as its exit code.

**Turn limit**: the inner tool-use loop currently has no bound. In interactive mode, someone watching can press Ctrl-C when the model gets stuck retrying the same command; in headless mode, nobody is there, so it can keep spending until the API refuses further calls. `--max-turns` costs only a counter but is essential for unattended runs, so it belongs in the CLI. Exhausting the budget still exits 0 under the table above.

**Headless system prompt**: the current prompt is written for a conversational assistant. In headless mode there is nobody to answer; if the model politely asks "should I continue?", that reply ends the run and the task scores zero. Headless mode therefore uses a separate system prompt that explicitly says not to ask the user questions, not to stop for confirmation, to make decisions independently through completion, and to state clearly when the work is done.

**Print API errors unchanged**: Harbor classifies errors by applying regular expressions to the agent's stdout and stderr — rate limits, usage limits, `Overloaded`, context exhaustion, authentication failures, connection drops, and so on. That classification, together with options such as `--max-retries 3 --retry-include ApiRateLimitError`, decides whether a retry is worthwhile. Error text should therefore be preserved rather than swallowed or rewritten, which removes half of the retry work from the agent itself.

**Acceptance**: one command is enough to show whether this step truly works:

```bash
printf "%s" "create hello.py that prints hi" | nanoPyCodeAgent; echo $?
```

If the task arrives through the pipe, the file is created, and the exit code is 0, the step is complete. This is the first point at which nanoPyCodeAgent can be called by a script, and only then can it be connected to a benchmark.

After this, the next work is: `--output-format stream-json` and `--trajectory` (step 4 of the minimum viable path, once failures need attribution), API retry and backoff (step 2), context compaction (step 5), `--workdir` (downgraded from P0 by the survey because all three benchmarks pass their working directory through the container's default `WORKDIR`; the agent only needs to operate in the current process directory), and `--timeout` (the harness owns the wall-clock timeout — Harbor stores it in `[agent].timeout_sec` in `task.toml` — so an agent-side timeout is a safeguard rather than an integration prerequisite).
52 changes: 52 additions & 0 deletions docs/dev_notes/zh-CN/0.8.x.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# 开发笔记 — 0.8.x

> 本文件为**手写中文源文件**(source of truth);英文版 [`../en/0.8.x.md`](../en/0.8.x.md) 由其生成。

## 0.8.0 - YYYY.MM.DD

目前已经开发了一些基本的 tool: bash, write, read, edit,类似 pi 的基本工具已经有了。下一步,为了从实践中了解,到底要优化哪些功能,新增哪些功能,可以真正完成 harness 层面的一些能力提升,需要跑一些公开的 benchmark,目标就是参考主流模型发布时评测的 benchmark,这样可以在相同模型下,对比不同 harness 的能力,以此找到优化目标。

首先,还是先做了调研,参考 [code_agent_benchmark](../../research/zh-CN/code_agent_benchmark.md),从调研结果看,不管要评测什么 benchmark,首先要实现的,就是非交互模式的 code agent 调用,只有这样,才能通过程序化的方式去运行 agent,端到端地执行任务,再根据任务结果去评测。然后,又根据首先推荐的几个 benchmark,整理了方便运行 benchmark 的非交互模式接口,参考 [benchmark_headless_interface](../../research/zh-CN/benchmark_headless_interface.md)。

第一个功能,先实现 **headless 模式的 CLI**,也就是调研里「最小可行路径」的第一步。它是唯一的硬阻塞项:三个 benchmark 的最小公共契约的头三条——一条命令拿到任务文本跑完退出、在进程当前 cwd 里干活、不交互不提问不等确认——全部卡在这里。当前唯一入口是 `agent.py` 的 `input("You> ")` 循环,容器里 stdin 是 EOF,进程会立刻打印 `Bye!` 退出,一件事都没干。这一步不通,后面任何 harness 层面的改动都无法被程序化地验证,也就无从谈优化目标。

### headless CLI 实现契约

**命令行形态**:

```text
nanoPyCodeAgent [-p/--prompt "<任务>" | --prompt-file <path> | (stdin 管道)]
[--max-turns N]
[--version]
```

`--version` 不是凑数的:Harbor 会用适配类的 `get_version_command()` / `parse_version()` 去探测并记录 agent 版本(best-effort,失败不报错)。`_package_version()` 已经写好了,只差挂到 CLI 上。

**headless 判定**:给了 `-p` 或 `--prompt-file` 就是 headless;两者都没给时,若 `sys.stdin.isatty()` 为 False,就把整段 stdin 读进来当任务;只有连着 tty 且没给任务,才进现有的 REPL。三种方式都要实现——Harbor 的两个官方范例正好各占一种:Claude Code 用 `printf … | claude --print` 走 stdin 管道,避开 shell 转义和命令行长度限制;mini-swe-agent 用 `--task=<shlex.quote(instruction)>` 走参数,并显式把 stdin 接到 `/dev/null`。顺带也就修掉了「容器里 stdin 是 EOF 就立刻 `Bye!` 退出」这个行为。

**退出码**:

| 退出码 | 场景 |
| :-: | --- |
| 0 | 模型声明完成;轮数用尽;任务没做成 |
| 非 0 | 无 API 凭证;参数错误;API 或传输错误 |

「没做完也 exit 0」比较反直觉:Harbor 把 agent 命令包在 `set -o pipefail` 下执行,非零退出码直接抛 `NonZeroAgentExitCodeError`,整个 trial 判成 agent 失败,还可能触发重试白烧钱。任务做没做成该由 verifier 从 `/logs/verifier/` 的 reward 文件判,不该由 agent 的退出码判。

这里还牵出一个现存 bug:没有 API 凭证时,`run()` 是 `print` 完直接 `return`,`main()` 又没有返回码的概念,最终退出码是 0——在 harness 眼里这等于「跑完了,只是没做出来」,一整批任务会静悄悄地全判 0 分,而且看不出是配置问题。所以 `main()` 要改成返回 int(console script 会拿返回值当退出码)。

**轮数上限**:内层的 tool_use 循环现在没有任何上限。交互模式下有人看着,模型卡在「反复试同一条命令」时可以 Ctrl-C;headless 下没人看着,会一路烧到 API 报错为止。`--max-turns` 只是一个计数器的成本,却是 headless 能不能无人值守跑的前提,所以和 CLI 一起做。轮数用尽按上表算 exit 0。

**headless 的系统提示词**:当前这份是写给对话助手的。headless 下模型没有人可问——它礼貌地问一句「要我继续吗」,这一轮就结束了,这题直接 0 分。所以 headless 用一份单独的系统提示词,显式要求:不要向用户提问、不要停下来等确认、自己决策到底、完成后明确声明结束。

**API 错误原样打出来**:Harbor 用正则扫 agent 的 stdout/stderr 做错误分类(rate limit、usage limit、Overloaded、上下文超限、未登录、网络中断……),分类结果配合 `--max-retries 3 --retry-include ApiRateLimitError` 决定要不要重试。所以错误信息宁可原文吐出去,也不要吞掉或改写——重试的活儿因此可以少干一半。

**验收**:一行命令就能说明这一步是否真的通了——

```bash
printf "%s" "create hello.py that prints hi" | nanoPyCodeAgent; echo $?
```

任务从管道输入、创建文件、退出码是 0,这一步就算完成。此后 nanoPyCodeAgent 才第一次具备「被脚本调用」的能力,也才谈得上接 benchmark。

上述完成后,后续要做的:`--output-format stream-json` 与 `--trajectory`(属最小可行路径第 4 步,等失败需要归因时再做)、API 重试与退避(第 2 步)、上下文压缩(第 5 步)、`--workdir`(调研已把它从 P0 降级:三家 benchmark 的工作目录都是靠容器默认 WORKDIR 传递的,agent 在进程当前 cwd 里干活即可)、`--timeout`(墙钟超时由 harness 侧管,Harbor 在 `task.toml` 的 `[agent].timeout_sec` 里,agent 自己的超时是保险而非接入前提)。
6 changes: 2 additions & 4 deletions src/nanopycodeagent/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
from nanopycodeagent.agent import run
from nanopycodeagent.cli import main


def main() -> None:
run()
__all__ = ["main"]
Loading