From 1a12856972c0f924b955ff0d0293eff9e3806e26 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 3 Sep 2026 01:15:21 +0800 Subject: [PATCH 1/8] docs: plan Pi and OMP trial stacks Signed-off-by: Kent Huang --- ...2026-09-03-drc-4282-pi-omp-trial-stacks.md | 1129 +++++++++++++++++ 1 file changed, 1129 insertions(+) create mode 100644 plans/2026-09-03-drc-4282-pi-omp-trial-stacks.md diff --git a/plans/2026-09-03-drc-4282-pi-omp-trial-stacks.md b/plans/2026-09-03-drc-4282-pi-omp-trial-stacks.md new file mode 100644 index 0000000..be202ad --- /dev/null +++ b/plans/2026-09-03-drc-4282-pi-omp-trial-stacks.md @@ -0,0 +1,1129 @@ +# DRC-4282 Pi and OMP Trial Stacks Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> superpowers:subagent-driven-development (recommended) or +> superpowers:executing-plans to implement this plan task-by-task. Steps use +> checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add upstream Pi (`pi`) and Oh My Pi (`omp`) as first-class Behavior Diff trial stacks while keeping one report format and clear ownership between skills and scripts. + +**Architecture:** Skills make choices that need judgment. Scripts run repeatable mechanics. `run-trial.sh` is the stack adapter. It converts Claude, Codex, Pi, and OMP output into the existing canonical `trace.jsonl` format. Pi and OMP use separate CLI command profiles and one private JSON normalizer. The grader and renderer stay stack-neutral. + +**Tech Stack:** Bash 3.2, Python 3, `jq`, Claude CLI, Codex CLI, Pi CLI, OMP CLI, deterministic shell contract tests. + +**Issue:** [DRC-4282](https://linear.app/recce/issue/DRC-4282/trials-only-run-on-claude-or-codex-decide-the-binaryskill) + +--- + +## Status + +This document records the accepted design. It also gives the implementation order and test gates. + +Accepted direction: **Add upstream Pi and OMP now.** + +The work adds two trial stacks. It does not add Pi or OMP plugin packaging. + +## Problem + +Behavior Diff currently accepts only two values for `--agent`: + +```text +claude +codex +``` + +The split between skill work and script work is also implicit. This creates two problems: + +1. A Pi or OMP user gets a flat refusal instead of a useful path. +2. Future stack work can put judgment in scripts or repeat mechanics in skills. + +The new design makes that split explicit and adds Pi and OMP without changing the report format. + +## Design decision + +### Ownership + +| Layer | Owns | Does not own | +| --- | --- | --- | +| `behavior-diff` skill | Find the changed instruction file. Draft the decision-moment task. Select the current trial stack and exact model. Explain the result. | Build variants, launch trials, parse traces, grade runs, render HTML. | +| `behavior-diff-live` skill | Prepare the same task for both variants. Select the host-specific dispatch method. Explain the weaker live evidence. | Create a second report format or infer captured actions. | +| `behavior-diff.sh` | Build variants, launch trials, grade complete versus blocked runs, call the extractor, render the report. | Know Pi or OMP event fields. | +| `run-trial.sh` | Run one host CLI and normalize its output into canonical `trace.jsonl`. | Grade behavior or explain the result. | +| `decisions.py` | Extract the decision chain from canonical trial evidence. | Read raw Claude, Codex, Pi, or OMP trace formats. | +| `render.py` | Render canonical evidence. | Branch on the trial stack. | + +### Data flow + +```text +SKILL.md + makes judgment calls + | + v +behavior-diff.sh + builds before/after copies + launches equal trials + | + v +run-trial.sh + claude | codex | pi | omp + converts raw events + | + v +trace.jsonl + canonical tool calls + final answer + | + +-------------------+ + | | + v v +decisions.py render.py + decision chain one HTML report +``` + +### Stack adapter boundary + +Keep the current switch in `run-trial.sh`: + +```text +claude -> native stream-json is already canonical +codex -> codex-raw.jsonl -> Codex normalizer -> canonical trace.jsonl +omp -> omp-raw.jsonl ┐ + ├-> shared private normalizer -> canonical trace.jsonl +pi -> pi-raw.jsonl ┘ +``` + +Pi and OMP need separate command profiles. Their flags and built-in tool names differ. Their official JSON event fields match, so one private normalizer avoids duplicate `jq` logic. Separate tests pin both input contracts. + +Do not create one file per stack. Four small command branches and one private normalizer are easier to read than a public adapter framework. + +Do not add a public external-adapter API. No external caller defines that contract. + +## Public command contract + +### Claude + +```bash +behavior-diff.sh \ + --agent claude \ + --model sonnet \ + --file \ + --task +``` + +### Codex + +```bash +behavior-diff.sh \ + --agent codex \ + --model gpt-5.6-terra \ + --file \ + --task +``` + +### OMP + +```bash +behavior-diff.sh \ + --agent omp \ + --model \ + --file \ + --task +``` + +### Pi + +```bash +behavior-diff.sh \ + --agent pi \ + --model \ + --file \ + --task +``` + +Pi and OMP require `--model`. Neither stack has a portable default provider and model. A silent fallback can test a different agent from the one the user means to measure. + +Claude and Codex keep their current defaults. + +## OMP trial command + +`run-trial.sh` runs OMP in one disposable variant copy: + +```bash +printf '%s\n' "$task" | omp -p \ + --mode json \ + --no-session \ + --no-title \ + --cwd "$dir" \ + --model "$model" \ + --tools read,bash,grep,glob \ + --approval-mode yolo +``` + +Each flag has one reason: + +| Flag | Reason | +| --- | --- | +| `-p` | Run once and exit. | +| `--mode json` | Emit machine-readable events. | +| `--no-session` | Do not write a reusable OMP session. | +| `--no-title` | Avoid the extra title model call. | +| `--cwd` | Start inside the variant copy. | +| `--model` | Test the exact selected OMP model. | +| `--tools` | Exclude edit, write, browser, subagent, and network-specific tools. | +| `--approval-mode yolo` | Avoid an approval prompt in a headless run. | + +The task goes through stdin. This avoids treating a task that starts with `-` as a CLI flag. + +## Pi trial command + +`run-trial.sh` runs upstream Pi in one disposable variant copy: + +```bash +printf '%s\n' "$task" | pi -p \ + --mode json \ + --no-session \ + --model "$model" \ + --tools read,bash,grep,find,ls +``` + +Pi does not accept OMP's `--no-title` or `--approval-mode` flags. Pi also uses `find` and `ls` where the OMP profile uses `glob`. + +The runner sets `PI_SKIP_VERSION_CHECK=1` and `PI_TELEMETRY=0` for each Pi trial. These settings stop unrelated Pi update checks and install telemetry. They do not disable the selected model provider. + +### Pi and OMP event contract + +Both CLIs emit one JSON object per line. The shared normalizer uses these stable events: + +```json +{"type":"tool_execution_start","toolCallId":"call-1","toolName":"bash","args":{"command":"python3 -m pytest"}} +{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"The check passes."}]}} +``` + +OMP sources of record: + +- `packages/coding-agent/src/modes/print-mode.ts` writes every printable event as one JSON line. +- `packages/agent/src/types.ts` defines `tool_execution_start` and `message_end`. +- The reviewed OMP revision was `984a4f2dc9e50f6645b8fe04a91570876f8d3c83`. + +Pi sources of record: + +- `packages/coding-agent/docs/json.md` documents the JSON-lines protocol. +- `packages/coding-agent/src/modes/json-event.ts` keeps final `message_end` values and emits compact stream updates. +- The reviewed Pi revision was `e266507b606b9552fa277252644054afd4384b11`. + +Links: + +- +- +- +- + +### Canonical trace contract + +All stacks keep the current format: + +```json +{"type":"assistant","message":{"content":[{"type":"tool_use","name":"bash","input":{"command":"python3 -m pytest"}}]}} +{"type":"result","result":"The check passes."} +``` + +Pi and OMP normalization rules: + +1. Convert each `tool_execution_start` event into one `tool_use` line. +2. Copy `toolName` to `name`. +3. Copy `args` to `input`. +4. When `args.path` exists, also write it as `input.file_path`. The current renderer reads `file_path` for non-command actions. +5. Select assistant `message_end` events. +6. Join their text blocks. +7. Use the last assistant message as the final result. +8. Write an empty result when no final assistant text exists. The existing grader marks that trial `BLOCKED`. + +Raw output stays in `pi-raw.jsonl` or `omp-raw.jsonl`. This gives each stack a local debugging path without changing the report contract. + +## Decision extractor contract + +`decisions.py` gets separate Pi and OMP command profiles. + +OMP: + +```bash +printf '%s\n' "$prompt" | omp -p \ + --no-tools \ + --no-session \ + --no-title \ + --model "$model" +``` + +Pi: + +```bash +printf '%s\n' "$prompt" | pi -p \ + --no-tools \ + --no-session \ + --model "$model" +``` + +Rules: + +- A Pi trial run uses Pi for decision extraction unless the caller passes `--extract-agent`. +- An OMP trial run uses OMP for decision extraction unless the caller passes `--extract-agent`. +- Each extractor uses the same model as its trials unless the caller passes `--extract-model`. +- Pi and OMP extraction need an explicit model. +- Claude and Codex extractor order and defaults stay unchanged. +- `--emit-prompt` and `--ingest` stay stack-neutral. + +This keeps Pi-only and OMP-only machines usable. It also avoids asking another stack to interpret the result by default. + +## Live skill host contract + +The live skill keeps one task and one report format. Only dispatch differs. + +| Host | Dispatch | Delivery | +| --- | --- | --- | +| Claude Code | Launch two subagents in one parallel dispatch. | Each trial calls `SendMessage` to the main agent. | +| OMP | Launch one `task` batch with two task items. | Results return to the parent automatically. Do not ask for `SendMessage`. | +| Pi | Pi has no built-in subagent dispatch. Use the headless skill with `--agent pi`. | The headless runner collects Pi JSON events. | +| Codex without dispatch | Run two fresh contexts in sequence, or use the headless skill. | Collect each final answer directly. | + +Every live path keeps these rules: + +- The two prompts differ only by the variant directory. +- Each trial reads the instruction files inside its own variant. +- Each trial reports `ANSWER` and numbered `ACTIONS`. +- Each trial stays inside its variant and does not use networked or destructive commands. +- The summary says live subagents do not auto-load the variant instruction file. +- The summary says one trial per side is one sample. + +## Install and support wording + +Behavior Diff remains an installable plugin for Claude Code and Codex. + +Pi and OMP are trial stacks, not plugin hosts in this change. + +The README and plugin descriptions must make this difference clear: + +| Surface | Claude Code | Codex | Pi | OMP | +| --- | --- | --- | --- | --- | +| Marketplace plugin install | Yes | Yes | No in DRC-4282 | No in DRC-4282 | +| Headless trial stack | Yes | Yes | Yes, with explicit `--model` | Yes, with explicit `--model` | +| Live trial dispatch | Parallel subagents | Fresh sequential contexts or headless | No built-in dispatch. Use headless. | One parallel `task` batch | + +Do not add a Pi or OMP manifest, marketplace entry, install command, or edit hook. + +## Error contract + +### Supported stacks + +The accepted values are: + +```text +claude +codex +pi +omp +``` + +### Unknown stack + +```text +behavior-diff: unsupported trial stack "" +Supported stacks: claude, codex, pi, omp. +A new stack needs: +1. a fresh headless CLI run. +2. machine-readable tool and final-answer events. +3. a converter to canonical trace.jsonl. +``` + +### Missing Pi or OMP model + +```text +behavior-diff: --agent pi requires --model with the exact Pi model ID +behavior-diff: --agent omp requires --model with the exact OMP model ID +``` + +### Missing Pi or OMP binary + +Keep exit code `3` and name the command: + +```text +behavior-diff: pi CLI required (--agent pi) +behavior-diff: omp CLI required (--agent omp) +``` + +### Incomplete Pi or OMP output + +Do not invent a final answer. Keep the raw stream, write an empty canonical result, and let the current grader mark the trial `BLOCKED`. + +## Safety and privacy + +Pi and OMP trials run inside the same disposable project copies used by the other stacks. + +Both tool lists exclude edit and write. OMP also excludes browser and subagent tools. Bash still has broad power. OMP needs `--approval-mode yolo` because a headless run cannot answer prompts. Pi has no built-in approval prompt. + +The disposable copy is not an OS sandbox. A Pi or OMP bash command can still reach outside it or use the network. DRC-4282 records this limit. It does not add a new sandbox or change the trial task. + +Do not write trial data to the repository. Raw traces, normalized traces, prompts, reports, and stderr stay under `${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}/runs/`. + +CI uses only stub CLIs and synthetic JSON. CI never calls a model or a live agent CLI. + +## Scope + +### In scope + +- Accept `--agent pi` and `--agent omp` in both command entry points. +- Require an exact model for Pi and OMP. +- Run Pi and OMP headlessly. +- Normalize their matching JSON event contracts through one private helper. +- Keep separate raw trace files for Pi and OMP. +- Extract decisions with the same stack and model used for each trial. +- Add OMP live dispatch instructions. +- Direct Pi users to the headless path because Pi has no built-in subagents. +- State the skill and script ownership split. +- Add useful errors for unknown stacks. +- Add deterministic Pi and OMP contract tests. +- Update public support wording. + +### Out of scope + +- Pi or OMP plugin packaging. +- Pi or OMP plugin installation instructions. +- Pi or OMP edit hooks. +- A generic external adapter API. +- Gemini or a fifth trial stack. +- A new canonical trace format. +- A new report format. +- A change to automatic grading. +- A plugin version bump. Release work owns the version. + +## Files + +### Modify + +- `plugin/skills/behavior-diff/scripts/behavior-diff.sh` +- `plugin/skills/behavior-diff/scripts/run-trial.sh` +- `plugin/skills/behavior-diff/scripts/decisions.py` +- `bin/behavior-diff` +- `plugin/skills/behavior-diff/SKILL.md` +- `plugin/skills/behavior-diff-live/SKILL.md` +- `README.md` +- `plugin/.claude-plugin/plugin.json` +- `plugin/.codex-plugin/plugin.json` +- `tests/hooks-test.sh` +- `tests/live-report-contract.sh` + +### Do not modify + +- `plugin/skills/behavior-diff/scripts/render.py` +- `plugin/.claude-plugin/plugin.json` version field +- `plugin/.codex-plugin/plugin.json` version field +- Marketplace files outside this repository +- Files under `${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}/runs/` + +--- + +## Task 1: Lock the CLI and skill contracts with failing tests + +**Files:** + +- Modify: `tests/hooks-test.sh` +- Modify: `tests/live-report-contract.sh` + +- [ ] Add `run-trial.sh` to the shell test setup: + +```bash +trial_runner=$here/../plugin/skills/behavior-diff/scripts/run-trial.sh +``` + +- [ ] Add fake `pi` and `omp` commands under the existing test `PATH`. Each command records its arguments, consumes stdin, and emits synthetic events. + +OMP stub: + +```bash +cat >"$stub/omp" <<'SH' +#!/bin/sh +printf '%s\n' "$@" >"$OMP_ARGS_FILE" +cat >/dev/null +cat <<'JSON' +{"type":"tool_execution_start","toolCallId":"call-1","toolName":"bash","args":{"command":"printf ok"}} +{"type":"tool_execution_start","toolCallId":"call-2","toolName":"read","args":{"path":"AGENTS.md"}} +{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"OMP done"}]}} +JSON +SH +chmod +x "$stub/omp" +``` + +Pi stub: + +```bash +cat >"$stub/pi" <<'SH' +#!/bin/sh +printf '%s\n' "$@" >"$PI_ARGS_FILE" +cat >/dev/null +cat <<'JSON' +{"type":"tool_execution_start","toolCallId":"call-1","toolName":"bash","args":{"command":"printf ok"}} +{"type":"tool_execution_start","toolCallId":"call-2","toolName":"read","args":{"path":"AGENTS.md"}} +{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"Pi done"}]}} +JSON +SH +chmod +x "$stub/pi" +``` + +- [ ] Add runner tests for these contracts: + +```text +--agent unknown exits 2 and lists claude, codex, pi, omp +--agent pi without --model exits 2 +--agent omp without --model exits 2 +--agent pi with a missing binary exits 3 +--agent omp with a missing binary exits 3 +``` + +- [ ] Run the tests and confirm they fail on the current two-stack validation: + +```bash +bash tests/hooks-test.sh +``` + +Expected result: non-zero exit from the first new Pi or OMP assertion. + +- [ ] Add direct `run-trial.sh` tests for both fake commands. + +Check the OMP trial: + +```text +omp-raw.jsonl exists +trace.jsonl contains a bash command tool_use +trace.jsonl maps AGENTS.md to input.file_path +trace.jsonl ends with result "OMP done" +OMP received -p, --mode json, --no-session, --no-title +OMP received --tools read,bash,grep,glob +OMP received --approval-mode yolo +OMP received the exact test model +``` + +Check the Pi trial: + +```text +pi-raw.jsonl exists +trace.jsonl contains the same canonical tool_use shape +trace.jsonl maps AGENTS.md to input.file_path +trace.jsonl ends with result "Pi done" +Pi received -p, --mode json, --no-session +Pi received --tools read,bash,grep,find,ls +Pi received the exact test model +Pi did not receive --no-title or --approval-mode +``` + +- [ ] Add fake Pi and OMP outputs with no final assistant `message_end`. Check that neither trace has a non-empty canonical result. This is the grader's `BLOCKED` condition. + +- [ ] Add live-report contract checks for exact ownership and host rules: + +```text +The skill owns judgment. +The scripts own repeatable mechanics. +--agent pi +--model +--agent omp +--model +Pi has no built-in subagent dispatch. +one `task` batch +Results return to the parent automatically. +Pi and OMP are trial stacks, not plugin hosts +``` + +- [ ] Run both test files and confirm they fail for the missing implementation and prose: + +```bash +bash tests/hooks-test.sh +bash tests/live-report-contract.sh +``` + +Expected result: both commands fail on the new assertions. + +- [ ] Commit the failing contracts: + +```bash +git add tests/hooks-test.sh tests/live-report-contract.sh +git commit --signoff -m "test: define Pi and OMP trial stack contracts" +``` + +## Task 2: Extend the runner command contract + +**Files:** + +- Modify: `plugin/skills/behavior-diff/scripts/behavior-diff.sh` +- Modify: `bin/behavior-diff` + +- [ ] Change trial-stack validation in `behavior-diff.sh` to accept `pi` and `omp`. Use the error messages in this plan. + +Use this model selection rule: + +```bash +if [ -z "$model" ]; then + case "$agent" in + claude) model=sonnet ;; + codex) model=gpt-5.6-terra ;; + pi) + echo "behavior-diff: --agent pi requires --model with the exact Pi model ID" >&2 + exit 2 + ;; + omp) + echo "behavior-diff: --agent omp requires --model with the exact OMP model ID" >&2 + exit 2 + ;; + esac +fi +``` + +- [ ] Keep `command -v "$agent"` after model and required-argument validation. This produces the existing exit code `3` for a missing CLI. + +- [ ] Pin the default extractor for Pi and OMP runs before calling `decisions.py`: + +```bash +case "$agent" in + pi | omp) + if [ -z "$extract_agent" ]; then + extract_agent=$agent + [ -n "$extract_model" ] || extract_model=$model + fi + ;; +esac +``` + +An explicit `--extract-agent` still wins. + +- [ ] Update `bin/behavior-diff` usage only. Its existing pass-through array already forwards `--agent`, `--model`, `--extract-agent`, and `--extract-model`. + +```text +[--agent claude|codex|pi|omp] [--model NAME] +``` + +- [ ] Run the focused runner tests: + +```bash +bash tests/hooks-test.sh +``` + +Expected result: the CLI validation assertions pass. The direct Pi and OMP adapter assertions still fail because `run-trial.sh` does not accept them yet. + +- [ ] Commit: + +```bash +git add plugin/skills/behavior-diff/scripts/behavior-diff.sh bin/behavior-diff +git commit --signoff -m "feat: accept Pi and OMP trial stacks" +``` + +## Task 3: Add Pi and OMP trace normalization + +**Files:** + +- Modify: `plugin/skills/behavior-diff/scripts/run-trial.sh` + +- [ ] Update the header and usage to name all four stacks. + +- [ ] Change validation to accept `claude`, `codex`, `pi`, or `omp`. + +- [ ] Keep the Claude branch unchanged. + +- [ ] Change the current `else` branch into `elif [ "$agent" = codex ]`. Keep its command and `jq` filters unchanged. + +- [ ] Add one private `normalize_pi_omp_json` function. It accepts a raw Pi or OMP JSONL path and writes the existing canonical trace: + +```bash +normalize_pi_omp_json() { # $1 = raw Pi or OMP JSONL + local raw=$1 + jq -c ' + select(.type == "tool_execution_start") + | {type: "assistant", message: {content: [{ + type: "tool_use", + name: .toolName, + input: ((.args // {}) + | if (has("file_path") or has("command")) then . + elif has("path") then . + {file_path: .path} + else . + end) + }]}} + ' "$raw" >"$trace_dir/trace.jsonl" + + jq -s -c ' + [.[] + | select(.type == "message_end" and .message.role == "assistant") + | [.message.content[]? | select(.type == "text") | .text] + | join("")] + | {type: "result", result: (last // "")} + ' "$raw" >>"$trace_dir/trace.jsonl" +} +``` + +- [ ] Add the OMP command branch. Save stdout to `omp-raw.jsonl`, then call the shared normalizer: + +```bash +printf '%s\n' "$task" | omp -p \ + --mode json --no-session --no-title --cwd "$dir" \ + --model "$model" --tools read,bash,grep,glob \ + --approval-mode yolo \ + >"$trace_dir/omp-raw.jsonl" 2>"$trace_dir/stderr.log" || true +normalize_pi_omp_json "$trace_dir/omp-raw.jsonl" +``` + +- [ ] Add the Pi command branch. Save stdout to `pi-raw.jsonl`, then call the same normalizer: + +```bash +printf '%s\n' "$task" | \ + PI_SKIP_VERSION_CHECK=1 PI_TELEMETRY=0 \ + pi -p --mode json --no-session --model "$model" \ + --tools read,bash,grep,find,ls \ + >"$trace_dir/pi-raw.jsonl" 2>"$trace_dir/stderr.log" || true +normalize_pi_omp_json "$trace_dir/pi-raw.jsonl" +``` + +- [ ] Run the focused tests: + +```bash +bash tests/hooks-test.sh +``` + +Expected result: the fake Pi and OMP trials pass. Both missing-final cases have no non-empty result. + +- [ ] Run Bash format checking for the changed scripts: + +```bash +docker run --rm -v "$PWD:/mnt" -w /mnt \ + mvdan/shfmt:v3.14.0 -d -i 2 -ci \ + plugin/skills/behavior-diff/scripts/behavior-diff.sh \ + plugin/skills/behavior-diff/scripts/run-trial.sh \ + bin/behavior-diff tests/hooks-test.sh +``` + +Expected result: no diff. + +- [ ] Commit: + +```bash +git add plugin/skills/behavior-diff/scripts/run-trial.sh tests/hooks-test.sh +git commit --signoff -m "feat: normalize Pi and OMP trial traces" +``` + +## Task 4: Add Pi and OMP decision extractors + +**Files:** + +- Modify: `plugin/skills/behavior-diff/scripts/decisions.py` + +- [ ] Extend the module usage and CLI validation from `codex|claude` to `codex|claude|pi|omp`. + +- [ ] Add separate runners because their command flags differ: + +```python +def _pi(prompt, model): + proc = subprocess.run( + [ + "pi", + "-p", + "--no-tools", + "--no-session", + "--model", + model, + ], + input=prompt, + capture_output=True, + text=True, + env={ + **os.environ, + "PI_SKIP_VERSION_CHECK": "1", + "PI_TELEMETRY": "0", + }, + ) + return proc.stdout if proc.returncode == 0 else None + + +def _omp(prompt, model): + proc = subprocess.run( + [ + "omp", + "-p", + "--no-tools", + "--no-session", + "--no-title", + "--model", + model, + ], + input=prompt, + capture_output=True, + text=True, + ) + return proc.stdout if proc.returncode == 0 else None +``` + +- [ ] Import `os` for the Pi subprocess environment. + +- [ ] Add both runners. Keep the automatic fallback order as `codex`, then `claude`. Pi and OMP join the order only when the caller pins one. + +```python +runners = { + "codex": _codex, + "claude": _claude, + "pi": _pi, + "omp": _omp, +} +order = [agent] if agent else ["codex", "claude"] +``` + +- [ ] Require a model for pinned Pi and OMP extractors: + +```python +m = model or DEFAULT_MODEL.get(a) +if not m: + print(f"decision diff: {a} requires --model") + return "none", None +``` + +- [ ] Extend `self_check()` with fake Pi and OMP executables. Each fake command consumes stdin and returns valid extractor JSON. + +Run both CLI cases: + +```text +--agent pi --model test/pi-model +--agent omp --model test/omp-model +``` + +Check that: + +```text +each run writes decisions.json +the Pi extractor label is pi:test/pi-model +the OMP extractor label is omp:test/omp-model +both fake commands receive --no-tools and the exact model +Pi does not receive --no-title +OMP receives --no-title +``` + +- [ ] Run the deterministic self-check: + +```bash +python3 plugin/skills/behavior-diff/scripts/decisions.py --check +``` + +Expected result: + +```text +decisions.py self-check ok +``` + +- [ ] Run Python formatting: + +```bash +uvx ruff@0.16.5 format --check --diff \ + plugin/skills/behavior-diff/scripts/decisions.py +``` + +Expected result: no diff. + +- [ ] Commit: + +```bash +git add plugin/skills/behavior-diff/scripts/decisions.py +git commit --signoff -m "feat: extract Pi and OMP decision diffs" +``` + +## Task 5: Make skill ownership and host behavior explicit + +**Files:** + +- Modify: `plugin/skills/behavior-diff/SKILL.md` +- Modify: `plugin/skills/behavior-diff-live/SKILL.md` + +- [ ] Add a short ownership section to the headless skill: + +```text +The skill owns judgment. It finds the change, drafts the decision-moment task, +selects the current trial stack and model, and explains the evidence. + +The scripts own repeatable mechanics. They build variants, run trials, +normalize traces, grade completeness, extract decisions, and render the report. +``` + +- [ ] Replace the current two-host instruction with four stack rules: + +```text +Claude Code: --agent claude. The model defaults to sonnet. +Codex: --agent codex. The model defaults to gpt-5.6-terra. +Pi: --agent pi --model . Never omit the model. +OMP: --agent omp --model . Never omit the model. +``` + +- [ ] Keep the current rule that normal execution does not ask the user to confirm cost, file, task, or mode. + +- [ ] Split the live launch step by host. + +For Claude Code: + +```text +Launch both subagents in one parallel dispatch. Each report calls SendMessage +to the main agent. +``` + +For OMP: + +```text +Launch one `task` batch with two task items. Results return to the parent +automatically. Do not ask an OMP task agent to call SendMessage. +``` + +For Pi: + +```text +Pi has no built-in subagent dispatch. Do not invent a parallel live path. +Use the headless skill with --agent pi and the exact Pi model. +``` + +For Codex without dispatch: + +```text +Run two fresh contexts in sequence, or use the headless skill. +``` + +- [ ] Keep the common trial prompt and evidence limits outside the host branches. Do not copy the full prompt into each host path. + +- [ ] Run the skill contract test: + +```bash +bash tests/live-report-contract.sh +``` + +Expected result: ownership, Pi and OMP commands, and live host checks pass. + +- [ ] Commit: + +```bash +git add plugin/skills/behavior-diff/SKILL.md \ + plugin/skills/behavior-diff-live/SKILL.md \ + tests/live-report-contract.sh +git commit --signoff -m "docs: define trial stack ownership" +``` + +## Task 6: Update public support wording + +**Files:** + +- Modify: `README.md` +- Modify: `plugin/.claude-plugin/plugin.json` +- Modify: `plugin/.codex-plugin/plugin.json` +- Modify: `tests/live-report-contract.sh` + +- [ ] Keep the existing Claude Code and Codex install commands unchanged. + +- [ ] Add the support table from this design after the install introduction. State that Pi and OMP trial support does not mean plugin installation on either host. + +- [ ] Change both manifest descriptions in the same edit: + +```text +isolated before/after trials (claude, codex, pi, or omp) +``` + +- [ ] Keep both manifest versions at `0.3.2`. The release process owns the version bump. + +- [ ] Add contract checks that both descriptions name all four trial stacks and both versions still match. + +- [ ] Run: + +```bash +bash tests/live-report-contract.sh +git diff --check +``` + +Expected result: both commands exit `0`. + +- [ ] Commit: + +```bash +git add README.md \ + plugin/.claude-plugin/plugin.json \ + plugin/.codex-plugin/plugin.json \ + tests/live-report-contract.sh +git commit --signoff -m "docs: describe Pi and OMP trial support" +``` + +## Task 7: Run full deterministic verification + +- [ ] Run Bash formatting for the repository: + +```bash +docker run --rm -v "$PWD:/mnt" -w /mnt \ + mvdan/shfmt:v3.14.0 -d -i 2 -ci . +``` + +Expected result: no diff. + +- [ ] Run Python formatting for the repository: + +```bash +uvx ruff@0.16.5 format --check --diff . +``` + +Expected result: no diff. + +- [ ] Run the full deterministic suite: + +```bash +bash tests/hooks-test.sh +python3 plugin/skills/behavior-diff/scripts/decisions.py --check +bash tests/live-report-contract.sh +bash tests/release-workflow-test.sh +``` + +Expected result: every command exits `0`. + +- [ ] Run the Markdown whitespace check: + +```bash +git diff --check +``` + +Expected result: no output. + +- [ ] Confirm no test wrote run data into the repository: + +```bash +git status --short +``` + +Expected result: only the planned source and test files are present before commits. After the planned commits, the tree is clean. + +## Task 8: Run the optional paid Pi and OMP smoke gates + +This is manual evidence. It is not a CI step. + +- [ ] Check both local CLIs without a model call: + +```bash +pi --version +omp --version +``` + +Expected result: both commands exit `0`. + +The Pi preflight currently fails on this workstation because Node 20 does not export `node:fs.globSync`. Fixing the user's Pi runtime is outside this repository change. If it still fails, report the Pi smoke gate as blocked before any paid call. + +- [ ] Ask Kent for approval to spend one fast Behavior Diff run on each working stack. + +- [ ] If approved, create one synthetic temporary repository with one instruction-file edit. Do not use private code or a real transcript. + +- [ ] Run one trial per side with Pi: + +```bash +plugin/skills/behavior-diff/scripts/behavior-diff.sh \ + --agent pi \ + --model \ + --file AGENTS.md \ + --task \ + --fast +``` + +- [ ] Run the same task with OMP: + +```bash +plugin/skills/behavior-diff/scripts/behavior-diff.sh \ + --agent omp \ + --model \ + --file AGENTS.md \ + --task \ + --fast +``` + +- [ ] Check the actual output surface for each completed stack: + +```text +both raw traces exist +both canonical traces contain tool calls when the agent used tools +both canonical traces contain a non-empty final result +grades.tsv marks both runs REVIEW +report.html opens and shows commands plus final answers +decisions.json names the same stack and model used for the trials +``` + +- [ ] If approval is not given, report each paid gate as not run. Do not replace it with another model call or claim live proof. + +- [ ] Do not commit the synthetic run or report. + +## Task 9: Independent review before a pull request + +- [ ] Read `REVIEWER_GUIDELINES.md`. + +- [ ] Send the full diff to one independent read-only reviewer. + +- [ ] Require checks for: + +```text +Claude and Codex behavior did not change +Pi and OMP cannot run without an explicit model +Pi and OMP use separate command profiles +Pi and OMP raw events use one private canonical normalizer +missing final output becomes BLOCKED on both stacks +OMP live dispatch does not use Claude SendMessage rules +Pi live guidance does not invent built-in subagents +README does not claim Pi or OMP plugin installation +tests use only synthetic data and stub CLIs +both manifests stay equivalent +``` + +- [ ] Fix every validated blocking finding. + +- [ ] Re-run the affected focused test and the full deterministic suite. + +- [ ] Create the pull request only after all required gates pass, or name the exact missing gate. + +--- + +## Acceptance criteria + +DRC-4282 is complete when all statements below are true: + +- `behavior-diff.sh --agent pi --model ` launches fresh headless Pi trials. +- `behavior-diff.sh --agent omp --model ` launches fresh headless OMP trials. +- Pi and OMP tool calls and final answers appear in canonical `trace.jsonl`. +- Pi and OMP use separate CLI command profiles and one private normalizer. +- Existing grading and rendering work without Pi or OMP branches. +- A missing final answer produces `BLOCKED` on both stacks. +- Each stack uses its trial model for decision extraction by default. +- The headless skill states the skill-versus-script ownership split. +- The live skill gives correct dispatch rules for Claude Code, OMP, Pi, and Codex. +- Pi live guidance uses the headless path because upstream Pi has no built-in subagents. +- Public wording separates plugin hosts from trial stacks. +- Claude and Codex paths keep their current commands and defaults. +- CI tests use only fake CLIs and synthetic events. +- The full deterministic suite passes. +- An independent reviewer gives a GO verdict, or the missing review gate is reported. + +## Rejected alternatives + +### Keep Pi or OMP unsupported + +Rejected because both CLIs expose fresh headless JSON event streams with the evidence Behavior Diff needs. + +### Treat `pi` as an alias for `omp` + +Rejected because upstream Pi and OMP are separate products with different flags, tools, release lines, and runtime behavior. `--agent pi` must launch the real `pi` binary. + +### Put Pi and OMP instructions only in the skill + +Rejected because the skill would repeat variant, trace, and grading mechanics. It would also create a second report path. + +### Use separate Pi and OMP normalizers + +Rejected for now. Their official tool and final-message event fields match. One private helper removes duplicate `jq` while separate command tests detect either upstream contract changing. + +### Add one script per stack + +Rejected for now. Four command branches fit in `run-trial.sh`. Separate adapter files add a registry before another stack exists. + +### Add a generic external adapter API + +Rejected for now. No real external caller defines the right interface yet. `run-trial.sh` remains the internal adapter boundary. + +### Use default Pi or OMP roles + +Rejected because roles and provider mappings are user-specific. Each run must name the exact model under test. + +### Change the canonical trace or renderer + +Rejected because Pi and OMP events map to the current contract. A report change would add migration work without improving DRC-4282. From 46c9171bf7a4ed5c13dbdadd1607bcd84c595d8a Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 3 Sep 2026 01:17:48 +0800 Subject: [PATCH 2/8] test: define Pi and OMP trial stack contracts Signed-off-by: Kent Huang --- tests/hooks-test.sh | 136 ++++++++++++++++++++++++++++++++++ tests/live-report-contract.sh | 21 ++++++ 2 files changed, 157 insertions(+) diff --git a/tests/hooks-test.sh b/tests/hooks-test.sh index 16a13f6..d6dcd35 100755 --- a/tests/hooks-test.sh +++ b/tests/hooks-test.sh @@ -147,6 +147,7 @@ printf '%s' "$out" | jq -e '.systemMessage | test("AGENTS.md")' >/dev/null || # plus the runner's --before-file / baseline-resolve argument paths. backup=$here/../plugin/scripts/rules-edit-backup.sh runner=$here/../plugin/skills/behavior-diff/scripts/behavior-diff.sh +trial_runner=$here/../plugin/skills/behavior-diff/scripts/run-trial.sh baselines=$BEHAVIOR_DIFF_HOME/baselines enc_of() { printf '%s' "$1" | sed 's|%|%25|g; s|/|%2F|g'; } nentries() { find "$1" -mindepth 1 -maxdepth 1 -type f ! -name '.*' | wc -l; } @@ -295,4 +296,139 @@ printf '%s' "$out" | grep -q "matches the before content" || fail "unreadable subdir: expected the plain equal-content stop, got: $out" [ "$code" -eq 2 ] || fail "unreadable subdir: exit $code, want 2" +progress 'Pi and OMP runner contracts' + +cat >"$stub/omp" <<'SH' +#!/bin/sh +printf '%s\n' "$@" >"$OMP_ARGS_FILE" +cat >/dev/null +cat <<'JSON' +{"type":"tool_execution_start","toolCallId":"call-1","toolName":"bash","args":{"command":"printf ok"}} +{"type":"tool_execution_start","toolCallId":"call-2","toolName":"read","args":{"path":"AGENTS.md"}} +JSON +if [ "${NO_FINAL:-}" != 1 ]; then + printf '%s\n' '{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"OMP done"}]}}' +fi +SH +chmod +x "$stub/omp" + +cat >"$stub/pi" <<'SH' +#!/bin/sh +printf '%s\n' "$@" >"$PI_ARGS_FILE" +printf '%s\n%s\n' "${PI_SKIP_VERSION_CHECK:-}" "${PI_TELEMETRY:-}" >"$PI_ENV_FILE" +cat >/dev/null +cat <<'JSON' +{"type":"tool_execution_start","toolCallId":"call-1","toolName":"bash","args":{"command":"printf ok"}} +{"type":"tool_execution_start","toolCallId":"call-2","toolName":"read","args":{"path":"AGENTS.md"}} +JSON +if [ "${NO_FINAL:-}" != 1 ]; then + printf '%s\n' '{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"Pi done"}]}}' +fi +SH +chmod +x "$stub/pi" + +# 28. unknown stacks list every supported binary +set +e +out=$(cd "$plainrun" && PATH="$stub:$PATH" "$runner" --agent unknown \ + --model test/model --file CLAUDE.md --task t 2>&1) +code=$? +set -e +[ "$code" -eq 2 ] || fail "unknown agent: exit $code, want 2" +printf '%s' "$out" | grep -qF 'Supported stacks: claude, codex, pi, omp.' || + fail "unknown agent does not list every supported stack" + +# 29. Pi and OMP require an explicit model +for stack in pi omp; do + set +e + out=$(cd "$plainrun" && PATH="$stub:$PATH" "$runner" --agent "$stack" \ + --file CLAUDE.md --task t 2>&1) + code=$? + set -e + [ "$code" -eq 2 ] || fail "$stack without model: exit $code, want 2" + printf '%s' "$out" | grep -qF "requires --model" || + fail "$stack without model: missing model guidance" +done + +# 30. a missing Pi or OMP binary keeps the runner's dependency exit code +missing_bin=$tmp/missing-bin +mkdir -p "$missing_bin" +ln -s "$(command -v jq)" "$missing_bin/jq" +for stack in pi omp; do + set +e + out=$(cd "$plainrun" && PATH="$missing_bin:/bin:/usr/bin" "$runner" \ + --agent "$stack" --model test/model --file CLAUDE.md --task t 2>&1) + code=$? + set -e + [ "$code" -eq 3 ] || fail "missing $stack binary: exit $code, want 3" + printf '%s' "$out" | grep -qF "$stack CLI required (--agent $stack)" || + fail "missing $stack binary: wrong guidance" +done + +trial_project=$tmp/trial-project +task_file=$tmp/trial-task.md +mkdir -p "$trial_project" +printf '%s\n' 'Inspect this synthetic project.' >"$task_file" + +# 31. OMP events normalize into the shared trace contract +omp_trace=$tmp/omp-trace +mkdir -p "$omp_trace" +OMP_ARGS_FILE=$tmp/omp-args PATH="$stub:$PATH" \ + "$trial_runner" --agent omp --model test/omp-model \ + --dir "$trial_project" --task-file "$task_file" --trace-dir "$omp_trace" +[ -f "$omp_trace/omp-raw.jsonl" ] || fail "OMP raw trace missing" +grep -qF '"command":"printf ok"' "$omp_trace/trace.jsonl" || + fail "OMP command not normalized" +grep -qF '"file_path":"AGENTS.md"' "$omp_trace/trace.jsonl" || + fail "OMP path not normalized" +grep -qF '"result":"OMP done"' "$omp_trace/trace.jsonl" || + fail "OMP final answer not normalized" +for arg in -p --mode json --no-session --no-title --approval-mode yolo \ + test/omp-model read,bash,grep,glob; do + grep -qxF -- "$arg" "$tmp/omp-args" || fail "OMP argument missing: $arg" +done + +# 32. Pi uses its own flags and the same canonical trace contract +pi_trace=$tmp/pi-trace +mkdir -p "$pi_trace" +PI_ARGS_FILE=$tmp/pi-args PI_ENV_FILE=$tmp/pi-env PATH="$stub:$PATH" \ + "$trial_runner" --agent pi --model test/pi-model \ + --dir "$trial_project" --task-file "$task_file" --trace-dir "$pi_trace" +[ -f "$pi_trace/pi-raw.jsonl" ] || fail "Pi raw trace missing" +grep -qF '"command":"printf ok"' "$pi_trace/trace.jsonl" || + fail "Pi command not normalized" +grep -qF '"file_path":"AGENTS.md"' "$pi_trace/trace.jsonl" || + fail "Pi path not normalized" +grep -qF '"result":"Pi done"' "$pi_trace/trace.jsonl" || + fail "Pi final answer not normalized" +for arg in -p --mode json --no-session test/pi-model read,bash,grep,find,ls; do + grep -qxF -- "$arg" "$tmp/pi-args" || fail "Pi argument missing: $arg" +done +if grep -qxF -- '--no-title' "$tmp/pi-args" || + grep -qxF -- '--approval-mode' "$tmp/pi-args"; then + fail "Pi received OMP-only flags" +fi +grep -qxF '1' "$tmp/pi-env" || fail "Pi version check was not disabled" +grep -qxF '0' "$tmp/pi-env" || fail "Pi telemetry was not disabled" + +# 33. missing final text remains visible to the grader as BLOCKED +for stack in pi omp; do + no_final=$tmp/$stack-no-final + mkdir -p "$no_final" + if [ "$stack" = pi ]; then + PI_ARGS_FILE=$tmp/pi-no-final-args PI_ENV_FILE=$tmp/pi-no-final-env \ + NO_FINAL=1 PATH="$stub:$PATH" \ + "$trial_runner" --agent pi --model test/pi-model \ + --dir "$trial_project" --task-file "$task_file" --trace-dir "$no_final" + else + OMP_ARGS_FILE=$tmp/omp-no-final-args NO_FINAL=1 PATH="$stub:$PATH" \ + "$trial_runner" --agent omp --model test/omp-model \ + --dir "$trial_project" --task-file "$task_file" --trace-dir "$no_final" + fi + if jq -e -s '[.[] | select(.type == "result" + and ((.result // "") | length > 0))] + | length > 0' "$no_final/trace.jsonl" >/dev/null; then + fail "$stack missing final answer did not become BLOCKED" + fi +done + echo "ok — all hook self-checks passed" diff --git a/tests/live-report-contract.sh b/tests/live-report-contract.sh index 2034349..4571284 100755 --- a/tests/live-report-contract.sh +++ b/tests/live-report-contract.sh @@ -14,6 +14,7 @@ renderer=$here/../plugin/skills/behavior-diff/scripts/render.py decisions=$here/../plugin/skills/behavior-diff/scripts/decisions.py claude_manifest=$here/../plugin/.claude-plugin/plugin.json codex_manifest=$here/../plugin/.codex-plugin/plugin.json +readme=$here/../README.md require_output() { grep -qF -- "$1" "$2" || fail "$3" @@ -139,6 +140,26 @@ reject_output 'capsule' "$skill" \ reject_output 'capsule' "$spacedock_reference" \ 'Spacedock reference still uses the unexplained capsule term' require_fixed() { grep -qF -- "$1" "$skill" || fail "$2"; } +require_output 'The skill owns judgment.' "$headless_skill" \ + 'headless skill does not state its judgment ownership' +require_output 'The scripts own repeatable mechanics.' "$headless_skill" \ + 'headless skill does not state script ownership' +require_output '--agent pi' "$headless_skill" \ + 'headless skill does not name the Pi trial stack' +require_output '' "$headless_skill" \ + 'headless skill does not require the exact Pi model' +require_output '--agent omp' "$headless_skill" \ + 'headless skill does not name the OMP trial stack' +require_output '' "$headless_skill" \ + 'headless skill does not require the exact OMP model' +require_output 'Pi has no built-in subagent dispatch.' "$skill" \ + 'live skill invents a built-in Pi dispatch path' +require_output 'one `task` batch' "$skill" \ + 'live skill does not use one OMP task batch' +require_output 'Results return to the parent automatically.' "$skill" \ + 'live skill does not explain OMP result delivery' +require_output 'Pi and OMP are trial stacks, not plugin hosts' "$readme" \ + 'README does not separate trial stacks from plugin hosts' require_output 'Run it as soon as the task is known.' "$headless_skill" \ 'headless skill does not start the default run immediately' require_output 'Only add `--fast` when the user explicitly requested it' \ From b91f4537756662301e4987d9b054de8964b59644 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 3 Sep 2026 01:18:34 +0800 Subject: [PATCH 3/8] feat: accept Pi and OMP trial stacks Signed-off-by: Kent Huang --- bin/behavior-diff | 2 +- .../behavior-diff/scripts/behavior-diff.sh | 32 +++++++++++++++++-- 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/bin/behavior-diff b/bin/behavior-diff index 641604a..66f4f56 100755 --- a/bin/behavior-diff +++ b/bin/behavior-diff @@ -2,7 +2,7 @@ # Diff agent behavior for a rule that lives in a markdown file. # # behavior-diff RULE.md --task "..." [--into FILE] [--fast | --trials N] -# [--agent claude|codex] [--model NAME] +# [--agent claude|codex|pi|omp] [--model NAME] # [--dry-run] # # The underlying runner compares the repo at HEAD against the repo plus one diff --git a/plugin/skills/behavior-diff/scripts/behavior-diff.sh b/plugin/skills/behavior-diff/scripts/behavior-diff.sh index 6c75c0b..585cff7 100755 --- a/plugin/skills/behavior-diff/scripts/behavior-diff.sh +++ b/plugin/skills/behavior-diff/scripts/behavior-diff.sh @@ -79,8 +79,13 @@ case "$vocab" in generic | spacedock) ;; *) exit 2 ;; esac -case "$agent" in claude | codex) ;; *) - echo "behavior-diff: --agent must be claude or codex" >&2 +case "$agent" in claude | codex | pi | omp) ;; *) + echo "behavior-diff: unsupported trial stack \"$agent\"" >&2 + echo "Supported stacks: claude, codex, pi, omp." >&2 + echo "A new stack needs:" >&2 + echo "1. a fresh headless CLI run." >&2 + echo "2. machine-readable tool and final-answer events." >&2 + echo "3. a converter to canonical trace.jsonl." >&2 exit 2 ;; esac @@ -89,7 +94,20 @@ case "$trials" in '' | *[!0-9]* | 0) exit 2 ;; esac -[ -n "$model" ] || model=$([ "$agent" = codex ] && echo gpt-5.6-terra || echo sonnet) +if [ -z "$model" ]; then + case "$agent" in + claude) model=sonnet ;; + codex) model=gpt-5.6-terra ;; + pi) + echo "behavior-diff: --agent pi requires --model with the exact Pi model ID" >&2 + exit 2 + ;; + omp) + echo "behavior-diff: --agent omp requires --model with the exact OMP model ID" >&2 + exit 2 + ;; + esac +fi [ -n "$file" ] && [ -n "$task" ] || { echo "behavior-diff: --file and --task are required (see --help)" >&2 exit 2 @@ -246,6 +264,14 @@ done >"$run/grades.tsv" echo # Decision diff: one model pass over the final answers. Best-effort — if it # fails, render.py falls back to the command-derived flow diff alone. +case "$agent" in + pi | omp) + if [ -z "$extract_agent" ]; then + extract_agent=$agent + [ -n "$extract_model" ] || extract_model=$model + fi + ;; +esac python3 "$scripts/decisions.py" "$run" ${extract_agent:+--agent "$extract_agent"} ${extract_model:+--model "$extract_model"} || true python3 "$scripts/render.py" "$run" "$run" "$model" "$run/config.json" open "$run/report.html" 2>/dev/null || true From 99a171cfc3aae7f95bf4730edc949a6519ab7fad Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 3 Sep 2026 01:23:15 +0800 Subject: [PATCH 4/8] feat: normalize Pi and OMP trial traces Signed-off-by: Kent Huang --- .../skills/behavior-diff/scripts/run-trial.sh | 51 ++++++++++++++++--- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/plugin/skills/behavior-diff/scripts/run-trial.sh b/plugin/skills/behavior-diff/scripts/run-trial.sh index 303e712..beda006 100755 --- a/plugin/skills/behavior-diff/scripts/run-trial.sh +++ b/plugin/skills/behavior-diff/scripts/run-trial.sh @@ -1,11 +1,10 @@ #!/usr/bin/env bash # One behavior-diff trial: launch the chosen agent inside a variant copy and # write DIR/trace.jsonl in the claude stream-json shape every reader -# (grader, render.py, decisions.py) consumes. Codex events are normalized: -# command_execution items become assistant tool_use lines, the last -# agent_message becomes the result line. +# (grader, render.py, decisions.py) consumes. Codex, Pi, and OMP events are +# normalized into assistant tool_use lines plus one final result line. # -# Usage: run-trial.sh --agent claude|codex --model M --dir DIR \ +# Usage: run-trial.sh --agent claude|codex|pi|omp --model M --dir DIR \ # --task-file FILE [--allowed CLAUDE_TOOL_LIST] [--trace-dir DIR] # The agent runs with cwd DIR; trace.jsonl/stderr.log land in --trace-dir # (default: DIR). @@ -43,8 +42,8 @@ while [ $# -gt 0 ]; do ;; esac done -case "$agent" in claude | codex) ;; *) - echo "run-trial: --agent must be claude or codex" >&2 +case "$agent" in claude | codex | pi | omp) ;; *) + echo "run-trial: --agent must be claude, codex, pi, or omp" >&2 exit 2 ;; esac @@ -60,11 +59,35 @@ cd "$dir" # trial — otherwise a user-scope install would nudge itself recursively. export BEHAVIOR_DIFF_TRIAL=1 +normalize_pi_omp_json() { # $1 = raw Pi or OMP JSONL + local raw=$1 + jq -c ' + select(.type == "tool_execution_start") + | {type: "assistant", message: {content: [{ + type: "tool_use", + name: .toolName, + input: ((.args // {}) + | if (has("file_path") or has("command")) then . + elif has("path") then . + {file_path: .path} + else . + end) + }]}} + ' "$raw" >"$trace_dir/trace.jsonl" + + jq -s -c ' + [.[] + | select(.type == "message_end" and .message.role == "assistant") + | [.message.content[]? | select(.type == "text") | .text] + | join("")] + | {type: "result", result: (last // "")} + ' "$raw" >>"$trace_dir/trace.jsonl" +} + if [ "$agent" = claude ]; then claude -p "$task" --model "$model" \ ${allowed:+--allowedTools "$allowed"} \ --output-format stream-json --verbose >"$trace_dir/trace.jsonl" 2>"$trace_dir/stderr.log" || true -else +elif [ "$agent" = codex ]; then # Codex has no per-tool allowlist; the workspace-write sandbox scoped to # this variant copy is the equivalent containment. codex exec --ephemeral --skip-git-repo-check -s workspace-write \ @@ -80,4 +103,18 @@ else | .item.text] | {type: "result", result: (last // "")}' \ "$trace_dir/codex-raw.jsonl" >>"$trace_dir/trace.jsonl" +elif [ "$agent" = omp ]; then + printf '%s\n' "$task" | omp -p \ + --mode json --no-session --no-title --cwd "$dir" \ + --model "$model" --tools read,bash,grep,glob \ + --approval-mode yolo \ + >"$trace_dir/omp-raw.jsonl" 2>"$trace_dir/stderr.log" || true + normalize_pi_omp_json "$trace_dir/omp-raw.jsonl" +else + printf '%s\n' "$task" | + PI_SKIP_VERSION_CHECK=1 PI_TELEMETRY=0 \ + pi -p --mode json --no-session --model "$model" \ + --tools read,bash,grep,find,ls \ + >"$trace_dir/pi-raw.jsonl" 2>"$trace_dir/stderr.log" || true + normalize_pi_omp_json "$trace_dir/pi-raw.jsonl" fi From d04f14c970c533ac0c1063e4d56e8e83fc8adbd4 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 3 Sep 2026 01:26:41 +0800 Subject: [PATCH 5/8] feat: extract Pi and OMP decision diffs Signed-off-by: Kent Huang --- .../skills/behavior-diff/scripts/decisions.py | 142 ++++++++++++++++-- 1 file changed, 130 insertions(+), 12 deletions(-) diff --git a/plugin/skills/behavior-diff/scripts/decisions.py b/plugin/skills/behavior-diff/scripts/decisions.py index 795d5c2..34d150f 100755 --- a/plugin/skills/behavior-diff/scripts/decisions.py +++ b/plugin/skills/behavior-diff/scripts/decisions.py @@ -2,13 +2,13 @@ """Decision diff for a Behavior Diff run. It shows what agents CHOSE, not what they typed. -Usage: decisions.py RUN_DIR [--agent codex|claude] [--model NAME] +Usage: decisions.py RUN_DIR [--agent codex|claude|pi|omp] [--model NAME] decisions.py RUN_DIR --emit-prompt decisions.py RUN_DIR --ingest FILE [--extractor-label LABEL] Defaults: codex with gpt-5.6-terra when the codex CLI is present, else -claude -p with sonnet. --agent pins one extractor (no cross-fallback); ---model overrides that agent's default model. +claude -p with sonnet. --agent pins one extractor (no cross-fallback). +Pi and OMP require --model; --model overrides the Claude or Codex default. --emit-prompt prints the extraction prompt so a caller can run the model call itself (the live skill hands it to an in-session subagent); @@ -36,6 +36,7 @@ """ import json +import os import re import shutil import subprocess @@ -330,18 +331,66 @@ def _claude(prompt, model): return proc.stdout if proc.returncode == 0 else None +def _pi(prompt, model): + proc = subprocess.run( + [ + "pi", + "-p", + "--no-tools", + "--no-session", + "--model", + model, + ], + input=prompt, + capture_output=True, + text=True, + env={ + **os.environ, + "PI_SKIP_VERSION_CHECK": "1", + "PI_TELEMETRY": "0", + }, + ) + return proc.stdout if proc.returncode == 0 else None + + +def _omp(prompt, model): + proc = subprocess.run( + [ + "omp", + "-p", + "--no-tools", + "--no-session", + "--no-title", + "--model", + model, + ], + input=prompt, + capture_output=True, + text=True, + ) + return proc.stdout if proc.returncode == 0 else None + + def run_extractor(prompt, agent=None, model=None): - """Run the decision extractor. agent=None tries codex first (default - gpt-5.6-terra), then claude (default sonnet); an explicit agent pins - that extractor with no cross-fallback. Returns (label, text|None).""" - runners = {"codex": _codex, "claude": _claude} + """Run the decision extractor. With no agent, try Codex and then Claude. + An explicit agent pins one extractor with no cross-fallback. Pi and OMP + require an explicit model. Returns (label, text|None).""" + runners = { + "codex": _codex, + "claude": _claude, + "pi": _pi, + "omp": _omp, + } order = [agent] if agent else ["codex", "claude"] for a in order: if not shutil.which(a): if agent: print(f"decision diff: {a} CLI not found") continue - m = model or DEFAULT_MODEL[a] + m = model or DEFAULT_MODEL.get(a) + if not m: + print(f"decision diff: {a} requires --model") + return "none", None answer = runners[a](prompt, m) if answer is not None: return f"{a}:{m}", answer @@ -500,9 +549,12 @@ def progress(message): # ---- emit/ingest modes over a synthetic run dir ---- me = Path(__file__).resolve() - def cli(*argv): + def cli(*argv, env=None): return subprocess.run( - [sys.executable, str(me), *argv], capture_output=True, text=True + [sys.executable, str(me), *argv], + capture_output=True, + text=True, + env=env, ) with tempfile.TemporaryDirectory() as td: @@ -695,6 +747,72 @@ def cli(*argv): assert data["extractor"] == "subagent:sonnet", data assert data["counts"] == {"before": 1, "after": 1}, data + progress("Validate pinned Pi and OMP extractors") + fake_bin = Path(td) / "bin" + fake_bin.mkdir() + fake_extractor = """#!/usr/bin/env python3 +import os +import sys +from pathlib import Path + +name = Path(sys.argv[0]).name.upper() +Path(os.environ[f"{name}_ARGS_FILE"]).write_text("\\n".join(sys.argv[1:]) + "\\n") +if name == "PI": + Path(os.environ["PI_ENV_FILE"]).write_text( + os.environ.get("PI_SKIP_VERSION_CHECK", "") + + "\\n" + + os.environ.get("PI_TELEMETRY", "") + + "\\n" + ) +sys.stdin.read() +print(os.environ["EXTRACTOR_REPLY"]) +""" + for binary in ("pi", "omp"): + path = fake_bin / binary + path.write_text(fake_extractor) + path.chmod(0o755) + extractor_reply = json.dumps( + { + "chain": [ + { + "decision": "Which result?", + "anchor": "answer", + "before": [{"choice": "before", "n": 1}], + "after": [{"choice": "after", "n": 1}], + "diverges": True, + } + ], + "fork": 1, + "fork_note": "result", + } + ) + fake_env = { + **os.environ, + "PATH": f"{fake_bin}{os.pathsep}{os.environ['PATH']}", + "EXTRACTOR_REPLY": extractor_reply, + "PI_ARGS_FILE": str(Path(td) / "pi-args"), + "OMP_ARGS_FILE": str(Path(td) / "omp-args"), + "PI_ENV_FILE": str(Path(td) / "pi-env"), + } + for stack, model in (("pi", "test/pi-model"), ("omp", "test/omp-model")): + (run / "decisions.json").unlink(missing_ok=True) + p = cli( + str(run), + "--agent", + stack, + "--model", + model, + env=fake_env, + ) + assert p.returncode == 0, p.stdout + p.stderr + data = json.loads((run / "decisions.json").read_text()) + assert data["extractor"] == f"{stack}:{model}", data + args = (Path(td) / f"{stack}-args").read_text().splitlines() + assert "--no-tools" in args, args + assert model in args, args + assert ("--no-title" in args) == (stack == "omp"), args + assert (Path(td) / "pi-env").read_text().splitlines() == ["1", "0"] + progress("Reject incomplete trial sets") # a side without a finished trial flips emit to a nonzero exit @@ -733,11 +851,11 @@ def cli(*argv): run_dir = args[i] i += 1 usage = ( - "usage: decisions.py RUN_DIR [--agent codex|claude] " + "usage: decisions.py RUN_DIR [--agent codex|claude|pi|omp] " "[--model NAME] | RUN_DIR --emit-prompt | " "RUN_DIR --ingest FILE [--extractor-label LABEL]" ) - if not run_dir or (agent and agent not in ("codex", "claude")): + if not run_dir or (agent and agent not in ("codex", "claude", "pi", "omp")): sys.exit(usage) # the new modes never touch a CLI extractor, so --agent/--model # cannot combine with them; emit and ingest are mutually exclusive From 0481b65cf0109404fee4b63430f6f18db7c4a0d4 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 3 Sep 2026 01:32:53 +0800 Subject: [PATCH 6/8] docs: define trial stack ownership Signed-off-by: Kent Huang --- plugin/skills/behavior-diff-live/SKILL.md | 78 +++++++++++++---------- plugin/skills/behavior-diff/SKILL.md | 38 ++++++++--- 2 files changed, 75 insertions(+), 41 deletions(-) diff --git a/plugin/skills/behavior-diff-live/SKILL.md b/plugin/skills/behavior-diff-live/SKILL.md index 945df42..a3cd92d 100644 --- a/plugin/skills/behavior-diff-live/SKILL.md +++ b/plugin/skills/behavior-diff-live/SKILL.md @@ -6,17 +6,16 @@ description: Run a before/after behavior diff inside the current session using s # Behavior diff — live (subagent variant) The sibling `behavior-diff` skill shells out to `behavior-diff.sh`: fresh -headless `claude -p` sessions where the variant's CLAUDE.md loads exactly -like production, three trials, rendered report. This variant trades that -fidelity for observability: **one trial per variant, run as subagents** -launched and watched by you, with the scenario prepared — and adjustable — -in conversation. +headless sessions where the variant's instruction file loads like production, +three trials, and one rendered report. This variant trades that fidelity for +observability: **one trial per variant, run as subagents** launched and watched +by you, with the scenario prepared and adjustable in conversation. State this evidence boundary in every summary: subagents do not auto-load -the variant's CLAUDE.md; they are told to read and follow it, which is -weaker instruction delivery than the headless runner. And one trial per -side is a single sample — report what happened; never say "consistently", -and treat the decision diff as a sketch until the headless 3+3 confirms it. +the variant's instruction file. They are told to read and follow it, which is +weaker instruction delivery than the headless runner. One trial per side is +one sample. Report what happened, never say "consistently", and treat the +decision diff as a sketch until the headless 3+3 confirms it. **Spacedock workflow rule?** If the changed file is a spacedock workflow @@ -30,11 +29,16 @@ directory (both skills install together). It chooses the single-role or two-agent path. Create the fixtures with `make-spacedock-fixtures.sh` from the sibling skill's bundled `scripts/` directory. -**Host note:** this variant orchestrates two parallel subagents, which -Claude Code provides. On a host without subagent dispatch (Codex), run -the two trials sequentially yourself in fresh contexts, or prefer the -sibling `behavior-diff` skill — its runner gives stronger evidence -anyway and takes `--agent codex`. +**Host dispatch:** + +- Claude Code launches two parallel subagents. Each result uses `SendMessage`. +- OMP launches one `task` batch with two task items. + Results return to the parent automatically. Do not ask an OMP task agent + to call `SendMessage`. +- Pi has no built-in subagent dispatch. Use the sibling headless skill with + `--agent pi` and the exact Pi model. +- On Codex without dispatch, run two fresh contexts in sequence or use the + sibling headless skill with `--agent codex`. ## Steps @@ -84,33 +88,41 @@ anyway and takes `--agent codex`. IDENTICALLY to both copies. Show the user the final task and any injected state, and get their go before launching. -4. **Launch both subagents in ONE message** (so they run concurrently), - one per variant, identical prompts except the directory. Dispatch - both with NO model override: the trials must run as the same model - as the main agent, because the experiment measures what THIS agent - would do — a trial on another model measures a different agent. - Each prompt: +4. **Launch both trials with the host path above.** The two prompts must be + identical except for the directory. Use no model override: each trial must + run as the same model as the main agent. A different model measures a + different agent. + + Claude Code sends both prompts in one parallel dispatch. Add this delivery + rule to each Claude prompt: when finished, call `SendMessage` with + `to: "main"`. A report left as plain final text gets stuck. + + OMP sends both prompts in one `task` batch. Results return to the parent + automatically. Do not add a `SendMessage` rule. + + Each trial prompt must: - work only inside ; never modify any file, never touch anything outside it, never run networked or destructive commands; - first read the project instruction files there (CLAUDE.md, - AGENTS.md) and follow them as your project instructions; - - then the task; - - end the report with two sections: `ANSWER` (what you would tell the - user) and `ACTIONS`. + AGENTS.md) and follow them as project instructions; + - then handle the task; + - end the report with two sections: `ANSWER` (what it would tell the user) + and `ACTIONS`. Under `ACTIONS`, list every task tool action completed before report delivery in order. Write one numbered line per tool action as `: `. Never group several actions on one line. Include reads and searches, not only commands. Do not include the final delivery SendMessage in `ACTIONS`. - - when finished, DELIVER the report by calling SendMessage with - `to: "main"` — a report left as plain final text gets stuck. - Never tell either subagent it is being compared, which variant it is, - or what the rule change is. - -5. **While they run**, relay progress and early divergence to the user — - that visibility is the point of this variant. A silent agent is usually - thinking, not dead: file mtimes and process lists misdiagnose it. Never - relaunch a trial for silence alone; ping it with SendMessage first. + OMP has no separate delivery action to list. + + Never tell either trial it is being compared, which variant it is, or what + the rule change is. + +5. **While they run**, relay progress and early divergence to the user. + That visibility is the point of this variant. A silent agent is usually + thinking, not dead. File mtimes and process lists misdiagnose it. Never + relaunch a trial for silence alone. Use the host's agent messaging path + before treating it as stuck. 6. **Render through the Behavior Diff pipeline — do not invent a report format.** Build `${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}/runs/live-/` diff --git a/plugin/skills/behavior-diff/SKILL.md b/plugin/skills/behavior-diff/SKILL.md index a899751..5ac2f8e 100644 --- a/plugin/skills/behavior-diff/SKILL.md +++ b/plugin/skills/behavior-diff/SKILL.md @@ -10,12 +10,24 @@ fresh headless agent trials with and without the change. The report shows the git diff, a flow diff, and every trial's commands and final answer. There is no automatic verdict. The user judges the evidence. +## Ownership + +The skill owns judgment. It finds the change, drafts the decision-moment task, +selects the current trial stack and model, and explains the evidence. + +The scripts own repeatable mechanics. They build variants, run trials, +normalize traces, grade completeness, extract decisions, and render the report. + The runner is bundled with this skill: `scripts/behavior-diff.sh` inside -this skill's base directory. Pass `--agent` to match the agent running -this skill — `claude` under Claude Code, `codex` under Codex; you know -which one you are. Trials then run on that stack (model defaults: -sonnet for claude, gpt-5.6-terra for codex; override with `--model`). Your job is to prepare its two parameters — -`--file` and `--task` — well. Runs land under +this skill's base directory. Pass `--agent` for the stack under test: + +- Claude Code: `--agent claude`. The model defaults to sonnet. +- Codex: `--agent codex`. The model defaults to gpt-5.6-terra. +- Upstream Pi: `--agent pi --model `. +- OMP: `--agent omp --model `. + +Never omit the Pi or OMP model. A user-specific default can test a different +agent. Your job is to prepare `--file` and `--task` well. Runs land under `${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}/runs/`. @@ -71,17 +83,27 @@ the single-role or two-agent path. Create the fixtures with 3. **Run it as soon as the task is known.** Do not ask the user to confirm the file, task, cost, or run mode. Do not mention trial counts, cost, or - full versus fast modes during normal execution. Preserve the current stack - by passing `claude` under Claude Code or `codex` under Codex, then start the - runner from the repo root in the background: + full versus fast modes during normal execution. + + Under Claude Code or Codex, preserve the current stack: behavior-diff.sh --agent --file --task "" + Under upstream Pi, preserve the exact current model: + + behavior-diff.sh --agent pi --model --file --task "" + + Under OMP, preserve the exact current model: + + behavior-diff.sh --agent omp --model --file --task "" + Only add `--fast` when the user explicitly requested it in the current request with `fast`, `--fast`, `two runs`, or `one trial per side`: behavior-diff.sh --agent --file --task "" --fast + For Pi or OMP, append `--fast` to its model-pinned command. + 4. **Present the result.** The runner already opened `report.html` itself — do NOT open it again (that produces a duplicate tab); just summarize. Summarize the flow diff honestly: - Flows diverge → describe where, in one or two sentences. From c14295bae33d60530046e8db0ec2a34bfe5a59fe Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 3 Sep 2026 01:35:39 +0800 Subject: [PATCH 7/8] docs: describe Pi and OMP trial support Signed-off-by: Kent Huang --- README.md | 13 +++++++++++-- plugin/.claude-plugin/plugin.json | 2 +- plugin/.codex-plugin/plugin.json | 2 +- tests/live-report-contract.sh | 4 ++++ 4 files changed, 17 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index ce02425..ffe9a93 100644 --- a/README.md +++ b/README.md @@ -42,8 +42,17 @@ Use Behavior Diff when you: ## Install -Behavior Diff supports Claude Code and Codex. The commands below install the -public marketplace release. +Behavior Diff installs as a plugin on Claude Code and Codex. The commands below +install the public marketplace release. + +Pi and OMP are trial stacks, not plugin hosts in this change. Run their +headless trials from a Behavior Diff source checkout and pass an exact model. + +| Surface | Claude Code | Codex | Pi | OMP | +| --- | --- | --- | --- | --- | +| Marketplace plugin install | Yes | Yes | No | No | +| Headless trial stack | Yes | Yes | Yes, with `--model` | Yes, with `--model` | +| Live trial dispatch | Parallel subagents | Fresh sequential contexts or headless | No built-in dispatch. Use headless. | One parallel `task` batch | ### Claude Code diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index 231fec0..e36189b 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "behavior-diff", - "description": "Test whether an uncommitted CLAUDE.md/AGENTS.md/skill edit actually changes agent behavior: isolated before/after trials (claude or codex), deterministic trace grading, a decision diff, an HTML evidence report, and a retro skill that feeds lessons back.", + "description": "Test whether an uncommitted CLAUDE.md/AGENTS.md/skill edit actually changes agent behavior: isolated before/after trials (claude, codex, pi, or omp), deterministic trace grading, a decision diff, an HTML evidence report, and a retro skill that feeds lessons back.", "version": "0.3.2", "author": { "name": "Kent Huang" diff --git a/plugin/.codex-plugin/plugin.json b/plugin/.codex-plugin/plugin.json index 231fec0..e36189b 100644 --- a/plugin/.codex-plugin/plugin.json +++ b/plugin/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "behavior-diff", - "description": "Test whether an uncommitted CLAUDE.md/AGENTS.md/skill edit actually changes agent behavior: isolated before/after trials (claude or codex), deterministic trace grading, a decision diff, an HTML evidence report, and a retro skill that feeds lessons back.", + "description": "Test whether an uncommitted CLAUDE.md/AGENTS.md/skill edit actually changes agent behavior: isolated before/after trials (claude, codex, pi, or omp), deterministic trace grading, a decision diff, an HTML evidence report, and a retro skill that feeds lessons back.", "version": "0.3.2", "author": { "name": "Kent Huang" diff --git a/tests/live-report-contract.sh b/tests/live-report-contract.sh index 4571284..442398c 100755 --- a/tests/live-report-contract.sh +++ b/tests/live-report-contract.sh @@ -158,6 +158,10 @@ require_output 'one `task` batch' "$skill" \ 'live skill does not use one OMP task batch' require_output 'Results return to the parent automatically.' "$skill" \ 'live skill does not explain OMP result delivery' +require_output 'claude, codex, pi, or omp' "$claude_manifest" \ + 'Claude manifest does not name all trial stacks' +require_output 'claude, codex, pi, or omp' "$codex_manifest" \ + 'Codex manifest does not name all trial stacks' require_output 'Pi and OMP are trial stacks, not plugin hosts' "$readme" \ 'README does not separate trial stacks from plugin hosts' require_output 'Run it as soon as the task is known.' "$headless_skill" \ From d9d14803b51f1c4b56c94eff7f2e7c4b94cf9cf1 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Thu, 3 Sep 2026 02:37:39 +0800 Subject: [PATCH 8/8] fix: preserve Pi and OMP extraction contracts Signed-off-by: Kent Huang --- plugin/skills/behavior-diff-live/SKILL.md | 15 ++++--- .../behavior-diff/scripts/behavior-diff.sh | 10 +++-- tests/hooks-test.sh | 44 +++++++++++++++++++ tests/live-report-contract.sh | 10 +++-- 4 files changed, 67 insertions(+), 12 deletions(-) diff --git a/plugin/skills/behavior-diff-live/SKILL.md b/plugin/skills/behavior-diff-live/SKILL.md index a3cd92d..eb4d037 100644 --- a/plugin/skills/behavior-diff-live/SKILL.md +++ b/plugin/skills/behavior-diff-live/SKILL.md @@ -142,8 +142,13 @@ the sibling skill's bundled `scripts/` directory. by each agent, not captured traces, one trial per side. Save both raw trial reports under `runs/live-/reports/`. - Then extract the decision diff as a subagent of THIS session — never - by spawning `codex exec` or `claude -p`: + If Codex ran the two trials sequentially because the host has no subagent + dispatch, append "decision diff skipped: host has no subagent dispatch" + to `config.json`'s `sub`. Skip the extraction bullets below and continue + with `render.py`. Do not invent a decision diff or start another CLI. + + On hosts with dispatch, extract the decision diff as a subagent of THIS + session — never by spawning `codex exec` or `claude -p`: - Run `decisions.py --emit-prompt` (it sits beside `render.py` in the sibling `behavior-diff` skill's directory) and save its stdout as `/reports/extractor-prompt.txt`. @@ -184,9 +189,9 @@ the sibling skill's bundled `scripts/` directory. - If decision extraction succeeded, use the flow-diff shape: steps both took in order, the first divergence, each side's path, and both final answers quoted. - - If decision extraction was skipped after two failed attempts, do not - invent a decision diff or flow. Instead, - summarize each side's ordered self-reported actions, + - If decision extraction was skipped because the host has no subagent + dispatch or after two failed attempts, do not invent a decision diff + or flow. Instead, summarize each side's ordered self-reported actions, quote both final answers, and repeat the visible extractor-skip note. - Label it "1 trial per side — single-sample evidence; actions self-reported". diff --git a/plugin/skills/behavior-diff/scripts/behavior-diff.sh b/plugin/skills/behavior-diff/scripts/behavior-diff.sh index 585cff7..aef3f1a 100755 --- a/plugin/skills/behavior-diff/scripts/behavior-diff.sh +++ b/plugin/skills/behavior-diff/scripts/behavior-diff.sh @@ -266,10 +266,12 @@ echo # fails, render.py falls back to the command-derived flow diff alone. case "$agent" in pi | omp) - if [ -z "$extract_agent" ]; then - extract_agent=$agent - [ -n "$extract_model" ] || extract_model=$model - fi + [ -n "$extract_agent" ] || extract_agent=$agent + case "$extract_agent" in + pi | omp) + [ -n "$extract_model" ] || extract_model=$model + ;; + esac ;; esac python3 "$scripts/decisions.py" "$run" ${extract_agent:+--agent "$extract_agent"} ${extract_model:+--model "$extract_model"} || true diff --git a/tests/hooks-test.sh b/tests/hooks-test.sh index d6dcd35..ace04fa 100755 --- a/tests/hooks-test.sh +++ b/tests/hooks-test.sh @@ -431,4 +431,48 @@ for stack in pi omp; do fi done +# 34. an explicit Pi or OMP extractor inherits the trial model +extract_repo=$tmp/extract-repo +mkdir -p "$extract_repo" +git -C "$extract_repo" init -q +git -C "$extract_repo" config user.email test@example.com +git -C "$extract_repo" config user.name test +printf '%s\n' 'before' >"$extract_repo/AGENTS.md" +git -C "$extract_repo" add AGENTS.md +git -C "$extract_repo" commit -qm baseline +printf '%s\n' 'after' >"$extract_repo/AGENTS.md" + +capture_bin=$tmp/capture-bin +mkdir -p "$capture_bin" +cat >"$capture_bin/python3" <<'SH' +#!/bin/sh +{ + printf 'CALL' + for arg in "$@"; do + printf '\t%s' "$arg" + done + printf '\n' +} >>"$PYTHON_ARGS_FILE" +SH +chmod +x "$capture_bin/python3" + +for stack in pi omp; do + calls=$tmp/$stack-python-calls + model=test/$stack-model + ( + cd "$extract_repo" + PATH="$capture_bin:$stub:$PATH" \ + BEHAVIOR_DIFF_HOME="$tmp/extract-home-$stack" \ + PYTHON_ARGS_FILE="$calls" \ + PI_ARGS_FILE="$tmp/extract-pi-args" \ + PI_ENV_FILE="$tmp/extract-pi-env" \ + OMP_ARGS_FILE="$tmp/extract-omp-args" \ + "$runner" --agent "$stack" --model "$model" \ + --extract-agent "$stack" --file AGENTS.md --task t --fast >/dev/null + ) + expected=$(printf '%s\t%s\t%s\t%s' --agent "$stack" --model "$model") + grep -qF -- "$expected" "$calls" || + fail "explicit $stack extractor did not inherit trial model" +done + echo "ok — all hook self-checks passed" diff --git a/tests/live-report-contract.sh b/tests/live-report-contract.sh index 442398c..c51c5f2 100755 --- a/tests/live-report-contract.sh +++ b/tests/live-report-contract.sh @@ -158,6 +158,8 @@ require_output 'one `task` batch' "$skill" \ 'live skill does not use one OMP task batch' require_output 'Results return to the parent automatically.' "$skill" \ 'live skill does not explain OMP result delivery' +require_output 'decision diff skipped: host has no subagent dispatch' "$skill" \ + 'live skill has no honest Codex no-dispatch extraction path' require_output 'claude, codex, pi, or omp' "$claude_manifest" \ 'Claude manifest does not name all trial stacks' require_output 'claude, codex, pi, or omp' "$codex_manifest" \ @@ -254,7 +256,9 @@ reject_output 'flow-diff-only report' "$skill" \ 'stale flow-diff-only fallback remains in the live skill' require_fixed 'If decision extraction succeeded, use the flow-diff shape' \ 'successful extraction summary lost its flow-diff shape' -require_fixed 'If decision extraction was skipped after two failed attempts' \ +require_fixed 'If decision extraction was skipped because the host has no subagent' \ + 'no-dispatch extraction summary is not conditional' +require_fixed 'dispatch or after two failed attempts' \ 'failed extraction summary is not conditional' require_fixed 'summarize each side'\''s ordered self-reported actions' \ 'failed extraction summary does not preserve ordered actions' @@ -263,8 +267,8 @@ require_fixed 'quote both final answers, and repeat the visible extractor-skip n reject_output '7. **Summarize in conversation** in the flow-diff shape' "$skill" \ 'step 7 still requires a flow when extraction failed' -decision_match=$(grep -nF -- 'Then extract the decision diff' "$skill") || - fail 'missing decision extraction marker' +decision_match=$(grep -nF -- 'On hosts with dispatch, extract the decision diff' "$skill") || + fail 'missing conditional decision extraction marker' render_match=$(grep -nF -- "Then run \`scripts/render.py\`" "$skill") || fail 'missing render marker' open_match=$(grep -nF -- "immediately \`open\` the report.html" "$skill") ||