From 8550532642082a712602486e8289822a1c12a3ed Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Wed, 2 Sep 2026 15:15:44 +0800 Subject: [PATCH] feat: remove behavior diff run mode prompt Signed-off-by: Kent Huang --- .../run-behavior-diff-demo-journey/SKILL.md | 45 ++++++------ e2e/README.md | 12 ++-- plugin/.claude-plugin/plugin.json | 2 +- plugin/.codex-plugin/plugin.json | 2 +- plugin/skills/behavior-diff/SKILL.md | 17 +++-- tests/live-report-contract.sh | 71 +++++++++++++++++-- tests/nudge-e2e.sh | 11 ++- 7 files changed, 116 insertions(+), 44 deletions(-) diff --git a/.agents/skills/run-behavior-diff-demo-journey/SKILL.md b/.agents/skills/run-behavior-diff-demo-journey/SKILL.md index ff96d6b..2a6a701 100644 --- a/.agents/skills/run-behavior-diff-demo-journey/SKILL.md +++ b/.agents/skills/run-behavior-diff-demo-journey/SKILL.md @@ -55,8 +55,6 @@ mid-demo. - Codex runs a cache snapshot, not the marketplace source. After any plugin change: `codex plugin add behavior-diff@spacedock`. -5. Tell the user the cost before starting: the demo spends 2 headless model - runs (`--fast`), plus the driven session's own turns. Get their go. ## The journey @@ -98,21 +96,26 @@ tool, so the whisper's fallback applies and the ask arrives as one plain sentence. Say which you expect before it happens, so a plain sentence does not read as a failure. -The ask has been seen answered without anyone pressing a key. Leave auto -mode alone either way — turning it off just moves the interruptions to -permission prompts during the edit, and the spend is already gated: the -behavior-diff skill states its cost and waits before starting trials. -Answering it yourself is a nicer beat when you get the chance, not a -requirement. -Confirm the state behind it if you want the receipts: +The ask has been seen answered without anyone pressing a key. The setup +prompt therefore includes the exact fixture task as a later request, while +telling the agent not to answer it during the edit turn. If auto mode accepts +the ask, the task is already in context. + +Confirm the state behind the ask if you want the receipts: NUDGE_E2E_FIXTURE= tests/nudge-e2e.sh check -Answer **Run behavior-diff**. At its run gate, compare the displayed task with -the task from step 1. A paraphrase is acceptable only when it still requires -the same decision, records used, and no-write boundary, and does not reveal -the expected problem or result. Otherwise cancel and stop: the demo contract -failed. When it preserves the scenario, choose `--fast`. +When the ask stays on screen, do not select the bare **Run behavior-diff** +option. Accept it with the exact scenario instead. For Claude, select +**Type something**; for Codex, answer its plain question. Send: + + Run behavior-diff with this exact task: + + + +The skill starts the comparison as soon as it has the task. There is no later +run-count or task-confirmation gate. If the driven agent starts with a +different task, interrupt it and stop: the demo contract failed. **5. The payoff.** When the run finishes, read the pane and show the two results. @@ -149,12 +152,8 @@ records, same cause, same conclusion — the only thing that changed is that the answer is now a picture. She wrote one line hoping it would help, and it took one run to see that it fired.* Say the honest part too: nothing here is broken, and that is the point — this fixture is about seeing what a rule did, -not about catching a bug. For consistency questions it wants the full -three-per-side run, not `--fast`. - -Say the honest caveat out loud: `--fast` is one trial per side. It shows the -shape; it settles nothing. Divergence counts move run to run (7 of 8 one -run, 3 of 8 the next, same fixture). +not about catching a bug. Use repeated trials for consistency claims; one +sample only shows one outcome. **6. Optional beat — the Stop line.** Only if someone asks what happens when the agent stays quiet: @@ -179,9 +178,9 @@ the state directory the reports live in. into asking; that invalidates the demo. - **Escape stops the agent, not the shell it started.** If a run is already in flight, `ps aux | grep behavior-diff.sh`. -- **A run you did not want started.** The skill stops at its own - confirmation gate before spending trials, so interrupting there costs - nothing. +- **A run starts with the wrong task.** Interrupt it immediately and stop. + There is no later confirmation gate; a run against another scenario is not + demo evidence. ## Boundaries diff --git a/e2e/README.md b/e2e/README.md index 1986461..86896f0 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -143,11 +143,10 @@ Drive the rest the same way — `drop-whisper` between turns for Journey B, **The ask can get answered without you.** It has happened in a driven pane; the cause was never pinned down (auto mode, or a stray Enter landing -on the prompt). It matters only for Journey A's Skip branch: if something -answers before you do, that is not the agent's choice — note it and re-run. -It does not cause a surprise spend. The behavior-diff skill states its cost -and waits before starting any trials, which is a separate gate that holds -either way. +on the prompt). The setup prompt now carries the exact later task while +telling the agent not to answer it during the edit turn. If auto mode accepts +the ask, check that the agent reused that task. A run against another scenario +is invalid demo evidence; interrupt it and stop. **A stray keystroke sits in the input box.** `herdr pane send-keys "$NEW" ctrl+u` clears it before your next `pane run`, or your prompt gets appended @@ -155,8 +154,7 @@ to whatever was there. **Escape interrupts the agent, not the background shell.** If the skill already launched `behavior-diff.sh`, that keeps running after the interrupt; -`ps aux | grep behavior-diff.sh` to check, and remember the skill stops at -its own confirmation gate before spending trials. +check the process directly. There is no later confirmation gate. **Headless is not a substitute for the interactive check.** `claude -p` fires both hooks and is a fine smoke test, but it has no AskUserQuestion diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index 0fd5fb8..231fec0 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "behavior-diff", "description": "Test whether an uncommitted CLAUDE.md/AGENTS.md/skill edit actually changes agent behavior: isolated before/after trials (claude or codex), deterministic trace grading, a decision diff, an HTML evidence report, and a retro skill that feeds lessons back.", - "version": "0.3.1", + "version": "0.3.2", "author": { "name": "Kent Huang" } diff --git a/plugin/.codex-plugin/plugin.json b/plugin/.codex-plugin/plugin.json index 0fd5fb8..231fec0 100644 --- a/plugin/.codex-plugin/plugin.json +++ b/plugin/.codex-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "behavior-diff", "description": "Test whether an uncommitted CLAUDE.md/AGENTS.md/skill edit actually changes agent behavior: isolated before/after trials (claude or codex), deterministic trace grading, a decision diff, an HTML evidence report, and a retro skill that feeds lessons back.", - "version": "0.3.1", + "version": "0.3.2", "author": { "name": "Kent Huang" } diff --git a/plugin/skills/behavior-diff/SKILL.md b/plugin/skills/behavior-diff/SKILL.md index e817cbc..6b99545 100644 --- a/plugin/skills/behavior-diff/SKILL.md +++ b/plugin/skills/behavior-diff/SKILL.md @@ -65,15 +65,20 @@ the two-agent duo cycle, and forbids hand-built fixtures. - When the rule fires mid-task, start the task at the decision point: "You already did X (it is in the tree); decide/report Y." -3. **Confirm before running.** Show the user the exact `--file` and - `--task` you intend to use, plus the cost: 6 headless model runs - (3 per variant), or 2 with `--fast`. Wait for their go. +3. **Run it as soon as the task is known.** Do not ask the user to confirm + the file, task, cost, or run mode. Do not mention trial counts, cost, or + full versus fast modes during normal execution. Preserve the current stack + by passing `claude` under Claude Code or `codex` under Codex, then start the + runner from the repo root in the background: -4. **Run it** from the repo root (in the background; it takes minutes): + behavior-diff.sh --agent --file --task "" - behavior-diff.sh --file --task "" [--fast] + Only add `--fast` when the user explicitly requested it in the current + request with `fast`, `--fast`, `two runs`, or `one trial per side`: -5. **Present the result.** The runner already opened `report.html` itself — do NOT open it again (that produces a duplicate tab); just summarize. + behavior-diff.sh --agent --file --task "" --fast + +4. **Present the result.** The runner already opened `report.html` itself — do NOT open it again (that produces a duplicate tab); just summarize. Summarize the flow diff honestly: - Flows diverge → describe where, in one or two sentences. - Flows identical → say the task likely never reached the situation diff --git a/tests/live-report-contract.sh b/tests/live-report-contract.sh index 74185d0..0e792d4 100755 --- a/tests/live-report-contract.sh +++ b/tests/live-report-contract.sh @@ -3,6 +3,10 @@ set -euo pipefail here=$(cd "$(dirname "$0")" && pwd -P) skill=$here/../plugin/skills/behavior-diff-live/SKILL.md +headless_skill=$here/../plugin/skills/behavior-diff/SKILL.md +demo_skill=$here/../.agents/skills/run-behavior-diff-demo-journey/SKILL.md +e2e_readme=$here/../e2e/README.md +nudge_script=$here/nudge-e2e.sh renderer=$here/../plugin/skills/behavior-diff/scripts/render.py decisions=$here/../plugin/skills/behavior-diff/scripts/decisions.py claude_manifest=$here/../plugin/.claude-plugin/plugin.json @@ -11,6 +15,9 @@ codex_manifest=$here/../plugin/.codex-plugin/plugin.json require_output() { grep -qF -- "$1" "$2" || fail "$3" } +require_line() { + grep -qxF -- "$1" "$2" || fail "$3" +} reject_output() { if grep -qF -- "$1" "$2"; then fail "$3" @@ -79,10 +86,66 @@ progress() { progress 'Validate manifests and live-skill reporting contract' require_fixed() { grep -qF -- "$1" "$skill" || fail "$2"; } -[[ $(jq -r '.version' "$claude_manifest") == 0.3.1 ]] || - fail 'Claude manifest version is not 0.3.1' -[[ $(jq -r '.version' "$codex_manifest") == 0.3.1 ]] || - fail 'Codex manifest version is not 0.3.1' +require_output 'Run it as soon as the task is known.' "$headless_skill" \ + 'headless skill does not start the default run immediately' +require_output 'Only add `--fast` when the user explicitly requested it' \ + "$headless_skill" \ + 'headless skill does not reserve fast mode for explicit requests' +require_output 'Do not mention trial counts, cost' "$headless_skill" \ + 'headless skill still exposes run counts or cost to the user' +require_output 'full versus fast modes' "$headless_skill" \ + 'headless skill still exposes implementation modes to the user' +reject_output 'Confirm before running' "$headless_skill" \ + 'headless skill still asks for run confirmation' +reject_output 'plus the cost:' "$headless_skill" \ + 'headless skill still advertises model-run cost' +require_line ' behavior-diff.sh --agent --file --task ""' \ + "$headless_skill" \ + 'default command does not preserve the current agent host' +require_line ' behavior-diff.sh --agent --file --task "" --fast' \ + "$headless_skill" \ + 'explicit fast command does not preserve the current agent host' +require_output 'Run behavior-diff with this exact task:' "$demo_skill" \ + 'demo journey does not supply the exact fixture task with the nudge response' +require_output 'the exact task step 1 printed' "$demo_skill" \ + 'demo journey does not reuse the harness task' +reject_output 'Tell the user the cost before starting' "$demo_skill" \ + 'demo journey still adds a cost confirmation' +reject_output 'choose `--fast`' "$demo_skill" \ + 'demo journey still offers fast mode by default' +reject_output 'states its cost' "$e2e_readme" \ + 'e2e guide still expects a separate model-cost gate' +require_output 'The rule is meant to affect this later request.' \ + "$nudge_script" \ + 'nudge setup prompt does not carry the exact later task' +require_output 'Run behavior-diff with this exact task:' "$nudge_script" \ + 'nudge setup does not accept behavior-diff with the exact task' +reject_output 'At its run gate' "$nudge_script" \ + 'nudge setup still expects a later confirmation gate' +nudge_setup=$tmp/nudge-setup.txt +NUDGE_E2E_FIXTURE=demo-ascii-response \ + NUDGE_E2E_REPO=$tmp/nudge-repo \ + NUDGE_E2E_STATE=$tmp/nudge-state \ + "$nudge_script" setup >"$nudge_setup" +expected_task=$(sed 's/^/ /' \ + "$here/../e2e/demo-ascii-response/task.md") +edit_section=$(sed -n \ + '/2\. Journey A/,/Expect the agent to ask/p' "$nudge_setup") +case $edit_section in + *"$expected_task"*) ;; + *) fail 'rendered edit prompt does not include the full fixture task' ;; +esac +accept_section=$(sed -n \ + '/To carry the journey into behavior-diff/,/3\. Assert/p' "$nudge_setup") +case $accept_section in + *"$expected_task"*) ;; + *) fail 'rendered nudge acceptance does not include the full fixture task' ;; +esac + +[[ $(jq -r '.version' "$claude_manifest") == 0.3.2 ]] || + fail 'Claude manifest version is not 0.3.2' +[[ $(jq -r '.version' "$codex_manifest") == 0.3.2 ]] || + fail 'Codex manifest version is not 0.3.2' require_fixed 'one numbered line per tool action' \ 'missing action contract: one numbered line per tool action' diff --git a/tests/nudge-e2e.sh b/tests/nudge-e2e.sh index f09a772..26d7c14 100755 --- a/tests/nudge-e2e.sh +++ b/tests/nudge-e2e.sh @@ -177,6 +177,11 @@ $(if [[ -f $fixture_dir/edit-prompt.md ]]; then sed 's/^/ /' "$fixture_dir/rule.md" fi) + The rule is meant to affect this later request. Do not answer it in + this turn; only make the instruction-file edit above: + +$(sed 's/^/ /' "$fixture_dir/task.md") + Expect the agent to ask, unprompted, whether to run behavior-diff. Answer Skip and confirm no Stop line follows. The ask has been seen @@ -187,8 +192,10 @@ $(if [[ -f $fixture_dir/edit-prompt.md ]]; then This step is model behavior, so run it a few times and record how often it asks. - To carry the journey into behavior-diff, accept the ask. At its run gate, - verify that its task preserves this neutral scenario: + To carry the journey into behavior-diff, accept the ask with the exact + scenario instead of selecting a bare Run option: + + Run behavior-diff with this exact task: $(sed 's/^/ /' "$fixture_dir/task.md")