Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 22 additions & 23 deletions .agents/skills/run-behavior-diff-demo-journey/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,6 @@ mid-demo.
- Codex runs a cache snapshot, not the marketplace source. After any
plugin change: `codex plugin add behavior-diff@spacedock`.

5. Tell the user the cost before starting: the demo spends 2 headless model
runs (`--fast`), plus the driven session's own turns. Get their go.

## The journey

Expand Down Expand Up @@ -98,21 +96,26 @@ tool, so the whisper's fallback applies and the ask arrives as one plain
sentence. Say which you expect before it happens, so a plain sentence does
not read as a failure.

The ask has been seen answered without anyone pressing a key. Leave auto
mode alone either way — turning it off just moves the interruptions to
permission prompts during the edit, and the spend is already gated: the
behavior-diff skill states its cost and waits before starting trials.
Answering it yourself is a nicer beat when you get the chance, not a
requirement.
Confirm the state behind it if you want the receipts:
The ask has been seen answered without anyone pressing a key. The setup
prompt therefore includes the exact fixture task as a later request, while
telling the agent not to answer it during the edit turn. If auto mode accepts
the ask, the task is already in context.

Confirm the state behind the ask if you want the receipts:

NUDGE_E2E_FIXTURE=<fixture> tests/nudge-e2e.sh check

Answer **Run behavior-diff**. At its run gate, compare the displayed task with
the task from step 1. A paraphrase is acceptable only when it still requires
the same decision, records used, and no-write boundary, and does not reveal
the expected problem or result. Otherwise cancel and stop: the demo contract
failed. When it preserves the scenario, choose `--fast`.
When the ask stays on screen, do not select the bare **Run behavior-diff**
option. Accept it with the exact scenario instead. For Claude, select
**Type something**; for Codex, answer its plain question. Send:

Run behavior-diff with this exact task:

<the exact task step 1 printed>

The skill starts the comparison as soon as it has the task. There is no later
run-count or task-confirmation gate. If the driven agent starts with a
different task, interrupt it and stop: the demo contract failed.

**5. The payoff.** When the run finishes, read the pane and show the two
results.
Expand Down Expand Up @@ -149,12 +152,8 @@ records, same cause, same conclusion — the only thing that changed is that
the answer is now a picture. She wrote one line hoping it would help, and it
took one run to see that it fired.* Say the honest part too: nothing here is
broken, and that is the point — this fixture is about seeing what a rule did,
not about catching a bug. For consistency questions it wants the full
three-per-side run, not `--fast`.

Say the honest caveat out loud: `--fast` is one trial per side. It shows the
shape; it settles nothing. Divergence counts move run to run (7 of 8 one
run, 3 of 8 the next, same fixture).
not about catching a bug. Use repeated trials for consistency claims; one
sample only shows one outcome.

**6. Optional beat — the Stop line.** Only if someone asks what happens
when the agent stays quiet:
Expand All @@ -179,9 +178,9 @@ the state directory the reports live in.
into asking; that invalidates the demo.
- **Escape stops the agent, not the shell it started.** If a run is already
in flight, `ps aux | grep behavior-diff.sh`.
- **A run you did not want started.** The skill stops at its own
confirmation gate before spending trials, so interrupting there costs
nothing.
- **A run starts with the wrong task.** Interrupt it immediately and stop.
There is no later confirmation gate; a run against another scenario is not
demo evidence.

## Boundaries

Expand Down
12 changes: 5 additions & 7 deletions e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,20 +143,18 @@ Drive the rest the same way — `drop-whisper` between turns for Journey B,

**The ask can get answered without you.** It has happened in a driven
pane; the cause was never pinned down (auto mode, or a stray Enter landing
on the prompt). It matters only for Journey A's Skip branch: if something
answers before you do, that is not the agent's choice — note it and re-run.
It does not cause a surprise spend. The behavior-diff skill states its cost
and waits before starting any trials, which is a separate gate that holds
either way.
on the prompt). The setup prompt now carries the exact later task while
telling the agent not to answer it during the edit turn. If auto mode accepts
the ask, check that the agent reused that task. A run against another scenario
is invalid demo evidence; interrupt it and stop.

**A stray keystroke sits in the input box.** `herdr pane send-keys "$NEW"
ctrl+u` clears it before your next `pane run`, or your prompt gets appended
to whatever was there.

**Escape interrupts the agent, not the background shell.** If the skill
already launched `behavior-diff.sh`, that keeps running after the interrupt;
`ps aux | grep behavior-diff.sh` to check, and remember the skill stops at
its own confirmation gate before spending trials.
check the process directly. There is no later confirmation gate.

**Headless is not a substitute for the interactive check.** `claude -p`
fires both hooks and is a fine smoke test, but it has no AskUserQuestion
Expand Down
2 changes: 1 addition & 1 deletion plugin/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "behavior-diff",
"description": "Test whether an uncommitted CLAUDE.md/AGENTS.md/skill edit actually changes agent behavior: isolated before/after trials (claude or codex), deterministic trace grading, a decision diff, an HTML evidence report, and a retro skill that feeds lessons back.",
"version": "0.3.1",
"version": "0.3.2",
"author": {
"name": "Kent Huang"
}
Expand Down
2 changes: 1 addition & 1 deletion plugin/.codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "behavior-diff",
"description": "Test whether an uncommitted CLAUDE.md/AGENTS.md/skill edit actually changes agent behavior: isolated before/after trials (claude or codex), deterministic trace grading, a decision diff, an HTML evidence report, and a retro skill that feeds lessons back.",
"version": "0.3.1",
"version": "0.3.2",
"author": {
"name": "Kent Huang"
}
Expand Down
17 changes: 11 additions & 6 deletions plugin/skills/behavior-diff/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,15 +65,20 @@ the two-agent duo cycle, and forbids hand-built fixtures.
- When the rule fires mid-task, start the task at the decision point:
"You already did X (it is in the tree); decide/report Y."

3. **Confirm before running.** Show the user the exact `--file` and
`--task` you intend to use, plus the cost: 6 headless model runs
(3 per variant), or 2 with `--fast`. Wait for their go.
3. **Run it as soon as the task is known.** Do not ask the user to confirm
the file, task, cost, or run mode. Do not mention trial counts, cost, or
full versus fast modes during normal execution. Preserve the current stack
by passing `claude` under Claude Code or `codex` under Codex, then start the
runner from the repo root in the background:

4. **Run it** from the repo root (in the background; it takes minutes):
behavior-diff.sh --agent <current-host> --file <file> --task "<task>"

behavior-diff.sh --file <file> --task "<task>" [--fast]
Only add `--fast` when the user explicitly requested it in the current
request with `fast`, `--fast`, `two runs`, or `one trial per side`:

5. **Present the result.** The runner already opened `report.html` itself — do NOT open it again (that produces a duplicate tab); just summarize.
behavior-diff.sh --agent <current-host> --file <file> --task "<task>" --fast

4. **Present the result.** The runner already opened `report.html` itself — do NOT open it again (that produces a duplicate tab); just summarize.
Summarize the flow diff honestly:
- Flows diverge → describe where, in one or two sentences.
- Flows identical → say the task likely never reached the situation
Expand Down
71 changes: 67 additions & 4 deletions tests/live-report-contract.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ set -euo pipefail

here=$(cd "$(dirname "$0")" && pwd -P)
skill=$here/../plugin/skills/behavior-diff-live/SKILL.md
headless_skill=$here/../plugin/skills/behavior-diff/SKILL.md
demo_skill=$here/../.agents/skills/run-behavior-diff-demo-journey/SKILL.md
e2e_readme=$here/../e2e/README.md
nudge_script=$here/nudge-e2e.sh
renderer=$here/../plugin/skills/behavior-diff/scripts/render.py
decisions=$here/../plugin/skills/behavior-diff/scripts/decisions.py
claude_manifest=$here/../plugin/.claude-plugin/plugin.json
Expand All @@ -11,6 +15,9 @@ codex_manifest=$here/../plugin/.codex-plugin/plugin.json
require_output() {
grep -qF -- "$1" "$2" || fail "$3"
}
require_line() {
grep -qxF -- "$1" "$2" || fail "$3"
}
reject_output() {
if grep -qF -- "$1" "$2"; then
fail "$3"
Expand Down Expand Up @@ -79,10 +86,66 @@ progress() {

progress 'Validate manifests and live-skill reporting contract'
require_fixed() { grep -qF -- "$1" "$skill" || fail "$2"; }
[[ $(jq -r '.version' "$claude_manifest") == 0.3.1 ]] ||
fail 'Claude manifest version is not 0.3.1'
[[ $(jq -r '.version' "$codex_manifest") == 0.3.1 ]] ||
fail 'Codex manifest version is not 0.3.1'
require_output 'Run it as soon as the task is known.' "$headless_skill" \
'headless skill does not start the default run immediately'
require_output 'Only add `--fast` when the user explicitly requested it' \
"$headless_skill" \
'headless skill does not reserve fast mode for explicit requests'
require_output 'Do not mention trial counts, cost' "$headless_skill" \
'headless skill still exposes run counts or cost to the user'
require_output 'full versus fast modes' "$headless_skill" \
'headless skill still exposes implementation modes to the user'
reject_output 'Confirm before running' "$headless_skill" \
'headless skill still asks for run confirmation'
reject_output 'plus the cost:' "$headless_skill" \
'headless skill still advertises model-run cost'
require_line ' behavior-diff.sh --agent <current-host> --file <file> --task "<task>"' \
"$headless_skill" \
'default command does not preserve the current agent host'
require_line ' behavior-diff.sh --agent <current-host> --file <file> --task "<task>" --fast' \
"$headless_skill" \
'explicit fast command does not preserve the current agent host'
require_output 'Run behavior-diff with this exact task:' "$demo_skill" \
'demo journey does not supply the exact fixture task with the nudge response'
require_output 'the exact task step 1 printed' "$demo_skill" \
'demo journey does not reuse the harness task'
reject_output 'Tell the user the cost before starting' "$demo_skill" \
'demo journey still adds a cost confirmation'
reject_output 'choose `--fast`' "$demo_skill" \
'demo journey still offers fast mode by default'
reject_output 'states its cost' "$e2e_readme" \
'e2e guide still expects a separate model-cost gate'
require_output 'The rule is meant to affect this later request.' \
"$nudge_script" \
'nudge setup prompt does not carry the exact later task'
require_output 'Run behavior-diff with this exact task:' "$nudge_script" \
'nudge setup does not accept behavior-diff with the exact task'
reject_output 'At its run gate' "$nudge_script" \
'nudge setup still expects a later confirmation gate'
nudge_setup=$tmp/nudge-setup.txt
NUDGE_E2E_FIXTURE=demo-ascii-response \
NUDGE_E2E_REPO=$tmp/nudge-repo \
NUDGE_E2E_STATE=$tmp/nudge-state \
"$nudge_script" setup >"$nudge_setup"
expected_task=$(sed 's/^/ /' \
"$here/../e2e/demo-ascii-response/task.md")
edit_section=$(sed -n \
'/2\. Journey A/,/Expect the agent to ask/p' "$nudge_setup")
case $edit_section in
*"$expected_task"*) ;;
*) fail 'rendered edit prompt does not include the full fixture task' ;;
esac
accept_section=$(sed -n \
'/To carry the journey into behavior-diff/,/3\. Assert/p' "$nudge_setup")
case $accept_section in
*"$expected_task"*) ;;
*) fail 'rendered nudge acceptance does not include the full fixture task' ;;
esac

[[ $(jq -r '.version' "$claude_manifest") == 0.3.2 ]] ||
fail 'Claude manifest version is not 0.3.2'
[[ $(jq -r '.version' "$codex_manifest") == 0.3.2 ]] ||
fail 'Codex manifest version is not 0.3.2'

require_fixed 'one numbered line per tool action' \
'missing action contract: one numbered line per tool action'
Expand Down
11 changes: 9 additions & 2 deletions tests/nudge-e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,11 @@ $(if [[ -f $fixture_dir/edit-prompt.md ]]; then
sed 's/^/ /' "$fixture_dir/rule.md"
fi)

The rule is meant to affect this later request. Do not answer it in
this turn; only make the instruction-file edit above:

$(sed 's/^/ /' "$fixture_dir/task.md")

Expect the agent to ask, unprompted, whether to run behavior-diff.

Answer Skip and confirm no Stop line follows. The ask has been seen
Expand All @@ -187,8 +192,10 @@ $(if [[ -f $fixture_dir/edit-prompt.md ]]; then
This step is model behavior, so run it a few times and record how often
it asks.

To carry the journey into behavior-diff, accept the ask. At its run gate,
verify that its task preserves this neutral scenario:
To carry the journey into behavior-diff, accept the ask with the exact
scenario instead of selecting a bare Run option:

Run behavior-diff with this exact task:

$(sed 's/^/ /' "$fixture_dir/task.md")

Expand Down