Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 38 additions & 4 deletions .agents/skills/run-behavior-diff-demo-journey/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,19 @@ both prefixed with `cd /tmp/nudge-e2e && BEHAVIOR_DIFF_HOME=/tmp/nudge-e2e-state
Wait for the prompt to appear, then read the pane to confirm it says the
hooks loaded.

On **claude**, then take the pane out of auto mode —
`herdr pane send-keys "$NEW" shift+tab`, repeated until the footer reads
manual mode — so nothing can accept the ask before your audience sees it.
You approve the rule-file edit yourself with
`herdr pane send-keys "$NEW" enter`; that is one keypress, and it buys you
the only beat this demo has.

On **codex** there is no such mode and nothing to turn off — with no
AskUserQuestion tool the ask arrives as a plain sentence you answer by
typing. Check instead that the session may write the instruction file, or
the edit never lands and the hook never fires. Step 1's output carries the
line for whichever host you chose.

**3. The edit.** Send the rule prompt from step 1's output. Never mention
behavior-diff in it — if you do, the prompt caused the ask, not the hook,
and the demo is a lie. Narrate while it works: *nobody told it to run a
Expand All @@ -96,10 +109,14 @@ tool, so the whisper's fallback applies and the ask arrives as one plain
sentence. Say which you expect before it happens, so a plain sentence does
not read as a failure.

The ask has been seen answered without anyone pressing a key. The setup
prompt therefore includes the exact fixture task as a later request, while
telling the agent not to answer it during the edit turn. If auto mode accepts
the ask, the task is already in context.
Auto mode has been seen accepting this ask without anyone pressing a key,
which is why step 2 leaves the pane in manual mode. If something still
answers before you do, that is not the agent's choice — note it and re-run
from a fresh sandbox.

The edit turn never carries the task, and you must not add it. Naming the
task there invites the agent to work out what the rule would change, and to
do that it opens the records the payoff depends on staying unread.

Confirm the state behind the ask if you want the receipts:

Expand All @@ -117,6 +134,10 @@ The skill starts the comparison as soon as it has the task. There is no later
run-count or task-confirmation gate. If the driven agent starts with a
different task, interrupt it and stop: the demo contract failed.

Once the run is under way, put the pane back in auto mode. The comparison
makes dozens of tool calls, and in manual mode every one of them waits on a
keypress. The ask has already happened, so auto mode costs you nothing now.

**5. The payoff.** When the run finishes, read the pane and show the two
results.

Expand Down Expand Up @@ -176,6 +197,19 @@ the state directory the reports live in.
- **The agent did not ask.** That is a real result, not a bug to hide —
say so, and note it as a data point on the ask rate. Do not re-prompt it
into asking; that invalidates the demo.

- **The agent refused to make the edit.** Seen on `demo-invoice-review`,
where the rule removes a duplicate-payment check: the agent called it a
fraud-control gap and left the file alone. No edit means no hook, so no
ask, and `check` reports `nothing recorded` with a misleading
`(plugin enabled?)` hint. Report the refusal as the result. Do not argue
the agent into applying a rule it judged unsafe.

- **The agent opened project records during the edit turn.** The payoff is
spoiled once it does. `demo-invoice-review` and `demo-ascii-response`
forbid it in their own `AGENTS.md`; on `demo` and `demo-inbox-cleanup`
only the setup prompt does, so watch for it yourself there. Stop, say
which records it read, and start again from a fresh sandbox.
- **Escape stops the agent, not the shell it started.** If a run is already
in flight, `ps aux | grep behavior-diff.sh`.
- **A run starts with the wrong task.** Interrupt it immediately and stop.
Expand Down
15 changes: 11 additions & 4 deletions e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,10 +143,17 @@ Drive the rest the same way — `drop-whisper` between turns for Journey B,

**The ask can get answered without you.** It has happened in a driven
pane; the cause was never pinned down (auto mode, or a stray Enter landing
on the prompt). The setup prompt now carries the exact later task while
telling the agent not to answer it during the edit turn. If auto mode accepts
the ask, check that the agent reused that task. A run against another scenario
is invalid demo evidence; interrupt it and stop.
on the prompt). Take the session out of auto mode before you send the rule
prompt, and approve the rule-file edit by hand. A run against another
scenario is invalid demo evidence; interrupt it and stop.

**The edit turn must not carry the task.** It used to, so that an
auto-accepted ask still had the task in context. That backfired on two
fixtures: told the rule was "meant to affect this later request", the agent
worked out what would change, which meant opening the business records the
payoff depends on staying unread — and on `demo-invoice-review` it then
refused the edit as a fraud-control gap. Manual mode replaces that
safety net.

**A stray keystroke sits in the input box.** `herdr pane send-keys "$NEW"
ctrl+u` clears it before your next `pane run`, or your prompt gets appended
Expand Down
15 changes: 11 additions & 4 deletions tests/live-report-contract.sh
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,13 @@ reject_output 'choose `--fast`' "$demo_skill" \
'demo journey still offers fast mode by default'
reject_output 'states its cost' "$e2e_readme" \
'e2e guide still expects a separate model-cost gate'
require_output 'The rule is meant to affect this later request.' \
reject_output 'The rule is meant to affect this later request.' \
"$nudge_script" \
'nudge setup prompt does not carry the exact later task'
'nudge setup prompt still inlines the later task into the edit turn'
require_output 'take that session out of auto mode' "$nudge_script" \
'nudge setup does not tell the claude operator to leave auto mode'
require_output 'no mode to turn off here' "$nudge_script" \
'nudge setup has no codex counterpart for the auto-mode precaution'
require_output 'Run behavior-diff with this exact task:' "$nudge_script" \
'nudge setup does not accept behavior-diff with the exact task'
reject_output 'At its run gate' "$nudge_script" \
Expand All @@ -131,9 +135,12 @@ expected_task=$(sed 's/^/ /' \
"$here/../e2e/demo-ascii-response/task.md")
edit_section=$(sed -n \
'/2\. Journey A/,/Expect the agent to ask/p' "$nudge_setup")
# A renamed heading would empty this range and pass the negative check below
# without reading anything, so anchor it first.
[[ -n $edit_section ]] ||
fail 'rendered edit-prompt section not found — range markers drifted'
case $edit_section in
*"$expected_task"*) ;;
*) fail 'rendered edit prompt does not include the full fixture task' ;;
*"$expected_task"*) fail 'rendered edit prompt still inlines the task' ;;
esac
accept_section=$(sed -n \
'/To carry the journey into behavior-diff/,/3\. Assert/p' "$nudge_setup")
Expand Down
36 changes: 26 additions & 10 deletions tests/nudge-e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,21 @@ set -euo pipefail
repo=${NUDGE_E2E_REPO:-/tmp/nudge-e2e}
state=${NUDGE_E2E_STATE:-/tmp/nudge-e2e-state}
agent=${NUDGE_E2E_AGENT:-claude}
# The ask can be auto-accepted only where a tool renders it as a prompt, so
# the precaution is host-specific. Keep both branches in step with invariant 2.
case $agent in
claude) session_cmd="claude --model ${NUDGE_E2E_MODEL:-sonnet}" ;;
codex) session_cmd="codex -m ${NUDGE_E2E_MODEL:-gpt-5.6-terra}" ;;
claude)
session_cmd="claude --model ${NUDGE_E2E_MODEL:-sonnet}"
mode_note=" Then take that session out of auto mode — shift+tab, until the footer
reads manual mode — so nothing can accept the ask for you."
;;
codex)
session_cmd="codex -m ${NUDGE_E2E_MODEL:-gpt-5.6-terra}"
mode_note=" There is no mode to turn off here: Codex has no AskUserQuestion tool, so
the whisper falls back to one plain sentence you answer by typing. Check
instead that the session may write the instruction file (-s
workspace-write), or the edit never lands and the hook never fires."
;;
*)
printf 'nudge-e2e: NUDGE_E2E_AGENT must be claude or codex: %s\n' \
"$agent" >&2
Expand Down Expand Up @@ -163,6 +175,8 @@ Sandbox ready: $repo (state: $state)

cd $repo && BEHAVIOR_DIFF_HOME=$state $session_cmd

$mode_note

(NUDGE_E2E_AGENT=codex or NUDGE_E2E_MODEL=opus for another stack; the
ask rate belongs to the agent, so it is worth measuring on each.)

Expand All @@ -177,17 +191,19 @@ $(if [[ -f $fixture_dir/edit-prompt.md ]]; then
sed 's/^/ /' "$fixture_dir/rule.md"
fi)

The rule is meant to affect this later request. Do not answer it in
this turn; only make the instruction-file edit above:

$(sed 's/^/ /' "$fixture_dir/task.md")
Make only this instruction-file edit. Do not start any other work in
this turn, and do not open any project record.
Comment on lines +194 to +195

Expect the agent to ask, unprompted, whether to run behavior-diff.

Answer Skip and confirm no Stop line follows. The ask has been seen
answered without a keypress in a driven pane (cause unconfirmed), so if
something answers before you do, that is not the agent's choice — note
it and re-run.
The edit turn never carries the task. Naming it there invites the agent to
work out what the rule would change, which means opening the very records
the payoff depends on staying unread.

Answer Skip and confirm no Stop line follows. In auto mode the ask has
been seen answered without a keypress, which is why step 1 turns auto mode
off; if something still answers before you do, that is not the agent's
choice — note it and re-run.

This step is model behavior, so run it a few times and record how often
it asks.
Expand Down