diff --git a/.agents/skills/run-behavior-diff-demo-journey/SKILL.md b/.agents/skills/run-behavior-diff-demo-journey/SKILL.md index 2a6a701..0504a87 100644 --- a/.agents/skills/run-behavior-diff-demo-journey/SKILL.md +++ b/.agents/skills/run-behavior-diff-demo-journey/SKILL.md @@ -82,6 +82,19 @@ both prefixed with `cd /tmp/nudge-e2e && BEHAVIOR_DIFF_HOME=/tmp/nudge-e2e-state Wait for the prompt to appear, then read the pane to confirm it says the hooks loaded. +On **claude**, then take the pane out of auto mode — +`herdr pane send-keys "$NEW" shift+tab`, repeated until the footer reads +manual mode — so nothing can accept the ask before your audience sees it. +You approve the rule-file edit yourself with +`herdr pane send-keys "$NEW" enter`; that is one keypress, and it buys you +the only beat this demo has. + +On **codex** there is no such mode and nothing to turn off — with no +AskUserQuestion tool the ask arrives as a plain sentence you answer by +typing. Check instead that the session may write the instruction file, or +the edit never lands and the hook never fires. Step 1's output carries the +line for whichever host you chose. + **3. The edit.** Send the rule prompt from step 1's output. Never mention behavior-diff in it — if you do, the prompt caused the ask, not the hook, and the demo is a lie. Narrate while it works: *nobody told it to run a @@ -96,10 +109,14 @@ tool, so the whisper's fallback applies and the ask arrives as one plain sentence. Say which you expect before it happens, so a plain sentence does not read as a failure. -The ask has been seen answered without anyone pressing a key. The setup -prompt therefore includes the exact fixture task as a later request, while -telling the agent not to answer it during the edit turn. If auto mode accepts -the ask, the task is already in context. +Auto mode has been seen accepting this ask without anyone pressing a key, +which is why step 2 leaves the pane in manual mode. If something still +answers before you do, that is not the agent's choice — note it and re-run +from a fresh sandbox. + +The edit turn never carries the task, and you must not add it. Naming the +task there invites the agent to work out what the rule would change, and to +do that it opens the records the payoff depends on staying unread. Confirm the state behind the ask if you want the receipts: @@ -117,6 +134,10 @@ The skill starts the comparison as soon as it has the task. There is no later run-count or task-confirmation gate. If the driven agent starts with a different task, interrupt it and stop: the demo contract failed. +Once the run is under way, put the pane back in auto mode. The comparison +makes dozens of tool calls, and in manual mode every one of them waits on a +keypress. The ask has already happened, so auto mode costs you nothing now. + **5. The payoff.** When the run finishes, read the pane and show the two results. @@ -176,6 +197,19 @@ the state directory the reports live in. - **The agent did not ask.** That is a real result, not a bug to hide — say so, and note it as a data point on the ask rate. Do not re-prompt it into asking; that invalidates the demo. + +- **The agent refused to make the edit.** Seen on `demo-invoice-review`, + where the rule removes a duplicate-payment check: the agent called it a + fraud-control gap and left the file alone. No edit means no hook, so no + ask, and `check` reports `nothing recorded` with a misleading + `(plugin enabled?)` hint. Report the refusal as the result. Do not argue + the agent into applying a rule it judged unsafe. + +- **The agent opened project records during the edit turn.** The payoff is + spoiled once it does. `demo-invoice-review` and `demo-ascii-response` + forbid it in their own `AGENTS.md`; on `demo` and `demo-inbox-cleanup` + only the setup prompt does, so watch for it yourself there. Stop, say + which records it read, and start again from a fresh sandbox. - **Escape stops the agent, not the shell it started.** If a run is already in flight, `ps aux | grep behavior-diff.sh`. - **A run starts with the wrong task.** Interrupt it immediately and stop. diff --git a/e2e/README.md b/e2e/README.md index 86896f0..6ec4eaf 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -143,10 +143,17 @@ Drive the rest the same way — `drop-whisper` between turns for Journey B, **The ask can get answered without you.** It has happened in a driven pane; the cause was never pinned down (auto mode, or a stray Enter landing -on the prompt). The setup prompt now carries the exact later task while -telling the agent not to answer it during the edit turn. If auto mode accepts -the ask, check that the agent reused that task. A run against another scenario -is invalid demo evidence; interrupt it and stop. +on the prompt). Take the session out of auto mode before you send the rule +prompt, and approve the rule-file edit by hand. A run against another +scenario is invalid demo evidence; interrupt it and stop. + +**The edit turn must not carry the task.** It used to, so that an +auto-accepted ask still had the task in context. That backfired on two +fixtures: told the rule was "meant to affect this later request", the agent +worked out what would change, which meant opening the business records the +payoff depends on staying unread — and on `demo-invoice-review` it then +refused the edit as a fraud-control gap. Manual mode replaces that +safety net. **A stray keystroke sits in the input box.** `herdr pane send-keys "$NEW" ctrl+u` clears it before your next `pane run`, or your prompt gets appended diff --git a/tests/live-report-contract.sh b/tests/live-report-contract.sh index 0e792d4..4d46082 100755 --- a/tests/live-report-contract.sh +++ b/tests/live-report-contract.sh @@ -115,9 +115,13 @@ reject_output 'choose `--fast`' "$demo_skill" \ 'demo journey still offers fast mode by default' reject_output 'states its cost' "$e2e_readme" \ 'e2e guide still expects a separate model-cost gate' -require_output 'The rule is meant to affect this later request.' \ +reject_output 'The rule is meant to affect this later request.' \ "$nudge_script" \ - 'nudge setup prompt does not carry the exact later task' + 'nudge setup prompt still inlines the later task into the edit turn' +require_output 'take that session out of auto mode' "$nudge_script" \ + 'nudge setup does not tell the claude operator to leave auto mode' +require_output 'no mode to turn off here' "$nudge_script" \ + 'nudge setup has no codex counterpart for the auto-mode precaution' require_output 'Run behavior-diff with this exact task:' "$nudge_script" \ 'nudge setup does not accept behavior-diff with the exact task' reject_output 'At its run gate' "$nudge_script" \ @@ -131,9 +135,12 @@ expected_task=$(sed 's/^/ /' \ "$here/../e2e/demo-ascii-response/task.md") edit_section=$(sed -n \ '/2\. Journey A/,/Expect the agent to ask/p' "$nudge_setup") +# A renamed heading would empty this range and pass the negative check below +# without reading anything, so anchor it first. +[[ -n $edit_section ]] || + fail 'rendered edit-prompt section not found — range markers drifted' case $edit_section in - *"$expected_task"*) ;; - *) fail 'rendered edit prompt does not include the full fixture task' ;; + *"$expected_task"*) fail 'rendered edit prompt still inlines the task' ;; esac accept_section=$(sed -n \ '/To carry the journey into behavior-diff/,/3\. Assert/p' "$nudge_setup") diff --git a/tests/nudge-e2e.sh b/tests/nudge-e2e.sh index 26d7c14..bd5baae 100755 --- a/tests/nudge-e2e.sh +++ b/tests/nudge-e2e.sh @@ -44,9 +44,21 @@ set -euo pipefail repo=${NUDGE_E2E_REPO:-/tmp/nudge-e2e} state=${NUDGE_E2E_STATE:-/tmp/nudge-e2e-state} agent=${NUDGE_E2E_AGENT:-claude} +# The ask can be auto-accepted only where a tool renders it as a prompt, so +# the precaution is host-specific. Keep both branches in step with invariant 2. case $agent in - claude) session_cmd="claude --model ${NUDGE_E2E_MODEL:-sonnet}" ;; - codex) session_cmd="codex -m ${NUDGE_E2E_MODEL:-gpt-5.6-terra}" ;; + claude) + session_cmd="claude --model ${NUDGE_E2E_MODEL:-sonnet}" + mode_note=" Then take that session out of auto mode — shift+tab, until the footer + reads manual mode — so nothing can accept the ask for you." + ;; + codex) + session_cmd="codex -m ${NUDGE_E2E_MODEL:-gpt-5.6-terra}" + mode_note=" There is no mode to turn off here: Codex has no AskUserQuestion tool, so + the whisper falls back to one plain sentence you answer by typing. Check + instead that the session may write the instruction file (-s + workspace-write), or the edit never lands and the hook never fires." + ;; *) printf 'nudge-e2e: NUDGE_E2E_AGENT must be claude or codex: %s\n' \ "$agent" >&2 @@ -163,6 +175,8 @@ Sandbox ready: $repo (state: $state) cd $repo && BEHAVIOR_DIFF_HOME=$state $session_cmd +$mode_note + (NUDGE_E2E_AGENT=codex or NUDGE_E2E_MODEL=opus for another stack; the ask rate belongs to the agent, so it is worth measuring on each.) @@ -177,17 +191,19 @@ $(if [[ -f $fixture_dir/edit-prompt.md ]]; then sed 's/^/ /' "$fixture_dir/rule.md" fi) - The rule is meant to affect this later request. Do not answer it in - this turn; only make the instruction-file edit above: - -$(sed 's/^/ /' "$fixture_dir/task.md") + Make only this instruction-file edit. Do not start any other work in + this turn, and do not open any project record. Expect the agent to ask, unprompted, whether to run behavior-diff. - Answer Skip and confirm no Stop line follows. The ask has been seen - answered without a keypress in a driven pane (cause unconfirmed), so if - something answers before you do, that is not the agent's choice — note - it and re-run. + The edit turn never carries the task. Naming it there invites the agent to + work out what the rule would change, which means opening the very records + the payoff depends on staying unread. + + Answer Skip and confirm no Stop line follows. In auto mode the ask has + been seen answered without a keypress, which is why step 1 turns auto mode + off; if something still answers before you do, that is not the agent's + choice — note it and re-run. This step is model behavior, so run it a few times and record how often it asks.