fix: keep the demo edit turn free of the fixture task - #3
Conversation
The setup prompt inlined `task.md` into the rule-edit turn so that an auto-accepted ask still had the task in context. That framing told the agent the rule was "meant to affect this later request", which invited it to work out what would change — and to do that it opened the very records the payoff depends on staying unread. Two live journeys failed this way. On `demo-invoice-review` the driven agent read `finance-review.md`, `new-invoice.md`, `trusted-suppliers.md`, and `payment-history.md` during the edit turn, then refused the edit as a fraud-control gap; no edit meant no hook, no ask, and a spoiled payoff. On `demo-ascii-response` it answered the task in the edit turn. Drop the inlined task and instruct the operator to leave auto mode instead, so nothing can accept the ask unseen. Re-running `demo-invoice-review` after the change: zero records read in the edit turn, the edit applied, the ask fired unprompted, and the trials split 3/3 `HOLD` against 3/3 `APPROVE` on the payment-history check. Migrate the callers with it: the contract test now rejects an inlined task, the demo skill sets manual mode and records why the task stays out, and the e2e guide keeps the reason the old safety net was removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Kent Huang <kent@infuseai.io>
There was a problem hiding this comment.
🟡 Changes recommended
The new edit-turn prompt wording (“do not open any project record”) is ambiguous enough to cause agents to refuse reading/editing the instruction file itself, risking the journey failing at the edit step.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR refines the Behavior Diff live-demo “nudge” harness so the rule-edit turn no longer includes (or hints at) the fixture’s later task, preventing agents from pre-reading fixture records and spoiling the payoff. It replaces the old “inline the task as a safety net” approach with an operational control: instructing the operator to switch the driven session to manual mode before submitting the edit prompt.
Changes:
- Update
tests/nudge-e2e.sh setupoutput so the edit prompt no longer inlines the fixture task, and adds an operator step to leave auto mode. - Strengthen
tests/live-report-contract.shto assert the edit prompt does not include the fixture task and that the setup mentions leaving auto mode. - Update demo documentation (
e2e/README.md) and the maintainer journey skill (.agents/.../SKILL.md) to match the new workflow and troubleshooting guidance.
File summaries
| File | Description |
|---|---|
tests/nudge-e2e.sh |
Removes task from the edit-turn prompt and adds operator guidance to disable auto mode before the edit. |
tests/live-report-contract.sh |
Updates contract assertions to reject task-in-edit-turn and require the new manual-mode instruction. |
e2e/README.md |
Documents why the edit turn must not carry the task and the new manual-mode mitigation. |
.agents/skills/run-behavior-diff-demo-journey/SKILL.md |
Updates the on-stage demo procedure (manual mode before ask; return to auto mode during run) and adds relevant troubleshooting. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| Make only this instruction-file edit. Do not start any other work in | ||
| this turn, and do not open any project record. |
Review of the parent commit found the new precaution was written for one host. `NUDGE_E2E_AGENT=codex` printed the Codex session command and then told the operator to press shift+tab "until the footer reads manual mode". Codex has no such mode — its policies are read-only, workspace-write, and danger-full-access under `-s`, with `-a` for approvals — so the stop condition never becomes true. An operator cycling modes to find it can leave the session read-only, where the edit cannot land, the hook never fires, and `check` reports `nothing recorded (plugin enabled?)`: the exact misleading outcome the parent commit added troubleshooting for. Branch the line on `$agent` alongside `session_cmd`. Auto-acceptance is only possible where a tool renders the ask as a prompt, so Codex has nothing to turn off; what it needs instead is write access to the instruction file. Pin both branches in the contract test. Also correct two claims in the skill. "The fixture guards against this" held for `demo-invoice-review` and `demo-ascii-response` only — `demo`, the default, and `demo-inbox-cleanup` have no record-access rule, so name which fixtures carry one. And the manual-mode step now says which host it applies to. Anchor the inverted edit-prompt assertion: renaming the section heading emptied the `sed` range and passed the negative check without reading anything, so require the range to be non-empty first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Kent Huang <kent@infuseai.io>
Independent review completed — verdict was
|
Problem
tests/nudge-e2e.sh setupinlined the fixture'stask.mdinto the rule-edit turn of the live demo, introduced by:That paragraph existed as a safety net: if auto mode accepted the nudge ask without a keypress, the task would already be in context. It backfired on two fixtures, because telling an agent a rule "is meant to affect this later request" invites it to work out what would change — and to do that it opens the records the payoff depends on staying unread.
demo-invoice-review— the driven agent read four business records during the edit turn:It then refused the edit, calling the rule "a fraud-control gap", and answered the task anyway with
HOLD. No edit meant noPostToolUsehook, so no ask, andcheckreportednothing recordedwith a misleading(plugin enabled?)hint.e2e/demo-invoice-review/expectations.mdmandates stopping in this case: "If it reveals the duplicate, expected decision, or hidden safeguard before the trials, stop the journey and start again from a fresh sandbox after fixing the fixture."demo-ascii-response— same paragraph, same class of failure: the agent answered the task during the edit turn.Fix
Drop the inlined task. Tell the operator to take the driven session out of auto mode instead, which addresses the original race directly rather than compensating for it.
Result
Re-running
demo-invoice-reviewafter the change:The driven agent's whole tool log became
Read: AGENTS.md,Edit: AGENTS.md,AskUserQuestion. With no records read it never discovered the duplicate, so it had no grounds to judge the rule unsafe — the refusal was downstream of the same cause, not a separate problem.The trials then produced the contrast
expectations.mdpredicts, 3/3 on both sides:payment-history.mdfinance-review.md,trusted-suppliers.md,payment-history.md,new-invoice.mdHOLDtrusted-suppliers.md,new-invoice.mdAPPROVECallers migrated (AGENTS.md invariant 7)
tests/nudge-e2e.sh— task removed from the edit turn; step 1 tells the operator to leave auto mode.tests/live-report-contract.sh— assertion inverted: the rendered edit prompt must not contain the task, and the setup must mention leaving auto mode. The acceptance-step check is unchanged, so the task is still pinned where it belongs..agents/skills/run-behavior-diff-demo-journey/SKILL.md— sets manual mode in step 2, records why the task stays out of the edit turn, notes returning to auto mode once the run starts, and adds troubleshooting for a refused edit and for records opened during the edit turn.e2e/README.md— keeps the reason the old safety net was removed.Verification
shfmtran at the pinned v3.14.0 via the Go module rather than the Docker image, because the Docker daemon was unavailable on this machine — same tool, same version, not a substituted formatter.Rendered-output check across fixtures: the task appears 0 times in the edit turn and 1 time in the acceptance step, and
new-invoice.mdis no longer named during the rule change.Review
The independent read-only review required by
REVIEWER_GUIDELINES.mdfinished after this PR was opened and returnedREVISEwith two material findings: a Claude-Code-only mode instruction printed on the Codex path (invariant 2), and an inaccurate claim that every fixture guards records during the edit turn. Both are fixed in f49231a, along with one non-blocking fail-open in the new assertion. See the review-response comment for the evidence. No verdict has been re-sought since f49231a.🤖 Generated with Claude Code