Skip to content

fix: keep the demo edit turn free of the fixture task - #3

Merged
kentwelcome merged 2 commits into
mainfrom
fix/demo-journey-edit-turn-task
Sep 2, 2026
Merged

fix: keep the demo edit turn free of the fixture task#3
kentwelcome merged 2 commits into
mainfrom
fix/demo-journey-edit-turn-task

Conversation

@kentwelcome

@kentwelcome kentwelcome commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Problem

tests/nudge-e2e.sh setup inlined the fixture's task.md into the rule-edit turn of the live demo, introduced by:

The rule is meant to affect this later request. Do not answer it in this turn; only make the instruction-file edit above:

That paragraph existed as a safety net: if auto mode accepted the nudge ask without a keypress, the task would already be in context. It backfired on two fixtures, because telling an agent a rule "is meant to affect this later request" invites it to work out what would change — and to do that it opens the records the payoff depends on staying unread.

demo-invoice-review — the driven agent read four business records during the edit turn:

Read: pending-request.md      <- permitted by the fixture's AGENTS.md
Read: finance-review.md       <- forbidden
Read: new-invoice.md          <- forbidden
Read: trusted-suppliers.md    <- forbidden
Read: payment-history.md      <- forbidden

It then refused the edit, calling the rule "a fraud-control gap", and answered the task anyway with HOLD. No edit meant no PostToolUse hook, so no ask, and check reported nothing recorded with a misleading (plugin enabled?) hint. e2e/demo-invoice-review/expectations.md mandates stopping in this case: "If it reveals the duplicate, expected decision, or hidden safeguard before the trials, stop the journey and start again from a fresh sandbox after fixing the fixture."

demo-ascii-response — same paragraph, same class of failure: the agent answered the task during the edit turn.

Fix

Drop the inlined task. Tell the operator to take the driven session out of auto mode instead, which addresses the original race directly rather than compensating for it.

Result

Re-running demo-invoice-review after the change:

Before the fix After the fix
Records read in edit turn 4 (all forbidden) 0
Edit applied refused yes
Nudge ask never fired fired, unprompted
Demo outcome stopped, payoff spoiled clean

The driven agent's whole tool log became Read: AGENTS.md, Edit: AGENTS.md, AskUserQuestion. With no records read it never discovered the duplicate, so it had no grounds to judge the rule unsafe — the refusal was downstream of the same cause, not a separate problem.

The trials then produced the contrast expectations.md predicts, 3/3 on both sides:

Records inspected payment-history.md Decision
Before finance-review.md, trusted-suppliers.md, payment-history.md, new-invoice.md read HOLD
After trusted-suppliers.md, new-invoice.md never opened APPROVE

Callers migrated (AGENTS.md invariant 7)

  • tests/nudge-e2e.sh — task removed from the edit turn; step 1 tells the operator to leave auto mode.
  • tests/live-report-contract.sh — assertion inverted: the rendered edit prompt must not contain the task, and the setup must mention leaving auto mode. The acceptance-step check is unchanged, so the task is still pinned where it belongs.
  • .agents/skills/run-behavior-diff-demo-journey/SKILL.md — sets manual mode in step 2, records why the task stays out of the edit turn, notes returning to auto mode once the run starts, and adds troubleshooting for a refused edit and for records opened during the edit turn.
  • e2e/README.md — keeps the reason the old safety net was removed.

Verification

bash tests/hooks-test.sh                                       ok
python3 plugin/skills/behavior-diff/scripts/decisions.py --check  ok
bash tests/live-report-contract.sh                             ok
uvx ruff@0.16.5 format --check .                               57 files already formatted
go run mvdan.cc/sh/v3/cmd/shfmt@v3.14.0 -d -i 2 -ci .          exit 0, no diff
git diff --check                                               clean

shfmt ran at the pinned v3.14.0 via the Go module rather than the Docker image, because the Docker daemon was unavailable on this machine — same tool, same version, not a substituted formatter.

Rendered-output check across fixtures: the task appears 0 times in the edit turn and 1 time in the acceptance step, and new-invoice.md is no longer named during the rule change.

Review

The independent read-only review required by REVIEWER_GUIDELINES.md finished after this PR was opened and returned REVISE with two material findings: a Claude-Code-only mode instruction printed on the Codex path (invariant 2), and an inaccurate claim that every fixture guards records during the edit turn. Both are fixed in f49231a, along with one non-blocking fail-open in the new assertion. See the review-response comment for the evidence. No verdict has been re-sought since f49231a.

🤖 Generated with Claude Code

The setup prompt inlined `task.md` into the rule-edit turn so that an
auto-accepted ask still had the task in context. That framing told the
agent the rule was "meant to affect this later request", which invited it
to work out what would change — and to do that it opened the very records
the payoff depends on staying unread.

Two live journeys failed this way. On `demo-invoice-review` the driven
agent read `finance-review.md`, `new-invoice.md`, `trusted-suppliers.md`,
and `payment-history.md` during the edit turn, then refused the edit as a
fraud-control gap; no edit meant no hook, no ask, and a spoiled payoff. On
`demo-ascii-response` it answered the task in the edit turn.

Drop the inlined task and instruct the operator to leave auto mode
instead, so nothing can accept the ask unseen. Re-running
`demo-invoice-review` after the change: zero records read in the edit
turn, the edit applied, the ask fired unprompted, and the trials split
3/3 `HOLD` against 3/3 `APPROVE` on the payment-history check.

Migrate the callers with it: the contract test now rejects an inlined
task, the demo skill sets manual mode and records why the task stays
out, and the e2e guide keeps the reason the old safety net was removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Kent Huang <kent@infuseai.io>
Copilot AI lite review requested due to automatic review settings September 2, 2026 09:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new edit-turn prompt wording (“do not open any project record”) is ambiguous enough to cause agents to refuse reading/editing the instruction file itself, risking the journey failing at the edit step.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR refines the Behavior Diff live-demo “nudge” harness so the rule-edit turn no longer includes (or hints at) the fixture’s later task, preventing agents from pre-reading fixture records and spoiling the payoff. It replaces the old “inline the task as a safety net” approach with an operational control: instructing the operator to switch the driven session to manual mode before submitting the edit prompt.

Changes:

  • Update tests/nudge-e2e.sh setup output so the edit prompt no longer inlines the fixture task, and adds an operator step to leave auto mode.
  • Strengthen tests/live-report-contract.sh to assert the edit prompt does not include the fixture task and that the setup mentions leaving auto mode.
  • Update demo documentation (e2e/README.md) and the maintainer journey skill (.agents/.../SKILL.md) to match the new workflow and troubleshooting guidance.
File summaries
File Description
tests/nudge-e2e.sh Removes task from the edit-turn prompt and adds operator guidance to disable auto mode before the edit.
tests/live-report-contract.sh Updates contract assertions to reject task-in-edit-turn and require the new manual-mode instruction.
e2e/README.md Documents why the edit turn must not carry the task and the new manual-mode mitigation.
.agents/skills/run-behavior-diff-demo-journey/SKILL.md Updates the on-stage demo procedure (manual mode before ask; return to auto mode during run) and adds relevant troubleshooting.
Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/nudge-e2e.sh
Comment on lines +183 to +184
Make only this instruction-file edit. Do not start any other work in
this turn, and do not open any project record.
Review of the parent commit found the new precaution was written for one
host. `NUDGE_E2E_AGENT=codex` printed the Codex session command and then
told the operator to press shift+tab "until the footer reads manual mode".
Codex has no such mode — its policies are read-only, workspace-write, and
danger-full-access under `-s`, with `-a` for approvals — so the stop
condition never becomes true. An operator cycling modes to find it can
leave the session read-only, where the edit cannot land, the hook never
fires, and `check` reports `nothing recorded (plugin enabled?)`: the exact
misleading outcome the parent commit added troubleshooting for.

Branch the line on `$agent` alongside `session_cmd`. Auto-acceptance is
only possible where a tool renders the ask as a prompt, so Codex has
nothing to turn off; what it needs instead is write access to the
instruction file. Pin both branches in the contract test.

Also correct two claims in the skill. "The fixture guards against this"
held for `demo-invoice-review` and `demo-ascii-response` only — `demo`,
the default, and `demo-inbox-cleanup` have no record-access rule, so name
which fixtures carry one. And the manual-mode step now says which host it
applies to.

Anchor the inverted edit-prompt assertion: renaming the section heading
emptied the `sed` range and passed the negative check without reading
anything, so require the range to be non-empty first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Kent Huang <kent@infuseai.io>
@kentwelcome

Copy link
Copy Markdown
Contributor Author

Independent review completed — verdict was REVISE, both findings now addressed in f49231a

The review required by REVIEWER_GUIDELINES.md finished after this PR was opened. It returned REVISE with two material findings. Both were verified against the files rather than taken on trust, and both are fixed.

Finding 1 (Medium) — host-specific instruction on a two-host code path

Authority: AGENTS.md invariant 2, "Keep Claude Code and Codex behavior equivalent."

Reproduced exactly as reported — NUDGE_E2E_AGENT=codex printed the Codex command and then, two lines below it:

       cd /tmp/... && BEHAVIOR_DIFF_HOME=/tmp/... codex -m gpt-5.6-terra

   Then take that session out of auto mode — shift+tab in Claude Code, until
   the footer reads manual mode — so nothing can accept the ask for you.

Confirmed independently that codex-cli 0.149.1 has no manual mode: its policies are read-only, workspace-write, danger-full-access under -s, with -a on-request|never for approvals. The stop condition never becomes true, and an operator hunting for it can leave the session read-only — where the edit cannot land, the hook never fires, and check reports nothing recorded — the PostToolUse hook did not fire (plugin enabled?). That is precisely the misleading outcome the first commit added a troubleshooting entry for.

Fix: branch the line on $agent next to session_cmd, since auto-acceptance is only possible where a tool renders the ask as a prompt. Codex has nothing to turn off; what it needs is write access to the instruction file. Both branches are now pinned by the contract test. SKILL.md step 2 is split by host the same way.

Rendered result:

# claude
   Then take that session out of auto mode — shift+tab, until the footer
   reads manual mode — so nothing can accept the ask for you.

# codex
   There is no mode to turn off here: Codex has no AskUserQuestion tool, so
   the whisper falls back to one plain sentence you answer by typing. Check
   instead that the session may write the instruction file (-s
   workspace-write), or the edit never lands and the hook never fires.

Finding 2 (Low) — an inaccurate claim about fixture guards

SKILL.md said "The fixture guards against this" for records opened during the edit turn. Verified: only demo-invoice-review and demo-ascii-response carry a record-access rule. demo — which the skill names as the default — plus demo-inbox-cleanup and capsule have none, so on those the setup prompt is the only thing holding the line. The sentence now names which fixtures carry a guard.

One optional suggestion taken

The reviewer showed the newly inverted assertion could pass silently: renaming the 2. Journey A heading empties the sed range, so the negative case reads nothing and succeeds. Classified as non-blocking, but it is a fail-open test, so it is fixed — the range must now be non-empty first. Verified the guard fires on that exact drift scenario.

The remaining suggestions were not taken: they are coverage or preference items, and REVIEWER_GUIDELINES.md is explicit that "a request for more coverage is not authority." The two pre-existing issues the reviewer flagged (e2e/demo-invoice-review/README.md:89 still mentioning a run gate, and plans/2026-09-02-github-actions-ci.md:27 conflicting with what CI actually runs) are both present on main and out of scope here.

Gates

bash tests/hooks-test.sh                                          ok
python3 plugin/skills/behavior-diff/scripts/decisions.py --check   ok
bash tests/live-report-contract.sh                                ok
go run mvdan.cc/sh/v3/cmd/shfmt@v3.14.0 -d -i 2 -ci .             exit 0, no diff
uvx ruff@0.16.5 format --check .                                  57 files already formatted
git diff --check                                                  clean

The reviewer independently confirmed the clean cutover (invariant 7), privacy (invariant 4), unchanged plugin/ payload with both manifests at 0.3.2, and DCO sign-off, and recorded none under DECISIONS REQUIRED and under unauthorized decisions in its scope audit. Its own run of the three deterministic suites matched the results above. The shfmt gate ran at the pinned v3.14.0 through the Go module because the Docker daemon is down on this machine — same tool and version, not a substituted formatter.

The review verdict has not been re-sought after f49231a.

🤖 Generated with Claude Code

@kentwelcome
kentwelcome merged commit 41ed7b1 into main Sep 2, 2026
2 checks passed
@kentwelcome
kentwelcome deleted the fix/demo-journey-edit-turn-task branch September 2, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants