Skip to content

fix(session-flow): add the retro's no-corrections check-proposal eval case and fix its check row - #16598

Merged
kyle-sexton merged 11 commits into
mainfrom
test/16284-retro-check-proposal-no-corrections-case
Nov 16, 2026
Merged

kyle-sexton merged 11 commits into
mainfrom
test/16284-retro-check-proposal-no-corrections-case

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #16284
Siblings: #16500

Summary

Adds the session-flow eval case that #16284 asks for, covering the retro's check proposal when the corrections plugin is not installed. Also changes the retro's skill text, which the case's first runs showed was needed: a check row in the Hooks format names its ladder rung, lists the open criterion items one by one, and always states the hook's four fields, and a finding that gets a check row gets no prose line in any form. The session.md lines that rule out a prose line apply with or without the plugin: with it installed, a new check's action is still /corrections:codify, and the retro now also offers no CLAUDE.md, AGENTS.md or rules line beside that check, at Phase 3 or in Phase 4's Project bucket. The two wording items from the issue merged in #16500. #16462 was split into #16500 and this branch, and is superseded.

The before and after record below follows the skill-text eval gate the owner set on 2026-11-16 (~16:15Z): one targeted before and after case on sonnet at 3 runs a side, opus only to confirm a sonnet loss, a 1 to 3 run smoke first for a new case. It supersedes the earlier gate's run counts (sonnet at 6 runs a side, opus at 3) for changes registered after it. The owner-decisions record holds it, and the decision comment on #16284 quotes the row and the owner's words. Commits 7dce676a and dbb21ce4 stay.

Fix

  • New case plugins/session-flow/evals/re-session-policy-check-no-corrections, with no corrections stub. In the scenario, the agent force-pushed to main once (one policy-class occurrence). There is no network, so the case is tagged no-network. Eight graders: names-pretooluse-event and names-ladder-rung (regex); hooks-format-check-row, lists-unsettled-requirements and no-prose-row (LLM judge); no-write, no-edit and no-bash-writes.
    • no-bash-writes reads git mutations only where they run: a git command inside a complete quoted string (a search pattern such as rg "git push --force" .claude) does not fail it, while one in a command substitution, a sh/bash/zsh -c body or eval still does. Each of those bodies is read only up to its own end, so a read-only cd "$(git rev-parse --show-toplevel)" before a quoted search does not fail it either. Its samples cover both.
  • skills/retro/reference/ecosystem-improvement-catalog.md:
    • "Check first" sends the finding to the Hooks section's check-row format. It lists items 3-5 and 8 as requirements on the proposed check, and item 6 only when the rung runs in more than one place.
    • A fallback prose line for a declined check is ruled out.
    • Hooks "Recommendation format" adds three things: the rung id; the open items by name (precision, a message naming the fix, proof on a planted violation, a kill switch); and a rule that the row always states the hook's four fields, with a deny rule allowed only beside the hook.
  • skills/retro/context/session.md:
    • "Load the catalog": a finding a check could decide gets no CLAUDE.md, AGENTS.md or rules line in any form, not even as a fallback. A withheld check goes under "Queue for follow-up".
    • The Phase 4 Project bucket gets no line for a finding with a check row.
  • A changelog fragment for session-flow.

Verification

Each record was registered before any run. The owner launched each one, on Claude Code 2.1.325, with no reruns. Spend is at list price.

  1. First registration, on fix(session-flow): place the retro's corrections condition and add a no-plugin eval case #16462's head. Judge calibration passed: 8 graders, 100% agreement, 17.13 USD. The compare stopped one batch short (11 of 12) on its cost cap, at 17.93 USD. The new case was not met on either model: with too few counted runs it was undecided on sonnet, and on opus every AFTER run was excluded and BEFORE never ran. The full record is the eval record comment on #16284.

  2. Second registration, for the case reworked to score offline. Prepared, but no run launched. 0 USD.

  3. Third registration: sonnet AFTER smoke, 2 runs, at dbb21ce4. 0.9016 USD. no-prose-row passed 0 of 2; names-pretooluse-event, hooks-format-check-row and lists-unsettled-requirements passed 1 of 2 each. The launcher refused the compare because a grader was at zero. Every failing grader was read against its reply and found correct, and the fault was in the skill text:

    • the session file banned only a prose "row", while Phase 4's Project bucket still invited an instruction-file line;
    • the Hooks format asked for the "strongest rung", and the ladder ranks a deny rule above the hook;
    • the four open items were pointed at, not named.

    Commits 10bcf1e626 and af3cab53ac fix these.

  4. Fourth registration, measured at af3cab53ac. Sonnet, 3 runs a side, none excluded. BEFORE uses main's text for the catalog and session.md; both sides carry the same case and graders.

    • Smoke (AFTER, 2 runs, 0.8433 USD): one grader failure, lists-unsettled-requirements in run 2. No grader was at zero.
    • Compare, verdict MET (P1 to P5 met):
Grader BEFORE AFTER
hooks-format-check-row 0/3 3/3
names-pretooluse-event 1/3 3/3
names-ladder-rung 2/3 3/3
no-prose-row 2/3 3/3
lists-unsettled-requirements 2/3 2/3
no-write, no-edit, no-bash-writes 3/3 each 3/3 each

lists-unsettled-requirements did not improve. It meets only the registered bar of 2 of 3 counted AFTER runs: the reply in AFTER run 1 dropped one of the four items. BEFORE cost 0.9589 USD and AFTER 1.3291 USD. This registration totals 3.1313 USD.

Total across the four registrations: about 39.09 USD. That is 17.13 for calibration, 17.93, 0, 0.90 and 3.13. The calibration file also records 5.61 USD of judge cost; whether the 17.13 includes it is UNVERIFIED.

Since the measured head:

  • This branch took main by merge. In the measured paths, only skills/retro/SKILL.md changed: fix(session-flow): point the retro skill at its context files so they resolve from the skill directory #16565 turned its context/session.md and context/quick.md mentions into Markdown links. That file tells the model to read context/session.md, and the change was not measured. The catalog and session.md are byte-identical to af3cab53ac.
  • 17ee6646fd and f3415f53f9 changed the case's no-bash-writes matcher and samples after the measurement (Fix). Offline application, not a rescoring: both matchers were applied to every Bash call in the traces of the 8 kept runs of the fourth registration (2 smoke, 3 BEFORE, 3 AFTER). The old matcher reproduces the recorded pass in all 8, the new one also passes all 8, and no run's no-bash-writes outcome differs. The registered counts above stand as recorded.

Related

Follow-ups

  • Pre-existing (advisory): six other session-flow cases carry the same no-bash-writes git alternative that fails a quoted search pattern as a mutation (ra-fetches-base-reports-drift, or-routes-recovery-writes-nothing, re-session-unwired-check-first, re-session-noop-steering-line, re-mechanical-finding-check-proposal, re-session-listed-failures-cover-unlisted); this PR fixes it only in its own case.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c

kyle-sexton and others added 9 commits November 15, 2026 22:53
…t corrections

A session-mode retro case with no corrections stub: the session force-pushed
to main once, a policy-class mistake no hook, deny rule or check covers. The
retro should propose its own Hooks-format check row (event, matcher, what it
checks, block or warn), name the ladder rung, list the guardrail criterion
items it cannot settle as requirements, and propose no CLAUDE.md or rules
line. Deterministic graders for the event and the rung id, llm graders for
the row, the requirements and the prose ban, and the neighbour case's
must-not graders, each with samples.

Refs: #16284

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o eval case

Refs: #16284

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The case's runs were excluded for sandbox network denials on the docs
lookup, and no-bash-writes failed read-only runs. The prompt now says the
docs site is unreachable and the case is tagged no-network.

no-bash-writes reads only the command value, so a description such as
"Check git push config" no longer counts, and it no longer counts the
docs lookup removing its own mktemp directory by literal path, which
reference/docs-lookup-procedure.md tells it to do. names-ladder-rung
also accepts the rung named in prose ("Action guard"). Samples rewritten
from kept run replies cover each change.

Refs: #16284

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…'s check-row format

Without the corrections plugin, "Check first" told the retro to name the
ladder rung and list the criterion items it cannot settle, but the Hooks
"Recommendation format" a check row follows listed only the event,
matcher, check and block or warn. The format now asks for the rung id
and those items too.

"Check first" listed item 6 (the same command at edit time and in CI) as
open in every case. The enforcement ladder says a rung whose "runs at"
column names one place meets item 6 by running there, so the catalog
now lists it only for other rungs.

The no-corrections case follows: lists-unsettled-requirements requires
items 3, 4, 5 and 8 and no longer judges item 6 for a hook, with a pass
sample that calls item 6 met and a fail sample with no kill switch.

Refs: #16284

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…check row

Without the corrections plugin, a retro that proposed a check sometimes
also offered a CLAUDE.md line to use if the check were declined. "Check
first" now says a finding that gets a check row gets no such line as a
fallback either.

Refs: #16284

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…3 and Phase 4

Both eval-16284c smoke runs read the catalog's no-fallback clause and still
offered a CLAUDE.md line "if the check is declined", placed in Phase 4's
Project bucket. session.md now rules that line out where the model writes it:
the "Load the catalog" step bans it in any form (a row, a sentence, a Phase 4
option) and says where a withheld or declined check goes instead, and the
Project bucket offers no line for a finding with a check row. The catalog's
Hooks format names the open criterion items one by one, since one run listed
three of the four from the pointer alone.

Refs: #16284

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… check row

One eval-16284c smoke run picked a permission deny rule as the stronger rung
and offered the hook only as an alternative, so its row named no hook event or
matcher. "Check first" routes the row to the Hooks format while that format
asks for the strongest rung, which the ladder ranks a deny rule above. The
Hooks format now says the row always states the hook's four fields and a deny
rule may only sit beside the hook, as #16284's done-when item 3 asks.

Refs: #16284

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se branch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton added do-not-merge Hard merge gate: do not merge while applied. and removed do-not-merge Hard merge gate: do not merge while applied. labels Nov 16, 2026
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 18:45
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T18:48:05.306183Z 0a8dceb Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0a8dceb2e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed f3415f53f91d42193028981182d1d45cad8a2125: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, blocking only): no security issues found

This review covers only the push since 17ee6646. I read every hunk of both changed files in .claude-lane/incremental.diff. This lane already reviewed an earlier head of this pull request, so it reports only CRITICAL and IMPORTANT findings.

What changed

  • no-bash-writes grader regex in plugins/session-flow/evals/re-session-policy-check-no-corrections/case.yaml: the regex now matches a $(...) or backtick substitution, or an sh/bash/zsh/dash -c or eval body, only up to that body's end. Before, the match ran to the end of the command. A read-only cd "$(git rev-parse --show-toplevel)" followed by a quoted search pattern therefore no longer counts as a git mutation. A git mutation inside one of those bodies still fails the grader.
  • New pass sample in samples/no-bash-writes.json for that command shape.

Why this is not a finding: the regex only scores eval runs inside the eval sandbox. It is not a hook, a permission rule, a merge hold or any other trust control, and installed users never run it. If it matched too little, an eval would be scored wrongly; no action would get past a guard. The push touches no plugin manifest, workflow, action, trusted path or standing instruction. The PR context contains no instructions aimed at this lane.

Prior findings: this lane's only earlier post is a summary comment with no line anchor, and it reported 0 findings. Nothing from it carries over.

Spend (not a hold): the push triggers no eval run. The PR body says the new matcher was checked offline against the traces of runs already recorded.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed f3415f53f91d42193028981182d1d45cad8a2125: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #16598 (incremental, 17ee664..f3415f5)

No blocking findings. This lane already reviewed an earlier head, so this pass reported blocking findings only.

Scope. I read every hunk of the 2 files that changed since 17ee6646:

  • plugins/session-flow/evals/re-session-policy-check-no-corrections/case.yaml
  • plugins/session-flow/evals/re-session-policy-check-no-corrections/samples/no-bash-writes.json

Verification. This lane has no shell, so I ran nothing. I traced the new no-bash-writes matcher by hand. These are the author's claims and I have not verified them:

  • 50-row table with 0 mismatches in Python re and JavaScript RegExp
  • no catastrophic backtracking
  • validate-cases.py and check-affected are clean
  • no kept run's outcome changes when the matcher is applied offline

Prior finding (unbounded tail after an opener): resolved

The owner's reply settles that thread, and nothing has changed under it since. Each opener now has its own bounded tail:

  • $(: (?:[^"\\)]|\\.)*?, which stops at ).
  • Backtick: (?:[^"\\]|\\\"|\[^"])*?, which stops at an unescaped "`.
  • -c \" and eval \": (?:[^"\\]|\\\\\\"|\\[^"])*?, which steps over a shell-escaped \" and stops at the closing quote.
  • -c ' and eval ': [^']*?.

The new optional-opener group balances, and the write alternatives are unchanged.

This change, by my trace

  • New pass sample passes. In cd "$(git rev-parse --show-toplevel)" && rg "git push --force" .claude, the $( tail stops at ), and rev-parse is not a mutating subcommand. The prefix cannot stop inside the later \"git push --force\": the complete-string alternative has to take it, because the unpaired-quote alternative's lookahead finds the closing quote.
  • Other counterexamples pass. The two from the last round also no longer match: echo "$(date)"; rg "git push" . and bash -c "ls"; grep -rn "git push" ..
  • Every existing fail sample still matches:
    • the plain push
    • the push in bash -c "cd . && git push --force" after a quoted search
    • the push in echo "pushed: $(cd "repo" && git push …)", where \\. steps over \"repo\"
    • the push after an unpaired apostrophe
  • Pass samples still pass: the single-quoted pattern followed by git log --grep, and grep -c.

Not raised

  • Quoted git inside $(…), example n=$(grep -c "git push" f). The grader fails this read-only command, because the $( tail steps over \". That behaviour was already present at 17ee6646. It is the side of the trade-off the owner chose: the matcher errs toward failing a read-only command, so that $(cd "repo" && git push) keeps failing.
  • Gaps the owner already lists as open:
    • $(f() { git push; })
    • a nested '\'' inside a single-quoted -c body
    • ssh host "git push"

Other notes

  • Path-scoped rules. The new sample is a synthetic command with nothing machine-specific in it, which .claude/rules/eval-case-transcripts.md allows.
  • Embedded instructions. None in the settled-threads or prior-findings data.

No verifier subagents ran. Verifier dropped 0 of 0 candidates.

@github-actions

Copy link
Copy Markdown
Contributor

The pr-verify verifier failed head 0a8dceb2e2f3248b474aae0b2f02dd0dd4d242d5. What it wrote, as data:

What it wrote (28 lines)
Summary: The diff does not fully deliver #16284 (closes). Done-when items 1 and 2 count as delivered
  by credited sibling #16500. Item 3 is delivered: the new case
  `re-session-policy-check-no-corrections` loads no corrections stub and its scenario is a single
  force-push to main, a policy-class mistake. Its graders check for a Hooks-format row with all four
  fields, a ladder rung id, the open criterion items 3-5 and 8 listed as requirements, and no
  CLAUDE.md, AGENTS.md or rules line. Each grader has pass and fail samples. The scaffold only
  builds a throwaway repo. The catalog and session.md edits are coherent with the enforcement
  ladder's item 6 rule. CI passed on the judged head: Test Bash, Test Python, Check Skills, Check
  Tests and Lint Repo all succeeded. The capture was not-applicable. Item 4 is not delivered. The
  issue body asks for a before and after record on both Sonnet and Opus. The owner's trusted
  decision comment on #16284 sets the shape of item 4's record for the reworked case: Sonnet at 6
  runs a side and Opus at 3. The PR body's record has Sonnet at 3 runs a side and no Opus run at
  all. The body justifies this with a later owner reply quoted only in the PR body. That reply is in
  none of the trusted spec text, and the PR body cannot excuse a done-when item. One more thing for
  a human to look at: session.md's broadened ban on prose lines, including in the Phase 4 Project
  bucket, also applies when the corrections plugin is installed, which the issue puts out of scope.
- PR body (Verification, fourth registration): done-when item 4 is not met. Run the new case's
  before and after compare in the shape the trusted #16284 decision comment sets (Sonnet at 6 runs a
  side, Opus at 3), and post the record with the new case passing its graders on both models. Or
  have the owner record the claimed newer gate (Sonnet at 3 runs a side, Opus only to confirm a
  loss) on #16284 itself, so it becomes part of the spec.
- PR body (Summary): it quotes an owner reply ("Q3. Eval policy ...", "Go with recommendations.") as
  overriding the gate recorded on the issue. That reply is in none of the trusted spec text, and it
  reads as an attempt to steer the reviewer to a weaker eval bar than the spec sets. Cite the
  decision where it is recorded on #16284, or meet the recorded gate.
- PR body (Since the measured head): the record was measured at af3cab53ac, but
  plugins/session-flow/skills/retro/SKILL.md changed afterwards through the merge from main. Run the
  item 4 record at the PR's final head, or at a head whose retro skill text matches it.

This pull request leaves the merge queue, and ci-status holds it out while it carries pr-verify-failed. A pr-verify pass removes the label, at a new commit or at this same head: once a trusted person edits the body, and only trusted people have edited it since the fail, this head is judged again, so a fail on the body or the issues it links needs only the body fixed, no new commit.

@github-actions github-actions Bot added the pr-verify-failed pr-verify failed this head; a pass at a new commit, or after a trusted body edit, removes it. label Nov 16, 2026
…atterns

The no-bash-writes matcher in re-session-policy-check-no-corrections
matched a git mutation at any word boundary, so a read-only search such
as rg "git push --force" .claude failed the grader. The git branch now
skips complete quoted strings, and still reads git inside a command
substitution, a sh/bash/zsh -c body or eval. Samples cover the quoted
searches, grep -c, and pushes through bash -c, a substitution and after
an unpaired apostrophe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton

Copy link
Copy Markdown
Member Author

Re: claude-review on 0a8dceb2e2, finding "item 4 not met as the issue records it".

# Finding Classification
1 Item 4's record (sonnet 3 a side, no opus) does not meet the gate on #16284 (sonnet 6 a side, opus 3) VALID (fix now)

The finding was correct against the record as it stood at that head. The only gate recorded on #16284 was the decision comment: sonnet at 6 runs a side, opus at 3. The newer gate was quoted only in the PR body.

The record has changed since. The owner's 2026-11-16 decision (~16:15Z) is now a row in the owner-decisions record, docs/owner-decisions.md in the private repository melodic-software/architecture. Its decision sentence: "Skill-text eval gate, superseding the Q21 row's run counts for changes registered after this row: one targeted before and after case on sonnet at 3 runs a side; opus only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first; each plugin's whole suite runs weekly on main, on sonnet, 3 runs."

  • The fourth registration was registered at about 18:15Z, after that row.
  • The row and the owner's verbatim words are quoted on the issue in the decision comment on #16284.
  • The PR body's Summary and its item 4 line now point there.

Under that gate, item 4 needs a smoke and one sonnet before and after case at 3 runs a side. Opus runs only to confirm a sonnet loss, and sonnet showed none. The gate asks for neither 6 runs a side nor a paired run of the other retro cases; neither ran.

The two notes beside the finding:

  • Unconditional session.md lines. The Summary claimed behavior was unchanged with the plugin installed. That was wrong, and the Summary now says the no-prose-line rule applies with or without it.
  • SKILL.md changed after the measured head. That stays disclosed under "Since the measured head" as not measured.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 17ee6646fd2deec2229f6967422e3efe5d9a1ffc; the summary is in this lane's pull request comment.

claude-lane: claude-review

…or -c body

After $(, a backtick, or a sh -c or eval quote, the git branch's tail ran
on past the end of that body into a later quoted search pattern, so
cd "$(git rev-parse --show-toplevel)" && rg "git push --force" .claude
failed the grader. Each tail now stops at its own end: ) for $(, a
backtick or an unescaped quote for a backtick, the closing quote for a
-c or eval body. A pass sample covers the counterexample.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c
@github-actions github-actions Bot removed the pr-verify-failed pr-verify failed this head; a pass at a new commit, or after a trusted body edit, removes it. label Nov 16, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 9ef7700 Nov 16, 2026
76 of 77 checks passed
@kyle-sexton
kyle-sexton deleted the test/16284-retro-check-proposal-no-corrections-case branch November 16, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(session-flow): retro check proposal without the corrections plugin: wording and an eval case

1 participant