Repository navigation
fix(session-flow): add the retro's no-corrections check-proposal eval case and fix its check row - #16598
Conversation
…t corrections A session-mode retro case with no corrections stub: the session force-pushed to main once, a policy-class mistake no hook, deny rule or check covers. The retro should propose its own Hooks-format check row (event, matcher, what it checks, block or warn), name the ladder rung, list the guardrail criterion items it cannot settle as requirements, and propose no CLAUDE.md or rules line. Deterministic graders for the event and the rung id, llm graders for the row, the requirements and the prose ban, and the neighbour case's must-not graders, each with samples. Refs: #16284 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o eval case Refs: #16284 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k-proposal-no-corrections-case
The case's runs were excluded for sandbox network denials on the docs
lookup, and no-bash-writes failed read-only runs. The prompt now says the
docs site is unreachable and the case is tagged no-network.
no-bash-writes reads only the command value, so a description such as
"Check git push config" no longer counts, and it no longer counts the
docs lookup removing its own mktemp directory by literal path, which
reference/docs-lookup-procedure.md tells it to do. names-ladder-rung
also accepts the rung named in prose ("Action guard"). Samples rewritten
from kept run replies cover each change.
Refs: #16284
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…'s check-row format Without the corrections plugin, "Check first" told the retro to name the ladder rung and list the criterion items it cannot settle, but the Hooks "Recommendation format" a check row follows listed only the event, matcher, check and block or warn. The format now asks for the rung id and those items too. "Check first" listed item 6 (the same command at edit time and in CI) as open in every case. The enforcement ladder says a rung whose "runs at" column names one place meets item 6 by running there, so the catalog now lists it only for other rungs. The no-corrections case follows: lists-unsettled-requirements requires items 3, 4, 5 and 8 and no longer judges item 6 for a hook, with a pass sample that calls item 6 met and a fail sample with no kill switch. Refs: #16284 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…check row Without the corrections plugin, a retro that proposed a check sometimes also offered a CLAUDE.md line to use if the check were declined. "Check first" now says a finding that gets a check row gets no such line as a fallback either. Refs: #16284 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…3 and Phase 4 Both eval-16284c smoke runs read the catalog's no-fallback clause and still offered a CLAUDE.md line "if the check is declined", placed in Phase 4's Project bucket. session.md now rules that line out where the model writes it: the "Load the catalog" step bans it in any form (a row, a sentence, a Phase 4 option) and says where a withheld or declined check goes instead, and the Project bucket offers no line for a finding with a check row. The catalog's Hooks format names the open criterion items one by one, since one run listed three of the four from the pointer alone. Refs: #16284 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… check row One eval-16284c smoke run picked a permission deny rule as the stronger rung and offered the hook only as an alternative, so its row named no hook event or matcher. "Check first" routes the row to the Hooks format while that format asks for the strongest rung, which the ladder ranks a deny rule above. The Hooks format now says the row always states the hook's four fields and a deny rule may only sit beside the hook, as #16284's done-when item 3 asks. Refs: #16284 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se branch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a8dceb2e2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
claude-security-review reviewed Security review (incremental, blocking only): no security issues foundThis review covers only the push since What changed
Why this is not a finding: the regex only scores eval runs inside the eval sandbox. It is not a hook, a permission rule, a merge hold or any other trust control, and installed users never run it. If it matched too little, an eval would be scored wrongly; no action would get past a guard. The push touches no plugin manifest, workflow, action, trusted path or standing instruction. The PR context contains no instructions aimed at this lane. Prior findings: this lane's only earlier post is a summary comment with no line anchor, and it reported 0 findings. Nothing from it carries over. Spend (not a hold): the push triggers no eval run. The PR body says the new matcher was checked offline against the traces of runs already recorded. critical-findings: 0 |
|
claude-review reviewed Code review: PR #16598 (incremental, 17ee664..f3415f5)No blocking findings. This lane already reviewed an earlier head, so this pass reported blocking findings only. Scope. I read every hunk of the 2 files that changed since
Verification. This lane has no shell, so I ran nothing. I traced the new
Prior finding (unbounded tail after an opener): resolvedThe owner's reply settles that thread, and nothing has changed under it since. Each opener now has its own bounded tail:
The new optional-opener group balances, and the write alternatives are unchanged. This change, by my trace
Not raised
Other notes
No verifier subagents ran. Verifier dropped 0 of 0 candidates. |
|
The pr-verify verifier failed head What it wrote (28 lines)This pull request leaves the merge queue, and ci-status holds it out while it carries |
…atterns The no-bash-writes matcher in re-session-policy-check-no-corrections matched a git mutation at any word boundary, so a read-only search such as rg "git push --force" .claude failed the grader. The git branch now skips complete quoted strings, and still reads git inside a command substitution, a sh/bash/zsh -c body or eval. Samples cover the quoted searches, grep -c, and pushes through bash -c, a substitution and after an unpaired apostrophe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re: claude-review on
The finding was correct against the record as it stood at that head. The only gate recorded on #16284 was the decision comment: sonnet at 6 runs a side, opus at 3. The newer gate was quoted only in the PR body. The record has changed since. The owner's 2026-11-16 decision (~16:15Z) is now a row in the owner-decisions record,
Under that gate, item 4 needs a smoke and one sonnet before and after case at 3 runs a side. Opus runs only to confirm a sonnet loss, and sonnet showed none. The gate asks for neither 6 runs a side nor a paired run of the other retro cases; neither ran. The two notes beside the finding:
|
…or -c body After $(, a backtick, or a sh -c or eval quote, the git branch's tail ran on past the end of that body into a later quoted search pattern, so cd "$(git rev-parse --show-toplevel)" && rg "git push --force" .claude failed the grader. Each tail now stops at its own end: ) for $(, a backtick or an unescaped quote for a backtick, the closing quote for a -c or eval body. A pass sample covers the counterexample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c
Closes #16284
Siblings: #16500
Summary
Adds the session-flow eval case that #16284 asks for, covering the retro's check proposal when the
correctionsplugin is not installed. Also changes the retro's skill text, which the case's first runs showed was needed: a check row in the Hooks format names its ladder rung, lists the open criterion items one by one, and always states the hook's four fields, and a finding that gets a check row gets no prose line in any form. The session.md lines that rule out a prose line apply with or without the plugin: with it installed, a new check's action is still/corrections:codify, and the retro now also offers noCLAUDE.md,AGENTS.mdor rules line beside that check, at Phase 3 or in Phase 4's Project bucket. The two wording items from the issue merged in #16500. #16462 was split into #16500 and this branch, and is superseded.The before and after record below follows the skill-text eval gate the owner set on 2026-11-16 (~16:15Z): one targeted before and after case on sonnet at 3 runs a side, opus only to confirm a sonnet loss, a 1 to 3 run smoke first for a new case. It supersedes the earlier gate's run counts (sonnet at 6 runs a side, opus at 3) for changes registered after it. The owner-decisions record holds it, and the decision comment on #16284 quotes the row and the owner's words. Commits
7dce676aanddbb21ce4stay.Fix
plugins/session-flow/evals/re-session-policy-check-no-corrections, with no corrections stub. In the scenario, the agent force-pushed to main once (one policy-class occurrence). There is no network, so the case is taggedno-network. Eight graders:names-pretooluse-eventandnames-ladder-rung(regex);hooks-format-check-row,lists-unsettled-requirementsandno-prose-row(LLM judge);no-write,no-editandno-bash-writes.no-bash-writesreads git mutations only where they run: a git command inside a complete quoted string (a search pattern such asrg "git push --force" .claude) does not fail it, while one in a command substitution, ash/bash/zsh -cbody orevalstill does. Each of those bodies is read only up to its own end, so a read-onlycd "$(git rev-parse --show-toplevel)"before a quoted search does not fail it either. Its samples cover both.skills/retro/reference/ecosystem-improvement-catalog.md:skills/retro/context/session.md:CLAUDE.md,AGENTS.mdor rules line in any form, not even as a fallback. A withheld check goes under "Queue for follow-up".Verification
Each record was registered before any run. The owner launched each one, on Claude Code 2.1.325, with no reruns. Spend is at list price.
First registration, on fix(session-flow): place the retro's corrections condition and add a no-plugin eval case #16462's head. Judge calibration passed: 8 graders, 100% agreement, 17.13 USD. The compare stopped one batch short (11 of 12) on its cost cap, at 17.93 USD. The new case was not met on either model: with too few counted runs it was undecided on sonnet, and on opus every AFTER run was excluded and BEFORE never ran. The full record is the eval record comment on #16284.
Second registration, for the case reworked to score offline. Prepared, but no run launched. 0 USD.
Third registration: sonnet AFTER smoke, 2 runs, at
dbb21ce4. 0.9016 USD.no-prose-rowpassed 0 of 2;names-pretooluse-event,hooks-format-check-rowandlists-unsettled-requirementspassed 1 of 2 each. The launcher refused the compare because a grader was at zero. Every failing grader was read against its reply and found correct, and the fault was in the skill text:Commits
10bcf1e626andaf3cab53acfix these.Fourth registration, measured at
af3cab53ac. Sonnet, 3 runs a side, none excluded. BEFORE uses main's text for the catalog and session.md; both sides carry the same case and graders.lists-unsettled-requirementsin run 2. No grader was at zero.hooks-format-check-rownames-pretooluse-eventnames-ladder-rungno-prose-rowlists-unsettled-requirementsno-write,no-edit,no-bash-writeslists-unsettled-requirementsdid not improve. It meets only the registered bar of 2 of 3 counted AFTER runs: the reply in AFTER run 1 dropped one of the four items. BEFORE cost 0.9589 USD and AFTER 1.3291 USD. This registration totals 3.1313 USD.Total across the four registrations: about 39.09 USD. That is 17.13 for calibration, 17.93, 0, 0.90 and 3.13. The calibration file also records 5.61 USD of judge cost; whether the 17.13 includes it is UNVERIFIED.
Since the measured head:
skills/retro/SKILL.mdchanged: fix(session-flow): point the retro skill at its context files so they resolve from the skill directory #16565 turned itscontext/session.mdandcontext/quick.mdmentions into Markdown links. That file tells the model to readcontext/session.md, and the change was not measured. The catalog and session.md are byte-identical toaf3cab53ac.17ee6646fdandf3415f53f9changed the case'sno-bash-writesmatcher and samples after the measurement (Fix). Offline application, not a rescoring: both matchers were applied to every Bash call in the traces of the 8 kept runs of the fourth registration (2 smoke, 3 BEFORE, 3 AFTER). The old matcher reproduces the recorded pass in all 8, the new one also passes all 8, and no run'sno-bash-writesoutcome differs. The registered counts above stand as recorded.Related
hooks-format-check-rowcovers event, matcher, what it checks, and block or warn), the ladder rung (names-ladder-rung), the unsettled criterion items as requirements (lists-unsettled-requirements), and noCLAUDE.mdor rules row (no-prose-row).Follow-ups
no-bash-writesgit alternative that fails a quoted search pattern as a mutation (ra-fetches-base-reports-drift,or-routes-recovery-writes-nothing,re-session-unwired-check-first,re-session-noop-steering-line,re-mechanical-finding-check-proposal,re-session-listed-failures-cover-unlisted); this PR fixes it only in its own case.🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c