Repository navigation
Description
Problem
The retro's check-proposal change for #15935 routes a mechanical finding to /corrections:codify when the corrections plugin is installed, and to the Hooks format and the retro's own ladder copy when it is not. The review of its pull request left three findings on the branch taken without the plugin. None changes behavior with the plugin installed, which is the only branch the two-version eval compare measured.
- The gotcha's install condition sits on the wrong clause.
plugins/session-flow/skills/retro/SKILL.mdreads "A mechanical finding goes to/corrections:codify, which proves its own check (when thecorrectionsplugin is installed), run once the user picks it at Phase 4". The condition belongs on the routing, where it sat before the change.skills/retro/context/session.mdand the catalog's "Check first" section still state the condition correctly, so routing does not depend on this sentence. - Item 2's pointer is narrower than the rule.
plugins/session-flow/skills/retro/reference/ecosystem-improvement-catalog.md, "Check first", judges item 2 "by the rule 'When to propose a check' ... gives for when the plugin is missing". A literal reader can take that as only the "repeat with the same cause" sentence and drop the policy-class one-occurrence exception, which the same section andreference/enforcement-ladder.mditem 2 both keep. - No eval case takes the branch without the plugin. Every session-flow case where a new check is proposed loads the
corrections-stubplugin.re-session-unwired-check-firstruns without it but covers "wire the existing check first", not the Hooks-format proposal.
Done when
- The gotcha in
skills/retro/SKILL.mdputs "(when thecorrectionsplugin is installed)" on the routing clause. - "Check first" item 2 in the catalog states the policy-class one-occurrence exception inline.
- One new session-flow eval case with no corrections stub: a mechanical finding gets a Hooks-format check row (event, matcher, what it checks, block or warn) that names the ladder rung and lists the criterion items a retro cannot settle as requirements, and gets no CLAUDE.md or rules row. A single policy-class occurrence is part of the scenario.
- A before and after eval on Sonnet and Opus is in the pull request body: the new case passes its graders on both models, and the retro cases that load the changed text run paired and show no loss. The pull request opens with the do-not-merge label and keeps it until that record is posted.
Out of scope
- Any change to the branch taken with the plugin installed.
Related
- Ask whether a check could decide a retro finding before proposing a prose rule #15935 and its pull request feat(session-flow): route a retro's mechanical finding to a check proposal #16282 (the three review findings are on that pull request's threads).
Metadata
Metadata
Assignees
Labels
No labels
Type
Fields
Work status
confirmed
Priority
medium
Activity
github-actions commentedon Nov 15, 2026
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: two wording fixes in the session-flow retro text (the install condition on the routing clause, and the policy-class one-occurrence exception stated inline in "Check first" item 2), plus one eval case for the branch without the
correctionsplugin, with a before and after record.The newest recorded decision is:
#16284 (comment)
That decision settles the open questions about the eval gate for item 4. Nothing is left to ask.
Status as the comments record it: items 1 and 2 are in #16500, and item 4's record for the reworked case is in #16598. I found no merged pull request that fixes this yet, so I'm naming none.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 4 (eval: "A before and after eval on Sonnet and Opus is in the pull request body: the n..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).kyle-sexton commentedon Nov 16, 2026
Eval record for done-when item 4. Result: not met. The new case does not pass its graders on either model, and too many runs were excluded to decide the rest. #16462 stays a draft with
do-not-mergeuntil the owner decides.Registered before any run, in the shape the owner chose on 2026-11-15 (sonnet first at 6 runs a side on the paired cases; opus on the new case only); judge calibration passed first (8 graders, 100% agreement, 17.13 USD); launched by the owner; no usage-limit stop; no rerun; Claude Code 2.1.325; 11 of 12 batches ran, 17.93 USD list price. Before = the branch's merge base; after = #16462's head 172c5e7.
The launcher stopped before batch 12 (the new case's before side on opus): the registered cost test adds the next batch's worst case to what is spent, and that passed the 20 USD stage cap and the 5 USD opus cap. That stop alone makes the opus rule not met.
The new case,
re-session-policy-check-no-corrections, after side, all runs launched:names-pretooluse-eventhooks-format-check-rownames-ladder-runglists-unsettled-requirementsno-prose-rowno-bash-writesno-write,no-editThe pass rule asks each grader to pass in at least 2 of 3 counted after runs. On top of that, only 1 of 3 sonnet runs and 0 of 3 opus runs were counted: the others were excluded under the registered rule for a permission denial in the trace (most often a blocked network call to the Claude Code docs site from the eval sandbox).
The four paired retro cases, sonnet, 6 runs a side: every launched run on both sides scored 1.00, so no loss was seen. Three cases are decided with no loss;
re-mechanical-finding-check-proposalis not decided, because 4 of its 6 before runs were excluded for a denied directory listing of the plugin folder, which leaves 2 counted against a minimum of 4. By the registered rule that makes the paired set not met.The opus paired set (stage 2) did not run: its gate opens only on a decided sonnet loss.
kyle-sexton commentedon Nov 16, 2026
Decision recorded.
Owner, 2026-11-16, asked what happens to #16462 with its eval not met (split it, with a wording-only pull request merging and the new case moving to a follow-up for rework; or hold it whole; or merge it as is): "Go with recommendations.", which is the split.
test/16284-retro-check-proposal-no-corrections-case, unchanged from the one measured in the record above, where it failed its own graders on both models. It needs rework before it gets a pull request: on the after sidenames-ladder-rungpassed in 1 of 3 sonnet and 0 of 3 opus runs,lists-unsettled-requirementsin 2 and 0, andno-bash-writesin 2 and 0.test(session-flow): make the no-corrections retro case scoreable offline
fix(session-flow): put the rung and open criterion items in the retro…
fix(session-flow): rule out a fallback prose line beside the retro's …
fix(session-flow): keep the retro's fallback prose line out of Phase …
fix(session-flow): keep the hook fields in the retro's no-corrections…
kyle-sexton commentedon Nov 16, 2026
Decision recorded.
The gate now in force for done-when item 4 is the owner's eval decision of 2026-11-16 (~16:15Z). It is recorded in the owner-decisions record,
docs/owner-decisions.mdin the private repositorymelodic-software/architecture(row "GitHub Work backlog drive, hop 7 Q3, Q9, Q11, Q12 and Q13", decisions D40 to D42, D44 and D50). That repository is private, so the row is quoted here.The question, as the row gives it: "adopt the eval policy the data supports? Options: (A, recommended) per change one targeted before and after case on sonnet at 3 runs a side, opus only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first; a weekly whole-suite run on main, sonnet, 3 runs; (B) as A at 6 runs a side (as the Q21 row decided); (C) as A without the weekly run."
The owner, verbatim as the row gives it: "Go with recommendations. Can you run these for me because I don't want to have to type them out each time? These are things that you shouldn't have to ask me to run."
The row's decision sentence: "Skill-text eval gate, superseding the Q21 row's run counts for changes registered after this row: one targeted before and after case on sonnet at 3 runs a side; opus only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first; each plugin's whole suite runs weekly on main, on sonnet, 3 runs."
The Q21 row it supersedes on run counts is the gate in the decision comment above (sonnet at 6 runs a side, opus at 3).
Item 4 under this gate. The record is eval-16284d, registered at about 18:15Z on 2026-11-16, after the row. It ran a 2-run sonnet smoke on the after side, then one targeted before and after case on sonnet at 3 runs a side, none excluded, measured at
af3cab53ac. Verdict MET. Sonnet showed no loss, so the gate asks for no opus run, and none ran. The record is in #16598's body.What this gate does not ask for, and what did not run for item 4:
The gate's weekly whole-suite run on main is separate from this item.