Skip to content

fix(session-flow): retro check proposal without the corrections plugin: wording and an eval case #16284

Description

@kyle-sexton

Problem

The retro's check-proposal change for #15935 routes a mechanical finding to /corrections:codify when the corrections plugin is installed, and to the Hooks format and the retro's own ladder copy when it is not. The review of its pull request left three findings on the branch taken without the plugin. None changes behavior with the plugin installed, which is the only branch the two-version eval compare measured.

  • The gotcha's install condition sits on the wrong clause. plugins/session-flow/skills/retro/SKILL.md reads "A mechanical finding goes to /corrections:codify, which proves its own check (when the corrections plugin is installed), run once the user picks it at Phase 4". The condition belongs on the routing, where it sat before the change. skills/retro/context/session.md and the catalog's "Check first" section still state the condition correctly, so routing does not depend on this sentence.
  • Item 2's pointer is narrower than the rule. plugins/session-flow/skills/retro/reference/ecosystem-improvement-catalog.md, "Check first", judges item 2 "by the rule 'When to propose a check' ... gives for when the plugin is missing". A literal reader can take that as only the "repeat with the same cause" sentence and drop the policy-class one-occurrence exception, which the same section and reference/enforcement-ladder.md item 2 both keep.
  • No eval case takes the branch without the plugin. Every session-flow case where a new check is proposed loads the corrections-stub plugin. re-session-unwired-check-first runs without it but covers "wire the existing check first", not the Hooks-format proposal.

Done when

  • The gotcha in skills/retro/SKILL.md puts "(when the corrections plugin is installed)" on the routing clause.
  • "Check first" item 2 in the catalog states the policy-class one-occurrence exception inline.
  • One new session-flow eval case with no corrections stub: a mechanical finding gets a Hooks-format check row (event, matcher, what it checks, block or warn) that names the ladder rung and lists the criterion items a retro cannot settle as requirements, and gets no CLAUDE.md or rules row. A single policy-class occurrence is part of the scenario.
  • A before and after eval on Sonnet and Opus is in the pull request body: the new case passes its graders on both models, and the retro cases that load the changed text run paired and show no loss. The pull request opens with the do-not-merge label and keeps it until that record is posted.

Out of scope

  • Any change to the branch taken with the plugin installed.

Related

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  3. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: two wording fixes in the session-flow retro text (the install condition on the routing clause, and the policy-class one-occurrence exception stated inline in "Check first" item 2), plus one eval case for the branch without the corrections plugin, with a before and after record.

    The newest recorded decision is:

    Decision recorded.

    The gate now in force for done-when item 4 is the owner's eval decision of 2026-11-16 (~16:15Z). It is recorded in the owner-decisions record, docs/owner-decisions.md in the private repository melodic-software/architecture (row "GitHub Work backlog drive, hop 7 Q3, Q9, Q11, Q12 and Q13", decisions D40 to D42, D44 and D50). That repository is private, so the row is quoted here.

    [quote truncated; the full decision is at the link]

    #16284 (comment)

    That decision settles the open questions about the eval gate for item 4. Nothing is left to ask.

    Status as the comments record it: items 1 and 2 are in #16500, and item 4's record for the reworked case is in #16598. I found no merged pull request that fixes this yet, so I'm naming none.

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 4 (eval: "A before and after eval on Sonnet and Opus is in the pull request body: the n..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Eval record for done-when item 4. Result: not met. The new case does not pass its graders on either model, and too many runs were excluded to decide the rest. #16462 stays a draft with do-not-merge until the owner decides.

    Registered before any run, in the shape the owner chose on 2026-11-15 (sonnet first at 6 runs a side on the paired cases; opus on the new case only); judge calibration passed first (8 graders, 100% agreement, 17.13 USD); launched by the owner; no usage-limit stop; no rerun; Claude Code 2.1.325; 11 of 12 batches ran, 17.93 USD list price. Before = the branch's merge base; after = #16462's head 172c5e7.

    The launcher stopped before batch 12 (the new case's before side on opus): the registered cost test adds the next batch's worst case to what is spent, and that passed the 20 USD stage cap and the 5 USD opus cap. That stop alone makes the opus rule not met.

    The new case, re-session-policy-check-no-corrections, after side, all runs launched:

    Grader sonnet (3 runs) opus (3 runs)
    names-pretooluse-event 3 3
    hooks-format-check-row 3 2
    names-ladder-rung 1 0
    lists-unsettled-requirements 2 0
    no-prose-row 1 2
    no-bash-writes 2 0
    no-write, no-edit 3, 3 3, 3

    The pass rule asks each grader to pass in at least 2 of 3 counted after runs. On top of that, only 1 of 3 sonnet runs and 0 of 3 opus runs were counted: the others were excluded under the registered rule for a permission denial in the trace (most often a blocked network call to the Claude Code docs site from the eval sandbox).

    The four paired retro cases, sonnet, 6 runs a side: every launched run on both sides scored 1.00, so no loss was seen. Three cases are decided with no loss; re-mechanical-finding-check-proposal is not decided, because 4 of its 6 before runs were excluded for a denied directory listing of the plugin folder, which leaves 2 counted against a minimum of 4. By the registered rule that makes the paired set not met.

    The opus paired set (stage 2) did not run: its gate opens only on a decided sonnet loss.

  7. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    Owner, 2026-11-16, asked what happens to #16462 with its eval not met (split it, with a wording-only pull request merging and the new case moving to a follow-up for rework; or hold it whole; or merge it as is): "Go with recommendations.", which is the split.

    • Done-when items 1 and 2 (the two wording fixes): fix(session-flow): place the retro's corrections condition and state the policy-class rule #16500.
    • Done-when items 3 and 4 (the eval case for the branch without the plugin, and its before and after record): still open. The case is on the branch test/16284-retro-check-proposal-no-corrections-case, unchanged from the one measured in the record above, where it failed its own graders on both models. It needs rework before it gets a pull request: on the after side names-ladder-rung passed in 1 of 3 sonnet and 0 of 3 opus runs, lists-unsettled-requirements in 2 and 0, and no-bash-writes in 2 and 0.
    • By a second decision of the same reply, a skill-text change is now gated on one targeted before and after case (sonnet at 6 runs a side, opus at 3), not a paired no-loss compare. Item 4's record for the reworked case follows that shape.
    • fix(session-flow): place the retro's corrections condition and add a no-plugin eval case #16462 is closed as superseded.
  8. updated Work status needs-triage and Priority medium on Nov 16, 2026
  9. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    The gate now in force for done-when item 4 is the owner's eval decision of 2026-11-16 (~16:15Z). It is recorded in the owner-decisions record, docs/owner-decisions.md in the private repository melodic-software/architecture (row "GitHub Work backlog drive, hop 7 Q3, Q9, Q11, Q12 and Q13", decisions D40 to D42, D44 and D50). That repository is private, so the row is quoted here.

    The question, as the row gives it: "adopt the eval policy the data supports? Options: (A, recommended) per change one targeted before and after case on sonnet at 3 runs a side, opus only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first; a weekly whole-suite run on main, sonnet, 3 runs; (B) as A at 6 runs a side (as the Q21 row decided); (C) as A without the weekly run."

    The owner, verbatim as the row gives it: "Go with recommendations. Can you run these for me because I don't want to have to type them out each time? These are things that you shouldn't have to ask me to run."

    The row's decision sentence: "Skill-text eval gate, superseding the Q21 row's run counts for changes registered after this row: one targeted before and after case on sonnet at 3 runs a side; opus only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first; each plugin's whole suite runs weekly on main, on sonnet, 3 runs."

    The Q21 row it supersedes on run counts is the gate in the decision comment above (sonnet at 6 runs a side, opus at 3).

    Item 4 under this gate. The record is eval-16284d, registered at about 18:15Z on 2026-11-16, after the row. It ran a 2-run sonnet smoke on the after side, then one targeted before and after case on sonnet at 3 runs a side, none excluded, measured at af3cab53ac. Verdict MET. Sonnet showed no loss, so the gate asks for no opus run, and none ran. The record is in #16598's body.

    What this gate does not ask for, and what did not run for item 4:

    • an opus run (opus only confirms a sonnet loss);
    • 6 runs a side;
    • the paired no-loss run of the other retro cases that load the changed text (the earlier gate had already dropped that paired compare, and this one keeps it dropped).

    The gate's weekly whole-suite run on main is separate from this item.

  10. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions