You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(session-flow): no Sonnet re-session eval run reads the enforcement ladder, so a ladder change is not exercised #16325
In the recorded ladder 2.1.1 against 3.0.0 compare (#16078), the session-flow plugin's Sonnet result does not exercise the change: none of the 24 counted after-side runs read skills/retro/reference/enforcement-ladder.md (the registration's ladder-read count is 0 of 24). The scores read non-inferior (+0.01, interval -0.01 to +0.03), but a run that never loads the changed text cannot show whether the change matters. On Opus the same cases read the ladder in 5 after-side runs.
The cases are the three re-session cases the compare used: re-session-listed-failures-cover-unlisted, re-session-noop-steering-line and re-session-unwired-check-first (8 runs per side each).
Likely cause (UNVERIFIED)
Not established. Either the retro skill on Sonnet answers these prompts without opening its ladder reference, or the cases' prompts do not reach the step that reads it.
Done when
The numeric bars in this list (runs per side, pass counts, allowances) are the filing session's judgment. They are not measured and not owner-approved. Whoever registers the eval sets them from the case's own measured pass rate and says so in the registration.
The traces of at least four Sonnet after-side runs are read and the reason no run opens the ladder reference is stated: the skill text, the case prompt, or the model skipping the read.
Either a case is added or changed so that a passing Sonnet run must use text that differs between ladder 2.1.1 and 3.0.0, or the record states why session-flow's use of the ladder cannot be exercised by an eval.
With such a case, a before and after eval on Sonnet, 8 runs per side, both sides run together, is recorded with the ladder-read count per side and a non-inferiority reading at margin 0.05.
Out of scope
Changing the ladder text.
Related
Refs: #16078 (the record and the owner's decision Q171), #16246.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: in the #16078 ladder 2.1.1 vs 3.0.0 compare, none of the 24 Sonnet after-side runs on the three session-flow re-session cases read the enforcement-ladder reference. That means the non-inferior Sonnet score doesn't test the ladder change. The work is to read at least four Sonnet traces and state why the ladder isn't opened (skill text, case prompt, or the model skipping it). Then either add or change a case that needs text differing between 2.1.1 and 3.0.0, or record why an eval can't exercise it. With such a case, record an 8-runs-per-side Sonnet before/after with the ladder-read count and a margin-0.05 reading. Changing the ladder text is out of scope.
Nothing is missing from the issue. Related but not duplicates: #16284 (retro eval case in session-flow) and #16172 (running-retro eval cases INVALID on main). Check whether those change the same cases before starting. I found no merged pull request that already covers this.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "With such a case, a before and after eval on Sonnet, 8 runs per side, both si..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Record: why session-flow's use of the enforcement ladder cannot be exercised on main
Done-when item 2 offers two branches: add or change a case so a passing Sonnet run must use text that differs between ladder 2.1.1 and 3.0.0, or record why that cannot be done. This is the record.
Item 1, the traces. Read from the #16078 compare's Sonnet AFTER runs: all 8 re-session-listed-failures-cover-unlisted runs (tool calls), 8 re-session-unwired-check-first runs (tool calls and ladder-text hits), and 8 re-session-noop-steering-line runs (last-message lines). Ladder tool calls: 0 of 24 AFTER runs (BEFORE unwired: 1 of 8). Every run read skills/retro/context/session.md and reference/ecosystem-improvement-catalog.md, and most read reference/gather.md.
Why no run opens the ladder: each prompt routes to a row that does not need it.
unwired: every reply took the "Unwired or broken check" row (context/session.md:318), e.g. "The unwired check is reported first, as the catalog requires ... I'm not proposing a new one".
noop: every reply took the "No-op in a steering file" row (context/session.md:321).
listed: a pre-existing-failure finding, which takes no check row.
The ladder is named in the "No guardrail" row (context/session.md:319), whose route is a report naming the "Guardrail criterion" section, which needs no read; the other link (SKILL.md, the Phase 4 gotcha) is in codification, which these read-only cases never reach.
Why no case can make the version matter. Retro hands rung choice to codify: context/session.md:306-309 ("routed to /corrections:codify ..., which counts it and picks its rung") and the catalog's owner routing. Outside the ladder file itself, no retro file on main names any rung or tier id that 3.0.0 added or renamed (git grep over plugins/session-flow/skills/retro for syntax-pattern-rule, semgrep-rule, static-analysis tier, Verification tier: no hit). On retro's path the only line that differs is the Guardrail criterion's intro: 2.1.1 "A finding earns a deterministic rung (tier 1 or 2) only if all eight hold." against 3.0.0 "A finding earns a deterministic rung (the codebase or static-analysis tier) only if all eight hold." The eight conditions themselves are unchanged, so no passing verdict on main can depend on the version.
A case that would exercise it needs a retro path that names a rung from the ladder. That path exists only in open #16282's branch, and #16284 asks for a case on it ("names the ladder rung"); a rung-id grader belongs there once #16282 merges, not in this issue.
Item 3 applies only when a case is added under item 2; none is, so no eval is owed.
This record delivers the done-when on its second branch.
Activity
github-actions commentedon Nov 15, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: in the #16078 ladder 2.1.1 vs 3.0.0 compare, none of the 24 Sonnet after-side runs on the three session-flow re-session cases read the enforcement-ladder reference. That means the non-inferior Sonnet score doesn't test the ladder change. The work is to read at least four Sonnet traces and state why the ladder isn't opened (skill text, case prompt, or the model skipping it). Then either add or change a case that needs text differing between 2.1.1 and 3.0.0, or record why an eval can't exercise it. With such a case, record an 8-runs-per-side Sonnet before/after with the
ladder-readcount and a margin-0.05 reading. Changing the ladder text is out of scope.Nothing is missing from the issue. Related but not duplicates: #16284 (retro eval case in session-flow) and #16172 (running-retro eval cases INVALID on main). Check whether those change the same cases before starting. I found no merged pull request that already covers this.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "With such a case, a before and after eval on Sonnet, 8 runs per side, both si..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).kyle-sexton commentedon Nov 15, 2026
Record: why session-flow's use of the enforcement ladder cannot be exercised on main
Done-when item 2 offers two branches: add or change a case so a passing Sonnet run must use text that differs between ladder 2.1.1 and 3.0.0, or record why that cannot be done. This is the record.
Item 1, the traces. Read from the #16078 compare's Sonnet AFTER runs: all 8
re-session-listed-failures-cover-unlistedruns (tool calls), 8re-session-unwired-check-firstruns (tool calls and ladder-text hits), and 8re-session-noop-steering-lineruns (last-message lines). Ladder tool calls: 0 of 24 AFTER runs (BEFOREunwired: 1 of 8). Every run readskills/retro/context/session.mdandreference/ecosystem-improvement-catalog.md, and most readreference/gather.md.Why no run opens the ladder: each prompt routes to a row that does not need it.
unwired: every reply took the "Unwired or broken check" row (context/session.md:318), e.g. "The unwired check is reported first, as the catalog requires ... I'm not proposing a new one".noop: every reply took the "No-op in a steering file" row (context/session.md:321).listed: a pre-existing-failure finding, which takes no check row.context/session.md:319), whose route is a report naming the "Guardrail criterion" section, which needs no read; the other link (SKILL.md, the Phase 4 gotcha) is in codification, which these read-only cases never reach.Why no case can make the version matter. Retro hands rung choice to codify:
context/session.md:306-309("routed to/corrections:codify ..., which counts it and picks its rung") and the catalog's owner routing. Outside the ladder file itself, no retro file on main names any rung or tier id that 3.0.0 added or renamed (git grepoverplugins/session-flow/skills/retroforsyntax-pattern-rule,semgrep-rule,static-analysis tier,Verification tier: no hit). On retro's path the only line that differs is the Guardrail criterion's intro: 2.1.1 "A finding earns a deterministic rung (tier 1 or 2) only if all eight hold." against 3.0.0 "A finding earns a deterministic rung (the codebase or static-analysis tier) only if all eight hold." The eight conditions themselves are unchanged, so no passing verdict on main can depend on the version.A case that would exercise it needs a retro path that names a rung from the ladder. That path exists only in open #16282's branch, and #16284 asks for a case on it ("names the ladder rung"); a rung-id grader belongs there once #16282 merges, not in this issue.
Item 3 applies only when a case is added under item 2; none is, so no eval is owed.
This record delivers the done-when on its second branch.
session: GitHub Support