Skip to content

test(session-flow): no Sonnet re-session eval run reads the enforcement ladder, so a ladder change is not exercised #16325

Closed

Description

@kyle-sexton

Problem

In the recorded ladder 2.1.1 against 3.0.0 compare (#16078), the session-flow plugin's Sonnet result does not exercise the change: none of the 24 counted after-side runs read skills/retro/reference/enforcement-ladder.md (the registration's ladder-read count is 0 of 24). The scores read non-inferior (+0.01, interval -0.01 to +0.03), but a run that never loads the changed text cannot show whether the change matters. On Opus the same cases read the ladder in 5 after-side runs.

The cases are the three re-session cases the compare used: re-session-listed-failures-cover-unlisted, re-session-noop-steering-line and re-session-unwired-check-first (8 runs per side each).

Likely cause (UNVERIFIED)

Not established. Either the retro skill on Sonnet answers these prompts without opening its ladder reference, or the cases' prompts do not reach the step that reads it.

Done when

The numeric bars in this list (runs per side, pass counts, allowances) are the filing session's judgment. They are not measured and not owner-approved. Whoever registers the eval sets them from the case's own measured pass rate and says so in the registration.

  • The traces of at least four Sonnet after-side runs are read and the reason no run opens the ladder reference is stated: the skill text, the case prompt, or the model skipping the read.
  • Either a case is added or changed so that a passing Sonnet run must use text that differs between ladder 2.1.1 and 3.0.0, or the record states why session-flow's use of the ladder cannot be exercised by an eval.
  • With such a case, a before and after eval on Sonnet, 8 runs per side, both sides run together, is recorded with the ladder-read count per side and a non-inferiority reading at margin 0.05.

Out of scope

Changing the ladder text.

Related

Refs: #16078 (the record and the owner's decision Q171), #16246.

session: GitHub Support

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  2. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: in the #16078 ladder 2.1.1 vs 3.0.0 compare, none of the 24 Sonnet after-side runs on the three session-flow re-session cases read the enforcement-ladder reference. That means the non-inferior Sonnet score doesn't test the ladder change. The work is to read at least four Sonnet traces and state why the ladder isn't opened (skill text, case prompt, or the model skipping it). Then either add or change a case that needs text differing between 2.1.1 and 3.0.0, or record why an eval can't exercise it. With such a case, record an 8-runs-per-side Sonnet before/after with the ladder-read count and a margin-0.05 reading. Changing the ladder text is out of scope.

    Nothing is missing from the issue. Related but not duplicates: #16284 (retro eval case in session-flow) and #16172 (running-retro eval cases INVALID on main). Check whether those change the same cases before starting. I found no merged pull request that already covers this.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  4. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "With such a case, a before and after eval on Sonnet, 8 runs per side, both si..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  5. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  6. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    MemberAuthor

    Record: why session-flow's use of the enforcement ladder cannot be exercised on main

    Done-when item 2 offers two branches: add or change a case so a passing Sonnet run must use text that differs between ladder 2.1.1 and 3.0.0, or record why that cannot be done. This is the record.

    Item 1, the traces. Read from the #16078 compare's Sonnet AFTER runs: all 8 re-session-listed-failures-cover-unlisted runs (tool calls), 8 re-session-unwired-check-first runs (tool calls and ladder-text hits), and 8 re-session-noop-steering-line runs (last-message lines). Ladder tool calls: 0 of 24 AFTER runs (BEFORE unwired: 1 of 8). Every run read skills/retro/context/session.md and reference/ecosystem-improvement-catalog.md, and most read reference/gather.md.

    Why no run opens the ladder: each prompt routes to a row that does not need it.

    • unwired: every reply took the "Unwired or broken check" row (context/session.md:318), e.g. "The unwired check is reported first, as the catalog requires ... I'm not proposing a new one".
    • noop: every reply took the "No-op in a steering file" row (context/session.md:321).
    • listed: a pre-existing-failure finding, which takes no check row.
    • The ladder is named in the "No guardrail" row (context/session.md:319), whose route is a report naming the "Guardrail criterion" section, which needs no read; the other link (SKILL.md, the Phase 4 gotcha) is in codification, which these read-only cases never reach.

    Why no case can make the version matter. Retro hands rung choice to codify: context/session.md:306-309 ("routed to /corrections:codify ..., which counts it and picks its rung") and the catalog's owner routing. Outside the ladder file itself, no retro file on main names any rung or tier id that 3.0.0 added or renamed (git grep over plugins/session-flow/skills/retro for syntax-pattern-rule, semgrep-rule, static-analysis tier, Verification tier: no hit). On retro's path the only line that differs is the Guardrail criterion's intro: 2.1.1 "A finding earns a deterministic rung (tier 1 or 2) only if all eight hold." against 3.0.0 "A finding earns a deterministic rung (the codebase or static-analysis tier) only if all eight hold." The eight conditions themselves are unchanged, so no passing verdict on main can depend on the version.

    A case that would exercise it needs a retro path that names a rung from the ladder. That path exists only in open #16282's branch, and #16284 asks for a case on it ("names the ladder rung"); a rung-id grader belongs there once #16282 merges, not in this issue.

    Item 3 applies only when a case is added under item 2; none is, so no eval is owed.

    This record delivers the done-when on its second branch.

    session: GitHub Support

  7. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions