Skip to content

fix(harness-config): ag-formatter-already-enforced reads a loss on Sonnet under ladder 3.0.0 on an uncalibrated judge grader #16324

Closed

Description

@kyle-sexton

Problem

In the recorded ladder 2.1.1 against 3.0.0 compare (#16078), the harness-config plugin reads a loss on Sonnet: the paired change is -0.06 with a 95% interval of -0.13 to +0.02, against a margin of -0.05. One judge grader decides it: posttooluse-not-a-gate in the case ag-formatter-already-enforced passed 6 of 6 runs before and 2 of 6 after.

The record's hand read of the four failing after-side runs found:

  • The judge passed and failed replies of the same shape ("unverified, leans REJECT", citing pre-commit and CI) on both sides. Its explanation field holds only the vote tally, so no reason is on record. One failing reply has a defensible fail: it offers a format-only PostToolUse hook.
  • Two of the four failing runs never read the after-side skills/audit-automation-gaps/context/enforcement-ladder.md; every before-side run read its ladder.
  • The grader is not calibrated.

So the loss stands as registered, and whether the 3.0.0 ladder text causes it is not shown.

Done when

The numeric bars in this list (runs per side, pass counts, allowances) are the filing session's judgment. They are not measured and not owner-approved. Whoever registers the eval sets them from the case's own measured pass rate and says so in the registration.

  • posttooluse-not-a-gate is calibrated against labelled pass and fail samples (the repository's grader-calibration step), and its criteria say how a reply that reports every candidate as unverified under light verification is judged.
  • The judge's reasons are kept for this grader, or the record states why they cannot be.
  • A before and after eval of ag-formatter-already-enforced on Sonnet, 12 runs per side, ladder 2.1.1 against 3.0.0, both sides run together, is recorded: the after-side pass rate of posttooluse-not-a-gate is not below the before-side rate by more than 2 of 12, and the count of runs that read the ladder file is given per side.
  • If the calibrated grader still shows a loss, the ladder text or the skill's read of it is changed in its own pull request and that eval is repeated.

Out of scope

The other harness-config cases, which read non-inferior.

Related

Refs: #16078 (the record and the owner's decision Q171), #16246.

session: GitHub Support

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  2. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: in the #16078 ladder 2.1.1 vs 3.0.0 compare, the posttooluse-not-a-gate judge grader in ag-formatter-already-enforced drives the Sonnet loss (6/6 before, 2/6 after). The grader is uncalibrated, it keeps no reasons, and it is not shown that the 3.0.0 ladder text causes the loss. The work is to calibrate the grader, keep its reasons, re-run a 12-per-side eval, and change the ladder only if a loss remains.

    Nothing is missing that blocks triage. The 12-run Sonnet eval may need an attended or host run, so it may be routed separately from the calibration and criteria edits.

    Related, not duplicates: #16325 (Sonnet runs not reading the enforcement ladder) and #16078. No merged pull request already fixes this.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  4. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of ag-formatter-already-enforced on Sonnet, 12 runs p..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); its body was edited after the triage verdict, which judged the body as it stood before. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  5. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  6. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions