You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(harness-config): ag-formatter-already-enforced reads a loss on Sonnet under ladder 3.0.0 on an uncalibrated judge grader #16324
In the recorded ladder 2.1.1 against 3.0.0 compare (#16078), the harness-config plugin reads a loss on Sonnet: the paired change is -0.06 with a 95% interval of -0.13 to +0.02, against a margin of -0.05. One judge grader decides it: posttooluse-not-a-gate in the case ag-formatter-already-enforced passed 6 of 6 runs before and 2 of 6 after.
The record's hand read of the four failing after-side runs found:
The judge passed and failed replies of the same shape ("unverified, leans REJECT", citing pre-commit and CI) on both sides. Its explanation field holds only the vote tally, so no reason is on record. One failing reply has a defensible fail: it offers a format-only PostToolUse hook.
Two of the four failing runs never read the after-side skills/audit-automation-gaps/context/enforcement-ladder.md; every before-side run read its ladder.
The grader is not calibrated.
So the loss stands as registered, and whether the 3.0.0 ladder text causes it is not shown.
Done when
The numeric bars in this list (runs per side, pass counts, allowances) are the filing session's judgment. They are not measured and not owner-approved. Whoever registers the eval sets them from the case's own measured pass rate and says so in the registration.
posttooluse-not-a-gate is calibrated against labelled pass and fail samples (the repository's grader-calibration step), and its criteria say how a reply that reports every candidate as unverified under light verification is judged.
The judge's reasons are kept for this grader, or the record states why they cannot be.
A before and after eval of ag-formatter-already-enforced on Sonnet, 12 runs per side, ladder 2.1.1 against 3.0.0, both sides run together, is recorded: the after-side pass rate of posttooluse-not-a-gate is not below the before-side rate by more than 2 of 12, and the count of runs that read the ladder file is given per side.
If the calibrated grader still shows a loss, the ladder text or the skill's read of it is changed in its own pull request and that eval is repeated.
Out of scope
The other harness-config cases, which read non-inferior.
Related
Refs: #16078 (the record and the owner's decision Q171), #16246.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: in the #16078 ladder 2.1.1 vs 3.0.0 compare, the posttooluse-not-a-gate judge grader in ag-formatter-already-enforced drives the Sonnet loss (6/6 before, 2/6 after). The grader is uncalibrated, it keeps no reasons, and it is not shown that the 3.0.0 ladder text causes the loss. The work is to calibrate the grader, keep its reasons, re-run a 12-per-side eval, and change the ladder only if a loss remains.
Nothing is missing that blocks triage. The 12-run Sonnet eval may need an attended or host run, so it may be routed separately from the calibration and criteria edits.
Related, not duplicates: #16325 (Sonnet runs not reading the enforcement ladder) and #16078. No merged pull request already fixes this.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of ag-formatter-already-enforced on Sonnet, 12 runs p..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); its body was edited after the triage verdict, which judged the body as it stood before. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Activity
github-actions commentedon Nov 15, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: in the #16078 ladder 2.1.1 vs 3.0.0 compare, the
posttooluse-not-a-gatejudge grader inag-formatter-already-enforceddrives the Sonnet loss (6/6 before, 2/6 after). The grader is uncalibrated, it keeps no reasons, and it is not shown that the 3.0.0 ladder text causes the loss. The work is to calibrate the grader, keep its reasons, re-run a 12-per-side eval, and change the ladder only if a loss remains.Nothing is missing that blocks triage. The 12-run Sonnet eval may need an attended or host run, so it may be routed separately from the calibration and criteria edits.
Related, not duplicates: #16325 (Sonnet runs not reading the enforcement ladder) and #16078. No merged pull request already fixes this.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of ag-formatter-already-enforced on Sonnet, 12 runs p..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); its body was edited after the triage verdict, which judged the body as it stood before. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).