You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(toolchain): #16228's text reads a Sonnet loss in the #16063 compare (not shown non-inferior) #16355
The #16063 two-version compare of #16228 reads NOT MET for toolchain on Sonnet: the paired with-arm change over 19 cases is -0.01 with a 95% interval of -0.06 to +0.04, so the lower end is below the -0.05 margin (noise-report.py, not shown non-inferior). Opus is non-inferior on the same cases. Per-case reads in the record on #16063: the after side lost single grader runs in ck-malformed-config-degrades (1.00 to .78), ck-missing-tool-incomplete (1.00 to .83), ck-skips-lint-only-surfaces (1.00 to .83) and ck-detects-changed-ecosystems (1.00 to .92), and gained in ln-missing-tool-incomplete (.67 to 1.00).
The numeric bars below (runs per side, margins) are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.
A before and after eval of toolchain on Sonnet over the same 19 cases, 6 runs per side, both sides run together, is recorded in that pull request's body or here, and reads non-inferior at margin 0.05; or the record states that the loss is noise, with the per-case evidence.
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: in the #16063 compare of #16228, toolchain on Sonnet is not shown non-inferior (paired change -0.01, 95% interval -0.06 to +0.04, below the -0.05 margin over 19 cases). Four cases lost single grader runs. The work is to read those traces and state the cause of each loss. If the #16228 text is the cause, fix the toolchain skill text in its own PR. Then record a before/after Sonnet eval, or show per-case evidence that the loss is noise.
Nothing is missing for triage. The cause is not yet known, so the approach depends on what the traces show. The 6-runs-per-side eval is host or paid-run work, and the issue already carries host-run for that. The numeric bars are noted in the issue as unapproved, so whoever registers the eval will set them.
Related, not duplicates: #16229 and the verification issue about #16228 (a Sonnet case drop in the same #16063 compare).
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of toolchain on Sonnet over the same 19 cases, 6 runs..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Activity
github-actions commentedon Nov 15, 2026
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: in the #16063 compare of #16228, toolchain on Sonnet is not shown non-inferior (paired change -0.01, 95% interval -0.06 to +0.04, below the -0.05 margin over 19 cases). Four cases lost single grader runs. The work is to read those traces and state the cause of each loss. If the #16228 text is the cause, fix the toolchain skill text in its own PR. Then record a before/after Sonnet eval, or show per-case evidence that the loss is noise.
Nothing is missing for triage. The cause is not yet known, so the approach depends on what the traces show. The 6-runs-per-side eval is host or paid-run work, and the issue already carries
host-runfor that. The numeric bars are noted in the issue as unapproved, so whoever registers the eval will set them.Related, not duplicates: #16229 and the verification issue about #16228 (a Sonnet case drop in the same #16063 compare).
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of toolchain on Sonnet over the same 19 cases, 6 runs..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).