You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(mutation-testing): #16229's Sonnet compare cannot decide non-inferiority on 3 cases in #16063 #16357
The #16063 two-version compare of #16229 reads NOT MET for mutation-testing on Sonnet: over 3 cases the paired with-arm change is +0.04 with a 95% interval of -0.28 to +0.35 (noise-report.py, not shown non-inferior). The interval is wide because there are only 3 cases at 3 runs per side; au-light-survivors-unclassified fell .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2 failing runs before/after) and au-no-test-command-asks rose .67 to 1.00. Opus is non-inferior (+0.00).
The numeric bars below are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.
A before and after eval of mutation-testing on Sonnet, sized so the interval can decide the 0.05 margin (more runs per case, or more cases that load the changed text), both sides run together, is recorded and reads non-inferior at margin 0.05; or the record states why that cannot be shown with the cases available.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the #16063 two-version compare of #16229 could not show mutation-testing non-inferior on Sonnet. Over 3 cases the paired change is +0.04, with a 95% interval of -0.28 to +0.35. The issue asks for two things. First, read the after-side au-light-survivors-unclassified runs against both graders and state the cause with quotes. Second, record a larger before and after Sonnet eval that reads non-inferior at margin 0.05, or record why the available cases cannot show that.
Nothing is missing that blocks triage. The eval runs may need a host run, which the lane routes separately.
Possibly related, but it targets different cases (the 12 se- cases): #16358. I found no merged pull request that already covers this.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 2 (eval: "A before and after eval of mutation-testing on Sonnet, sized so the interval ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Activity
github-actions commentedon Nov 15, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the #16063 two-version compare of #16229 could not show mutation-testing non-inferior on Sonnet. Over 3 cases the paired change is +0.04, with a 95% interval of -0.28 to +0.35. The issue asks for two things. First, read the after-side
au-light-survivors-unclassifiedruns against both graders and state the cause with quotes. Second, record a larger before and after Sonnet eval that reads non-inferior at margin 0.05, or record why the available cases cannot show that.Nothing is missing that blocks triage. The eval runs may need a host run, which the lane routes separately.
Possibly related, but it targets different cases (the 12
se-cases): #16358. I found no merged pull request that already covers this.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 2 (eval: "A before and after eval of mutation-testing on Sonnet, sized so the interval ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).