Skip to content

test(mutation-testing): #16229's Sonnet compare cannot decide non-inferiority on 3 cases in #16063 #16357

Open

Description

@kyle-sexton

Refs: #16063

The #16063 two-version compare of #16229 reads NOT MET for mutation-testing on Sonnet: over 3 cases the paired with-arm change is +0.04 with a 95% interval of -0.28 to +0.35 (noise-report.py, not shown non-inferior). The interval is wide because there are only 3 cases at 3 runs per side; au-light-survivors-unclassified fell .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2 failing runs before/after) and au-no-test-command-asks rose .67 to 1.00. Opus is non-inferior (+0.00).

The numeric bars below are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.

Done when

  • The after-side runs of au-light-survivors-unclassified on Sonnet are read against the two graders, and the cause is stated with quotes: the feat(mutation-testing): accept any mutation tool id and name one with no shipped adapter #16229 text, the case, or the grader.
  • A before and after eval of mutation-testing on Sonnet, sized so the interval can decide the 0.05 margin (more runs per case, or more cases that load the changed text), both sides run together, is recorded and reads non-inferior at margin 0.05; or the record states why that cannot be shown with the cases available.

session: GitHub Support

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  3. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: the #16063 two-version compare of #16229 could not show mutation-testing non-inferior on Sonnet. Over 3 cases the paired change is +0.04, with a 95% interval of -0.28 to +0.35. The issue asks for two things. First, read the after-side au-light-survivors-unclassified runs against both graders and state the cause with quotes. Second, record a larger before and after Sonnet eval that reads non-inferior at margin 0.05, or record why the available cases cannot show that.

    Nothing is missing that blocks triage. The eval runs may need a host run, which the lane routes separately.

    Possibly related, but it targets different cases (the 12 se- cases): #16358. I found no merged pull request that already covers this.

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 2 (eval: "A before and after eval of mutation-testing on Sonnet, sized so the interval ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions