Skip to content

test(mutation-testing): run a before and after eval of #16229's setup-skill text over the 12 se- cases #16358

Open

Description

@kyle-sexton

Refs: #15710

#16229 changed the mutation-testing setup skill's text (plugins/mutation-testing/skills/setup/SKILL.md lines 73-76 and 96-98, skills/setup/templates/config-template.md, and the config schema description). By the owner's choice in the #16063 compare (D303), that text was not compared and the 12 se-* setup cases were not run, so its effect is not measured.

The numeric bars below are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.

Done when

  • A before and after eval of the 12 se-* setup cases, on Sonnet and Opus, before 59d6f9d110 against after 08fcb9062e (or the current main text against its parent where it has changed), both sides run together, 3 runs per side, is recorded here with the per-grader pass counts per side.
  • The record reads non-inferior at margin 0.05 with no case drop on each model; or each loss is named with its cause and carried in its own issue.

session: GitHub Support

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  3. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: #16229 changed the mutation-testing setup skill text, and its effect was never measured because the 12 se-* setup cases were skipped in the #16063 compare. This issue asks for a paired before/after eval (Sonnet and Opus, 3 runs per side) and a non-inferiority record at margin 0.05. The numeric bars are stated as the filing session's judgment, to be set by whoever registers the eval from measured rates. That does not leave an outcome open.

    The eval runs themselves may need a host or an attended run, so the issue keeps its host-run label.

    Related, not a duplicate: #16357 covers the Sonnet compare for 3 cases in #16063. No merged pull request in the recent list appears to already do this.

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (eval: "A before and after eval of the 12 se-* setup cases, on Sonnet and Opus, befor..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions