Skip to content

fix(toolchain): #16228's text reads a Sonnet loss in the #16063 compare (not shown non-inferior) #16355

Open

Description

@kyle-sexton

Refs: #16063

The #16063 two-version compare of #16228 reads NOT MET for toolchain on Sonnet: the paired with-arm change over 19 cases is -0.01 with a 95% interval of -0.06 to +0.04, so the lower end is below the -0.05 margin (noise-report.py, not shown non-inferior). Opus is non-inferior on the same cases. Per-case reads in the record on #16063: the after side lost single grader runs in ck-malformed-config-degrades (1.00 to .78), ck-missing-tool-incomplete (1.00 to .83), ck-skips-lint-only-surfaces (1.00 to .83) and ck-detects-changed-ecosystems (1.00 to .92), and gained in ln-missing-tool-incomplete (.67 to 1.00).

The numeric bars below (runs per side, margins) are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.

Done when

session: GitHub Support

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  3. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: in the #16063 compare of #16228, toolchain on Sonnet is not shown non-inferior (paired change -0.01, 95% interval -0.06 to +0.04, below the -0.05 margin over 19 cases). Four cases lost single grader runs. The work is to read those traces and state the cause of each loss. If the #16228 text is the cause, fix the toolchain skill text in its own PR. Then record a before/after Sonnet eval, or show per-case evidence that the loss is noise.

    Nothing is missing for triage. The cause is not yet known, so the approach depends on what the traces show. The 6-runs-per-side eval is host or paid-run work, and the issue already carries host-run for that. The numeric bars are noted in the issue as unapproved, so whoever registers the eval will set them.

    Related, not duplicates: #16229 and the verification issue about #16228 (a Sonnet case drop in the same #16063 compare).

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of toolchain on Sonnet over the same 19 cases, 6 runs..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions