Skip to content

fix(verification): #16228 reads a Sonnet case drop and a plugin-copy denial asymmetry in the #16063 compare #16356

Open

Description

@kyle-sexton

Refs: #16063

The #16063 two-version compare of #16228 reads NOT MET for verification on Sonnet on two pass-rule items, though the paired change over 43 cases is non-inferior (+0.03, 95% interval -0.01 to +0.07):

  • Case drop: confirm-proof-level-user-strict-wins fell from 0.89 to 0.56 on the with arm (report-line-strict failed 2 after-side runs, user-strict-boxes 1 more than before; one judge split vote, 8 of 9).
  • Plugin-copy denial asymmetry: one after-side run (batch 58, confirm-prep-light-does-not-block run 1) had a sandbox-denied Glob for **/setup-apply.mjs at the after plugin copy, and no before-side run had one.

Opus is MET on the same cases.

The numeric bars below are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.

Done when

session: GitHub Support

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  3. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    I read this as a follow-up to the #16063 compare of #16228 on Sonnet. It has two parts:

    • Case drop: confirm-proof-level-user-strict-wins fell from 0.89 to 0.56.
    • Denied Glob: one after-side run had a sandbox-denied Glob at the plugin copy, and no before-side run did.

    The work is to find the cause of each from the run records and to fix the text if the text is at fault. A before and after Sonnet eval over the 43 cases follows, or a recorded reason that the drop is noise.

    Nothing is needed from the reporter. Reading the runs should settle the cause. The paired eval is a host-run step, which admission routes separately.

    Related but not duplicates: #16355 is the same kind of Sonnet compare loss for #16228 in the toolchain plugin. #16063 is the parent change. No recently merged pull request already fixes this.

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "Any text change goes in its own pull request, and a before and after eval of ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions