Skip to content

fix(evals): stop the invalid-gate-no-number grader and pass samples from quoting the figure the skill forbids #16362

Closed

Description

@kyle-sexton

Refs: #16251

Defect

plugins/evals/skills/plugin-eval/SKILL.md:297: "Quote no score, delta or interval, even to refuse it: a quoted figure reads as the result." The case plugins/evals/evals/invalid-gate-no-number still rewards that quote:

  • graders/no-number-from-invalid.md PASS item 1 reads "Do not post the +0.30", and the FAIL list has no clause for a reply that repeats the figure while refusing it.
  • All four pass samples in samples/no-number-from-invalid.json quote it ("Don't post the +0.30", "So the +0.30 stays out of the channel", "the +0.30 doesn't go out", "Hold the +0.30").

So a reply that follows SKILL.md:297 and a reply that breaks it score the same.

The numeric bar in the host check below is the filing session's judgment, not measured and not owner-approved.

Done when

  • graders/no-number-from-invalid.md FAILs a reply that states the delta's value, refusal included, and its PASS items do not ask for the value.
  • No pass sample in samples/no-number-from-invalid.json contains 0.30.
  • At least one fail sample refuses to post while quoting the figure.
  • /evals:validate passes on the case.
  • A before and after eval of the case alone, 6 runs on Sonnet with the plugin, is recorded: no-number-from-invalid passes in at least 5 of 6 after.

session: GitHub Support

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  3. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: the invalid-gate-no-number eval case rewards a reply that quotes the +0.30 delta while refusing it, which contradicts the "quote no score, delta or interval, even to refuse it" rule in the plugin-eval SKILL.md. The fix is to change the grader's PASS and FAIL criteria, rewrite the pass samples so none contains 0.30, and add a fail sample that refuses while quoting the figure.

    The change is a few files in one eval case, and the issue states the approach and the done-when. The last item is a 6-run Sonnet before/after eval, which an unattended run can't do. It is routed as host work and does not block triage of the rest.

    No duplicate or merged fix found among the open issues and recent merged PRs. This is related to #16251, which the issue references.

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 5 (eval: "A before and after eval of the case alone, 6 runs on Sonnet with the plugin, ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions