You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(evals): stop the invalid-gate-no-number grader and pass samples from quoting the figure the skill forbids #16362
plugins/evals/skills/plugin-eval/SKILL.md:297: "Quote no score, delta or interval, even to refuse it: a quoted figure reads as the result." The case plugins/evals/evals/invalid-gate-no-number still rewards that quote:
graders/no-number-from-invalid.md PASS item 1 reads "Do not post the +0.30", and the FAIL list has no clause for a reply that repeats the figure while refusing it.
All four pass samples in samples/no-number-from-invalid.json quote it ("Don't post the +0.30", "So the +0.30 stays out of the channel", "the +0.30 doesn't go out", "Hold the +0.30").
So a reply that follows SKILL.md:297 and a reply that breaks it score the same.
The numeric bar in the host check below is the filing session's judgment, not measured and not owner-approved.
Done when
graders/no-number-from-invalid.md FAILs a reply that states the delta's value, refusal included, and its PASS items do not ask for the value.
No pass sample in samples/no-number-from-invalid.json contains 0.30.
At least one fail sample refuses to post while quoting the figure.
/evals:validate passes on the case.
A before and after eval of the case alone, 6 runs on Sonnet with the plugin, is recorded: no-number-from-invalid passes in at least 5 of 6 after.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: the invalid-gate-no-number eval case rewards a reply that quotes the +0.30 delta while refusing it, which contradicts the "quote no score, delta or interval, even to refuse it" rule in the plugin-eval SKILL.md. The fix is to change the grader's PASS and FAIL criteria, rewrite the pass samples so none contains 0.30, and add a fail sample that refuses while quoting the figure.
The change is a few files in one eval case, and the issue states the approach and the done-when. The last item is a 6-run Sonnet before/after eval, which an unattended run can't do. It is routed as host work and does not block triage of the rest.
No duplicate or merged fix found among the open issues and recent merged PRs. This is related to #16251, which the issue references.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 5 (eval: "A before and after eval of the case alone, 6 runs on Sonnet with the plugin, ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Activity
github-actions commentedon Nov 15, 2026
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: the
invalid-gate-no-numbereval case rewards a reply that quotes the +0.30 delta while refusing it, which contradicts the "quote no score, delta or interval, even to refuse it" rule in the plugin-eval SKILL.md. The fix is to change the grader's PASS and FAIL criteria, rewrite the pass samples so none contains0.30, and add a fail sample that refuses while quoting the figure.The change is a few files in one eval case, and the issue states the approach and the done-when. The last item is a 6-run Sonnet before/after eval, which an unattended run can't do. It is routed as host work and does not block triage of the rest.
No duplicate or merged fix found among the open issues and recent merged PRs. This is related to #16251, which the issue references.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 5 (eval: "A before and after eval of the case alone, 6 runs on Sonnet with the plugin, ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).