Repository navigation
test(evals): fail invalid-gate-no-number replies that quote the refused figure - #16611
Open
kyle-sexton wants to merge 2 commits into
Open
kyle-sexton wants to merge 2 commits into
kyle-sexton wants to merge 2 commits into
Conversation
…ed figure
The no-number-from-invalid grader of the invalid-gate-no-number case asked
for "Do not post the +0.30" and had no FAIL clause for a reply that repeats
the figure while refusing it, so it rewarded what plugin-eval SKILL.md:297
forbids ("Quote no score, delta or interval, even to refuse it").
- The grader's PASS item 1 now asks not to post the delta by reference, and
the FAIL list fails a reply that states the value anywhere, a refusal,
a quote of the user's message or an explanation included.
- No pass sample holds the figure; two new fail samples are right in every
other point and quote it, one in the refusal and one in the explanation.
F-partial no longer quotes it, so it tests only the missing rerun.
- The case's expected_outcome no longer states the value.
- A bump-none changelog fragment for the evals plugin.
Refs: #16362
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e-no-number-grader
kyle-sexton
marked this pull request as ready for review
November 16, 2026 19:20
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #16362
Summary
The
invalid-gate-no-numbercase rewarded what the plugin-eval skill forbids.plugins/evals/skills/plugin-eval/SKILL.md:297says "Quote no score, delta or interval, even torefuse it: a quoted figure reads as the result." The case's
no-number-from-invalidgrader askedfor "Do not post the +0.30", had no FAIL clause for a reply that repeats the figure while refusing
it, and all four pass samples quoted it. A reply that follows the rule and one that breaks it scored
the same.
Fix
graders/no-number-from-invalid.md: PASS item 1 asks not to post the delta by reference ("thedelta", "that number" or "it" meets it). The FAIL list now fails a reply that states the value
anywhere (
+0.30,0.30,0.3,.30or in words), including in a refusal, a quote of theuser's message, or an explanation of why the figure cannot be trusted.
samples/no-number-from-invalid.json: no pass sample contains0.30. Two new fail samples areright on every other point and quote the figure, one in the refusal ("No, don't post the +0.30,
with or without a footnote") and one in the explanation. The F-partial sample no longer quotes
it, so it tests only the missing rerun.
prompt.md:expected_outcomeno longer states the value. The user turn is unchanged.bump: nonechangelog fragment for the evals plugin. No skill, script or shipped behaviorchanges.
Done-when items, and what delivers each:
the grader's FAIL clause and PASS item 1 above.
0.30: 0 of 4 pass samples do.refusal, one in the explanation).
/evals:validatepasses: see Verification.no-number-from-invalidat least 5 of 6 after: the evalrecord below, 6 of 6.
Verification
validate-cases.py plugins/evals/evalson the merged head: 0 FAIL. This case's lines are WARNsthat its sample answers need judge calibration (no model call is made by the check).
validate-cases.test.sh: 105 tests OK.check-changelog-fragments.sh --checkand--check-required origin/main,check-cost-figures.sh --check, markdownlint on the changedmarkdown, and the pre-push
check-affected: all pass.Eval record
Pre-registered before launch: case
invalid-gate-no-numberalone,--model sonnet, with theplugin, judge
claude-opus-5-5, Claude Code 2.1.325. BEFORE =f6dfd1d5c2(main's grader), AFTER=
83aa05c887(this branch's grader). A 2-run AFTER smoke, then 6 runs per side. The bar is theissue's (at least 5 of 6 AFTER), used as written. Before launch, the registration estimated only
about a 0.17 chance of meeting it, from 18 stored replies; the owner chose to launch as registered.
The skill text is byte-identical on both sides; only the grader, its samples and
expected_outcomediffer. Both sides draw replies from the same distribution, so the compare measures the grader, not
the skill.
no recorded stop.
no-number-from-invalidwhy-invalidskill-firedtoken-keep-tempFigure check (a regex for
+0.30,0.30,0.3,.30or "thirty" over each full reply; printed,not a pass rule):
state none.
What BEFORE shows about the judge grader: main's grader passed all 4 replies that state the figure,
so that grader on its own does not detect the behavior. The figure check is what separates the
sides. The AFTER judge was not shown a reply that quotes the figure in this run, so this record
does not test that it fails one; the two new fail samples are its uncalibrated check (judge
calibration was registered as optional and not run).
Cost: 1.8682 USD across the 14 runs (sum of the result files'
costUsd, which already includesthe judge). The run log's running total of 2.3808 adds
judgeCostUsda second time.Main merged after the eval
mainwas merged into the branch after the measured head. Against83aa05c887, the measuredpaths (
plugins/evals/skills/andplugins/evals/evals/invalid-gate-no-number/) are unchanged.Under
plugins/evals, main changed only the release version inplugin.jsonandCHANGELOG.md(#16537) and
lib/prerequisites.{sh,ps1,mjs}(#16519), none of which a run of this case reads.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c