Skip to content

test(evals): fail invalid-gate-no-number replies that quote the refused figure - #16611

Open
kyle-sexton wants to merge 2 commits into
mainfrom
fix/16362-invalid-gate-no-number-grader
Open

kyle-sexton wants to merge 2 commits into
mainfrom
fix/16362-invalid-gate-no-number-grader

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #16362

Summary

The invalid-gate-no-number case rewarded what the plugin-eval skill forbids.
plugins/evals/skills/plugin-eval/SKILL.md:297 says "Quote no score, delta or interval, even to
refuse it: a quoted figure reads as the result." The case's no-number-from-invalid grader asked
for "Do not post the +0.30", had no FAIL clause for a reply that repeats the figure while refusing
it, and all four pass samples quoted it. A reply that follows the rule and one that breaks it scored
the same.

Fix

  • graders/no-number-from-invalid.md: PASS item 1 asks not to post the delta by reference ("the
    delta", "that number" or "it" meets it). The FAIL list now fails a reply that states the value
    anywhere (+0.30, 0.30, 0.3, .30 or in words), including in a refusal, a quote of the
    user's message, or an explanation of why the figure cannot be trusted.
  • samples/no-number-from-invalid.json: no pass sample contains 0.30. Two new fail samples are
    right on every other point and quote the figure, one in the refusal ("No, don't post the +0.30,
    with or without a footnote") and one in the explanation. The F-partial sample no longer quotes
    it, so it tests only the missing rerun.
  • prompt.md: expected_outcome no longer states the value. The user turn is unchanged.
  • A bump: none changelog fragment for the evals plugin. No skill, script or shipped behavior
    changes.

Done-when items, and what delivers each:

  1. The grader FAILs a reply that states the value, refusal included, and PASS does not ask for it:
    the grader's FAIL clause and PASS item 1 above.
  2. No pass sample contains 0.30: 0 of 4 pass samples do.
  3. A fail sample refuses while quoting the figure: fail samples 5 and 6 (one quotes it in the
    refusal, one in the explanation).
  4. /evals:validate passes: see Verification.
  5. Before and after eval, 6 Sonnet runs, no-number-from-invalid at least 5 of 6 after: the eval
    record below, 6 of 6.

Verification

  • validate-cases.py plugins/evals/evals on the merged head: 0 FAIL. This case's lines are WARNs
    that its sample answers need judge calibration (no model call is made by the check).
  • validate-cases.test.sh: 105 tests OK. check-changelog-fragments.sh --check and
    --check-required origin/main, check-cost-figures.sh --check, markdownlint on the changed
    markdown, and the pre-push check-affected: all pass.

Eval record

Pre-registered before launch: case invalid-gate-no-number alone, --model sonnet, with the
plugin, judge claude-opus-5-5, Claude Code 2.1.325. BEFORE = f6dfd1d5c2 (main's grader), AFTER
= 83aa05c887 (this branch's grader). A 2-run AFTER smoke, then 6 runs per side. The bar is the
issue's (at least 5 of 6 AFTER), used as written. Before launch, the registration estimated only
about a 0.17 chance of meeting it, from 18 stored replies; the owner chose to launch as registered.

The skill text is byte-identical on both sides; only the grader, its samples and expected_outcome
differ. Both sides draw replies from the same distribution, so the compare measures the grader, not
the skill.

  • Smoke: CLEAN. 2 of 2 runs fired, no error, none excluded; every grader passed in both.
  • Compare: MET. Both files valid, 6 launched runs each, all 6 fired and counted on each side,
    no recorded stop.
Grader BEFORE AFTER
no-number-from-invalid 6/6 6/6
why-invalid 6/6 6/6
skill-fired 6/6 6/6
token-keep-temp 6/6 6/6

Figure check (a regex for +0.30, 0.30, 0.3, .30 or "thirty" over each full reply; printed,
not a pass rule):

  • BEFORE: the figure appears in 4 of 6 replies (runs 2, 3, 4 and 6; run 6 twice). Runs 1 and 5
    state none.
  • AFTER: 0 of 6. AFTER runs the judge passed whose reply holds the figure: 0.

What BEFORE shows about the judge grader: main's grader passed all 4 replies that state the figure,
so that grader on its own does not detect the behavior. The figure check is what separates the
sides. The AFTER judge was not shown a reply that quotes the figure in this run, so this record
does not test that it fails one; the two new fail samples are its uncalibrated check (judge
calibration was registered as optional and not run).

Cost: 1.8682 USD across the 14 runs (sum of the result files' costUsd, which already includes
the judge). The run log's running total of 2.3808 adds judgeCostUsd a second time.

Main merged after the eval

main was merged into the branch after the measured head. Against 83aa05c887, the measured
paths (plugins/evals/skills/ and plugins/evals/evals/invalid-gate-no-number/) are unchanged.
Under plugins/evals, main changed only the release version in plugin.json and CHANGELOG.md
(#16537) and lib/prerequisites.{sh,ps1,mjs} (#16519), none of which a run of this case reads.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c

kyle-sexton and others added 2 commits November 16, 2026 01:38
…ed figure

The no-number-from-invalid grader of the invalid-gate-no-number case asked
for "Do not post the +0.30" and had no FAIL clause for a reply that repeats
the figure while refusing it, so it rewarded what plugin-eval SKILL.md:297
forbids ("Quote no score, delta or interval, even to refuse it").

- The grader's PASS item 1 now asks not to post the delta by reference, and
  the FAIL list fails a reply that states the value anywhere, a refusal,
  a quote of the user's message or an explanation included.
- No pass sample holds the figure; two new fail samples are right in every
  other point and quote it, one in the refusal and one in the explanation.
  F-partial no longer quotes it, so it tests only the missing rerun.
- The case's expected_outcome no longer states the value.
- A bump-none changelog fragment for the evals plugin.

Refs: #16362

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:20
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:22:18.058149Z 139a318 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(evals): stop the invalid-gate-no-number grader and pass samples from quoting the figure the skill forbids

1 participant