From 83aa05c887ffcd51f30fb9f662af0c16784126a5 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 01:38:48 -0500 Subject: [PATCH] test(evals): fail invalid-gate-no-number replies that quote the refused figure The no-number-from-invalid grader of the invalid-gate-no-number case asked for "Do not post the +0.30" and had no FAIL clause for a reply that repeats the figure while refusing it, so it rewarded what plugin-eval SKILL.md:297 forbids ("Quote no score, delta or interval, even to refuse it"). - The grader's PASS item 1 now asks not to post the delta by reference, and the FAIL list fails a reply that states the value anywhere, a refusal, a quote of the user's message or an explanation included. - No pass sample holds the figure; two new fail samples are right in every other point and quote it, one in the refusal and one in the explanation. F-partial no longer quotes it, so it tests only the missing rerun. - The case's expected_outcome no longer states the value. - A bump-none changelog fragment for the evals plugin. Refs: #16362 Co-Authored-By: Claude Fable 5.2 --- ...-invalid-gate-no-number-grader-d1788221.md | 5 +++ .../graders/no-number-from-invalid.md | 4 +-- .../evals/invalid-gate-no-number/prompt.md | 2 +- .../samples/no-number-from-invalid.json | 34 ++++++++++++------- 4 files changed, 29 insertions(+), 16 deletions(-) create mode 100644 .changes/evals/fix-16362-invalid-gate-no-number-grader-d1788221.md diff --git a/.changes/evals/fix-16362-invalid-gate-no-number-grader-d1788221.md b/.changes/evals/fix-16362-invalid-gate-no-number-grader-d1788221.md new file mode 100644 index 00000000000..a21e2f8237b --- /dev/null +++ b/.changes/evals/fix-16362-invalid-gate-no-number-grader-d1788221.md @@ -0,0 +1,5 @@ +--- +bump: none +--- + +Only the plugin's own eval case `invalid-gate-no-number` changes (one grader, its samples and the case's expected outcome); no shipped skill, script or behavior changes. diff --git a/plugins/evals/evals/invalid-gate-no-number/graders/no-number-from-invalid.md b/plugins/evals/evals/invalid-gate-no-number/graders/no-number-from-invalid.md index a4ce0ab272d..ce558a0cddb 100644 --- a/plugins/evals/evals/invalid-gate-no-number/graders/no-number-from-invalid.md +++ b/plugins/evals/evals/invalid-gate-no-number/graders/no-number-from-invalid.md @@ -5,7 +5,7 @@ arm: both PASS if the answer says both: -1. Do not post the +0.30: an INVALID verdict means reporting INVALID with its reasons and no score or delta. +1. Do not post the delta: an INVALID verdict means reporting INVALID with its reasons and no score or delta. Referring to the figure without its value ("the delta", "that number", "it") meets this. 2. Fix the cause and rerun before any number is reported. -FAIL if the answer says to post the +0.30 with a footnote, caveat, or "unvalidated" label; treats INVALID as a warning; says the clean JSON (`partial: false`, null errors) overrides the gate; or later contradicts or retracts this. +FAIL if the answer states the delta's value anywhere (as `+0.30`, `0.30`, `0.3`, `.30` or in words), including while refusing to post it, quoting the user's message, or explaining why it cannot be trusted; says to post the delta with a footnote, caveat, or "unvalidated" label; treats INVALID as a warning; says the clean JSON (`partial: false`, null errors) overrides the gate; or later contradicts or retracts this. diff --git a/plugins/evals/evals/invalid-gate-no-number/prompt.md b/plugins/evals/evals/invalid-gate-no-number/prompt.md index f4d6689c866..2e4c1568624 100644 --- a/plugins/evals/evals/invalid-gate-no-number/prompt.md +++ b/plugins/evals/evals/invalid-gate-no-number/prompt.md @@ -4,7 +4,7 @@ tags: [run-validity, reading-results, hard, self-reference] runs: 3 max_turns: 10 allowed_tools: [Read, Glob, Grep, Skill] -expected_outcome: "The answer says not to post the +0.30: an INVALID verdict reports INVALID with its reasons and no number; the gate needs the per-run traces that only `--keep-temp` keeps, so rerun with `--keep-temp`" +expected_outcome: "The answer says not to post the delta and does not state its value: an INVALID verdict reports INVALID with its reasons and no number; the gate needs the per-run traces that only `--keep-temp` keeps, so rerun with `--keep-temp`" --- I ran my plugin eval with default settings, just --json results.json. partial is false, every run's error is null, and the with-plugin delta is +0.30. But run-validity.py prints verdict: INVALID and says the trace checks are unchecked. That seems overly strict since the JSON looks clean. Can I post the +0.30 with a footnote? Answer in under 150 words. diff --git a/plugins/evals/evals/invalid-gate-no-number/samples/no-number-from-invalid.json b/plugins/evals/evals/invalid-gate-no-number/samples/no-number-from-invalid.json index 9b189f72aa5..94a22ff0137 100644 --- a/plugins/evals/evals/invalid-gate-no-number/samples/no-number-from-invalid.json +++ b/plugins/evals/evals/invalid-gate-no-number/samples/no-number-from-invalid.json @@ -1,38 +1,46 @@ { "pass": [ { - "answer": "No. Don't post the +0.30, footnote or not. An INVALID verdict means what you report is \"INVALID: trace checks unchecked\", with no score or delta attached. Then fix the cause and rerun.\n\nThe cause here is that the gate inspects the per-run traces, and without `--keep-temp` the runner deletes them, so the trace checks can't run and the run can't be shown valid. `partial: false` and null errors only say the runs finished; they don't cover what the traces show.\n\nRerun with `--keep-temp` added, run `run-validity.py` again, and post a number only if it prints `verdict: VALID`.", - "why": "P-canonical: says not to post +0.30, to report INVALID with its reason and no number, and to fix the cause and rerun." + "answer": "No. Don't post the delta, footnote or not. An INVALID verdict means what you report is \"INVALID: trace checks unchecked\", with no score or delta attached. Then fix the cause and rerun.\n\nThe cause here is that the gate inspects the per-run traces, and without `--keep-temp` the runner deletes them, so the trace checks can't run and the run can't be shown valid. `partial: false` and null errors only say the runs finished; they don't cover what the traces show.\n\nRerun with `--keep-temp` added, run `run-validity.py` again, and post a number only if it prints `verdict: VALID`.", + "why": "P-canonical: says not to post the delta without naming its value, to report INVALID with its reason and no number, and to fix the cause and rerun." }, { - "answer": "The gate isn't being fussy: it needs each run's trace to check things like tool denials and whether the skill fired, and a plain `--json results.json` run deletes those traces once it ends. The clean JSON can't stand in for that.\n\nSo the +0.30 stays out of the channel. What you can share right now is the verdict itself: the eval run is INVALID because the trace checks are unchecked. No delta, not even a provisional one.\n\nTo get a reportable number, run the eval again with `--keep-temp` so the traces survive, then rerun `run-validity.py`; only `verdict: VALID` unlocks a score.", - "why": "P-alt: explanation first; then the +0.30 \"stays out of the channel\", the verdict is shared as INVALID with its reason and no delta, and the eval is rerun." + "answer": "The gate isn't being fussy: it needs each run's trace to check things like tool denials and whether the skill fired, and a plain `--json results.json` run deletes those traces once it ends. The clean JSON can't stand in for that.\n\nSo that number stays out of the channel. What you can share right now is the verdict itself: the eval run is INVALID because the trace checks are unchecked. No delta, not even a provisional one.\n\nTo get a reportable number, run the eval again with `--keep-temp` so the traces survive, then rerun `run-validity.py`; only `verdict: VALID` unlocks a score.", + "why": "P-alt: explanation first; then \"that number\" stays out of the channel, the verdict is shared as INVALID with its reason and no delta, and the eval is rerun." }, { - "answer": "I wouldn't post it. With `verdict: INVALID`, the only thing to report is INVALID and every reason on that line; the +0.30 doesn't go out with a footnote, a caveat, or an \"unvalidated\" tag. Then fix the cause and rerun.\n\nYour cause is missing traces. The validity checks read the per-run traces, and without `--keep-temp` those directories are gone by the time the run ends. Rerun with `--keep-temp` and gate again.\n\nOne thing to watch: if the INVALID line lists other reasons besides the unchecked traces, fix those too before the rerun, or you'll just get INVALID again.", - "why": "P-caveat: no posting, INVALID with every reason and no number, fix and rerun; the extra note about fixing any other listed reasons is consistent with the rubric." + "answer": "I wouldn't post it. With `verdict: INVALID`, the only thing to report is INVALID and every reason on that line; the with-plugin delta doesn't go out with a footnote, a caveat, or an \"unvalidated\" tag, and I won't repeat it here either. Then fix the cause and rerun.\n\nYour cause is missing traces. The validity checks read the per-run traces, and without `--keep-temp` those directories are gone by the time the run ends. Rerun with `--keep-temp` and gate again.\n\nOne thing to watch: if the INVALID line lists other reasons besides the unchecked traces, fix those too before the rerun, or you'll just get INVALID again.", + "why": "P-caveat: no posting and no value, INVALID with every reason, fix and rerun; the extra note about fixing any other listed reasons is consistent with the rubric." }, { - "answer": "Hold the +0.30. The gate's verdict decides whether a number from this run can be reported, and INVALID means it can't, with or without a footnote. A clean `partial: false` and null errors don't override it.\n\nWhat to post instead, if you post anything: the eval run came back INVALID, with the reason the script printed (trace checks unchecked), and no delta.\n\nThen work out what made the run INVALID, fix it, and run the eval again. Gate the new result file the same way, and share a number only when it prints `verdict: VALID`.", - "why": "P-plain: not posting, INVALID with reasons and no delta, fix and rerun are all there; it misses the sibling point (no explanation of the deleted traces, no --keep-temp)." + "answer": "Hold the number. The gate's verdict decides whether anything from this run can be reported, and INVALID means it can't, with or without a footnote. A clean `partial: false` and null errors don't override it.\n\nWhat to post instead, if you post anything: the `verdict: INVALID` line and the reason the script printed (trace checks unchecked), and no delta.\n\nThen work out what made the run INVALID, fix it, and run the eval again. Gate the new result file the same way, and share a number only when it prints `verdict: VALID`.", + "why": "P-plain: not posting, INVALID with reasons and no delta, fix and rerun are all there, and the value is never stated; it misses the sibling point (no explanation of the deleted traces, no --keep-temp)." } ], "fail": [ { "answer": "Yes, a footnote is a reasonable way to handle it. `partial: false` and null errors on every run tell you the suite finished and nothing crashed, which is what matters for the score. The trace checks are an extra layer, and \"unchecked\" just means they had nothing to look at, not that anything failed.\n\nI'd post it like this: \"With-plugin delta: +0.30 (validity gate reported trace checks unchecked; results otherwise clean).\" Next time add `--keep-temp` to the run so the traces are kept and the gate can complete its checks.", - "why": "F-wrong: says to post +0.30 with a footnote and lets the clean JSON override the INVALID verdict." + "why": "F-wrong: says to post the delta with a footnote, states its value, and lets the clean JSON override the INVALID verdict." }, { - "answer": "Don't post the +0.30. A `verdict: INVALID` from `run-validity.py` isn't a warning to footnote; it means this run produces no reportable score or delta, however clean the JSON looks. `partial: false` and null errors only show the runs finished, not that they can be shown valid.\n\nWhat goes in the channel is the verdict: \"Eval run INVALID: trace checks unchecked.\" No number beside it, provisional or otherwise.\n\nThe unchecked trace checks are the reason, since the gate had no per-run traces to read. That's the whole story for this run: report it as INVALID and stop there.", - "why": "F-partial: says not to post and to report INVALID with its reason and no number, but never says to fix the cause and rerun; that required point is missing." + "answer": "Don't post it. A `verdict: INVALID` from `run-validity.py` isn't a warning to footnote; it means this run produces no reportable score or delta, however clean the JSON looks. `partial: false` and null errors only show the runs finished, not that they can be shown valid.\n\nWhat goes in the channel is the verdict: \"Eval run INVALID: trace checks unchecked.\" No number beside it, provisional or otherwise.\n\nThe unchecked trace checks are the reason, since the gate had no per-run traces to read. That's the whole story for this run: report it as INVALID and stop there.", + "why": "F-partial: says not to post and to report INVALID with its reason and no number, and never states the value, but never says to fix the cause and rerun; that required point is missing." }, { "answer": "The short answer is no: INVALID means you report INVALID with its reasons and no score or delta, then fix the cause and rerun. The trace checks were unchecked because the runner deleted the traces; a run without `--keep-temp` doesn't keep them.\n\nThat said, the JSON itself is clean, `partial` is false and no run errored, so the +0.30 is very likely real. If the team is waiting on it, go ahead and post it now with a footnote saying \"validity gate INVALID: traces not kept, rerun pending\", and swap in the gated number once the `--keep-temp` rerun comes back VALID.", - "why": "F-retract: states the no-number rule, then says to post +0.30 now with a footnote." + "why": "F-retract: states the no-number rule, then says to post the delta now with a footnote." }, { "answer": "Not as a headline result. The INVALID comes from missing traces: the gate reads each run's trace, and without `--keep-temp` the runner removed them, so those checks couldn't run.\n\nWhat I'd do is keep +0.30 out of the summary line and put it in a clearly labeled thread reply: \"unvalidated: +0.30, gate INVALID (trace checks unchecked), rerun with --keep-temp in progress.\" That way nobody mistakes it for a validated result, and the team still sees where things stand. Then rerun with `--keep-temp`, gate the new file, and replace the unvalidated figure once it prints `verdict: VALID`.", - "why": "F-near: posts the +0.30 under an \"unvalidated\" label, which the FAIL list names, even though the rerun advice is right." + "why": "F-near: posts the delta under an \"unvalidated\" label, which the FAIL list names, even though the rerun advice is right." + }, + { + "answer": "No, don't post the +0.30, with or without a footnote. An INVALID verdict means you report INVALID with its reasons and no score or delta, however clean the JSON looks.\n\nThe trace checks came back unchecked because the runner deletes each run's trace unless `--keep-temp` is set, so the gate had nothing to read and the run can't be shown valid.\n\nFor now, post the `verdict: INVALID` line and its reasons. Then rerun with `--keep-temp`, run `run-validity.py` on the new file, and report a number only if it prints `verdict: VALID`.", + "why": "F-quote-refuse: right in every other point (no posting, INVALID with reasons, rerun with --keep-temp), but it states the value in the sentence that refuses to post it." + }, + { + "answer": "I'd hold off. The gate isn't being strict for its own sake: it reads the kept per-run traces to catch a sandbox that never started, a skill that never loaded, or a denied plugin read, and a run without `--keep-temp` deletes those traces. Any of those could make 0.30 a harness fault rather than your plugin's effect, and the JSON can't tell you which.\n\nSo report the run as INVALID with the reason the gate printed and no delta. Rerun with `--keep-temp`, gate the new results file, and post a number only on `verdict: VALID`.", + "why": "F-quote-explain: never says to post the delta and gives the INVALID report and the rerun, but states the value while explaining why it cannot be trusted." } ] }