Repository navigation
fix(playbooks): say style example pairs show their input, not a Bad/Good contrast - #16574
Conversation
…ood contrast The structured-output rule in skill-authoring's skill-criteria.md asked for labeled input/output pairs, but every failing Sonnet run of the skill-authoring-style-output-labeled-pairs case printed a Bad:/Good: pair of two titles with no input, the shape of the file's own worked example for another rule. The rule now says each pair shows the input beside the output, labeled apart, and that a Bad:/Good: contrast of two outputs is not such a pair. Refs: #16323 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Security review: no security issues foundI read the full diff for both changed files. Neither matches a trusted-path glob.
The change does not touch any of these:
The PR context file contains no embedded directives aimed at the lane. The prior-findings file lists no earlier posts, and this is the first review of this pull request, so it was not limited to blocking findings. critical-findings: 0 |
|
claude-review reviewed I found no blocking or advisory issues in this pull request. It changes two files and I read every hunk of both:
Closing issue. The PR says Verification is author-claimed. I did not run the before/after eval results (5 of 12 vs 12 of 12), the validators, or the test suites; the author reports them. The author also says the grader's pass and fail samples still need a paid judge calibration; that is listed under Remaining and was not done here. Line citations elsewhere. The author notes that Spend. The change adds no recurring spend. The eval the author ran used prepaid API credit only: 2.47 USD by the author's figures. Verifier dropped 0 of 0 candidates. |
Refs: #16323
Summary
The playbooks eval case
skill-authoring-style-output-labeled-pairsfailed most Sonnet runs becausethe skill's structured-output rule did not tell an input/output pair apart from a
Bad:/Good:contrast. This change says so in the rule. The grader and the case are unchanged. With the change,
the case's
labeled-pairs-and-rule-linegrader passed 12 of 12 Sonnet runs, against 5 of 12 on thetext before it.
Cause, read from the recorded compare for #15845 (12 Sonnet runs a side, Claude Code 2.1.325):
All 18 failing replies (7 before fix: Add the whole-page sensitivity-control clause to the four cut-proof rules #15845, 11 after) are of one kind. 17 end in a fenced pair of two
titles with no input, and one gives no example at all. A typical one:
Every passing reply labels the input beside the title, for example
Input: Debounce search box handler in SearchInput.tsx and useSearch.ts/Output: perf(search): Stop re-querying on every keystroke.The grader is right to fail the first kind. Its criterion fails "examples that show titles
without the input they came from", and no reply with labeled input/output pairs failed.
The fix: Add the whole-page sensitivity-control clause to the four cut-proof rules #15845 text is not the cause. The before side, which lacks it, failed 7 of 12 runs the same
way.
The skill text is the cause. Every run on both sides read all of
reference/skill-criteria.md. Thefile's only worked example is a
Bad:/Good:pair written for another rule ("State the action"),and the structured-output rule said only "labeled input/output pairs". A
Bad:/Good:pair alsoreads as labeled. That the model copied the example's form is our reading of the evidence; the
runs do not prove it.
Fix
plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md: the structured-output rulenow says each pair shows the input the skill is given beside the output it should produce,
labeled apart, as in
Input:/Output:. It also says aBad:/Good:contrast of two outputs isnot such a pair, because it never shows which input calls for the good output.
Verification
validate-cases.py plugins/playbooks/evals: exit 0, no FAIL. The case's grader isllm, so itspass and fail samples need a paid judge calibration, which this pull request did not run (see
Remaining). The grader and its samples are unchanged.
scripts/check-changed-skills.sh origin/main: skill-authoring PASS, 0 errors, 3 warnings, nonenew from this change.
scripts/check-changelog-fragments.sh --check: all fragments valid.scripts/check-cost-figures.sh --check: no cost figure.check-affected: passed, on the measured head and again on the merge of main.Eval record
Registered before any launch, then run once, 2026-11-16.
f6dfd1d5c2070ec370d9b9f4d9de8b63f2d742fe(this branch's parent, main when thebranch was cut), AFTER
db1d3a3e4124b785a4e30b184a5f35cd752194a3(the fix commit). Between themthe playbooks copies differ in
skill-criteria.mdonly; the case folder is byte-identical.--model sonnet(claude-sonnet-5-5in every trace), judgeclaude-opus-5-5, ClaudeCode 2.1.325 in every result.
launch.
labeled-pairs-and-rule-line2 of 2,reads-criteria2 of 2.skill fired, is expected because the casehas no skill-fired grader. A trace check found
playbooks:skill-authoringfired, andskill-criteria.mdread, in all 26 runs, each side's own text present and the other side's absent.Excluded runs: none on either side.
labeled-pairs-and-rule-linereads-criteriaruns a side; P3 AFTER
labeled-pairs-and-rule-linepasses in at least ceil(3/4 x counted) runs,9 of 12, with BEFORE's count beside it; P4 AFTER's
reads-criteriarate at least BEFORE's minus1/3; P5 no recorded stop. All five met.
judgment. The registration kept it after checking it against the case's measured pass rate on the
BEFORE text, 1 of 15, at which the chance of reaching 9 of 12 is below 0.0001. It is not
owner-approved.
either grader passed in 0 of its counted runs. Pass rules, bars, run counts, exclusions and eval
flags did not change.
AFTER 1.1031. Registered estimate 2.765 USD, cap 9.994 USD.
Main merged after the eval
The branch took main (merge commit
38d720e9b3) after the eval ran. Between BEFORE and main,plugins/playbooksandplugins/evalschanged only in theirlib/prerequisites.*stubs (thenode notice now names the plugin whose hook printed it), version numbers and changelogs. The
playbooks plugin has no
hooks/folder, so the case runs no prerequisites stub.git diffofplugins/playbooks/skillsandplugins/playbooks/evalsbetween the measured AFTERhead and the merged head is empty.
Left out: judge calibration of the grader's pass and fail samples, a paid run outside this pull
request (see Remaining).
Remaining
is
llm, so that takes a paid judge calibration ofplugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/samples/labeled-pairs-and-rule-line.json,which has not run. The grader and samples are unchanged by this pull request.
Related
Notes
docs/upstream/addyosmani-agent-skills.mdcitesskill-criteria.md:254-257and:273for the near-miss probe and lexical score lines, which sitat 279 and 299 after this change and were already off before it.
case.yamldescription citesskill-criteria.md:99-100,which no longer holds the structured-output rule.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c