Skip to content

fix(playbooks): say style example pairs show their input, not a Bad/Good contrast - #16574

Merged
kyle-sexton merged 2 commits into
mainfrom
fix/16323-skill-authoring-labeled-pairs
Nov 16, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
fix/16323-skill-authoring-labeled-pairs

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #16323

Summary

The playbooks eval case skill-authoring-style-output-labeled-pairs failed most Sonnet runs because
the skill's structured-output rule did not tell an input/output pair apart from a Bad:/Good:
contrast. This change says so in the rule. The grader and the case are unchanged. With the change,
the case's labeled-pairs-and-rule-line grader passed 12 of 12 Sonnet runs, against 5 of 12 on the
text before it.

Cause, read from the recorded compare for #15845 (12 Sonnet runs a side, Claude Code 2.1.325):

  • All 18 failing replies (7 before fix: Add the whole-page sensitivity-control clause to the four cut-proof rules #15845, 11 after) are of one kind. 17 end in a fenced pair of two
    titles with no input, and one gives no example at all. A typical one:

    Bad:  fix(auth): Updated session.ts and token.ts.
    Good: fix(auth): stop sessions expiring during active use
    
  • Every passing reply labels the input beside the title, for example
    Input: Debounce search box handler in SearchInput.tsx and useSearch.ts /
    Output: perf(search): Stop re-querying on every keystroke.

  • The grader is right to fail the first kind. Its criterion fails "examples that show titles
    without the input they came from", and no reply with labeled input/output pairs failed.

  • The fix: Add the whole-page sensitivity-control clause to the four cut-proof rules #15845 text is not the cause. The before side, which lacks it, failed 7 of 12 runs the same
    way.

  • The skill text is the cause. Every run on both sides read all of reference/skill-criteria.md. The
    file's only worked example is a Bad:/Good: pair written for another rule ("State the action"),
    and the structured-output rule said only "labeled input/output pairs". A Bad:/Good: pair also
    reads as labeled. That the model copied the example's form is our reading of the evidence; the
    runs do not prove it.

Fix

  • plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md: the structured-output rule
    now says each pair shows the input the skill is given beside the output it should produce,
    labeled apart, as in Input:/Output:. It also says a Bad:/Good: contrast of two outputs is
    not such a pair, because it never shows which input calls for the good output.
  • A patch changelog fragment for playbooks.

Verification

  • validate-cases.py plugins/playbooks/evals: exit 0, no FAIL. The case's grader is llm, so its
    pass and fail samples need a paid judge calibration, which this pull request did not run (see
    Remaining). The grader and its samples are unchanged.
  • scripts/check-changed-skills.sh origin/main: skill-authoring PASS, 0 errors, 3 warnings, none
    new from this change.
  • markdownlint on the two touched files: 0 issues.
  • scripts/check-changelog-fragments.sh --check: all fragments valid.
  • scripts/check-cost-figures.sh --check: no cost figure.
  • The playbooks plugin's 9 test suites: all passed.
  • Pre-push check-affected: passed, on the measured head and again on the merge of main.

Eval record

Registered before any launch, then run once, 2026-11-16.

  • Versions: BEFORE f6dfd1d5c2070ec370d9b9f4d9de8b63f2d742fe (this branch's parent, main when the
    branch was cut), AFTER db1d3a3e4124b785a4e30b184a5f35cd752194a3 (the fix commit). Between them
    the playbooks copies differ in skill-criteria.md only; the case folder is byte-identical.
  • Model: --model sonnet (claude-sonnet-5-5 in every trace), judge claude-opus-5-5, Claude
    Code 2.1.325 in every result.
  • Design: a 2-run AFTER smoke, then 12 runs BEFORE and 12 runs AFTER, run back to back in one
    launch.
  • Smoke: CLEAN. 2 runs, no error, no exclusion; labeled-pairs-and-rule-line 2 of 2,
    reads-criteria 2 of 2.
  • Validity: every result file VALID; its one warning, skill fired, is expected because the case
    has no skill-fired grader. A trace check found playbooks:skill-authoring fired, and
    skill-criteria.md read, in all 26 runs, each side's own text present and the other side's absent.
    Excluded runs: none on either side.
Grader BEFORE AFTER
labeled-pairs-and-rule-line 5 of 12 12 of 12
reads-criteria 12 of 12 12 of 12
  • Pass rules (fixed before launch): P1 both files hold 12 launched runs; P2 at least 8 counted
    runs a side; P3 AFTER labeled-pairs-and-rule-line passes in at least ceil(3/4 x counted) runs,
    9 of 12, with BEFORE's count beside it; P4 AFTER's reads-criteria rate at least BEFORE's minus
    1/3; P5 no recorded stop. All five met.
  • Verdict: MET.
  • The bar: 9 of 12 is the issue's number, which the issue marks as the filing session's
    judgment. The registration kept it after checking it against the case's measured pass rate on the
    BEFORE text, 1 of 15, at which the chance of reaching 9 of 12 is below 0.0001. It is not
    owner-approved.
  • Amendment (dated, before any launch): the compare launcher was made to refuse a smoke in which
    either grader passed in 0 of its counted runs. Pass rules, bars, run counts, exclusions and eval
    flags did not change.
  • Cost: 2.4691 USD for 26 runs, judge included (0.3461 USD of it): smoke 0.2376, BEFORE 1.1284,
    AFTER 1.1031. Registered estimate 2.765 USD, cap 9.994 USD.

Main merged after the eval

The branch took main (merge commit 38d720e9b3) after the eval ran. Between BEFORE and main,
plugins/playbooks and plugins/evals changed only in their lib/prerequisites.* stubs (the
node notice now names the plugin whose hook printed it), version numbers and changelogs. The
playbooks plugin has no hooks/ folder, so the case runs no prerequisites stub.
git diff of plugins/playbooks/skills and plugins/playbooks/evals between the measured AFTER
head and the merged head is empty.

Left out: judge calibration of the grader's pass and fail samples, a paid run outside this pull
request (see Remaining).

Remaining

  • Done-when item 2 also asks that the case's pass sample pass and its fail sample fail. The grader
    is llm, so that takes a paid judge calibration of
    plugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/samples/labeled-pairs-and-rule-line.json,
    which has not run. The grader and samples are unchanged by this pull request.

Related

Notes

  • Pre-existing: docs/upstream/addyosmani-agent-skills.md cites
    skill-criteria.md:254-257 and :273 for the near-miss probe and lexical score lines, which sit
    at 279 and 299 after this change and were already off before it.
  • Pre-existing: this case's case.yaml description cites skill-criteria.md:99-100,
    which no longer holds the structured-output rule.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c

kyle-sexton and others added 2 commits November 16, 2026 01:36
…ood contrast

The structured-output rule in skill-authoring's skill-criteria.md asked
for labeled input/output pairs, but every failing Sonnet run of the
skill-authoring-style-output-labeled-pairs case printed a Bad:/Good:
pair of two titles with no input, the shape of the file's own worked
example for another rule. The rule now says each pair shows the input
beside the output, labeled apart, and that a Bad:/Good: contrast of two
outputs is not such a pair.

Refs: #16323

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 17:04
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T17:05:34.491938Z 38d720e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 38d720e9b314fc4aa34edcce4c33c9530dd558d4: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

I read the full diff for both changed files. Neither matches a trusted-path glob.

  • .changes/playbooks/fix-16323-skill-authoring-labeled-pairs-a7a6b6e4.md: a new patch changelog fragment. It is prose only.
  • plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md: adds guidance to the structured-output rule. Each style example pair should show the input it came from beside the output, labeled apart (Input:/Output:). A Bad:/Good: contrast of two outputs does not count as such a pair. This adds to a standing instruction and deletes nothing. Under the instruction-surface lens, the change removes or weakens no guardrail and touches no protected class. The register file was not needed, because no rule was removed.

The change does not touch any of these:

  • a trust control or merge hold
  • a plugin manifest
  • a hook or workflow
  • token or secret handling
  • executable code

The PR context file contains no embedded directives aimed at the lane. The prior-findings file lists no earlier posts, and this is the first review of this pull request, so it was not limited to blocking findings.

critical-findings: 0

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 38d720e9b314fc4aa34edcce4c33c9530dd558d4: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

I found no blocking or advisory issues in this pull request. It changes two files and I read every hunk of both:

  • plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:161-169 now says each style example pair shows the input the skill is given and the output it should produce from it. The two are labeled apart, as in Input:/Output:, and a Bad:/Good: contrast of two outputs does not count as a pair.
    • I read the rule against the text around it. It does not contradict the "State the action" example at lines 125-130, which is a Bad:/Good: contrast for a different rule. Nor does it contradict the Bad:/Good: description examples in SKILL.md:104-106; those also serve a different rule.
    • The new text narrows the existing phrase "labeled input/output pairs" and does not replace it, so the rest of the rule still reads correctly.
  • .changes/playbooks/fix-16323-skill-authoring-labeled-pairs-a7a6b6e4.md is a patch fragment. Patch suits a wording fix in reference guidance, and the fragment describes the change accurately.

Closing issue. The PR says Refs: #16323 and closes no issue: the context file has no closing-issue section. So I checked no acceptance criteria.

Verification is author-claimed. I did not run the before/after eval results (5 of 12 vs 12 of 12), the validators, or the test suites; the author reports them. The author also says the grader's pass and fail samples still need a paid judge calibration; that is listed under Remaining and was not done here.

Line citations elsewhere. The author notes that docs/upstream/addyosmani-agent-skills.md cites skill-criteria.md line numbers that were already wrong before this change. This change shifts lines after 165 by 3 more. Those citations were already broken, so I did not report this as a finding.

Spend. The change adds no recurring spend. The eval the author ran used prepaid API credit only: 2.47 USD by the author's figures.

Verifier dropped 0 of 0 candidates.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 566a676 Nov 16, 2026
77 checks passed
@kyle-sexton
kyle-sexton deleted the fix/16323-skill-authoring-labeled-pairs branch November 16, 2026 17:19
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant