Skip to content

fix(playbooks): skill-authoring-style-output-labeled-pairs fails most Sonnet runs (examples show titles without their input) #16323

Closed

Description

@kyle-sexton

Problem

The playbooks eval case skill-authoring-style-output-labeled-pairs fails most of the time on Sonnet, before and after #15845. In the recorded compare (12 runs per side), its judge grader labeled-pairs-and-rule-line passed 5 of 12 before the change and 1 of 12 after; reads-criteria passed 12 of 12 on both sides.

The failing replies give Bad: / Good: pairs of two titles and never show the input the titles came from. The grader fails exactly that: "examples that show titles without the input they came from". Before the change 7 of 12 replies had that shape; after it, 10 of 12, and one gave no example pair.

Likely cause (UNVERIFIED)

Not established. Two readings to check:

  • The skill-authoring text does not tell the model clearly enough to label the input in each example pair, so the miss is the skill's on either version.
  • The text fix: Add the whole-page sensitivity-control clause to the four cut-proof rules #15845 added to reference/skill-criteria.md and reference/rewrite-a-plugin.md (read by all 12 after-side runs) crowds out the example guidance. The fall of 4 passes in 12 is one more than the registered allowance of 3; it is a small sample.

Done when

The numeric bars in this list (runs per side, pass counts, allowances) are the filing session's judgment. They are not measured and not owner-approved. Whoever registers the eval sets them from the case's own measured pass rate and says so in the registration.

  • The full last message of at least four failing runs on main is read against the grader, and the cause is stated with quotes: the skill's example guidance, the added text, or the grader.
  • The cause is fixed in its own pull request (the skill text, or the grader or its samples), and the case's pass sample passes and its fail sample fails.
  • A before and after eval of this case on Sonnet, 12 runs per side, both sides run together, is in that pull request's body: after the change labeled-pairs-and-rule-line passes in at least 9 of 12 runs, with the before count beside it.

Out of scope

Reverting #15845's clause; the owner accepted the recorded loss on this case (#15707, decision Q170).

Related

Refs: #15707 (the record and the decision), #15845.

session: GitHub Support

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  2. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: the playbooks eval case skill-authoring-style-output-labeled-pairs fails on most Sonnet runs. The labeled-pairs-and-rule-line grader passed 1 of 12 after #15845, against 5 of 12 before. The failing replies show Bad/Good title pairs without the input each title came from. The cause is not established. It is either the skill's example guidance, the text #15845 added, or the grader. Reading at least four failing runs against the grader should settle that before any fix.

    Nothing is missing from the issue. I found no open duplicate and no merged pull request that already fixes this. #15845 is the change it references, and reverting it is out of scope.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  4. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of this case on Sonnet, 12 runs per side, both sides ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Make the skill-authoring example guidance (or the grader and its samples) get the model to label the input in each Bad/Good title pair, once the cause is read from failing runs.

    Related work

    Open questions

    1. Where do the full last messages of the failing runs live, so an unattended run can read at least four against the grader? They are not in the checkout.
      RECOMMENDED: Attach or link the recorded compare outputs in the issue; otherwise a person does done-when item 1 and posts the quoted cause, and the fix is filed after that. Basis: judgment: a repo-wide search for the case name finds only case.yaml and CHANGELOG.md, no run transcripts.
    2. Which fix is wanted if the cause is the skill's guidance? The skill text already says 'make them labeled input/output pairs and add a line naming the rule' at reference/skill-criteria.md:160, yet runs still print title-only Bad/Good pairs.
      RECOMMENDED: Add an explicit instruction in reference/skill-criteria.md near line 160 that each example pair shows the input it came from, for example an Input: line above Bad:/Good:, and leave the grader unchanged. Basis: plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:160
    3. Is the grader too strict if the cause is the grader? It accepts only pairs where input and output are told apart and fails Bad:/Good: title pairs with no input. The prompt at prompt.md:5 gives only a style rule and no input, so Bad:/Good: pairs are a reasonable reading of it.
      RECOMMENDED: Keep the grader, since the skill criteria ask for input/output pairs and the grader enforces them. Change it only if the read runs show valid labeled pairs failing. Basis: plugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/graders/labeled-pairs-and-rule-line.md:5
    4. Who sets the pass bar of 9 of 12 after the change? The issue says it is the filing session's judgment and not owner-approved, and item 3 needs a person to run the eval.
      RECOMMENDED: The owner or the eval registrant sets the bar from the case's measured pass rate and says so in the registration. Treat item 3 as a check a person runs. Basis: judgment: the issue's own note on the numeric bars.

    Sources read

    • /home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:157-162: structured-output rule asks for diverse canonical examples, labeled input/output pairs where style matters, and a line naming the rule.
    • /home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/graders/labeled-pairs-and-rule-line.md:5: PASS needs at least two labeled input/output pairs plus a rule line; FAIL on titles shown without their input.
    • /home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/prompt.md:5: the prompt gives a house-style rule for PR titles and no sample input.
    • /home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/playbooks/CHANGELOG.md:423: the case was added to check the skill criteria's structured-output rules and that the run read reference/skill-criteria.md.

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  6. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  7. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. The last owed item, the judge run of the grader's samples, is recorded; the issue closes on it.

    Record: labeled-pairs-and-rule-line judged against its own samples, on main

    • Registered before the run: the skill's documented judge calibration over samples/labeled-pairs-and-rule-line.json (2 must-pass and 3 must-fail samples), 3 runs per sample, 15 runs in one batch, judge claude-opus-5-5. Pass rule: every must-pass sample judged PASS and every must-fail sample judged FAIL in every counted run.
    • Result: MET. All 5 samples were judged as labeled in 3 of 3 runs, with every judge vote unanimous (45 of 45 votes). Agreement 15 of 15 runs.
    • Cost: 0.22 USD, judge included.

    Done-when

    1. Failing runs read against the grader, cause stated with quotes: fix(playbooks): say style example pairs show their input, not a Bad/Good contrast #16574.
    2. Cause fixed in its own pull request: fix(playbooks): say style example pairs show their input, not a Bad/Good contrast #16574; the pass samples pass and the fail samples fail: this record.
    3. Before and after on Sonnet, 12 runs a side: in fix(playbooks): say style example pairs show their input, not a Bad/Good contrast #16574's body, labeled-pairs-and-rule-line 5 of 12 before and 12 of 12 after, against the bar of 9.

    The eval harness is local and not committed, per this repository's rule for one-off eval tooling.

  8. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    needs-owner-decision

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions