You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(playbooks): skill-authoring-style-output-labeled-pairs fails most Sonnet runs (examples show titles without their input) #16323
The playbooks eval case skill-authoring-style-output-labeled-pairs fails most of the time on Sonnet, before and after #15845. In the recorded compare (12 runs per side), its judge grader labeled-pairs-and-rule-line passed 5 of 12 before the change and 1 of 12 after; reads-criteria passed 12 of 12 on both sides.
The failing replies give Bad: / Good: pairs of two titles and never show the input the titles came from. The grader fails exactly that: "examples that show titles without the input they came from". Before the change 7 of 12 replies had that shape; after it, 10 of 12, and one gave no example pair.
Likely cause (UNVERIFIED)
Not established. Two readings to check:
The skill-authoring text does not tell the model clearly enough to label the input in each example pair, so the miss is the skill's on either version.
The numeric bars in this list (runs per side, pass counts, allowances) are the filing session's judgment. They are not measured and not owner-approved. Whoever registers the eval sets them from the case's own measured pass rate and says so in the registration.
The full last message of at least four failing runs on main is read against the grader, and the cause is stated with quotes: the skill's example guidance, the added text, or the grader.
The cause is fixed in its own pull request (the skill text, or the grader or its samples), and the case's pass sample passes and its fail sample fails.
A before and after eval of this case on Sonnet, 12 runs per side, both sides run together, is in that pull request's body: after the change labeled-pairs-and-rule-line passes in at least 9 of 12 runs, with the before count beside it.
Out of scope
Reverting #15845's clause; the owner accepted the recorded loss on this case (#15707, decision Q170).
Related
Refs: #15707 (the record and the decision), #15845.
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the playbooks eval case skill-authoring-style-output-labeled-pairs fails on most Sonnet runs. The labeled-pairs-and-rule-line grader passed 1 of 12 after #15845, against 5 of 12 before. The failing replies show Bad/Good title pairs without the input each title came from. The cause is not established. It is either the skill's example guidance, the text #15845 added, or the grader. Reading at least four failing runs against the grader should settle that before any fix.
Nothing is missing from the issue. I found no open duplicate and no merged pull request that already fixes this. #15845 is the change it references, and reverting it is out of scope.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of this case on Sonnet, 12 runs per side, both sides ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Make the skill-authoring example guidance (or the grader and its samples) get the model to label the input in each Bad/Good title pair, once the cause is read from failing runs.
Where do the full last messages of the failing runs live, so an unattended run can read at least four against the grader? They are not in the checkout.
RECOMMENDED: Attach or link the recorded compare outputs in the issue; otherwise a person does done-when item 1 and posts the quoted cause, and the fix is filed after that. Basis: judgment: a repo-wide search for the case name finds only case.yaml and CHANGELOG.md, no run transcripts.
Which fix is wanted if the cause is the skill's guidance? The skill text already says 'make them labeled input/output pairs and add a line naming the rule' at reference/skill-criteria.md:160, yet runs still print title-only Bad/Good pairs.
RECOMMENDED: Add an explicit instruction in reference/skill-criteria.md near line 160 that each example pair shows the input it came from, for example an Input: line above Bad:/Good:, and leave the grader unchanged. Basis: plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:160
Is the grader too strict if the cause is the grader? It accepts only pairs where input and output are told apart and fails Bad:/Good: title pairs with no input. The prompt at prompt.md:5 gives only a style rule and no input, so Bad:/Good: pairs are a reasonable reading of it.
RECOMMENDED: Keep the grader, since the skill criteria ask for input/output pairs and the grader enforces them. Change it only if the read runs show valid labeled pairs failing. Basis: plugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/graders/labeled-pairs-and-rule-line.md:5
Who sets the pass bar of 9 of 12 after the change? The issue says it is the filing session's judgment and not owner-approved, and item 3 needs a person to run the eval.
RECOMMENDED: The owner or the eval registrant sets the bar from the case's measured pass rate and says so in the registration. Treat item 3 as a check a person runs. Basis: judgment: the issue's own note on the numeric bars.
Sources read
/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:157-162: structured-output rule asks for diverse canonical examples, labeled input/output pairs where style matters, and a line naming the rule.
/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/graders/labeled-pairs-and-rule-line.md:5: PASS needs at least two labeled input/output pairs plus a rule line; FAIL on titles shown without their input.
/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/prompt.md:5: the prompt gives a house-style rule for PR titles and no sample input.
/home/runner/work/claude-code-plugins/claude-code-plugins/.base/plugins/playbooks/CHANGELOG.md:423: the case was added to check the skill criteria's structured-output rules and that the run read reference/skill-criteria.md.
To answer, post a comment that starts with this line:
**Decision recorded.** <answer>
Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.
Decision recorded. The last owed item, the judge run of the grader's samples, is recorded; the issue closes on it.
Record: labeled-pairs-and-rule-line judged against its own samples, on main
Registered before the run: the skill's documented judge calibration over samples/labeled-pairs-and-rule-line.json (2 must-pass and 3 must-fail samples), 3 runs per sample, 15 runs in one batch, judge claude-opus-5-5. Pass rule: every must-pass sample judged PASS and every must-fail sample judged FAIL in every counted run.
Result: MET. All 5 samples were judged as labeled in 3 of 3 runs, with every judge vote unanimous (45 of 45 votes). Agreement 15 of 15 runs.
Activity
github-actions commentedon Nov 15, 2026
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the playbooks eval case
skill-authoring-style-output-labeled-pairsfails on most Sonnet runs. Thelabeled-pairs-and-rule-linegrader passed 1 of 12 after #15845, against 5 of 12 before. The failing replies show Bad/Good title pairs without the input each title came from. The cause is not established. It is either the skill's example guidance, the text #15845 added, or the grader. Reading at least four failing runs against the grader should settle that before any fix.Nothing is missing from the issue. I found no open duplicate and no merged pull request that already fixes this. #15845 is the change it references, and reverting it is out of scope.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of this case on Sonnet, 12 runs per side, both sides ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 15, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Make the skill-authoring example guidance (or the grader and its samples) get the model to label the input in each Bad/Good title pair, once the cause is read from failing runs.
Related work
Open questions
RECOMMENDED: Attach or link the recorded compare outputs in the issue; otherwise a person does done-when item 1 and posts the quoted cause, and the fix is filed after that. Basis: judgment: a repo-wide search for the case name finds only case.yaml and CHANGELOG.md, no run transcripts.
RECOMMENDED: Add an explicit instruction in reference/skill-criteria.md near line 160 that each example pair shows the input it came from, for example an Input: line above Bad:/Good:, and leave the grader unchanged. Basis: plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:160
RECOMMENDED: Keep the grader, since the skill criteria ask for input/output pairs and the grader enforces them. Change it only if the read runs show valid labeled pairs failing. Basis: plugins/playbooks/evals/skill-authoring-style-output-labeled-pairs/graders/labeled-pairs-and-rule-line.md:5
RECOMMENDED: The owner or the eval registrant sets the bar from the case's measured pass rate and says so in the registration. Treat item 3 as a check a person runs. Basis: judgment: the issue's own note on the numeric bars.
Sources read
To answer, post a comment that starts with this line:
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.kyle-sexton commentedon Nov 16, 2026
Decision recorded. The last owed item, the judge run of the grader's samples, is recorded; the issue closes on it.
Record:
labeled-pairs-and-rule-linejudged against its own samples, on mainsamples/labeled-pairs-and-rule-line.json(2 must-pass and 3 must-fail samples), 3 runs per sample, 15 runs in one batch, judgeclaude-opus-5-5. Pass rule: every must-pass sample judged PASS and every must-fail sample judged FAIL in every counted run.Done-when
labeled-pairs-and-rule-line5 of 12 before and 12 of 12 after, against the bar of 9.The eval harness is local and not committed, per this repository's rule for one-off eval tooling.