From 541179c4ba131d3991beba480824f1221f1eb2c6 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:15:25 -0500 Subject: [PATCH 1/2] feat(playbooks): add reference-independence and config-before-generator skill rules skill-authoring's criteria gain two rules: - Reference files: a reference works without the skill's steps. Another skill or a person loads it through the skill's invocation or a documented action, never by path, and acts on it without running the step sequence, so it names its own inputs. - Body: a shipped skill generates a repository-specific skill (as /app-control:map does) only where repository configuration cannot express the difference; when config can carry it, keep one shipped skill and read the config. Each rule has an eval case under plugins/playbooks/evals. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- ...-reference-and-generator-rules-3d9f1b72.md | 7 ++++ .../case.yaml | 9 +++++ .../graders/reads-config-not-generates.md | 13 +++++++ .../graders/reads-criteria.md | 6 ++++ .../prompt.md | 9 +++++ .../samples/reads-config-not-generates.json | 18 ++++++++++ .../samples/reads-criteria.json | 34 +++++++++++++++++++ .../case.yaml | 9 +++++ .../graders/reads-criteria.md | 6 ++++ .../graders/stands-on-its-own.md | 13 +++++++ .../prompt.md | 13 +++++++ .../samples/reads-criteria.json | 34 +++++++++++++++++++ .../samples/stands-on-its-own.json | 26 ++++++++++++++ .../reference/skill-criteria.md | 11 ++++++ 14 files changed, 208 insertions(+) create mode 100644 .changes/playbooks/feat-skill-authoring-reference-and-generator-rules-3d9f1b72.md create mode 100644 plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml create mode 100644 plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md create mode 100644 plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-criteria.md create mode 100644 plugins/playbooks/evals/skill-authoring-config-before-generated-skill/prompt.md create mode 100644 plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json create mode 100644 plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-criteria.json create mode 100644 plugins/playbooks/evals/skill-authoring-reference-works-without-steps/case.yaml create mode 100644 plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/reads-criteria.md create mode 100644 plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/stands-on-its-own.md create mode 100644 plugins/playbooks/evals/skill-authoring-reference-works-without-steps/prompt.md create mode 100644 plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/reads-criteria.json create mode 100644 plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/stands-on-its-own.json diff --git a/.changes/playbooks/feat-skill-authoring-reference-and-generator-rules-3d9f1b72.md b/.changes/playbooks/feat-skill-authoring-reference-and-generator-rules-3d9f1b72.md new file mode 100644 index 00000000000..f92e4f11f0f --- /dev/null +++ b/.changes/playbooks/feat-skill-authoring-reference-and-generator-rules-3d9f1b72.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Added + +- skill-authoring's criteria add two rules: a reference file works without the skill's steps, so another skill or a person can load it through the skill's invocation and act on it without running the sequence; and a shipped skill generates a repository-specific skill only where repository configuration cannot express the difference, reading config when it can. diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml new file mode 100644 index 00000000000..561a763d6c2 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml @@ -0,0 +1,9 @@ +schema_version: "1.1" +name: skill-authoring-config-before-generated-skill +description: "Config before a generated skill: a setup action that writes a per-repository copy of the skill whose only difference is the linter's name is replaced by one shipped skill that reads the name from repository config. Written from plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, Body." +tags: [skill-authoring, composition] +expected_outcome: "The rewritten section no longer generates a per-repository skill; setup records the linter name in the repository's config, and the shipped skill reads it from there on each run." +execution: + allowed_tools: [Read, Skill] + max_turns: 20 + timeout_seconds: 600 diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md new file mode 100644 index 00000000000..aa37b61f67b --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md @@ -0,0 +1,13 @@ +--- +type: llm +arm: both +--- +The section being rewritten detected the repository's linter, then wrote a copy of the whole skill into the repository's `.claude/skills/lint-here/` with the linter's name filled in, and told the user to use that copy from then on. The copy differs from the shipped skill only in the linter's name, which repository configuration can hold. + +PASS only if all three hold in the rewritten section: + +1. It no longer writes a copy of the skill (or any generated skill) into the repository, and no longer sends the user to a `/lint-here` skill. +2. It records the detected linter name in repository configuration (a config file or config entry the skill reads), not in a skill file. +3. The shipped skill itself reads that configured name on later runs; detection still happens when no name is configured. + +FAIL if a per-repository skill is still generated, in full or as an option offered beside config; if the linter name is stored only in a generated skill; or if the section drops detection without saying where the name comes from. diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-criteria.md b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-criteria.md new file mode 100644 index 00000000000..20212fb171b --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-criteria.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Read +arm: both +input_match: '"file_path"\s*:\s*"[^"]*skill-authoring/reference/skill-criteria\.md"' +--- diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/prompt.md b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/prompt.md new file mode 100644 index 00000000000..7e2c655989c --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/prompt.md @@ -0,0 +1,9 @@ +/playbooks:skill-authoring. Rewrite this section of my plugin skill's SKILL.md so it meets our skill criteria. Print only the rewritten section in your reply; do not write any file. + +## setup + +1. Detect the repository's linter from its manifests (`eslint`, `ruff` or `golangci-lint`). +2. Copy this SKILL.md to `.claude/skills/lint-here/SKILL.md` in the repository, replacing every `` with the detected name. +3. Tell the user to run `/lint-here` from now on instead of this skill. + +Author's note, not part of the section: the generated copy is identical to this skill apart from the linter's name. Generating it gives each repository its own skill, so nothing has to look anything up at run time. diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json new file mode 100644 index 00000000000..a90a6126a83 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "## setup\n\n1. Read the linter name from `.claude/lint.md` (`linter:`). If it is set, stop: setup is done.\n2. Otherwise detect the linter from the repository's manifests (`eslint`, `ruff` or `golangci-lint`) and write it to `.claude/lint.md` as `linter: `.\n\nEvery run of this skill reads `linter:` from that file.", + "why": "stores the name in config, the shipped skill reads it, no generated copy" + } + ], + "fail": [ + { + "answer": "## setup\n\n1. Detect the repository's linter from its manifests.\n2. Write `.claude/skills/lint-here/SKILL.md`, a copy of this skill with `` replaced by the detected name.\n3. Tell the user to run `/lint-here` from now on.", + "why": "still generates a per-repository skill" + }, + { + "answer": "## setup\n\n1. Detect the repository's linter and write it to `.claude/lint.md` as `linter: `.\n2. Optionally also generate `.claude/skills/lint-here/SKILL.md` with the name filled in, for repositories that prefer their own skill.", + "why": "keeps the generated skill as an option beside config" + } + ] +} diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-criteria.json b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-criteria.json new file mode 100644 index 00000000000..8b2e55e03a4 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-criteria.json @@ -0,0 +1,34 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "playbooks:skill-authoring" + } + }, + { + "tool": "Read", + "input": { + "file_path": "/root/.claude/plugins/cache/melodic-software/playbooks/1.0.0/skills/skill-authoring/reference/skill-criteria.md" + } + } + ], + "why": "the skill fired and its criteria reference was read" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "playbooks:skill-authoring" + } + } + ], + "why": "criteria never read" + } + ] +} diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/case.yaml b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/case.yaml new file mode 100644 index 00000000000..a741a983253 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/case.yaml @@ -0,0 +1,9 @@ +schema_version: "1.1" +name: skill-authoring-reference-works-without-steps +description: "A reference works without the steps: a reference file that leans on a value an earlier step set is rewritten to name its own input, so another skill can load it and act on it without running the sequence. Written from plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, Reference files." +tags: [skill-authoring, progressive-disclosure] +expected_outcome: "The rewritten reference states where the ticket key comes from as its own input (given by the caller or read by the reference itself), keeps the branch-name rule, and no longer refers to step 2, step 3, or a value set earlier in the sequence." +execution: + allowed_tools: [Read, Skill] + max_turns: 20 + timeout_seconds: 600 diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/reads-criteria.md b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/reads-criteria.md new file mode 100644 index 00000000000..20212fb171b --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/reads-criteria.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Read +arm: both +input_match: '"file_path"\s*:\s*"[^"]*skill-authoring/reference/skill-criteria\.md"' +--- diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/stands-on-its-own.md b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/stands-on-its-own.md new file mode 100644 index 00000000000..e3499dd2f40 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/stands-on-its-own.md @@ -0,0 +1,13 @@ +--- +type: llm +arm: both +--- +The reference file being rewritten told the reader to take "the `$KEY` you looked up in step 2", build a branch name `/-` (lowercase; type `feat`, `fix` or `docs`), and "then continue with step 4". It depends on the owning skill's step sequence, and another skill wants to use the same naming rule. + +PASS only if all three hold in the rewritten file: + +1. It names the ticket key as the file's own input: the caller supplies it, or the file says where to get it. It does not say it was set or looked up in an earlier step. +2. It refers to no step of the owning skill's sequence: no "step 2", "step 3", "step 4", "continue with", or "the value you set earlier". +3. It keeps the naming rule: the `/-` shape, lowercase, and the three allowed types. + +FAIL if any step reference remains, if the key still comes from an earlier step, or if the naming rule is dropped or changed. diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/prompt.md b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/prompt.md new file mode 100644 index 00000000000..82f9561999f --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/prompt.md @@ -0,0 +1,13 @@ +/playbooks:skill-authoring. Rewrite this reference file of my skill so it meets our skill criteria. Print only the rewritten file in your reply; do not write any file. + +`reference/branch-names.md`: + +```markdown +# Branch names + +Now take the `$KEY` you looked up in step 2 and build the branch name from it: +`/-`, all lowercase, where `` is `feat`, `fix` or `docs`. +Then continue with step 4. +``` + +Author's note, not part of the file: the skill's SKILL.md runs five steps, and step 2 looks up the ticket key. This file is only ever read during step 3, so it can rely on step 2 having run. Our release skill also wants to name branches the same way. diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/reads-criteria.json b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/reads-criteria.json new file mode 100644 index 00000000000..8b2e55e03a4 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/reads-criteria.json @@ -0,0 +1,34 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "playbooks:skill-authoring" + } + }, + { + "tool": "Read", + "input": { + "file_path": "/root/.claude/plugins/cache/melodic-software/playbooks/1.0.0/skills/skill-authoring/reference/skill-criteria.md" + } + } + ], + "why": "the skill fired and its criteria reference was read" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "playbooks:skill-authoring" + } + } + ], + "why": "criteria never read" + } + ] +} diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/stands-on-its-own.json b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/stands-on-its-own.json new file mode 100644 index 00000000000..7669ab243bf --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/stands-on-its-own.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "# Branch names\n\nInput: the ticket key, given by the caller.\n\nBuild the branch name as `/-`, all lowercase, where `` is `feat`, `fix` or `docs`.", + "why": "names its own input, no step references, keeps the rule" + }, + { + "answer": "# Branch names\n\nGet the ticket key from the issue the work closes. Name the branch `/-`, lowercase; `` is one of `feat`, `fix`, `docs`.", + "why": "says where the key comes from without leaning on the sequence" + } + ], + "fail": [ + { + "answer": "# Branch names\n\nTake the `$KEY` you looked up in step 2 and build `/-`, all lowercase, where `` is `feat`, `fix` or `docs`.", + "why": "still leans on step 2" + }, + { + "answer": "# Branch names\n\nInput: the ticket key.\n\nBuild `/-`, all lowercase, where `` is `feat`, `fix` or `docs`. Then continue with step 4.", + "why": "still hands back into the step sequence" + }, + { + "answer": "# Branch names\n\nInput: the ticket key.\n\nName the branch after the ticket.", + "why": "drops the naming rule" + } + ] +} diff --git a/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md b/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md index 946bfdfa109..23a1d6f9c88 100644 --- a/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md +++ b/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md @@ -150,6 +150,12 @@ except the pointer swap that **Cutting text** exempts ([Measurement](#measuremen after the owning skill changes, so the two then disagree and nothing flags it. When the owning skill may not be installed, one line says what this skill does without it; that line is not a second copy of the procedure. +- **Config before a generated skill.** A shipped skill generates a repository-specific skill (as + `/app-control:map` generates a project's control skill) only where repository configuration + cannot express the difference. When config can carry it, such as a tool name the skill reads + from the repo's config (`docs/contracts/config-cascade/README.md`), keep one shipped skill and + read the config. Why: a generated skill is a copy that stops receiving the shipped skill's + fixes, while config is read by the current skill on every run. - **SKILL.md is a router, under 5,000 tokens.** It is the hub: it holds the standing rules and routes each action to its own spoke under `actions/`, one file per action, so a run loads the router and the one action it needs. The cap is a house setting; `/skill-quality:check` estimates the count at 4 @@ -209,6 +215,11 @@ build tool) takes these steps in order: practices in [Sources](#sources)). - Each pointer says what the file holds and when to read it; a bare "see X" is a missed connection. +- **A reference works without the steps.** Another skill or a person loads it through this + skill's invocation or a documented action, never by its path, and acts on it without running + the step sequence. So a reference names its own inputs and never leans on a value an earlier + step set or on steps having run. Why: a reference that only makes sense mid-sequence cannot be + reused, and a caller after one rule has to run steps it does not need. - A spoke is read, not rendered, so no substitution variable resolves in it: cite `/scripts/`. `scripts/check-spoke-plugin-root.sh` gates the plugin-root and skill-dir variables in this marketplace. From 9859cb9713c50adfa3cc45b4d112857a8887cf1b Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:17:03 -0500 Subject: [PATCH 2/2] test(playbooks): accept live repository config in the generator eval grader The config-before-generated-skill grader required an explicit config entry and failed rewrites that read the linter from the repository's own linter config on every run, which the rule also allows. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC --- .../case.yaml | 2 +- .../graders/reads-config-not-generates.md | 9 ++++----- .../samples/reads-config-not-generates.json | 6 +++++- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml index 561a763d6c2..8d2aaf21341 100644 --- a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml @@ -2,7 +2,7 @@ schema_version: "1.1" name: skill-authoring-config-before-generated-skill description: "Config before a generated skill: a setup action that writes a per-repository copy of the skill whose only difference is the linter's name is replaced by one shipped skill that reads the name from repository config. Written from plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, Body." tags: [skill-authoring, composition] -expected_outcome: "The rewritten section no longer generates a per-repository skill; setup records the linter name in the repository's config, and the shipped skill reads it from there on each run." +expected_outcome: "The rewritten section no longer generates a per-repository skill; the shipped skill gets the linter on each run from repository configuration, either the repository's own linter config or an entry setup records there." execution: allowed_tools: [Read, Skill] max_turns: 20 diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md index aa37b61f67b..a19abdc446b 100644 --- a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md @@ -2,12 +2,11 @@ type: llm arm: both --- -The section being rewritten detected the repository's linter, then wrote a copy of the whole skill into the repository's `.claude/skills/lint-here/` with the linter's name filled in, and told the user to use that copy from then on. The copy differs from the shipped skill only in the linter's name, which repository configuration can hold. +The section being rewritten detected the repository's linter, then wrote a copy of the whole skill into the repository's `.claude/skills/lint-here/` with the linter's name filled in, and told the user to use that copy from then on. The copy differs from the shipped skill only in the linter's name, which repository configuration can carry. -PASS only if all three hold in the rewritten section: +PASS only if both hold in the rewritten section: 1. It no longer writes a copy of the skill (or any generated skill) into the repository, and no longer sends the user to a `/lint-here` skill. -2. It records the detected linter name in repository configuration (a config file or config entry the skill reads), not in a skill file. -3. The shipped skill itself reads that configured name on later runs; detection still happens when no name is configured. +2. The shipped skill itself gets the linter on each run from repository configuration: the repository's own linter config files and manifests, or a config entry the skill records there and reads back. Either counts. -FAIL if a per-repository skill is still generated, in full or as an option offered beside config; if the linter name is stored only in a generated skill; or if the section drops detection without saying where the name comes from. +FAIL if a per-repository skill is still generated, in full or as an option offered beside config; if the linter name lives only in a generated skill; or if the section no longer says where the linter comes from. diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json index a90a6126a83..9c17e233c35 100644 --- a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json @@ -2,7 +2,11 @@ "pass": [ { "answer": "## setup\n\n1. Read the linter name from `.claude/lint.md` (`linter:`). If it is set, stop: setup is done.\n2. Otherwise detect the linter from the repository's manifests (`eslint`, `ruff` or `golangci-lint`) and write it to `.claude/lint.md` as `linter: `.\n\nEvery run of this skill reads `linter:` from that file.", - "why": "stores the name in config, the shipped skill reads it, no generated copy" + "why": "records the name in config the shipped skill reads, no generated copy" + }, + { + "answer": "## setup\n\nOn every run, find the linter from the repository's own config: `eslint.config.*` or `.eslintrc*`, `[tool.ruff]` in `pyproject.toml`, or `.golangci.yml`. Don't write a copy of this skill into the repository; a copy stops getting this skill's fixes.", + "why": "the shipped skill reads the repository's linter config live, no generated copy" } ], "fail": [