diff --git a/.changes/playbooks/feat-skill-authoring-reference-and-generator-rules-3d9f1b72.md b/.changes/playbooks/feat-skill-authoring-reference-and-generator-rules-3d9f1b72.md new file mode 100644 index 00000000000..f92e4f11f0f --- /dev/null +++ b/.changes/playbooks/feat-skill-authoring-reference-and-generator-rules-3d9f1b72.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Added + +- skill-authoring's criteria add two rules: a reference file works without the skill's steps, so another skill or a person can load it through the skill's invocation and act on it without running the sequence; and a shipped skill generates a repository-specific skill only where repository configuration cannot express the difference, reading config when it can. diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml new file mode 100644 index 00000000000..8d2aaf21341 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/case.yaml @@ -0,0 +1,9 @@ +schema_version: "1.1" +name: skill-authoring-config-before-generated-skill +description: "Config before a generated skill: a setup action that writes a per-repository copy of the skill whose only difference is the linter's name is replaced by one shipped skill that reads the name from repository config. Written from plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, Body." +tags: [skill-authoring, composition] +expected_outcome: "The rewritten section no longer generates a per-repository skill; the shipped skill gets the linter on each run from repository configuration, either the repository's own linter config or an entry setup records there." +execution: + allowed_tools: [Read, Skill] + max_turns: 20 + timeout_seconds: 600 diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md new file mode 100644 index 00000000000..a19abdc446b --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-config-not-generates.md @@ -0,0 +1,12 @@ +--- +type: llm +arm: both +--- +The section being rewritten detected the repository's linter, then wrote a copy of the whole skill into the repository's `.claude/skills/lint-here/` with the linter's name filled in, and told the user to use that copy from then on. The copy differs from the shipped skill only in the linter's name, which repository configuration can carry. + +PASS only if both hold in the rewritten section: + +1. It no longer writes a copy of the skill (or any generated skill) into the repository, and no longer sends the user to a `/lint-here` skill. +2. The shipped skill itself gets the linter on each run from repository configuration: the repository's own linter config files and manifests, or a config entry the skill records there and reads back. Either counts. + +FAIL if a per-repository skill is still generated, in full or as an option offered beside config; if the linter name lives only in a generated skill; or if the section no longer says where the linter comes from. diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-criteria.md b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-criteria.md new file mode 100644 index 00000000000..20212fb171b --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/graders/reads-criteria.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Read +arm: both +input_match: '"file_path"\s*:\s*"[^"]*skill-authoring/reference/skill-criteria\.md"' +--- diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/prompt.md b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/prompt.md new file mode 100644 index 00000000000..7e2c655989c --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/prompt.md @@ -0,0 +1,9 @@ +/playbooks:skill-authoring. Rewrite this section of my plugin skill's SKILL.md so it meets our skill criteria. Print only the rewritten section in your reply; do not write any file. + +## setup + +1. Detect the repository's linter from its manifests (`eslint`, `ruff` or `golangci-lint`). +2. Copy this SKILL.md to `.claude/skills/lint-here/SKILL.md` in the repository, replacing every `` with the detected name. +3. Tell the user to run `/lint-here` from now on instead of this skill. + +Author's note, not part of the section: the generated copy is identical to this skill apart from the linter's name. Generating it gives each repository its own skill, so nothing has to look anything up at run time. diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json new file mode 100644 index 00000000000..9c17e233c35 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-config-not-generates.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "## setup\n\n1. Read the linter name from `.claude/lint.md` (`linter:`). If it is set, stop: setup is done.\n2. Otherwise detect the linter from the repository's manifests (`eslint`, `ruff` or `golangci-lint`) and write it to `.claude/lint.md` as `linter: `.\n\nEvery run of this skill reads `linter:` from that file.", + "why": "records the name in config the shipped skill reads, no generated copy" + }, + { + "answer": "## setup\n\nOn every run, find the linter from the repository's own config: `eslint.config.*` or `.eslintrc*`, `[tool.ruff]` in `pyproject.toml`, or `.golangci.yml`. Don't write a copy of this skill into the repository; a copy stops getting this skill's fixes.", + "why": "the shipped skill reads the repository's linter config live, no generated copy" + } + ], + "fail": [ + { + "answer": "## setup\n\n1. Detect the repository's linter from its manifests.\n2. Write `.claude/skills/lint-here/SKILL.md`, a copy of this skill with `` replaced by the detected name.\n3. Tell the user to run `/lint-here` from now on.", + "why": "still generates a per-repository skill" + }, + { + "answer": "## setup\n\n1. Detect the repository's linter and write it to `.claude/lint.md` as `linter: `.\n2. Optionally also generate `.claude/skills/lint-here/SKILL.md` with the name filled in, for repositories that prefer their own skill.", + "why": "keeps the generated skill as an option beside config" + } + ] +} diff --git a/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-criteria.json b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-criteria.json new file mode 100644 index 00000000000..8b2e55e03a4 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-config-before-generated-skill/samples/reads-criteria.json @@ -0,0 +1,34 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "playbooks:skill-authoring" + } + }, + { + "tool": "Read", + "input": { + "file_path": "/root/.claude/plugins/cache/melodic-software/playbooks/1.0.0/skills/skill-authoring/reference/skill-criteria.md" + } + } + ], + "why": "the skill fired and its criteria reference was read" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "playbooks:skill-authoring" + } + } + ], + "why": "criteria never read" + } + ] +} diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/case.yaml b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/case.yaml new file mode 100644 index 00000000000..a741a983253 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/case.yaml @@ -0,0 +1,9 @@ +schema_version: "1.1" +name: skill-authoring-reference-works-without-steps +description: "A reference works without the steps: a reference file that leans on a value an earlier step set is rewritten to name its own input, so another skill can load it and act on it without running the sequence. Written from plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md, Reference files." +tags: [skill-authoring, progressive-disclosure] +expected_outcome: "The rewritten reference states where the ticket key comes from as its own input (given by the caller or read by the reference itself), keeps the branch-name rule, and no longer refers to step 2, step 3, or a value set earlier in the sequence." +execution: + allowed_tools: [Read, Skill] + max_turns: 20 + timeout_seconds: 600 diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/reads-criteria.md b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/reads-criteria.md new file mode 100644 index 00000000000..20212fb171b --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/reads-criteria.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Read +arm: both +input_match: '"file_path"\s*:\s*"[^"]*skill-authoring/reference/skill-criteria\.md"' +--- diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/stands-on-its-own.md b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/stands-on-its-own.md new file mode 100644 index 00000000000..e3499dd2f40 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/graders/stands-on-its-own.md @@ -0,0 +1,13 @@ +--- +type: llm +arm: both +--- +The reference file being rewritten told the reader to take "the `$KEY` you looked up in step 2", build a branch name `/-` (lowercase; type `feat`, `fix` or `docs`), and "then continue with step 4". It depends on the owning skill's step sequence, and another skill wants to use the same naming rule. + +PASS only if all three hold in the rewritten file: + +1. It names the ticket key as the file's own input: the caller supplies it, or the file says where to get it. It does not say it was set or looked up in an earlier step. +2. It refers to no step of the owning skill's sequence: no "step 2", "step 3", "step 4", "continue with", or "the value you set earlier". +3. It keeps the naming rule: the `/-` shape, lowercase, and the three allowed types. + +FAIL if any step reference remains, if the key still comes from an earlier step, or if the naming rule is dropped or changed. diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/prompt.md b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/prompt.md new file mode 100644 index 00000000000..82f9561999f --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/prompt.md @@ -0,0 +1,13 @@ +/playbooks:skill-authoring. Rewrite this reference file of my skill so it meets our skill criteria. Print only the rewritten file in your reply; do not write any file. + +`reference/branch-names.md`: + +```markdown +# Branch names + +Now take the `$KEY` you looked up in step 2 and build the branch name from it: +`/-`, all lowercase, where `` is `feat`, `fix` or `docs`. +Then continue with step 4. +``` + +Author's note, not part of the file: the skill's SKILL.md runs five steps, and step 2 looks up the ticket key. This file is only ever read during step 3, so it can rely on step 2 having run. Our release skill also wants to name branches the same way. diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/reads-criteria.json b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/reads-criteria.json new file mode 100644 index 00000000000..8b2e55e03a4 --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/reads-criteria.json @@ -0,0 +1,34 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "playbooks:skill-authoring" + } + }, + { + "tool": "Read", + "input": { + "file_path": "/root/.claude/plugins/cache/melodic-software/playbooks/1.0.0/skills/skill-authoring/reference/skill-criteria.md" + } + } + ], + "why": "the skill fired and its criteria reference was read" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "playbooks:skill-authoring" + } + } + ], + "why": "criteria never read" + } + ] +} diff --git a/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/stands-on-its-own.json b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/stands-on-its-own.json new file mode 100644 index 00000000000..7669ab243bf --- /dev/null +++ b/plugins/playbooks/evals/skill-authoring-reference-works-without-steps/samples/stands-on-its-own.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "# Branch names\n\nInput: the ticket key, given by the caller.\n\nBuild the branch name as `/-`, all lowercase, where `` is `feat`, `fix` or `docs`.", + "why": "names its own input, no step references, keeps the rule" + }, + { + "answer": "# Branch names\n\nGet the ticket key from the issue the work closes. Name the branch `/-`, lowercase; `` is one of `feat`, `fix`, `docs`.", + "why": "says where the key comes from without leaning on the sequence" + } + ], + "fail": [ + { + "answer": "# Branch names\n\nTake the `$KEY` you looked up in step 2 and build `/-`, all lowercase, where `` is `feat`, `fix` or `docs`.", + "why": "still leans on step 2" + }, + { + "answer": "# Branch names\n\nInput: the ticket key.\n\nBuild `/-`, all lowercase, where `` is `feat`, `fix` or `docs`. Then continue with step 4.", + "why": "still hands back into the step sequence" + }, + { + "answer": "# Branch names\n\nInput: the ticket key.\n\nName the branch after the ticket.", + "why": "drops the naming rule" + } + ] +} diff --git a/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md b/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md index 946bfdfa109..23a1d6f9c88 100644 --- a/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md +++ b/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md @@ -150,6 +150,12 @@ except the pointer swap that **Cutting text** exempts ([Measurement](#measuremen after the owning skill changes, so the two then disagree and nothing flags it. When the owning skill may not be installed, one line says what this skill does without it; that line is not a second copy of the procedure. +- **Config before a generated skill.** A shipped skill generates a repository-specific skill (as + `/app-control:map` generates a project's control skill) only where repository configuration + cannot express the difference. When config can carry it, such as a tool name the skill reads + from the repo's config (`docs/contracts/config-cascade/README.md`), keep one shipped skill and + read the config. Why: a generated skill is a copy that stops receiving the shipped skill's + fixes, while config is read by the current skill on every run. - **SKILL.md is a router, under 5,000 tokens.** It is the hub: it holds the standing rules and routes each action to its own spoke under `actions/`, one file per action, so a run loads the router and the one action it needs. The cap is a house setting; `/skill-quality:check` estimates the count at 4 @@ -209,6 +215,11 @@ build tool) takes these steps in order: practices in [Sources](#sources)). - Each pointer says what the file holds and when to read it; a bare "see X" is a missed connection. +- **A reference works without the steps.** Another skill or a person loads it through this + skill's invocation or a documented action, never by its path, and acts on it without running + the step sequence. So a reference names its own inputs and never leans on a value an earlier + step set or on steps having run. Why: a reference that only makes sense mid-sequence cannot be + reused, and a caller after one rule has to run steps it does not need. - A spoke is read, not rendered, so no substitution variable resolves in it: cite `/scripts/`. `scripts/check-spoke-plugin-root.sh` gates the plugin-root and skill-dir variables in this marketplace.