Skip to content

feat(playbooks): add reference-independence and config-before-generator skill rules - #16612

Open
kyle-sexton wants to merge 2 commits into
mainfrom
docs/skill-authoring-reference-generator-rules
Open

kyle-sexton wants to merge 2 commits into
mainfrom
docs/skill-authoring-reference-generator-rules

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: two authoring rules decided for the skill-authoring criteria, with no tracking issue.

Summary

playbooks:skill-authoring's criteria (reference/skill-criteria.md) had no rule that a reference file must work outside the skill's step sequence, and no rule for when a shipped skill may generate a repository-specific skill instead of reading repository config.

Fix

Two short rules, each with its reason, in the sections that already hold the neighbouring rules:

  • Reference files: "A reference works without the steps." Another skill or a person loads a reference through the skill's invocation or a documented action (never by path, which keeps the public-surface contract /docs-hygiene:audit-encapsulation checks) and acts on it without running the steps. So a reference names its own inputs and never leans on a value an earlier step set. No existing statement covered this; the nearest bullet ("Each pointer says what the file holds and when to read it") is about pointers, so the rule sits directly after it.
  • Body: "Config before a generated skill." A shipped skill generates a repository-specific skill (as /app-control:map generates a project's control skill) only where repository configuration cannot express the difference; when config can carry it, keep one shipped skill and read the config. Placed after "Layer on a skill, never restate it", the other rule against copies.

Two eval cases, one per rule, and a minor changelog fragment for playbooks.

Verification

Both new cases ran through claude plugin eval (2 runs per arm, with-plugin and no-plugin baseline, haiku judge). The plugin was copied outside the repository first, because the CLI refuses a plugin whose repository has many linked worktrees.

Case With Without Δ
skill-authoring-reference-works-without-steps 1.00 0.50 +0.50
skill-authoring-config-before-generated-skill 1.00 0.25 +0.75
  • reference-works-without-steps: the behavior grader passed in both arms. The base model already removes an obvious step dependency, so the Δ comes from the criteria-read grader. The case guards the rule against regression; it does not show the rule changes behavior.

  • config-before-generated-skill: the first run's grader required an explicit config entry, and it failed the with-plugin runs, which read the linter from the repository's own linter config on every run. That also satisfies the rule, so the grader now accepts either form. On the re-run, the behavior grader passed 2/2 with the plugin and 1/2 without it.

  • validate-cases.py plugins/playbooks/evals: no FAIL; the new cases add only the standard "judge samples need calibration" WARN every llm-graded case here carries.

  • check-skill.sh plugins/playbooks/skills: skill-authoring PASS, 0 errors, 3 warnings, all pre-existing (SKILL.md is unchanged).

  • plugins/playbooks/skills/skill-authoring/scripts/update.test.sh: 32/32 passed.

Related

None beyond the rules above: no other issue or PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

kyle-sexton and others added 2 commits November 16, 2026 14:15
…or skill rules

skill-authoring's criteria gain two rules:

- Reference files: a reference works without the skill's steps. Another
  skill or a person loads it through the skill's invocation or a
  documented action, never by path, and acts on it without running the
  step sequence, so it names its own inputs.
- Body: a shipped skill generates a repository-specific skill (as
  /app-control:map does) only where repository configuration cannot
  express the difference; when config can carry it, keep one shipped
  skill and read the config.

Each rule has an eval case under plugins/playbooks/evals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…grader

The config-before-generated-skill grader required an explicit config
entry and failed rewrites that read the linter from the repository's
own linter config on every run, which the rule also allows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:19
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:20:36.843530Z 9859cb9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant