Skip to content

ci(skills): ratchet stack-specific code fences in skill bodies and amend ADR 0101 - #16649

Open
kyle-sexton wants to merge 2 commits into
mainfrom
feat/no-per-stack-ratchet
Open

kyle-sexton wants to merge 2 commits into
mainfrom
feat/no-per-stack-ratchet

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Refs: #16634
No related issue: owner decision recorded in docs/plugin-philosophy.md (no per-stack examples in shipped skills)

Summary

Shipped skills carry no ecosystem-specific examples: no code sample written for one stack, and no link to one tool's documentation. This PR makes two follow-ups to that rule. ADR 0101's Source citation row still kept any documentation link, so it is amended. A count ratchet stops new stack-specific code fences from landing in skill and agent bodies while the existing ones are cut.

Fix

  • docs/adr/0101-...md: the status line, the Source citation row and a dated ## Amendment (2026-11-16) section. The amendment states the rule and its reason, points at the plugin philosophy's "Hardcoded consumer specifics", and names the ratchet. This amends the record in place, following the repository's own convention: ADRs 0018, 0028 and 0075 each carry an accepted; amended <date> status line with a dated amendment section. That convention wins over /architecture:record-decision's default of a new record.
  • scripts/skill-stack-fences.py: a per-file counter. It prints <path> fences=<n> for each tracked plugins/*/skills/**/*.md and plugins/*/agents/**/*.md file holding a counted fence, and exits 2 when git fails or no file is in scope, so a broken scope can't read as 0.
  • scripts/test_skill_stack_fences.py and scripts/skill-stack-fences.test.sh (the python_probe wrapper) are shaped like the js-complexity-share counter's. Twelve cases cover counted and excluded tags, tags with attributes, list-indented and tilde fences, fences behind blockquote and list-marker prefixes (and the next block a missed list-item opener would hide), fences nested inside another fence, closing-fence length, scope, sorted output, and exit 2 both with no file in scope and outside a repository.
  • .performance/ratchets.json: a new per-file skill-stack-fences row with 39 file ceilings and 174 fences in total, recorded by ratchet.py add --per-file --break-glass off. The existing Check performance counter ceilings step in check-plugins runs ratchet.py check --file .performance/ratchets.json, so no workflow edit is needed.

No plugin is touched, so no plugin needs a version bump or changelog fragment.

The counting rule

A fence counts when it opens a code block in a skill or agent body and the first word of its info string (lowercased, {, . and } stripped) is not in the excluded set. Every other tag counts, so a new language (rust, kotlin) is counted without an edit. Untagged fences do not count. A fence nested inside another fence is content and does not count.

Excluded group Tags Basis
Shells the agent runs commands through bash sh shell zsh console powershell pwsh ps1 batch bat cmd The skill runs these through its own shell tool. For example, plugins/machine-health/skills/audit/reference/windows/check-catalog.md holds 25 powershell checks the audit runs.
Data, prose and diagram formats a skill reads or emits text txt plaintext markdown md json jsonc jsonl yaml yml toml ini csv diff mermaid gitconfig Output templates and contracts, not stack samples.
Skill pre-compute block ! Commands the skill itself runs at load time, e.g. plugins/knowledge/skills/video-digest/SKILL.md.
Queries and patterns skills run against their own data sql regex plugins/debugging/skills/analyze-profile/reference/formats/cpuprofile.md:40-42 is a query the skill runs with --param. plugins/docs-hygiene/skills/rename-references/context/patterns.md:43-45 is the pattern the sweep uses.

Out of scope by directory: evals, tests, test, __tests__, fixtures, testdata, vendor. These hold eval inputs, test data, and upstream copies a sync replaces, such as plugins/playwright/skills/playwright/vendor/, which would otherwise fail CI on a sync PR. This scope follows scripts/js-complexity-share.py.

Basis for per-file over one total: /toolchain:ratchet "Per-file counters". A count that files own is held per file, so cutting a fence in one file cannot pay for a new one elsewhere.

Known imprecision: some counted js fences are code a skill executes, such as the course-digest adapters, the playwright references and the harness-ops extraction. The tag cannot tell those apart, so they count. A declared-ecosystem surface or executed adapter that needs a new fence raises its file's ceiling in the PR that adds it, with the reason in the body.

Today's count is 174 fences in 39 files at f552369. #16634 reported 179 in 51 files. The figures differ because of scope: #16634's count also took in plugins/*/reference/**, while this one covers skills and agents only, drops the eval, test, fixture and vendor directories, and excludes sql and regex.

Verification

  • python3 scripts/test_skill_stack_fences.py: written first and failed (7 of 9 cases, counter absent), then 9 of 9 pass. The review fix for container prefixes added three cases, which failed first (3 of 12) and now pass: 12 of 12. bash scripts/skill-stack-fences.test.sh: 12 pass. The count is unchanged after that fix: 174 fences in 39 files.
  • ratchet.py check on a file holding only the new row: OK skill-stack-fences: every file at or below its ceiling (fences=174 total). With a python fence appended to an uncounted skill file it reported ABOVE ... fences=1 > ceiling 0 and exited 1. With the file restored it exited 0. The full ratchets.json was not run locally because the hook-census rows need strace, so CI runs it.
  • scripts/check-affected.sh --no-fetch: 16 passed, 3 skipped over 5 changed files.
  • The following passed: scripts/run-ruff.sh check and format --check; shellcheck; check-fixture-git-isolation.sh --check; check-test-tmp-cleanup.sh --check; check-silent-skips.sh; check-purged-em-dashes.sh; check-vocabulary.py; check-adr-numbers.sh --check; check-discriminating-test-skips.sh; check-line-citations.py; check-lane-coverage.sh --check; check-cant-fail-scan.sh origin/main; and the commit-time comment lint.

Related

Not in this PR

  • Plugin-level plugins/*/reference/** files. They are model-read under ADR 0101 and docs(philosophy): ship no ecosystem-specific examples in skills #16634 proposed counting them; this ratchet covers skill and agent bodies only.
  • Matching links to tool documentation. That would need a per-stack domain list, the same maintenance the rule exists to avoid.
  • A scripts/suite-seconds.txt row for the new suite. The rows are measured from CI runs, and until one is added the lane planner sizes an unlisted suite at the mean.

🤖 Generated with Claude Code

…end ADR 0101

Shipped skills carry no ecosystem-specific examples. ADR 0101's Source
citation row still kept any documentation link, so it is amended in
place: a link to one ecosystem's tool documentation, or an example
written for one stack, is no longer a sanctioned citation.

scripts/skill-stack-fences.py counts, per file, the code fences in
tracked plugins/*/skills/**/*.md and plugins/*/agents/**/*.md whose tag
is not a shell, data or prose format, diagram, pre-compute block, or the
sql and regex skills run against their own data. Eval, test, fixture
and vendor directories are out of scope. A per-file skill-stack-fences
row in .performance/ratchets.json freezes today's 174 fences in 39
files; the existing ratchet step in check-plugins checks it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 21:52
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T21:54:21.052042Z 3f3b2fc Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f3b2fcd75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/skill-stack-fences.py Resolved
@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43964540026

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed fc3ee2f983879e95aa5d39010c5e6f1c862aba6e: read 2 of 2 changed file(s); 1 finding(s), 1 as inline comments.

claude-review: incremental review of 3f3b2fc..fc3ee2f, blocking only. This lane already reviewed an earlier head, so this pass reports only blocking findings. I read every hunk of both files in scope: scripts/skill-stack-fences.py and scripts/test_skill_stack_fences.py.

What changed: the counter now strips container prefixes (any mix of > blockquote markers, -/*/+/1./1) list markers, and indentation) before it matches a fence line. This fixes the settled thread on the old line 60. Three new tests cover blockquote fences, fences right after a list marker, and the next block that a missed list-item opener would hide.

Finding (1, inline): OPENING is shared between openers and closers. Inside an open top-level fence, a content line like > ```` now closes that fence. The file's real closer then opens an untagged fence that hides every later fence in the file, so stack fences there go uncounted by the ratchet. The input in the inline comment counted 1 on the base head and counts 0 now. See [scripts/skill-stack-fences.py#L91-L99`](

match = OPENING.match(line)
if fence:
if (
match
and match.group(1)[0] == fence[0]
and len(match.group(1)) >= len(fence)
and not match.group(2).strip()
):
fence = ""
).

What I checked:

  • Regex safety. Each repetition of CONTAINER_PREFIX must consume at least one character that is not whitespace (>, a bullet, or digits plus ./)). That rules out catastrophic backtracking.
  • Test expectations. The counts in the three new tests match the counter's logic, traced by hand: 3, 4 (bash is excluded) and 2.

Not verified by me: the PR body says 12 of 12 tests pass and the repository count stays at 174 fences in 39 files. Those are the author's claims; I have no shell, so I ran nothing.

Settled thread: the old-line-60 thread was answered by the owner with this fix. I raise nothing from it again. The finding above is a different problem, which the fix itself introduced.

Closing issue: none (Refs: #16634 only), so there are no acceptance criteria to check.

Spend: three more short unittest cases on each CI run. That is negligible and not a hold.

Verifier dropped 0 of 1 candidates. I checked the candidate myself by tracing it by hand; no separate verifier subagent ran.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed fc3ee2f983879e95aa5d39010c5e6f1c862aba6e: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16649 (incremental since 3f3b2fc), no security issues found

Scope. I read every hunk of the 2 files changed since 3f3b2fcd: scripts/skill-stack-fences.py and scripts/test_skill_stack_fences.py. Neither file matches a trusted-path glob. The prior-findings file says blocking only: yes, so this review reports only CRITICAL and IMPORTANT findings. It found none.

What changed

  • scripts/skill-stack-fences.py. The fence-opener regex now strips container prefixes before it matches a fence: blockquote markers (>), list markers (-, *, +, 1., 1)) and their indentation. The new CONTAINER_PREFIX pattern is anchored with ^, and every repetition of its group has to consume a non-whitespace marker character, so a crafted line cannot cause catastrophic backtracking. Its input is the repository's own tracked skill and agent .md files, read by a CI counter that handles no credentials and writes nothing. Nothing untrusted reaches a shell. The only effect is on the fence count that feeds the ratchet.
  • scripts/test_skill_stack_fences.py. Adds three unit cases for fences behind blockquote and list-marker prefixes. They run in the same isolated temporary git repositories as the existing cases.

Checks

  • Trust controls: none touched. No trusted-trigger check, token scope, kill switch, token broker, merge hold or .github/actions/ action changed.
  • Plugin manifests: none changed.
  • Instruction-surface deletions: none.
  • Embedded instructions: the PR context file holds none directed at this lane.
  • Spend: none beyond the existing local CI run of the counter.

Earlier posts: the prior summary comment (#16649 (comment)) reported 0 findings, so nothing carries forward.

critical-findings: 0

…fixes

The opening-fence pattern allowed only whitespace before the delimiter,
so a fence inside a blockquote (`> ```python`) or right after a list
marker (`- ```python`, `1. ```python`) was missed, and the missed
opener's closer then read as an untagged opener that hid the next real
block. Any mix of `>` markers, list markers and their indentation now
comes off before an opener or a closer is matched.

Three regression cases cover the blockquote form, the list-marker form
and the hidden next block. Today's count is unchanged: 174 fences in 39
files, so the ceilings stand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on fc3ee2f983879e95aa5d39010c5e6f1c862aba6e; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment on lines +62 to +63
CONTAINER_PREFIX = r"(?:\s*(?:>|[-*+](?=\s)|\d{1,9}[.)](?=\s)))*\s*"
OPENING = re.compile(rf"^{CONTAINER_PREFIX}(`{{3,}}|~{{3,}})(.*)$")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closers now strip container prefixes the open fence never had, so a top-level fence can close early and hide every fence after it.

OPENING is used for openers and closers alike (count_fences, lines 91-99). So inside an open top-level fence, a content line such as `> ```` now counts as its closer. Under CommonMark it does not: a closing fence may have only indentation before it. A blockquote marker on that line belongs to a container the fence is not inside, so the line is code content.

Reproduce with a skill body that shows a blockquoted code block inside a markdown example, then has a real stack fence:

```markdown
> ```python
> x = 1
> ```
```

```python
y
```

I traced this by hand; I did not run it:

  • line 1 opens a markdown fence (excluded tag).
  • `> ```` on line 4 now matches with an empty info string and closes it.
  • the real closer on line 5 then opens an untagged fence.
  • that fence swallows the python opener on line 7 (its info string is not empty, so it is not a closer) and closes on line 9.

count_fences returns 0 where CommonMark gives 1. On the base head this input counted 1, because ^\s* did not match > ````. So this push introduced the under-count. It is the same miss as the one test_list_item_opener_does_not_hide_the_next_block` guards against, but in the other direction. Any new fence later in that file passes the ratchet uncounted.

Fix: strip container prefixes only when matching an opener, and keep the opener's container. For example, record how many > markers the opener had, and accept a closer only when its prefix has the same > depth and no list marker. Add the input above as a regression case.

claude-lane: claude-review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant