Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .changes/corrections/codify-rules-ledger-5e1c7a42.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
bump: minor
---

### Added

- `/corrections:codify rules <file>...` turns the rules in instruction files (`AGENTS.md` and the in-repo files it points to, `CLAUDE.md`, `.claude/rules/`, `.cursor/rules/`, review criteria docs) into checks. Each rule becomes a check on its rung with its own proof and its prose flagged for deletion, stays prose with a reason, is deleted as derivable, or is held. A rule becomes a check only when it needs no judgment and the check covers its known cases; an unproven check is held. One file's rules may go through in one pass.
- `scripts/rules-ledger.mjs` reconciles the ledger of every rule read and exits 1 naming each rule dropped or left incomplete, so the run cannot finish with a rule unaccounted for.
- `reference/adding-a-team-rule.md`: survey the legitimate uses, scope the rule, warn before error, allow only exclusions with a reason, and audit suppressions afterwards with `/code-metrics:audit-suppressions`. A rule codify proposes to write ships with its own valid and invalid rule tests in the tool's own harness where one exists, and states what it does not catch. Every checking-rung proposal now points at it.
- Eval cases `codify-rules-ledger-accounts-for-every-rule` and `codify-custom-rule-ships-rule-tests`.
2 changes: 1 addition & 1 deletion docs/skill-cheat-sheet.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ From the stage definitions: A conditional stage runs only when its trigger holds

| Skill | Plugin | What it does | Actions | Arguments | Upstream | Next |
| --- | --- | --- | --- | --- | --- | --- |
| [`/corrections:codify`](../plugins/corrections/skills/codify/SKILL.md) | `corrections` | Count a recurring mistake, pick the strongest rung and prove the check | `findings`, `reviews`, `issues`, `note`, `preferences`, `group`, `count-checks` | `<<description> [#PR]\|findings <file>\|reviews\|issues\|note <text>\|preferences\|group\|count-checks…>` | [pstack `automate-me`](upstream/cursor-pstack.md#attribution-table), [pstack `principle-encode-lessons-in-structure`](upstream/cursor-pstack.md#attribution-table), [pstack `correct`](upstream/cursor-pstack.md#attribution-table) | `/harness-config:audit-automation-gaps`, `/architecture:improve`, `/corrections:codify`, `/implementation:implement` |
| [`/corrections:codify`](../plugins/corrections/skills/codify/SKILL.md) | `corrections` | Count a recurring mistake, pick the strongest rung and prove the check | `findings`, `rules`, `reviews`, `issues`, `note`, `preferences`, `group`, `count-checks` | `<<desc> [#PR]\|findings <file>\|rules <files>\|reviews\|issues\|note\|preferences\|group\|count-checks…>` | [pstack `automate-me`](upstream/cursor-pstack.md#attribution-table), [pstack `principle-encode-lessons-in-structure`](upstream/cursor-pstack.md#attribution-table), [pstack `correct`](upstream/cursor-pstack.md#attribution-table) | `/harness-config:audit-automation-gaps`, `/architecture:improve`, `/corrections:codify`, `/implementation:implement` |
| [`/session-flow:audit-friction`](../plugins/session-flow/skills/audit-friction/SKILL.md) | `session-flow` | Mine sessions for permission and autonomy friction and end in one decision brief | | `[remeasure] [--days <n>] [--project <text>] [--source <dir>] [--verify consequential\|full\|probes]` | | `/session-flow:audit-friction` |
| [`/session-flow:audit-sessions`](../plugins/session-flow/skills/audit-sessions/SKILL.md) | `session-flow` | Cross-session audit of transcripts with routed, never-applied findings | `sweep`, `pr`, `trends` | `[sweep\|pr\|trends…] [--scope machine\|project] [--since <day>] [--until <day>] [--write-report]` | [pstack `automate-me`](upstream/cursor-pstack.md#attribution-table) | `/corrections:codify`, `/session-flow:audit-friction` |
| [`/session-flow:retro`](../plugins/session-flow/skills/retro/SKILL.md) | `session-flow` | Structured session retrospective with codified learnings | `session`, `quick` | `[session [--pattern '<regex>']\|quick] [--unattended]` | [Pocock `retro`](upstream/mattpocock-skills.md#attribution-table), [Pocock `chief-of-staff`](upstream/mattpocock-skills.md#attribution-table) | `/session-flow:audit-sessions` |
Expand Down
2 changes: 2 additions & 0 deletions plugins/corrections/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ mistake it targets and to pass on the current code.
|---|---|
| `/corrections:codify "<description>" [#PR]` | Counts how often the mistake recurs, picks the strongest enforcement-ladder rung, places the lesson by scope, and proposes a check proven with `prove-check.sh` |
| `/corrections:codify findings <findings-file>` | Takes each row of one review findings file through the same steps, ending in one proposal per row |
| `/corrections:codify rules <file>...` | Turns the rules in the named instruction files (`AGENTS.md` and what it points to, `CLAUDE.md`, `.claude/rules/`, `.cursor/rules/`, review criteria docs) into checks: each rule becomes a proven check with its prose flagged for deletion, stays prose with a reason, is deleted as derivable, or is held, in a ledger `scripts/rules-ledger.mjs` reconciles |
| `/corrections:codify reviews [--pr <n> [--repo <owner/repo>]]` | Mines recent merged pull requests' review comments, through `/source-control:pull-request review-comments`, and takes each lesson that recurs through the same steps; with `--pr`, one chosen pull request (open ones too), whose every lesson goes on labeled as from one PR, still for your approval; with `--repo` naming another repository than the checkout, it lists the lessons and proposes nothing |
| `/corrections:codify issues [--label <name>] [--limit <n>]` | Mines the bound tracker's closed bug items, through `/work-items:track`, for mistakes that came back; needs the `work-items` plugin |
| `/corrections:codify preferences [--scope machine\|project] [--since <day>]` | With the `session-flow` plugin installed, reads your typed turns through `/session-flow:audit-sessions excerpts-only`, counts each preference by distinct session, and proposes each one you confirm as a line for a user-scope file, never a repository file |
Expand All @@ -30,6 +31,7 @@ many merged pull requests `reviews` reads without `--pr`. Each is a plugin optio
| `scripts/prove-check.sh` | Runs a proposed check in temporary worktrees at two refs and exits 0 only when it fails at the old ref and passes at the new one |
| `scripts/codify-settings.mjs` | Prints `recurrence.min_occurrences`, `ratchet_offer` and `review_mining_prs` and the layer that set each, resolved through the config cascade (schema `schemas/corrections.schema.json`) |
| `scripts/group-findings.mjs [--by <field>[,<field>...]] [--since <UTC ISO-8601>] [--rows]` | Runs `buffer.sh read` and prints its rows grouped by the named top-level fields (default `producer`): the keys first, then each group with its row and occurrence counts (distinct `source_id`) and its members |
| `scripts/rules-ledger.mjs <ledger.json>` | Reconciles a `rules` ledger from the repository root: every input file's declared rule count matches its rows, each rule's quoted text is at its `<file>:<line>`, and each row carries its disposition's fields (a `check` row a checking rung, a proof that exited 0, the known cases and its prose flagged for deletion); exits 1 naming each rule left unaccounted for |
| `scripts/count-checks.mjs [--since <YYYY-MM-DD>] [--until <YYYY-MM-DD>] [--bucket week\|month]` | Reads the `Codifies:` records and their reverts from the default branch's `git log` and prints one row per period and a window row, with the not-counted, reverted, promoted and alias lines listed after |

## prove-check
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
schema_version: "1.1"
name: codify-custom-rule-ships-rule-tests
description: 'Gain case: a custom rule codify proposes for a repository that already tests its ESLint rules with RuleTester ships with its own valid and invalid rule tests in that harness, and the proposal states what the rule does not catch.'
tags: [codify, rules]
plugins: ["../.."]
expected_outcome: 'Row 1 takes custom-analyzer, built in the repository''s own ESLint rules folder by /implementation:implement. The proposal requires RuleTester tests with valid and invalid cases (the Secret-typed logger call invalid; a non-Secret logger call valid) and lists what the rule does not catch, such as an aliased or re-exported logger, computed member access, or a Secret passed through a wrapper.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/corrections:codify findings findings.md What rung does row 1 get, and what must the proposed check ship with? This run has no Bash tool: skip the proof and report the check as unproven, and present the proposal in the report, writing nothing.'
allowed_tools: [Read, Glob, Grep, Skill]
max_turns: 15
timeout_seconds: 420
graders:
- name: rule-tests-in-the-harness
type: 'regex'
pattern: '^(?=[\s\S]*RuleTester)(?=[\s\S]*\binvalid\b)(?=[\s\S]*\bvalid\b)'
flags: 'i'
arm: 'both'
- name: states-what-it-misses
type: llm
focus: last_message
arm: 'both'
criteria: 'The reply proposes a custom lint rule (an analyzer) that fails a logger call receiving a value typed Secret. PASS only if the reply states, as part of the proposal, at least one concrete case the rule will not catch, such as a logger imported under another name or re-exported, a computed or dynamic member access (logger["info"]), a Secret passed through a wrapper function or another indirect path, or a value whose Secret type was cast away. FAIL if the reply presents the rule as catching every case or names no limit of the rule.'
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "Row 1: semantic-invariant on custom-analyzer. The rule ships with RuleTester tests beside eslint-rules/no-default-export.test.js: invalid, logger.info(token) where token is Secret<string>; valid, logger.info(last4)."
}
],
"fail": [
{
"answer": "Row 1: semantic-invariant on custom-analyzer, built by /implementation:implement."
},
{
"answer": "Row 1: custom-analyzer. Add RuleTester tests for the rule."
},
{
"answer": "Row 1: custom-analyzer. Test it with valid and invalid cases in vitest."
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"pass": [
{
"answer": "Row 1: custom-analyzer, a typed ESLint rule in eslint-rules/. Rule tests: invalid, logger.info(t) with t typed Secret<string>; valid, logger.info(card.last4). Does not catch: a logger imported under another name, logger[\"info\"](t), or a Secret passed through a helper that logs it."
}
],
"fail": [
{
"answer": "Row 1: custom-analyzer, a typed ESLint rule in eslint-rules/ with valid and invalid RuleTester cases. It catches every way a Secret can reach the logger.",
"why": "presents the rule as complete"
},
{
"answer": "Row 1: custom-analyzer, a typed ESLint rule with valid and invalid RuleTester cases, built by /implementation:implement.",
"why": "names no limit of the rule"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
#!/usr/bin/env bash
# A TypeScript repository that already writes its own ESLint rules, each with a RuleTester test,
# and a findings file whose one row needs a new rule that resolves a value's type.
set -euo pipefail
git init -q -b main .
git config user.email eval@example.invalid
git config user.name eval
git config commit.gpgsign false
mkdir -p src/billing eslint-rules
cat >package.json <<'EOF'
{ "name": "billing", "version": "1.0.0", "type": "module",
"scripts": { "lint": "eslint src", "test:rules": "node --test eslint-rules" } }
EOF
cat >eslint.config.js <<'EOF'
import tseslint from "typescript-eslint";
import noDefaultExport from "./eslint-rules/no-default-export.js";

export default tseslint.config({
files: ["src/**/*.ts"],
languageOptions: { parserOptions: { projectService: true } },
plugins: { local: { rules: { "no-default-export": noDefaultExport } } },
rules: { "local/no-default-export": "error" },
});
EOF
cat >eslint-rules/no-default-export.js <<'EOF'
export default {
meta: { type: "problem", messages: { named: "Use a named export." }, schema: [] },
create: (context) => ({
ExportDefaultDeclaration: (node) => context.report({ node, messageId: "named" }),
}),
};
EOF
cat >eslint-rules/no-default-export.test.js <<'EOF'
import { RuleTester } from "eslint";
import rule from "./no-default-export.js";

new RuleTester().run("no-default-export", rule, {
valid: ["export const a = 1;"],
invalid: [{ code: "export default 1;", errors: [{ messageId: "named" }] }],
});
EOF
cat >src/secret.ts <<'EOF'
export type Secret<T> = T & { readonly __secret: unique symbol };
EOF
cat >src/logger.ts <<'EOF'
export const logger = { info: (...args: unknown[]) => console.log(JSON.stringify(args)) };
EOF
cat >src/billing/charge.ts <<'EOF'
import type { Secret } from "../secret";
import { logger } from "../logger";

export const charge = (card: { token: Secret<string>; last4: string }) => {
const t = card.token;
logger.info("charging", t);
};
EOF
git add .
git commit -q -m "feat: charge"
cat >findings.md <<'EOF'
---
type: review-findings
date: 2026-01-01T00:00:00Z
branch: fixture
tier: medium
---

## Findings

| Rank | Tier | Confidence | Location | Surface(s) | Finding | Action |
|------|------|------------|----------|------------|---------|--------|
| 1 | CRITICAL | high | src/billing/charge.ts:6 | code-reviewer | Passes `t`, a value typed `Secret<string>`, to `logger.info`, which writes it in plain text. The variable's name gives no hint of its type. | Fail any logger call that receives a value whose type is `Secret`, whatever the variable is called. |

## Unparsed

None.

## Surfaces

Ran: code-reviewer.
EOF
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
schema_version: "1.1"
name: codify-rules-ledger-accounts-for-every-rule
description: 'Gain case: `rules` over two instruction files, one pointing at a third, accounts for every rule in a reconciled ledger: the console.log rule a past commit fixed becomes a proven check with its prose flagged for deletion, the boundary rule nothing ever broke is held as unproven, the naming and one-job rules stay prose, and the npm test line is deleted as derivable. No instruction file is edited.'
tags: [codify, rules]
plugins: ["../.."]
expected_outcome: 'Five rules from three files (.claude/rules/api.md added by following the AGENTS.md pointer). console.log: check, proven against the parent of the fix commit, prose flagged for deletion. Boundary rule: held, no old ref. Naming and one-job rules: prose, needs judgment. npm test: delete, package.json states it. rules-ledger.mjs exits 0 on the written ledger.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/corrections:codify rules AGENTS.md .cursor/rules/boundaries.mdc'
allowed_tools: [Bash, Read, Glob, Grep, Skill, Write]
max_turns: 40
timeout_seconds: 900
graders:
- name: ledger-reconciled
type: 'tool_used'
tool: 'Bash'
input_match: 'rules-ledger\.mjs'
min: 1
arm: 'with-only'
- name: writes-only-under-work-codify
type: 'tool_used'
tool: 'Write'
input_match: '"file_path":"(?![^"]*\.work/codify/)'
min: 0
max: 0
arm: 'both'
- name: every-rule-accounted
type: llm
focus: last_message
arm: 'both'
criteria: 'Five rules were planted across AGENTS.md, .cursor/rules/boundaries.mdc and .claude/rules/api.md (which AGENTS.md points to): (a) never call console.log in src/, broken and then fixed by a past commit; (b) only src/internal/ and src/widget.ts may import from src/internal/, never broken in history; (c) name functions so a reader understands them; (d) run the tests with npm test, which package.json scripts.test already states; (e) keep each exported API function focused on one job. PASS only if all hold: (1) every one of the five rules appears in the reply with an outcome, none left out; (2) rule (a) becomes a check on a named checking rung, with a proof result, and its prose line is flagged for deletion (or removal) once the check lands; (3) rule (b) is not reported as a proven check: it is held, unproven, or pending because no commit broke it; (4) rules (c) and (e) stay prose because they need judgment; (5) rule (d) is deleted, or proposed for deletion, as derivable from package.json. FAIL if any rule is missing, if (c) or (e) becomes a check, or if (b) is reported as a proven check.'
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"pass": [
{
"answer": "Inputs: AGENTS.md, .cursor/rules/boundaries.mdc, .claude/rules/api.md (added: AGENTS.md points to it). Ledger complete: 5 rules from 3 files accounted for.\n\n| Source | Rule | Disposition | Rung | Proof | Reason |\n|---|---|---|---|---|---|\n| AGENTS.md:3 | Never call console.log in src/ | check | syntax-pattern-rule | old a1b2c3 exit 1, HEAD exit 0 | prose flagged for deletion |\n| AGENTS.md:5 | Name functions so a reader understands them | prose | | | needs judgment |\n| AGENTS.md:7 | Run the tests with npm test | delete | | | package.json scripts.test states it |\n| .claude/rules/api.md:1 | Keep each exported API function focused on one job | prose | | | needs judgment |\n| .cursor/rules/boundaries.mdc:6 | Only src/internal and src/widget.ts import src/internal | held | architecture-test | not run | unproven: no commit ever broke it, so no old ref |"
}
],
"fail": [
{
"answer": "console.log rule: check (syntax-pattern-rule), proven, prose flagged for deletion. Boundary rule: held, no old ref. Naming rule: prose. npm test: delete, package.json states it.",
"why": "the one-job rule from the pointed-to file is missing"
},
{
"answer": "console.log: check, proven, prose flagged. Boundary: architecture-test, proven. Naming: prose. One-job rule: prose. npm test: delete.",
"why": "the boundary rule, which nothing ever broke, is reported as a proven check"
},
{
"answer": "console.log: check, proven, prose flagged. Boundary: held, no old ref. Naming: prose. One-job rule: custom-analyzer check counting statements. npm test: delete.",
"why": "a judgment rule became a check"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"pass": [
{
"answer": [
{
"tool": "Bash",
"input": {
"command": "node \"/p/scripts/rules-ledger.mjs\" .work/codify/main/20261116T000000Z-rules-ledger.json"
}
}
]
}
],
"fail": [
{
"answer": [
{
"tool": "Bash",
"input": {
"command": "git log --oneline"
}
}
]
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"pass": [
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/w/.work/codify/main/20261031T000000Z-proposal.md",
"content": "x"
}
}
]
}
],
"fail": [
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/w/.eslintrc.json",
"content": "x"
}
}
]
}
]
}
Loading
Loading