Repository navigation
feat(corrections): turn instruction-file rules into checks with a reconciled ledger - #16633
Conversation
…onciled ledger Add `/corrections:codify rules <file>...`. It reads every rule in the named instruction files (AGENTS.md and what it points to, CLAUDE.md, .claude/rules/, .cursor/rules/, review criteria docs) and gives each one disposition: a proven check with its prose flagged for deletion, prose with a reason, deleted as derivable, or held. A rule becomes a check only when it needs no judgment and the check covers its known cases; an unproven check is held. scripts/rules-ledger.mjs reconciles the ledger and exits 1 naming each rule dropped or incomplete, with a test suite. reference/adding-a-team-rule.md states how a checking rung's rule is added safely and that a custom rule ships with its own valid and invalid rule tests and states what it does not catch; the hub and finding-classes point at it for every input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…plugin-root placeholder The rules action names its script through <plugin-root>, as action files do. A proposed rule's tests run in the tool's own rule-test harness, which codify finds from the repository and the tool's current docs (/discovery:research when the repository does not settle it) and names in the proposal; the skill names no tool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…rate the cheat sheet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Hold lifted: the owner confirmed the decisions behind this change (2026-11-16), and the no-per-stack-examples rule is applied. Known gap: the ledger eval case was not model-run on this host; the scaffold was run by hand. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f739872493
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A quote only had to appear within ten lines of its cited line, so one rule quoted at two lines could stand in for the rule above it and the ledger still reported complete. The quote must now start on its cited line, and two rows resolving to the same start are one rule listed twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
|
claude-security-review reviewed Security review: PR #16633,
|
|
claude-review reviewed Code review:
|
No related issue: an owner-decided extension of
/corrections:codify, building on the capability routing that #16603 landed.Summary
Instruction files hold rules that a check could enforce, and nothing turns them into checks or shows which ones were dropped.
/corrections:codify rules <file>...reads every rule in the named instruction files and gives each one exactly one outcome, recorded in a ledger. A script reconciles the ledger and fails the run when a rule is unaccounted for. A new reference states how a checking rule is added safely, and that a custom rule ships with its own rule tests.Fix
actions/rules.md(new): the inputs areAGENTS.mdand the in-repo files it points to,CLAUDE.md,.claude/rules/,.cursor/rules/and review criteria docs. Each file is read as data. Every rule gets one of four dispositions:check: a proven check, with its prose flagged for deletion. The bar for that deletion is/harness-config:audit-instructions.prose: kept, with a reason.delete: derivable from the code, config or history, naming what states it.held: with a reason, for example an unproven check.A rule becomes a check only when it needs no judgment and the check catches every known case listed in its row. Each check has its own fail-on-old, pass-on-new proof, and one file's rules may be processed in one pass. The action writes one proposal per input file and edits no instruction file.
scripts/rules-ledger.mjsand its test (new) reconcile the ledger and exit 1 naming each rule that:file:line;checkrow missing any of: a current checking-rung id (an old id or a guidance rung is refused), an old ref with proof exit 0, its known cases, or the prose-deletion flag;prose,deleteorheldrow with no reason.reference/adding-a-team-rule.md(new) gives five steps for adding a rule:/code-metrics:audit-suppressions.A rule that has to be written ships with valid and invalid test cases in the tool's own rule-test harness. Codify finds that harness at run time, from the repository and the tool's current docs, through
/discovery:researchwhen the repository does not settle it. The rule also states what it does not catch: aliased imports, computed access, indirect paths and generated code. The skill names no tool.SKILL.md: an Actions row,rulesin the exact-word action list, the argument hint, a Boundary line, and section 6's proposal contents. The proposal contents point every checking-rung proposal, from any input, at the new reference.finding-classes.mdadds the rule-test requirement undercustom-analyzerandsyntax-pattern-rule.README rows, a changelog fragment (minor), and the regenerated
docs/skill-cheat-sheet.md. The argument hint was shortened to stay under its 100-character budget.Eval cases:
codify-rules-ledger-accounts-for-every-ruleandcodify-custom-rule-ships-rule-tests.Verification
plugins/corrections/scripts/rules-ledger.test.sh: 60/60 ok. It was written first and failed before the script existed (27 of 47 not ok).The plugin's other script suites (
prove-check,group-findings,codify-settings,count-checks): 0 not ok.validate-cases.pyonplugins/corrections/evals: no FAIL.markdownlint-cli2on the changed markdown: 0 issues.check-changed-skills.sh origin/main: codify PASS, 0 errors.validate-plugins.sh --only corrections: pass.check-affected.sh --base origin/mainran as the pre-push hook and passed.Eval
codify-custom-rule-ships-rule-tests: sonnet, 3 runs per arm, final skill text, 0.58 USD.states-what-it-missesrule-tests-in-the-harnessThe harness grader does not separate the arms: the fixture repository already has rule tests in that harness, so both arms name it. An earlier run on the generic wording, before the run-time harness lookup was added, scored with 0/3 and 3/3 on the same two graders (0.56 USD).
Eval
codify-rules-ledger-accounts-for-every-rule: not run. This host refuses any evaluation that grants Bash, because the Docker credential store holds a symbolic link the Bash sandbox cannot exclude, and the case needs Bash for the proof and the reconciler. A manual run on the case's own scaffold stood in for it:prove-check.shon theconsole.logcheck exited 0 (old ref exit 1, HEAD exit 0)..claude/rules/api.md) reconciled withledger complete: 5 rules from 3 files accounted for.Related
/harness-config:audit-instructionsdefines the bar for prose a check already enforces; this change points at that skill by name.🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC