diff --git a/.changes/harness-config/audit-instructions-enforced-prose-7c41e2a9.md b/.changes/harness-config/audit-instructions-enforced-prose-7c41e2a9.md new file mode 100644 index 00000000000..b687d895038 --- /dev/null +++ b/.changes/harness-config/audit-instructions-enforced-prose-7c41e2a9.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Changed + +- `audit-instructions`: prose restating a rule a deterministic check already enforces (a hook, a CI check, a linter rule, a test) is flagged for deletion with the enforcing check named, under a new enforced deletion tier that needs no watch. I5 drops the prose once a must-always rule is converted to a check instead of keeping both, I1 and I8 route an enforced line or step to I5, and protected-class rules in the instruction exception register still hold, with the check reported beside the hold. Eval case 6 now expects the CI-enforced PR-title step to be flagged for deletion rather than kept. diff --git a/.changes/harness-memory/audit-c8-enforced-prose-3b9d05f6.md b/.changes/harness-memory/audit-c8-enforced-prose-3b9d05f6.md new file mode 100644 index 00000000000..9684d8cd0d8 --- /dev/null +++ b/.changes/harness-memory/audit-c8-enforced-prose-3b9d05f6.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Changed + +- `audit`: C8 flags a CLAUDE.md instruction a deterministic check already enforces for deletion, naming the check, and holds an instruction matching a protected class in the instruction exception register. The cross-file pass that detects already-enforced instructions now reports them under C8. diff --git a/docs/contracts/instruction-exception-register/README.md b/docs/contracts/instruction-exception-register/README.md index 5ba08a83bf1..7ec4ff34ebc 100644 --- a/docs/contracts/instruction-exception-register/README.md +++ b/docs/contracts/instruction-exception-register/README.md @@ -52,9 +52,10 @@ over another is style. | Consumer | How it uses the register | |---|---| -| `harness-config:audit-instructions` | Deletion-class criteria (I1, I4, I5) hold back a candidate matching a protected class and report the hold rather than proposing the cut | +| `harness-config:audit-instructions` | Deletion-class criteria (I1, I4, I5, and I8's enforced-step route) hold back a candidate matching a protected class and report the hold rather than proposing the cut | +| `harness-memory:audit` | C8 holds a memory-layer instruction matching a protected class when a deterministic check enforces it, reporting the hold and the check rather than flagging the prose for deletion | | `harness-config:unhobble` | The bare-baseline experiment may strip a protected rule during the run, since the strip is reversible and branch-local, but Phase 4 restores it regardless of whether the ledger logged a stumble against it. A protected rule is never left deleted on the evidence of "no stumble was observed", and it never enters a deletion watch. A consequential rule outside the register is deletable only when a closed watch has zero attributed rows | -| `review:security-review` | The instruction-surface lens reads the register on any diff that deletes, narrows, or softens a standing instruction, and its skip gate may not wave such a diff through as prose-only. This is the review half of a guardrail deletion: the holds above act only inside the two trimming skills, while a deletion by hand or by any other route meets this lens when its pull request is reviewed (#3566) | +| `review:security-review` | The instruction-surface lens reads the register on any diff that deletes, narrows, or softens a standing instruction, and its skip gate may not wave such a diff through as prose-only. This is the review half of a guardrail deletion: the holds above act only inside the trimming skills listed above, while a deletion by hand or by any other route meets this lens when its pull request is reviewed (#3566) | | `review` plugin's `security-reviewer` agent | Behind `/review:quality-gate` security mode and `/review:fanout`. Applies the same lens whenever either dispatches it; `/review:fanout` dispatches it by its tier rule, so a small diff touching no security-sensitive path does not reach it | | `instruction-placement:*` | Unchanged. It owns the classes and the relocation verdict; this register is the deletion counterpart and defers to it on class membership | diff --git a/plugins/harness-config/evals/ai-enforced-step-flagged/case.yaml b/plugins/harness-config/evals/ai-enforced-step-flagged/case.yaml new file mode 100644 index 00000000000..d40d29f8257 --- /dev/null +++ b/plugins/harness-config/evals/ai-enforced-step-flagged/case.yaml @@ -0,0 +1,21 @@ +schema_version: "1.1" +name: ai-enforced-step-flagged +graders: + - name: step-1-cull + type: regex + pattern: '^[\s>*`_-]*step 1[*`_\s]*:[*`_\s]*cull\b' + flags: 'im' + target: 'last_message' + - name: step-5-keep + type: regex + pattern: '^[\s>*`_-]*step 5[*`_\s]*:[*`_\s]*keep\b' + flags: 'im' + target: 'last_message' + - name: step-8-delete-enforced + type: regex + pattern: '^[\s>*`_-]*step 8[*`_\s]*:[*`_\s]*delete-enforced\b' + flags: 'im' + target: 'last_message' + - name: check-named + type: llm + criteria: 'PASS only if the reply names the pr-title.yml lint-title CI job as what enforces step 8, and says step 8 is flagged for deletion rather than kept as a hard constraint. FAIL if step 8 is kept, or no enforcing check is named for it.' diff --git a/plugins/harness-config/evals/ai-enforced-step-flagged/prompt.md b/plugins/harness-config/evals/ai-enforced-step-flagged/prompt.md new file mode 100644 index 00000000000..df9ee977c6a --- /dev/null +++ b/plugins/harness-config/evals/ai-enforced-step-flagged/prompt.md @@ -0,0 +1,35 @@ +--- +description: "Medium. criteria.md Deletion tiers (enforced) and row I5: prose restating a rule a deterministic check already enforces is flagged for deletion with the check named, while I8's cull keeps a hard constraint no check enforces. In the 9-step PR procedure, step 8 (Conventional Commits PR title, which CI rejects otherwise) is enforced by the named pr-title.yml lint-title CI job, so it is flagged for deletion naming that job; step 5 (run tests and stop on failure) has no stated check, so it stays; step 1 (git status) is narration I8 culls. Read-only; no audit runs. Hard because a step CI enforces reads like an external contract a cull should keep." +tags: [audit-instructions, outcome, deletion-tiers] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Skill, Read, Glob, Grep] +expected_outcome: "The reply ends with step 1: cull, step 5: keep, step 8: delete-enforced, and names the pr-title.yml lint-title job as what enforces step 8." +--- + +Before we run /harness-config:audit-instructions on our repo, I need to know how its catalog treats one of our skills. It spells out a 9-step numbered procedure for opening a PR: + +1. run git status to see what changed +2. run git diff to review the changes +3. stage the files with git add +4. write a commit message +5. run the test suite and stop if it fails +6. commit +7. push the branch +8. title the PR per Conventional Commits (CI rejects other titles) +9. open the PR with gh pr create + +Our `.github/workflows/pr-title.yml` job `lint-title` fails any PR whose title is not in Conventional Commits form. Nothing checks that tests ran before a commit. + +Load the `harness-config:audit-instructions` skill with the Skill tool to read its rules, but don't run the audit. Read the skill's catalog (its `reference/criteria.md`: Deletion tiers, rows I5 and I8) and tell me, for steps 1, 5 and 8, which of these the catalog does with that step: + +- keep: stays in the procedure as a hard constraint +- cull: removed as narration a current model sequences on its own +- delete-enforced: flagged for deletion because a deterministic check already enforces it, with that check named + +Name the check for any step you mark delete-enforced. End your reply with exactly these three lines, each filled in with one of the three words: + +step 1: +step 5: +step 8: diff --git a/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-1-cull.json b/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-1-cull.json new file mode 100644 index 00000000000..d6c81bc9e52 --- /dev/null +++ b/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-1-cull.json @@ -0,0 +1,20 @@ +{ + "pass": [ + { + "answer": "step 1: cull\nstep 5: keep\nstep 8: delete-enforced" + }, + { + "answer": "`step 1`: cull" + } + ], + "fail": [ + { + "answer": "step 1: keep", + "why": "keeps narration" + }, + { + "answer": "step 10: cull", + "why": "names a different step" + } + ] +} diff --git a/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-5-keep.json b/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-5-keep.json new file mode 100644 index 00000000000..b15dc26e282 --- /dev/null +++ b/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-5-keep.json @@ -0,0 +1,20 @@ +{ + "pass": [ + { + "answer": "step 1: cull\nstep 5: keep\nstep 8: delete-enforced" + }, + { + "answer": "- step 5: keep" + } + ], + "fail": [ + { + "answer": "step 5: cull", + "why": "drops the test gate no check enforces" + }, + { + "answer": "step 5: delete-enforced", + "why": "no enforcing check is stated for step 5" + } + ] +} diff --git a/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-8-delete-enforced.json b/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-8-delete-enforced.json new file mode 100644 index 00000000000..710531b02ca --- /dev/null +++ b/plugins/harness-config/evals/ai-enforced-step-flagged/samples/step-8-delete-enforced.json @@ -0,0 +1,20 @@ +{ + "pass": [ + { + "answer": "step 1: cull\nstep 5: keep\nstep 8: delete-enforced" + }, + { + "answer": "**step 8**: delete-enforced" + } + ], + "fail": [ + { + "answer": "step 1: cull\nstep 5: keep\nstep 8: keep", + "why": "keeps the CI-enforced step as prose" + }, + { + "answer": "step 8: cull", + "why": "culls it as narration instead of naming the enforcing check" + } + ] +} diff --git a/plugins/harness-config/skills/audit-instructions/SKILL.md b/plugins/harness-config/skills/audit-instructions/SKILL.md index 6e3dd6aea8d..c3a3a784f6d 100644 --- a/plugins/harness-config/skills/audit-instructions/SKILL.md +++ b/plugins/harness-config/skills/audit-instructions/SKILL.md @@ -451,8 +451,8 @@ End with a **Routing** subsection listing every excluded upstream-owned or memory-layer surface and where its findings should go, plus the catalog's [AGENTS.md content-home advisory](reference/criteria.md#agentsmd-content-home-advisory), and a **Recommended follow-through** subsection. An editorial cut (removal would not change behavior, or the content is derivable) -may be applied from this report. A consequential deletion, a rule that governs a situation and -is outside the exception register, is applicable only when the commit cites a closed +or an enforced cut (prose a located check enforces) may be applied from this report. Any other +consequential deletion, a governing rule outside the exception register, applies only when the commit cites a closed `/harness-config:unhobble watch` (qualifying sessions met, zero attributed rows). Open the Sources line with the two official pages the paths and doctrine derive from (code.claude.com @@ -465,7 +465,7 @@ plainly that nothing has been applied. ## Next -- An editorial cut is applied from the report; a consequential cut cites a closed watch: `/harness-config:unhobble watch`. +- An editorial or enforced cut is applied from the report; another consequential cut cites a closed watch: `/harness-config:unhobble watch`. - A finding lands on the memory layer: `/harness-memory:audit`. - Project instructions still live in `CLAUDE.md`: `/instruction-placement:migrate plan`. - Posture guidance is absent rather than wrong: `/harness-config:audit-prompting-postures`. diff --git a/plugins/harness-config/skills/audit-instructions/evals/evals.json b/plugins/harness-config/skills/audit-instructions/evals/evals.json index 184c51ff6a5..b0b6945a264 100644 --- a/plugins/harness-config/skills/audit-instructions/evals/evals.json +++ b/plugins/harness-config/skills/audit-instructions/evals/evals.json @@ -65,12 +65,13 @@ "id": 6, "name": "step-list-culled-not-preserved", "prompt": "/harness-config:audit-instructions one of my skills spells out a 9-step numbered procedure for opening a PR: 1. run git status to see what changed; 2. run git diff to review the changes; 3. stage the files with git add; 4. write a commit message; 5. run the test suite and stop if it fails; 6. commit; 7. push the branch; 8. title the PR per Conventional Commits (CI rejects other titles); 9. open the PR with gh pr create. Should a capable model keep all the steps?", - "expected_output": "Flags the mechanical numbered procedure as an I8 model-era finding (over-prescriptive step list) and proposes culling it to the intent plus the hard constraints, removing the narration steps (status/diff/add/message/commit/push — a procedure a current model already sequences correctly) while keeping the steps that encode a real safety gate or external contract: run-tests-and-stop-on-failure (step 5) and the Conventional Commits PR-title requirement CI enforces (step 8). It does not preserve all 9 steps verbatim. Because I8 is behavioral-tier, the cull reaches the report as a verify-pass-vetted behavioral proposal, not an unverified confident removal.", + "expected_output": "Flags the mechanical numbered procedure as an I8 model-era finding (over-prescriptive step list) and proposes culling it to the intent plus the hard constraints, removing the narration steps (status/diff/add/message/commit/push — a procedure a current model already sequences correctly) while keeping the test gate, run-tests-and-stop-on-failure (step 5), which nothing in the prompt says a check enforces. Step 8, the Conventional Commits PR-title requirement, is not kept as prose: the prompt says CI enforces it, so it is flagged for deletion under I5's enforced case with the enforcing CI title check named, and the deletion clears once that check is located in the repository's CI rather than on the prose's own claim. It does not preserve all 9 steps verbatim. Because I8 is behavioral-tier, the cull reaches the report as a verify-pass-vetted behavioral proposal, not an unverified confident removal.", "files": [], "expectations": [ "Identifies the numbered step list as an I8 over-prescriptive finding", "Proposes culling to intent plus hard constraints rather than keeping all 9 steps verbatim", - "Keeps the test-gate step (run tests, stop on failure) and the CI-enforced Conventional Commits PR-title step as load-bearing", + "Keeps the test-gate step (run tests, stop on failure) as load-bearing", + "Flags the CI-enforced Conventional Commits PR-title step (step 8) for deletion rather than keeping it as prose, naming the CI title check as the enforcing check", "Culls narration steps (git status/diff/add/commit/push) that a current model sequences correctly from intent", "Surfaces the cull as a behavioral proposal that passed the fresh-context verify pass, not an unverified confident removal" ] diff --git a/plugins/harness-config/skills/audit-instructions/reference/criteria.md b/plugins/harness-config/skills/audit-instructions/reference/criteria.md index e11d1849bdd..5e31ed84c95 100644 --- a/plugins/harness-config/skills/audit-instructions/reference/criteria.md +++ b/plugins/harness-config/skills/audit-instructions/reference/criteria.md @@ -1,6 +1,6 @@ --- -version: 1.32.0 -last-updated: 2026-11-09 +version: 1.33.0 +last-updated: 2026-11-16 --- # Instruction-Audit Criteria @@ -280,17 +280,33 @@ each only on the surfaces its row names. ### Deletion tiers -Two deletion tiers, editorial and consequential, and one hold outside them, protected. One +Three deletion tiers, editorial, enforced and consequential, and one hold outside them, +protected. One grammar. The grammar is `unhobble`'s re-add gate: a ledger row, same-cause aggregation, and a commit that cites the rows. This catalog does not define a second grammar. The operational form is `/harness-config:unhobble watch`. - **Editorial.** I1 (removal would not change behavior), I4 (derivable or redundant), and stale scaffolding that restates the obvious. Propose the cut. No watch. +- **Enforced.** Prose outside the register restating a rule a deterministic check already + enforces: a hook, a CI check, a linter rule, or a test. The rule lives in the check, so the + prose is flagged for deletion even when it governs a situation, with the enforcing check named + by its location (the hook file, CI workflow and job, lint rule id, or test path). No watch: the + named check is what clears this tier. The check must be located in the repository or its CI, + must fail or block rather than only warn (a required CI check, an error-level lint rule, a hook + that denies), and must be active everywhere the prose loads; scope it does not reach keeps its + prose. The prose's own claim + that something enforces it ("CI rejects other titles") identifies the candidate but does not + clear it. Report such a candidate as flagged for deletion pending the check's location, never + as a kept hard constraint. Where the check covers less than the prose states, only the covered + part goes. - **Protected.** A candidate matching the instruction exception register. Hold. Never a watch - and never a deletion. Compression in place or hook conversion stays available. -- **Consequential.** A rule that governs a situation and is outside the register. Do not present - the deletion as applicable. Propose opening a watch for that rule. The cut becomes applicable + and never a deletion, even when a check enforces it: report the enforcing check beside the + hold. Compression in place or hook conversion stays available. When the register cannot be + read, hold a candidate that may match a class by consequence and say the register was unread. +- **Consequential.** A rule that governs a situation, outside the register, with no check that + meets the Enforced bar: no check at all, or one that only warns or reaches less than the prose + (the uncovered part). Do not present the deletion as applicable. Propose opening a watch for that rule. The cut becomes applicable only when a closed watch is cited: its qualifying-session count met and zero attributed rows. The recommended commit cites that watch. That citation is what clears this tier. @@ -302,7 +318,8 @@ Tier `mechanical` · Authority `ANTHROPIC-DOCS` · Severity `warning` · Surface something the model already does correctly. - **Remediate:** cut it. This check's cut is editorial (Deletion tiers) and does not require a watch. If the line enforces a governed situation, it is consequential: convert per I5 or open a - watch, and do not cut it on this check alone. + watch, and do not cut it on this check alone. If a deterministic check already enforces it, + flag it for deletion under I5 with the check named. - **Hold instead of delete** when the candidate matches a protected class in the [instruction exception register](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/instruction-exception-register/README.md), on I5's terms. This bar asks whether removal would change behavior *today*; a protected rail's @@ -383,7 +400,8 @@ Tier `mechanical` · Authority `ANTHROPIC-DOCS` · Severity `warning` · Surface knows, inlined API docs that should be a link, or self-evident practices. - **Remediate:** delete; link to the source of truth instead of inlining it. This cut is editorial (Deletion tiers) unless the line governs a situation, in which case it is - consequential and needs a closed watch before the deletion is applicable. + enforced when a located check already enforces it (I5), else consequential and needs a closed + watch before the deletion is applicable. - **Hold instead of delete** when the candidate matches a protected class in the [instruction exception register](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/instruction-exception-register/README.md) (the Gate 0 consequence classes, adopted there by reference for the deletion operation). Report @@ -398,17 +416,23 @@ Tier `mechanical` · Authority `ANTHROPIC-DOCS` · Severity `warning` · Surface Tier `mechanical` · Authority `ANTHROPIC-DOCS` · Severity `info` · Surfaces: I1–I5 partition. -- **Detect:** a rule the model already follows without it, or one that must fire every time with - zero exceptions. -- **Remediate:** an already-followed rule that does not govern a situation is an editorial - deletion. A rule that governs a situation is consequential: open a watch, and propose the - deletion only when a closed watch is cited (Deletion tiers). Convert a must-always rule to a - hook either way; conversion is not a deletion. +- **Detect:** a rule the model already follows without it, one that must fire every time with + zero exceptions, or one a deterministic check already enforces (a hook, a CI check, a linter + rule, a test). +- **Remediate:** a rule a deterministic check already enforces to the Enforced bar is not also + kept as prose: flag the prose for deletion and name the enforcing check (Deletion tiers, + enforced). Convert a must-always rule with no such check, including one whose check only warns + or reaches less than the prose, to a hook or another blocking check; once the check exists, + the prose goes in the same change, named against that check. An already-followed rule that does + not govern a situation is an editorial deletion. Any other rule that governs a situation, + including the part a warning-only or narrower check leaves uncovered, is consequential: open a + watch, and propose the deletion only when a closed watch is cited. - **Hold instead of delete** when the candidate matches a protected class in the [instruction exception register](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/instruction-exception-register/README.md). "The model already does this" is the weakest possible evidence against a rail whose absence is unrecoverable, and the hook conversion stays available: converting a protected rule to a - deterministic mechanism is a remediation, deleting it is not. + deterministic mechanism is a remediation, deleting it is not. A protected rule keeps its prose + even when a check enforces it; report the check beside the hold. - **Source:** best-practices, "Keep CLAUDE.md short" (already-followed rules) and "Enforce steps with hooks". @@ -474,7 +498,11 @@ disagrees with it. surface's to make, not this row's to override. - **Remediate:** propose removal or a briefer instruction; verify per Deletion tiers (a consequential removal needs a closed watch, an editorial one does not) that default performance - holds or improves. + holds or improves. When a cull keeps the hard constraints, a step a deterministic check already + enforces is not one of them: flag it for deletion with the check named, per I5 (on + memory-layer surfaces, the harness-memory audit's C8). A step matching a protected class in the + [instruction exception register](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/instruction-exception-register/README.md) + holds instead, with the check reported beside the hold. - **Bounded by:** the **Stopping condition** below, which is enabled by default. - **Source:** prompting best practices, "Leverage thinking & interleaved thinking capabilities" (the gate-meeting, model-agnostic one). Convergent model guide: Fable 5, "Recommended @@ -2681,7 +2709,8 @@ Unscoped. - **Adjacent rows:** on an agent definition, a line I1 or I4 would flag is reported once, under this row. I29 covers body text that restates the description. - **Remediate:** cut it. The cut is editorial (Deletion tiers) unless the line governs a situation, - in which case it is consequential and needs a closed watch. + in which case it is enforced when a located check already enforces it (I5), else consequential + and needs a closed watch. - **Source:** `/playbooks:skill-authoring` `## Agents`; skill authoring best practices, "Concise is key". Correlate: the vendor `prompt-audit` procedure's Step 3 deletion rule, read at the commit [bundled-claude-api.md](bundled-claude-api.md) pins. Pointer: [concise is @@ -2723,7 +2752,9 @@ more aggressive. - **Withhold** an I6 or I8 proposal where the instruction guards a high-consequence area: a safety gate, an irreversible or destructive action, a security or permission boundary, an external contract, or genuine ordering another step depends on. De-prescription is the default posture; it - is not the posture in the places where being wrong is expensive. + is not the posture in the places where being wrong is expensive. A step a located check already + enforces is not withheld on these grounds: the check guards it, so it goes per I5's enforced + route, and a protected class in the instruction exception register still holds there. - **Report every withholding** in the run's own section, naming the check it moderated and the ground. A silently suppressed finding reads as coverage. - **Source:** none. A high-consequence carve-out appears on no official page, and it is the diff --git a/plugins/harness-memory/evals/audit-c8-enforced-prose/case.yaml b/plugins/harness-memory/evals/audit-c8-enforced-prose/case.yaml new file mode 100644 index 00000000000..567a511c04e --- /dev/null +++ b/plugins/harness-memory/evals/audit-c8-enforced-prose/case.yaml @@ -0,0 +1,21 @@ +schema_version: "1.1" +name: audit-c8-enforced-prose +graders: + - name: 'pr-title-delete' + type: 'regex' + pattern: '^[\s>*`_-]*pr-title[*`_\s]*:[*`_\s]*delete\b' + flags: 'im' + target: 'last_message' + - name: 'force-push-hold' + type: 'regex' + pattern: '^[\s>*`_-]*force-push[*`_\s]*:[*`_\s]*hold\b' + flags: 'im' + target: 'last_message' + - name: 'meters-keep' + type: 'regex' + pattern: '^[\s>*`_-]*meters[*`_\s]*:[*`_\s]*keep\b' + flags: 'im' + target: 'last_message' + - name: 'checks-named' + type: 'llm' + criteria: 'PASS only if the reply names the pr-title.yml lint-title job as the check enforcing the PR-title line it flags for deletion, and names the block-force-push.sh hook beside the force-push hold, giving a protected class in the instruction exception register (such as a destructive-action or security rail) as the reason the force-push prose stays. FAIL if either check is not named, or the force-push line is flagged for deletion.' diff --git a/plugins/harness-memory/evals/audit-c8-enforced-prose/prompt.md b/plugins/harness-memory/evals/audit-c8-enforced-prose/prompt.md new file mode 100644 index 00000000000..1c882af9809 --- /dev/null +++ b/plugins/harness-memory/evals/audit-c8-enforced-prose/prompt.md @@ -0,0 +1,27 @@ +--- +description: 'Medium. criteria.md C8 items 4 and 5: a CLAUDE.md instruction a deterministic check already enforces is flagged for deletion with the check named, and one matching a protected class in the instruction exception register is held even when a check enforces it. The PR-title line is enforced by a CI job, so it is delete; the never-force-push line is enforced by a hook but is a protected rail (the register''s own example), so it is hold; the meters line has no check and is keep. Read-only; no audit runs. Bash is granted for the skill''s spine. Hard because both enforced lines look alike and only one may go.' +tags: [audit, harness-memory, enforcement] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Read, Glob, Grep, Skill, Bash] +expected_outcome: 'The reply ends with pr-title: delete, force-push: hold, meters: keep, naming the pr-title CI job for the first line and the force-push hook beside the hold.' +--- + +Before we run /harness-memory:audit on our repo, I need to know how its C8 check treats three lines in our project CLAUDE.md: + +- pr-title: "Title every pull request in Conventional Commits form." Our `.github/workflows/pr-title.yml` job `lint-title` fails any other title. +- force-push: "Never force-push to main." Our `.claude/hooks/block-force-push.sh` PreToolUse hook blocks every force push to main. +- meters: "Store distances in meters." Nothing checks this. + +Load the `harness-memory:audit` skill with the Skill tool to read its rules, but don't run the audit. Read its `reference/criteria.md`, check C8, and tell me, for each line, which of these C8 does: + +- delete: flags the prose for deletion because a check already enforces it +- hold: keeps the prose even though a check enforces it +- keep: no deletion finding + +Name the enforcing check for any line you mark delete or hold. End your reply with exactly these three lines, each filled in with one of the three words: + +pr-title: +force-push: +meters: diff --git a/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/force-push-hold.json b/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/force-push-hold.json new file mode 100644 index 00000000000..2c8bc930e57 --- /dev/null +++ b/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/force-push-hold.json @@ -0,0 +1,10 @@ +{ + "pass": [ + {"answer": "pr-title: delete\nforce-push: hold\nmeters: keep"}, + {"answer": "- force-push: hold"} + ], + "fail": [ + {"answer": "force-push: delete", "why": "deletes a protected rail"}, + {"answer": "force-push-main: hold", "why": "names a different line"} + ] +} diff --git a/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/meters-keep.json b/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/meters-keep.json new file mode 100644 index 00000000000..0586dda8f11 --- /dev/null +++ b/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/meters-keep.json @@ -0,0 +1,10 @@ +{ + "pass": [ + {"answer": "pr-title: delete\nforce-push: hold\nmeters: keep"}, + {"answer": "`meters`: keep"} + ], + "fail": [ + {"answer": "meters: delete", "why": "no check enforces it"}, + {"answer": "kilometers: keep", "why": "names a different line"} + ] +} diff --git a/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/pr-title-delete.json b/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/pr-title-delete.json new file mode 100644 index 00000000000..d351d8e9875 --- /dev/null +++ b/plugins/harness-memory/evals/audit-c8-enforced-prose/samples/pr-title-delete.json @@ -0,0 +1,10 @@ +{ + "pass": [ + {"answer": "pr-title: delete\nforce-push: hold\nmeters: keep"}, + {"answer": "**pr-title**: delete"} + ], + "fail": [ + {"answer": "pr-title: keep", "why": "keeps prose a CI job enforces"}, + {"answer": "pr-title: hold", "why": "not a protected class"} + ] +} diff --git a/plugins/harness-memory/skills/audit/context/audit.md b/plugins/harness-memory/skills/audit/context/audit.md index 795a39b9013..aeb1984895a 100644 --- a/plugins/harness-memory/skills/audit/context/audit.md +++ b/plugins/harness-memory/skills/audit/context/audit.md @@ -117,7 +117,9 @@ where a project CLAUDE.md would in every pass below. Concrete passes: 1. Compare each CLAUDE.md (any scope) against every co-resident rule for contradictions 2. Compare CLAUDE.md against CLAUDE.local.md (same or cross-scope) for contradictions and redundancy 3. Compare rules against other co-resident rules for contradictions (same-scope and cross-scope) -4. Check if any CLAUDE.md instruction is already enforced by rule, hook, or analyzer +4. Check if any CLAUDE.md instruction is already enforced by rule, hook, or analyzer; when a + deterministic check enforces it, report it under C8 (flag the prose for deletion, name the + check, hold a protected class) 5. Compare the **user**-scope surfaces against the project ones for any remaining pair not covered above. Both load together in every session here, so a user instruction that contradicts a project one is a live conflict rather than a layering choice, and a user instruction the project already diff --git a/plugins/harness-memory/skills/audit/reference/criteria.md b/plugins/harness-memory/skills/audit/reference/criteria.md index 524d675482c..48a9cd0115e 100644 --- a/plugins/harness-memory/skills/audit/reference/criteria.md +++ b/plugins/harness-memory/skills/audit/reference/criteria.md @@ -7,8 +7,8 @@ - [Checks for auto-memory (MEMORY.md + topic files)](#checks-for-auto-memory-memorymd--topic-files) - [Audit output format](#audit-output-format) -Version: 1.7.0 -Last updated: 2026-10-11 +Version: 1.8.0 +Last updated: 2026-11-16 Source: official Claude Code docs, chiefly [How CLAUDE.md files load](https://code.claude.com/docs/en/memory#how-claude-md-files-load), [Keep CLAUDE.md short](https://code.claude.com/docs/en/best-practices#keep-claude-md-short), @@ -306,9 +306,29 @@ that posture honest, so give C5-kept navigation entries particular attention her - A Claude Code hook (deterministic) 2. WARN per instruction that could move up the hierarchy 3. Include which enforcement level it could move to +4. For each instruction a deterministic check already enforces (a hook, a CI check, a linter + rule, a test), WARN and flag the prose for deletion, naming the enforcing check by its location + (the hook file, CI workflow and job, lint rule id, or test path). The check must be located in + the repository or its CI, must fail or block rather than only warn, and must be active + everywhere the prose loads (scope it does not reach keeps its prose); the prose's own claim + that something enforces it is not enough. + Where the check covers less than the prose states, flag only the covered part. A check that + only warns, and the part a narrower check leaves uncovered, are not enforced: they keep their + prose and fall under item 2 (WARN that they could move up the hierarchy). A move up the + hierarchy from item 2 ends the same way: once the check exists, the prose goes in that change. +5. **Hold instead of delete** when the instruction matches a protected class in the + [instruction exception register](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/instruction-exception-register/README.md): + it keeps its prose even when a check enforces it. Report the hold, its class and the + enforcing check. Membership goes by consequence, what breaks when the rule is absent (a + destructive-action or security rail, for example); when the register cannot be read, hold a + candidate that may match one and say the register was unread. + +**Provenance**: items 4 and 5 are this repo's rule, not doc-stated, so the `update` action must +not overwrite them. **Why**: Prefer deterministic enforcement over documentation. When a guideline can become a -compile-time or runtime check, that is the stronger default. +compile-time or runtime check, that is the stronger default, and once the check exists the prose +is a second copy of the rule that can drift from it. ### C9: Build and Test Commands Present [FAIL]