Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: minor
---

### Changed

- `audit-instructions`: prose restating a rule a deterministic check already enforces (a hook, a CI check, a linter rule, a test) is flagged for deletion with the enforcing check named, under a new enforced deletion tier that needs no watch. I5 drops the prose once a must-always rule is converted to a check instead of keeping both, I1 and I8 route an enforced line or step to I5, and protected-class rules in the instruction exception register still hold, with the check reported beside the hold. Eval case 6 now expects the CI-enforced PR-title step to be flagged for deletion rather than kept.
7 changes: 7 additions & 0 deletions .changes/harness-memory/audit-c8-enforced-prose-3b9d05f6.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: minor
---

### Changed

- `audit`: C8 flags a CLAUDE.md instruction a deterministic check already enforces for deletion, naming the check, and holds an instruction matching a protected class in the instruction exception register. The cross-file pass that detects already-enforced instructions now reports them under C8.
5 changes: 3 additions & 2 deletions docs/contracts/instruction-exception-register/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,10 @@ over another is style.

| Consumer | How it uses the register |
|---|---|
| `harness-config:audit-instructions` | Deletion-class criteria (I1, I4, I5) hold back a candidate matching a protected class and report the hold rather than proposing the cut |
| `harness-config:audit-instructions` | Deletion-class criteria (I1, I4, I5, and I8's enforced-step route) hold back a candidate matching a protected class and report the hold rather than proposing the cut |
| `harness-memory:audit` | C8 holds a memory-layer instruction matching a protected class when a deterministic check enforces it, reporting the hold and the check rather than flagging the prose for deletion |
| `harness-config:unhobble` | The bare-baseline experiment may strip a protected rule during the run, since the strip is reversible and branch-local, but Phase 4 restores it regardless of whether the ledger logged a stumble against it. A protected rule is never left deleted on the evidence of "no stumble was observed", and it never enters a deletion watch. A consequential rule outside the register is deletable only when a closed watch has zero attributed rows |
| `review:security-review` | The instruction-surface lens reads the register on any diff that deletes, narrows, or softens a standing instruction, and its skip gate may not wave such a diff through as prose-only. This is the review half of a guardrail deletion: the holds above act only inside the two trimming skills, while a deletion by hand or by any other route meets this lens when its pull request is reviewed (#3566) |
| `review:security-review` | The instruction-surface lens reads the register on any diff that deletes, narrows, or softens a standing instruction, and its skip gate may not wave such a diff through as prose-only. This is the review half of a guardrail deletion: the holds above act only inside the trimming skills listed above, while a deletion by hand or by any other route meets this lens when its pull request is reviewed (#3566) |
| `review` plugin's `security-reviewer` agent | Behind `/review:quality-gate` security mode and `/review:fanout`. Applies the same lens whenever either dispatches it; `/review:fanout` dispatches it by its tier rule, so a small diff touching no security-sensitive path does not reach it |
| `instruction-placement:*` | Unchanged. It owns the classes and the relocation verdict; this register is the deletion counterpart and defers to it on class membership |

Expand Down
21 changes: 21 additions & 0 deletions plugins/harness-config/evals/ai-enforced-step-flagged/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
schema_version: "1.1"
name: ai-enforced-step-flagged
graders:
- name: step-1-cull
type: regex
pattern: '^[\s>*`_-]*step 1[*`_\s]*:[*`_\s]*cull\b'
flags: 'im'
target: 'last_message'
- name: step-5-keep
type: regex
pattern: '^[\s>*`_-]*step 5[*`_\s]*:[*`_\s]*keep\b'
flags: 'im'
target: 'last_message'
- name: step-8-delete-enforced
type: regex
pattern: '^[\s>*`_-]*step 8[*`_\s]*:[*`_\s]*delete-enforced\b'
flags: 'im'
target: 'last_message'
- name: check-named
type: llm
criteria: 'PASS only if the reply names the pr-title.yml lint-title CI job as what enforces step 8, and says step 8 is flagged for deletion rather than kept as a hard constraint. FAIL if step 8 is kept, or no enforcing check is named for it.'
35 changes: 35 additions & 0 deletions plugins/harness-config/evals/ai-enforced-step-flagged/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
description: "Medium. criteria.md Deletion tiers (enforced) and row I5: prose restating a rule a deterministic check already enforces is flagged for deletion with the check named, while I8's cull keeps a hard constraint no check enforces. In the 9-step PR procedure, step 8 (Conventional Commits PR title, which CI rejects otherwise) is enforced by the named pr-title.yml lint-title CI job, so it is flagged for deletion naming that job; step 5 (run tests and stop on failure) has no stated check, so it stays; step 1 (git status) is narration I8 culls. Read-only; no audit runs. Hard because a step CI enforces reads like an external contract a cull should keep."
tags: [audit-instructions, outcome, deletion-tiers]
runs: 3
max_turns: 12
timeout_seconds: 600
allowed_tools: [Skill, Read, Glob, Grep]
expected_outcome: "The reply ends with step 1: cull, step 5: keep, step 8: delete-enforced, and names the pr-title.yml lint-title job as what enforces step 8."
---

Before we run /harness-config:audit-instructions on our repo, I need to know how its catalog treats one of our skills. It spells out a 9-step numbered procedure for opening a PR:

1. run git status to see what changed
2. run git diff to review the changes
3. stage the files with git add
4. write a commit message
5. run the test suite and stop if it fails
6. commit
7. push the branch
8. title the PR per Conventional Commits (CI rejects other titles)
9. open the PR with gh pr create

Our `.github/workflows/pr-title.yml` job `lint-title` fails any PR whose title is not in Conventional Commits form. Nothing checks that tests ran before a commit.

Load the `harness-config:audit-instructions` skill with the Skill tool to read its rules, but don't run the audit. Read the skill's catalog (its `reference/criteria.md`: Deletion tiers, rows I5 and I8) and tell me, for steps 1, 5 and 8, which of these the catalog does with that step:

- keep: stays in the procedure as a hard constraint
- cull: removed as narration a current model sequences on its own
- delete-enforced: flagged for deletion because a deterministic check already enforces it, with that check named

Name the check for any step you mark delete-enforced. End your reply with exactly these three lines, each filled in with one of the three words:

step 1: <keep|cull|delete-enforced>
step 5: <keep|cull|delete-enforced>
step 8: <keep|cull|delete-enforced>
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"pass": [
{
"answer": "step 1: cull\nstep 5: keep\nstep 8: delete-enforced"
},
{
"answer": "`step 1`: cull"
}
],
"fail": [
{
"answer": "step 1: keep",
"why": "keeps narration"
},
{
"answer": "step 10: cull",
"why": "names a different step"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"pass": [
{
"answer": "step 1: cull\nstep 5: keep\nstep 8: delete-enforced"
},
{
"answer": "- step 5: keep"
}
],
"fail": [
{
"answer": "step 5: cull",
"why": "drops the test gate no check enforces"
},
{
"answer": "step 5: delete-enforced",
"why": "no enforcing check is stated for step 5"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"pass": [
{
"answer": "step 1: cull\nstep 5: keep\nstep 8: delete-enforced"
},
{
"answer": "**step 8**: delete-enforced"
}
],
"fail": [
{
"answer": "step 1: cull\nstep 5: keep\nstep 8: keep",
"why": "keeps the CI-enforced step as prose"
},
{
"answer": "step 8: cull",
"why": "culls it as narration instead of naming the enforcing check"
}
]
}
6 changes: 3 additions & 3 deletions plugins/harness-config/skills/audit-instructions/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -451,8 +451,8 @@ End with a **Routing** subsection listing every excluded upstream-owned
or memory-layer surface and where its findings should go, plus the catalog's
[AGENTS.md content-home advisory](reference/criteria.md#agentsmd-content-home-advisory), and a **Recommended follow-through**
subsection. An editorial cut (removal would not change behavior, or the content is derivable)
may be applied from this report. A consequential deletion, a rule that governs a situation and
is outside the exception register, is applicable only when the commit cites a closed
or an enforced cut (prose a located check enforces) may be applied from this report. Any other
consequential deletion, a governing rule outside the exception register, applies only when the commit cites a closed
`/harness-config:unhobble watch` (qualifying sessions met, zero attributed rows).

Open the Sources line with the two official pages the paths and doctrine derive from (code.claude.com
Expand All @@ -465,7 +465,7 @@ plainly that nothing has been applied.

## Next

- An editorial cut is applied from the report; a consequential cut cites a closed watch: `/harness-config:unhobble watch`.
- An editorial or enforced cut is applied from the report; another consequential cut cites a closed watch: `/harness-config:unhobble watch`.
- A finding lands on the memory layer: `/harness-memory:audit`.
- Project instructions still live in `CLAUDE.md`: `/instruction-placement:migrate plan`.
- Posture guidance is absent rather than wrong: `/harness-config:audit-prompting-postures`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,13 @@
"id": 6,
"name": "step-list-culled-not-preserved",
"prompt": "/harness-config:audit-instructions one of my skills spells out a 9-step numbered procedure for opening a PR: 1. run git status to see what changed; 2. run git diff to review the changes; 3. stage the files with git add; 4. write a commit message; 5. run the test suite and stop if it fails; 6. commit; 7. push the branch; 8. title the PR per Conventional Commits (CI rejects other titles); 9. open the PR with gh pr create. Should a capable model keep all the steps?",
"expected_output": "Flags the mechanical numbered procedure as an I8 model-era finding (over-prescriptive step list) and proposes culling it to the intent plus the hard constraints, removing the narration steps (status/diff/add/message/commit/push — a procedure a current model already sequences correctly) while keeping the steps that encode a real safety gate or external contract: run-tests-and-stop-on-failure (step 5) and the Conventional Commits PR-title requirement CI enforces (step 8). It does not preserve all 9 steps verbatim. Because I8 is behavioral-tier, the cull reaches the report as a verify-pass-vetted behavioral proposal, not an unverified confident removal.",
"expected_output": "Flags the mechanical numbered procedure as an I8 model-era finding (over-prescriptive step list) and proposes culling it to the intent plus the hard constraints, removing the narration steps (status/diff/add/message/commit/push — a procedure a current model already sequences correctly) while keeping the test gate, run-tests-and-stop-on-failure (step 5), which nothing in the prompt says a check enforces. Step 8, the Conventional Commits PR-title requirement, is not kept as prose: the prompt says CI enforces it, so it is flagged for deletion under I5's enforced case with the enforcing CI title check named, and the deletion clears once that check is located in the repository's CI rather than on the prose's own claim. It does not preserve all 9 steps verbatim. Because I8 is behavioral-tier, the cull reaches the report as a verify-pass-vetted behavioral proposal, not an unverified confident removal.",
"files": [],
"expectations": [
"Identifies the numbered step list as an I8 over-prescriptive finding",
"Proposes culling to intent plus hard constraints rather than keeping all 9 steps verbatim",
"Keeps the test-gate step (run tests, stop on failure) and the CI-enforced Conventional Commits PR-title step as load-bearing",
"Keeps the test-gate step (run tests, stop on failure) as load-bearing",
"Flags the CI-enforced Conventional Commits PR-title step (step 8) for deletion rather than keeping it as prose, naming the CI title check as the enforcing check",
"Culls narration steps (git status/diff/add/commit/push) that a current model sequences correctly from intent",
"Surfaces the cull as a behavioral proposal that passed the fresh-context verify pass, not an unverified confident removal"
]
Expand Down
Loading
Loading