Repository navigation
fix(harness-config): flag instruction prose a deterministic check already enforces - #16615
Conversation
…eady enforces A rule a hook, CI check, linter rule or test enforces is no longer kept as prose beside the check. audit-instructions gains an enforced deletion tier: the prose is flagged for deletion with the enforcing check named by location, with no watch, and the prose's own claim of enforcement does not clear it. I5 drops the prose once a must-always rule becomes a check instead of keeping both; I1 and I8 route an enforced line or step to I5. Protected-class rules in the instruction exception register still hold, with the check reported beside the hold. harness-memory:audit C8 gets the same rule for memory-layer surfaces, where I1-I5 do not run, and becomes a listed register consumer. Legacy eval case 6 now expects the CI-enforced PR-title step to be flagged for deletion. New runnable cases ai-enforced-step-flagged and audit-c8-enforced-prose cover both criteria. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
- The enforced tier applies only outside the exception register, and the check must fail or block rather than warn and be active wherever the prose loads. - I8's enforced-step route, the Stopping condition, I4 and I41 now use the enforced tier instead of the old two-tier rule, and I8 names the register hold. SKILL.md's report rule names the enforced cut. - When the register cannot be read, a candidate that may match a protected class holds, in both audit-instructions and harness-memory C8. - Register consumer table names I8's route; last-updated dates bumped; C8 eval grader names the register class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Lane run report
Failed GitHub API calls
|
|
claude-security-review needs no new review of Security review (incremental since 342dff7): no security issues foundcritical-findings: 0 I read both files in the incremental diff in full:
This review was blocking only, because this lane already reviewed an earlier head. What changed. Both edits are prose. They change how the audit treats a rule whose check only warns, or whose check covers less than the prose does. Such a rule no longer counts as enforced:
Instruction-surface lens. This push makes the earlier guardrail stricter. Fewer rules can be flagged for deletion, since a warning-only or partial check no longer clears one. The push removes no rule, and every hold for a protected class in the register is unchanged. Nothing new reaches a model without a check. There is no code, workflow, manifest, token or trust-control change in this push. The PR context has no text aimed at the review lane. The incremental change adds no new spend. |
|
claude-review needs no new review of Code review (incremental, blocking only): PR #16615 at
|
…k to consequential A governing rule whose check only warns, or reaches less than the prose, met neither the Enforced tier nor Consequential, which required "enforced by no check". Consequential now covers any governing rule outside the register with no check meeting the Enforced bar. I5 and harness-memory C8 item 4 say the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
|
Held for the owner's review of the decisions behind this change. The hold comes off when the owner confirms them. |
|
Hold lifted: the owner reviewed and confirmed the decisions behind this change (2026-11-16). |
No related issue: owner decision that a rule a deterministic check enforces is not also kept as prose in instruction files
Summary
harness-config:audit-instructionstold the auditor to convert a must-always rule to a hook "either way; conversion is not a deletion", so prose stayed beside the check that enforces it, and legacy eval case 6 expected a CI-enforced step (the Conventional Commits PR title) to be kept. Two copies of one rule can drift apart. Now the audit flags such prose for deletion and names the enforcing check.harness-memory:auditC8, which covers CLAUDE.md where I1-I5 do not run, gets the same rule.Fix
audit-instructionscriteria (1.33.0): a new enforced deletion tier. Prose restating a rule a hook, CI check, linter rule or test already enforces is flagged for deletion, with the check named by its location and no watch needed. The prose's own claim of enforcement makes it a candidate pending the check's location, never a kept hard constraint. I5 drops the prose once a must-always rule becomes a check, and I1 and I8 route an enforced line or step to I5. Protected-class rules in the instruction exception register still hold, with the check reported beside the hold.harness-memory:auditcriteria (1.8.0): C8 items 4-5 flag already-enforced CLAUDE.md prose for deletion and hold protected classes. The cross-file pass that already detected enforced instructions now reports them under C8. The register's consumer table listsharness-memory:audit.Evals: legacy case 6 now expects step 8 to be flagged for deletion while the test gate (step 5) stays. New runnable cases
ai-enforced-step-flaggedandaudit-c8-enforced-prosecover both criteria. Changelog fragments (minor) for both plugins.Review fixes: the enforced tier applies only outside the register, and the check must fail or block and be active wherever the prose loads. I4, I41, I8 and the Stopping condition use the enforced tier, and I8 names the register hold. When the register cannot be read, a possible protected candidate holds.
Review fix (654485e): Consequential now covers a governing rule outside the register with no check that meets the Enforced bar, including a warning-only check or one that covers less than the prose, so such prose needs a watch before deletion; I5 and harness-memory C8 item 4 match. The eval cases were not re-run for this commit (neither involves a warning-only or narrower check).
Verification
claude plugin evalonai-enforced-step-flaggedagainst the final text (3 runs, with-plugin arm only): 3/3 runs scored 1.00. An earlier version, whose prompt named no CI job, scored 0.50 on all 3: the model applied the "check must be located" rule and kept step 8. That led to the pending-location wording, and the case now names the job.claude plugin evalonaudit-c8-enforced-proseagainst the final text (3 runs, with-plugin arm only): 3/3 runs scored 1.00. An earlier version scored 0.50 on one run: the model could not fetch the register from the sandbox and deleted the force-push line, which led to the fail-closed hold. The final run had no Bash grant, because a symlink in this host's Docker config now stops the Bash sandbox from starting.local failing), found no blocking defect. The fixes for their findings are listed above.validate-cases.pyon both eval suites: no FAIL.check-changelog-fragments.sh --check: all valid.check-changed-skills.sh origin/main: 2 skills, 0 failed, and both skills' script tests passed.jq emptyonevals.json: valid. Legacyevals.jsoncase 6 was not model-run, since that format has no CLI runner. The new runnable case covers the same scenario.Related
docs/contracts/instruction-exception-register/README.md🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC