Skip to content

fix(harness-config): flag instruction prose a deterministic check already enforces - #16615

Merged
kyle-sexton merged 4 commits into
mainfrom
fix/audit-instructions-enforced-prose
Nov 16, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
fix/audit-instructions-enforced-prose

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

No related issue: owner decision that a rule a deterministic check enforces is not also kept as prose in instruction files

Summary

harness-config:audit-instructions told the auditor to convert a must-always rule to a hook "either way; conversion is not a deletion", so prose stayed beside the check that enforces it, and legacy eval case 6 expected a CI-enforced step (the Conventional Commits PR title) to be kept. Two copies of one rule can drift apart. Now the audit flags such prose for deletion and names the enforcing check. harness-memory:audit C8, which covers CLAUDE.md where I1-I5 do not run, gets the same rule.

Fix

  • audit-instructions criteria (1.33.0): a new enforced deletion tier. Prose restating a rule a hook, CI check, linter rule or test already enforces is flagged for deletion, with the check named by its location and no watch needed. The prose's own claim of enforcement makes it a candidate pending the check's location, never a kept hard constraint. I5 drops the prose once a must-always rule becomes a check, and I1 and I8 route an enforced line or step to I5. Protected-class rules in the instruction exception register still hold, with the check reported beside the hold.

  • harness-memory:audit criteria (1.8.0): C8 items 4-5 flag already-enforced CLAUDE.md prose for deletion and hold protected classes. The cross-file pass that already detected enforced instructions now reports them under C8. The register's consumer table lists harness-memory:audit.

  • Evals: legacy case 6 now expects step 8 to be flagged for deletion while the test gate (step 5) stays. New runnable cases ai-enforced-step-flagged and audit-c8-enforced-prose cover both criteria. Changelog fragments (minor) for both plugins.

  • Review fixes: the enforced tier applies only outside the register, and the check must fail or block and be active wherever the prose loads. I4, I41, I8 and the Stopping condition use the enforced tier, and I8 names the register hold. When the register cannot be read, a possible protected candidate holds.

  • Review fix (654485e): Consequential now covers a governing rule outside the register with no check that meets the Enforced bar, including a warning-only check or one that covers less than the prose, so such prose needs a watch before deletion; I5 and harness-memory C8 item 4 match. The eval cases were not re-run for this commit (neither involves a warning-only or narrower check).

Verification

  • claude plugin eval on ai-enforced-step-flagged against the final text (3 runs, with-plugin arm only): 3/3 runs scored 1.00. An earlier version, whose prompt named no CI job, scored 0.50 on all 3: the model applied the "check must be located" rule and kept step 8. That led to the pending-location wording, and the case now names the job.
  • claude plugin eval on audit-c8-enforced-prose against the final text (3 runs, with-plugin arm only): 3/3 runs scored 1.00. An earlier version scored 0.50 on one run: the model could not fetch the register from the sandbox and deleted the force-push line, which led to the fail-closed hold. The final run had no Bash grant, because a symlink in this host's Docker config now stops the Bash sandbox from starting.
  • Local code and security reviews, standing in for the CI review lanes (their venue read local failing), found no blocking defect. The fixes for their findings are listed above.
  • validate-cases.py on both eval suites: no FAIL. check-changelog-fragments.sh --check: all valid. check-changed-skills.sh origin/main: 2 skills, 0 failed, and both skills' script tests passed. jq empty on evals.json: valid. Legacy evals.json case 6 was not model-run, since that format has no CLI runner. The new runnable case covers the same scenario.

Related

  • Instruction exception register: docs/contracts/instruction-exception-register/README.md

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

kyle-sexton and others added 2 commits November 16, 2026 14:30
…eady enforces

A rule a hook, CI check, linter rule or test enforces is no longer kept
as prose beside the check. audit-instructions gains an enforced deletion
tier: the prose is flagged for deletion with the enforcing check named
by location, with no watch, and the prose's own claim of enforcement does
not clear it. I5 drops the prose once a must-always rule becomes a check
instead of keeping both; I1 and I8 route an enforced line or step to I5.
Protected-class rules in the instruction exception register still hold,
with the check reported beside the hold.

harness-memory:audit C8 gets the same rule for memory-layer surfaces,
where I1-I5 do not run, and becomes a listed register consumer.

Legacy eval case 6 now expects the CI-enforced PR-title step to be
flagged for deletion. New runnable cases ai-enforced-step-flagged and
audit-c8-enforced-prose cover both criteria.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
- The enforced tier applies only outside the exception register, and the
  check must fail or block rather than warn and be active wherever the
  prose loads.
- I8's enforced-step route, the Stopping condition, I4 and I41 now use
  the enforced tier instead of the old two-tier rule, and I8 names the
  register hold. SKILL.md's report rule names the enforced cut.
- When the register cannot be read, a candidate that may match a
  protected class holds, in both audit-instructions and harness-memory C8.
- Register consumer table names I8's route; last-updated dates bumped;
  C8 eval grader names the register class.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:41
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:43:29.498644Z 342dff7 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43942159753

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review needs no new review of 080f092fc55c63a9d63dea02a191d7144cc0251f: no file of this pull request changed since the review of 654485e.

Security review (incremental since 342dff7): no security issues found

critical-findings: 0

I read both files in the incremental diff in full:

  • plugins/harness-config/skills/audit-instructions/reference/criteria.md
  • plugins/harness-memory/skills/audit/reference/criteria.md

This review was blocking only, because this lane already reviewed an earlier head.

What changed. Both edits are prose. They change how the audit treats a rule whose check only warns, or whose check covers less than the prose does. Such a rule no longer counts as enforced:

  • In audit-instructions, the Consequential tier and I5's Remediate step now send it to a watch before any deletion.
  • In harness-memory C8 item 4, it keeps its prose and falls under item 2.

Instruction-surface lens. This push makes the earlier guardrail stricter. Fewer rules can be flagged for deletion, since a warning-only or partial check no longer clears one. The push removes no rule, and every hold for a protected class in the register is unchanged. Nothing new reaches a model without a check.

There is no code, workflow, manifest, token or trust-control change in this push. The PR context has no text aimed at the review lane. The incremental change adds no new spend.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 342dff7daa3ab4bcbb505c63f40b97d417356d7b; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review needs no new review of 080f092fc55c63a9d63dea02a191d7144cc0251f: no file of this pull request changed since the review of 654485e.

Code review (incremental, blocking only): PR #16615 at 654485e5

Scope: the change since 342dff7d, covering two files. I read every hunk of both in .claude-lane/incremental.diff, and I read the surrounding sections of each file at head for context.

This lane already reviewed an earlier head, so this pass looks only for blocking findings and drops advisory ones.

What changed

  • plugins/harness-config/skills/audit-instructions/reference/criteria.md
    • The Consequential tier now covers a governing rule with no check that meets the Enforced bar: no check at all, a check that only warns, or one that reaches less than the prose (the part left uncovered).
    • I5's Remediate line now says the same thing. A must-always rule whose check only warns, or is narrower than the prose, gets converted to a blocking check. The part left uncovered is consequential and needs a watch.
  • plugins/harness-memory/skills/audit/reference/criteria.md
    • C8 item 4 now sends a warn-only check, and the part a narrower check leaves uncovered, to item 2: the prose stays, with a WARN that the rule could move up the hierarchy.

Consistency check

  • The new wording matches across the Deletion tiers block (Enforced bar at L291-302, Consequential at L307-311), I5's Remediate line and C8 item 4.
  • A rule backed by a warn-only or partial check now lands in exactly one tier. That closes the gap my earlier advisory finding described. The owner resolved that thread, and this change carries the fix, so I did not raise it again.
  • I found no contradiction with the Protected tier or with C8 item 5.

Findings: none blocking.

Closing issue: none. The PR cites an owner decision, so there were no acceptance criteria to check. I found no embedded instructions in the PR context, the settled threads or the prior findings.

Spend: none in this increment.

No verifier subagent ran because there were no candidate findings. Verifier dropped 0 of 0 candidates.

@kyle-sexton
kyle-sexton disabled auto-merge November 16, 2026 20:02
…k to consequential

A governing rule whose check only warns, or reaches less than the prose,
met neither the Enforced tier nor Consequential, which required
"enforced by no check". Consequential now covers any governing rule
outside the register with no check meeting the Enforced bar. I5 and
harness-memory C8 item 4 say the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton
kyle-sexton disabled auto-merge November 16, 2026 20:09
@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Held for the owner's review of the decisions behind this change. The hold comes off when the owner confirms them.

@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Hold lifted: the owner reviewed and confirmed the decisions behind this change (2026-11-16).

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit bfb9fd5 Nov 16, 2026
48 checks passed
@kyle-sexton
kyle-sexton deleted the fix/audit-instructions-enforced-prose branch November 16, 2026 21:58
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant