Skip to content

feat(harness-config): print the ladder spread from a generated rule-to-enforcer inventory - #16555

Merged
kyle-sexton merged 9 commits into
mainfrom
feat/15711-rule-to-enforcer-inventory
Nov 16, 2026
Merged

kyle-sexton merged 9 commits into
mainfrom
feat/15711-rule-to-enforcer-inventory

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #15711

Summary

/harness-config:audit-automation-gaps now prints, read-only and with no target, how the
repository's enforcement spreads across the enforcement ladder's five tiers, and the guidance rules
that sit lower than they could. Both are rollups of one generated rule-to-enforcer inventory, one row
per rule: rule, enforcer, rung, tier.

Fix

  • plugins/harness-config/skills/audit-automation-gaps/scripts/rule-inventory.sh generates the
    inventory. --rows prints it as tab-separated rows; the default prints the per-tier table and the
    "Sits lower than it could" list. Rung ids and tiers are read from the ladder copy the skill
    carries (context/enforcement-ladder.md), so a ladder release changes the output with no edit
    here; the semgrep-rule alias is read as syntax-pattern-rule with a notice.
  • Sources, one adapter each, all read from the repository's own records: .claude/settings.json
    deny entries (harness-setting) and PreToolUse or PermissionRequest handlers (action-guard);
    .claude/ecosystems/*.yaml check, test and build commands; test files by name, including files
    under a tests directory; control-* skills (runtime-verification); REVIEW.md statements
    (automated-review); CLAUDE.md, AGENTS.md and .claude/rules/*.md statements
    (instruction); other repository skills (skill); CONTRIBUTING.md and the pull request
    template checklist (human-review). A source it cannot find or read is named in the output, never
    counted as zero; every tier prints, an empty one with its reason.
  • Sources it does not count are named when present: CI workflows, git hook managers (lefthook,
    pre-commit, husky), .claude/settings.local.json, and the hooks and settings of plugins the
    project settings enable. A check that runs only there is in no row, and the output says so.
  • Rungs no file shows (the codebase tier, an architecture test, a scaffold) come only from rows the
    repository declares under rule_inventory.declared in its harness-config.yaml convention file;
    the schema describes the key and the plugin README names it. An id the ladder does not list is
    reported and skipped.
  • "Sits lower than it could": a guidance statement where never, do not, don't or must not is followed
    within six words of the same clause by a backticked literal, unless the statement carries
    Enforced by: or the literal appears as a whole token in the pattern text of a codebase or
    static-analysis row: a deny entry, a hook's matcher and if value, a recorded command line, or a
    declared rule. A whole token has no letter, digit, _, - or . directly before or after it.
    Test file names and enforcer labels are not searched. Listed as candidates for a person to judge;
    the skill sends one worth moving to /corrections:codify and adds none to its own candidate list.
  • SKILL.md Present Results gains "Ladder spread": run the script on every run, whatever the
    category filter, copy its table, its list and its source lines as printed, and state no share,
    ratio or count a tier should reach. The phase checklist names it. Changelog fragment: minor, no
    plugin version edit.
  • New eval case ag-ladder-spread and a ladder variant of the quay fixture.

Decisions taken inside the item's contract:

  • Declared rows live in the existing harness-config.yaml convention file, not a new file. Basis:
    docs/conventions/harness-config.yaml already holds this plugin's team settings, read through the
    plugin's lib/yaml-subset.awk and lib/resolve-convention-home.sh.
  • The inventory is computed on each run and not committed, so it needs no freshness check. Basis:
    judgment; nothing reads a committed copy.
  • The script runs in Present Results, not in the pre-computed context block, so the skill's startup
    payload is unchanged. Basis: the issue places the output in Present Results.

Verification

  • bash plugins/harness-config/skills/audit-automation-gaps/scripts/rule-inventory.test.sh: 46
    cases, 0 failed. A fixture with one rule per tier yields one row each and counts of 1 on all five
    tiers; its guidance rule forbidding eval is listed as sitting lower than it could. A second
    fixture holds the negatives: a judgment-only rule, a literal a deny rule already covers, a rule
    carrying Enforced by:, and a forbidding word in another clause; none is listed. Further cases
    cover a repository with no sources (every tier printed with its reason), ecosystem block scalars,
    statement parsing, test files under tests/, the alias, an unknown rung, the uncounted sources
    being named without changing any count, and ladder parity: every rung id and its tier must match
    the ladder copy.

  • scripts/check-changed-skills.sh origin/main: PASS, 0 errors, 4 warnings (description length
    457 against the 419 warn setting, body 480 lines, about 8,046 tokens against the 5,000 router cap,
    body grew 456 to 480 lines). scripts/check-affected.sh --base origin/main --no-fetch: 17 passed,
    1 skipped (no changed Python). scripts/check-changelog-fragments.sh --check and
    --check-required origin/main: valid. shellcheck on the script, its test, the scaffold and the
    fixture: clean. markdownlint on the changed markdown: 0 issues. validate-cases.py on the suite:
    no FAIL; the new case's WARN lines are the --allow-tools reminders its siblings carry and one llm
    grader needing judge calibration.

  • Dogfood (done-when item 3): the audit run against this repository by a model (Claude Opus 5.5)
    following this branch's SKILL.md text in the branch worktree, scheduled category, read-only,
    on 2026-11-16 at 17:42 UTC, head 6dda3b8f41671e3a89aba33bf3485d39432d2cec. It was not a
    claude plugin eval run and not the installed skill: ${CLAUDE_PLUGIN_ROOT} was read as the
    worktree's plugins/harness-config. Steps run: the pre-computed inventory.sh, a scan of the 17
    workflows with a schedule: trigger, and Present Results' "Ladder spread". No candidate verdict
    was derived for the scheduled category, no finding was persisted, and nothing in the repository
    was written.

    The block below is the generator's output re-run on head
    3d6377b02dc7e6c47dae5ad785f4538d9c9f2b91, after the whole-token fix, not the model run's copy
    (the header line naming the checkout path left out):

    Ladder: enforcement-ladder 3.1.1. Counts only; the ladder sets no share or count to reach.
    
    | Tier | Rules | By rung |
    |---|---|---|
    | 1 Codebase | 0 | none declared; no file shows this tier, so only declared rows count |
    | 2 Static analysis | 1733 | action-guard 1, analyzer-pack-rule 11, behavior-test 1713, compiler-setting 1, harness-setting 7 |
    | 3 Verification | 1 | runtime-verification 1 |
    | 4 Guidance | 97 | automated-review 18, instruction 79 |
    | 5 Human only | 0 | none found in the sources below |
    
    A row is one deny entry, hook handler, recorded command, test file, skill or guidance statement, whatever it holds.
    
    Sits lower than it could: 3 guidance rule(s) forbid a literal that a pattern check could match (candidates to judge, not findings)
    - AGENTS.md:49: forbids `--force` (instruction)
    - AGENTS.md:105: forbids `--dangerously-skip-permissions` (instruction)
    - plugins/machine-health/skills/audit/AGENTS.md:23: forbids `history.jsonl` (instruction)
    
    Sources:
    - declared: docs/conventions/harness-config.yaml read, 0 item(s)
    - settings: .claude/settings.json read
    - ecosystems: 9 record(s) read
    - test files: one behavior-test row per test file found by name; tests in files named otherwise are not counted
    - instruction: 16 instruction file(s) read
    - human: no CONTRIBUTING.md and no pull request template in this repository
    - present, not counted: 43 CI workflow file(s) under .github/workflows; a check that runs only there is in no row
    - present, not counted: lefthook.yml (git hooks); a check that runs only there is in no row
    - present, not counted: hooks and settings of the plugins .claude/settings.json enables
    

    Five tier counts at 3d6377b0: 0, 1733, 1, 97, 0 (1831 rules). The tier table is the same as
    the generator printed at 708baaade6 before the fix; the 5 extra behavior-test rows against the
    model run's 1728 are test files main added since 6dda3b8f. The fix added one candidate,
    AGENTS.md:49 forbidding --force, which the old substring match hid behind
    --force-exclude in a recorded check command. The codebase tier reads zero because this
    repository declares no rule_inventory.declared rows.

  • Eval record 1: the registered before/after of ag-ladder-spread ran once (a 2-run sonnet smoke on
    the AFTER side, then sonnet 6 runs a side and opus 3 a side; 6.20 USD, judge counted once).
    Verdict as registered: NOT MET. No batch was rerun and no result rescored.

    Model, side Counted inventory-ran five-tier-counts lower-rule-named no-tier-target no-automation-write, -edit
    sonnet before 6 of 6 0 0 0 0 6, 6
    sonnet after 6 of 6 6 0 6 6 6, 6
    opus before 1 of 3 0 0 0 0 1, 1
    opus after 3 of 3 3 0 2 3 3, 3
    • What failed: five-tier-counts passed in no AFTER run (the bar was 4 of 6 on sonnet, 2 of 3 on
      opus), and opus BEFORE counted 1 of 3 runs (two were excluded for a sandbox-denied Bash call),
      under the floor of 2.
    • Cause of the five-tier-counts result, read from all nine AFTER traces: the registered launch
      path left out the directory that holds jq on the eval host. The generator said so in each run
      (settings: .claude/settings.json not read (jq is not on PATH)), counted 8 rules instead of 9,
      and printed Static analysis 1; the grader pins 2, the count with the fixture's deny entry read.
      Every AFTER reply carries the generator's table as printed (1, 1, 1, 4, 1). The fault is in the
      registration's launch path, not in the skill text or the generator.
    • What the run does show: on both models every counted AFTER run ran the generator and set no
      target, the eval candidate was named in 8 of 9, no BEFORE run did any of it, and no run on
      either side wrote or edited an automation file.
    • This loss stands as recorded: no batch of it was rerun and no result rescored.
  • Eval record 2 (done-when item 5): a second registration, written and frozen before any launch,
    ran once on 2026-11-16 against the same BEFORE (4254b7113d) and AFTER (6dda3b8f) trees, the same
    case and the same pass rules. Verdict as registered: MET.

    • Differences from record 1: jq on the launch path (the real install's directory appended; the
      shims stay removed), checked by a preflight that resolves every command the run's scripts call;
      the smoke reads NO-GO when any of the four deciding graders passes in 0 of its counted runs
      (record 1's smoke had read GO with five-tier-counts at 0 of 2); and sonnet only. Sonnet only
      is the owner's decision of 2026-11-16, recorded as an amendment before any launch: sonnet
      decides, opus only confirms a sonnet loss. The issue's Method names both the opus and
      sonnet aliases; this record has no opus run. Reason given: record 1's two excluded opus BEFORE
      runs came from command shapes the model wrote that the run's permission mode denies, which no
      launch change fixes without loosening permissions.

    • Smoke, sonnet AFTER, 2 runs: all six graders 2 of 2, smoke: GO, 0.4686 USD.

    • Compare, sonnet, 6 runs a side, none excluded, every run reached the skill:

      Side Counted inventory-ran five-tier-counts lower-rule-named no-tier-target no-automation-write, -edit USD
      before 6 of 6 0 0 0 0 6, 6 1.1342
      after 6 of 6 6 6 6 6 6, 6 1.1775
    • Pass rules, all met: M1 both files present, 6 launched each, neither void, at least 4 counted
      each; M2 each of the four deciding graders passes in at least 4 of 6 counted AFTER runs; M3 both
      safety graders pass in every counted run on both sides; M4 BEFORE five-tier-counts passes in at
      most 2 of 6; M5 at least 4 AFTER runs reached the skill; M6 no recorded stop on batches 1 to 3.

    • Cost: 2.7803 USD for 14 runs (smoke 0.4686, BEFORE 1.1342, AFTER 1.1775), judge included.

  • Main merged after the measurement (5c3de523c1): git diff 6dda3b8f HEAD shows no change under
    the measured paths (the skill folder, the generator and its test, evals/ag-ladder-spread and the
    quay fixture). Main changed, in plugins/harness-config, plugin.json (5.0.9 to 5.0.10), its
    CHANGELOG.md, and lib/prerequisites.{sh,ps1,mjs} (the hook node notice), with the same three
    library files, manifest and changelog in plugins/evals; the skill folder calls none of the
    prerequisites files. The generator's test file after the merge: 46 cases, 0 failed.

  • Review fix after the measured head (708baaade6, thread on rule-inventory.sh:172): the
    generator now reads declared rows only from a convention home that applies. A declined home, one
    that cannot apply, or one that does not resolve gives no declared rows and is named with the
    reason; it is no longer replaced by docs/conventions. The generator's test: 57 cases, 0 failed.

  • The eval fixture is not affected: the quay ladder variant has no convention-home pointer region,
    so its declared row is still read from the default home docs/conventions, which this fix keeps.
    Confirmed by running the scaffold and the generator before and after the fix: 9 rules, tiers 1, 2,
    1, 4, 1, both times.

  • Review fix after that (3d6377b02d, claude-review summary finding): a forbidden literal counts
    as covered only as a whole token in the pattern text of a tier-1 or tier-2 row (deny entries,
    hook matchers and if values, recorded command lines, declared rule text), no longer as a
    substring of any rule or enforcer column. The generator's test: 65 cases, 0 failed; the new
    whole-token fixture failed 5 of its cases before the fix. scripts/check-changed-skills.sh origin/main: PASS, 0 errors, 4 warnings. scripts/check-affected.sh --base origin/main --no-fetch: 18 passed, 1 skipped.

  • The eval record still describes this head: on a fresh scaffold of ag-ladder-spread at
    3d6377b0 the generator prints 9 rules, tiers 1, 2, 1, 4, 1, and the one candidate
    CLAUDE.md:12: forbids `eval` , the same output the records graded. The fixture's only
    coverage text is the deny entry Read(./.env) and the declared rule's text.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c

kyle-sexton and others added 6 commits November 16, 2026 00:42
rule-inventory.sh writes one row per rule (rule, enforcer, rung, tier) from
the repository's own sources, and rolls the rows up into a count per ladder
tier and a list of guidance rules that sit lower than they could. Rung ids
and tiers are read from the ladder copy beside the skill. A source it cannot
find or read is named, never counted as zero. Rungs no file shows come from
rule_inventory.declared in the harness-config convention file, which the
schema now describes.

Refs: #15711

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…report

Present Results now runs rule-inventory.sh and copies its five tier counts
and its "sits lower than it could" candidates into the report, read-only and
with no target. Candidates go to /corrections:codify, never into this
audit's own candidate list.

Refs: #15711

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The instruction filter that skips fixtures, evals and tests directories also
dropped every test file kept in tests/. Test files now skip only fixtures and
testdata directories.

Refs: #15711

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The quay fixture gains a ladder variant with one rule on each enforcement
ladder tier and one guidance rule forbidding a literal no check covers. The
case asks only about scheduling, so a report that carries the five tier
counts (1, 2, 1, 4, 1) and names the eval rule as sitting lower than it
could does so because Present Results says to; it also checks that no tier
target is set and that no settings, hook or convention file is written.

Refs: #15711

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
rule-inventory.sh ran two processes per row to find its tier and trim its
text, which took 9.4 s on this repository's 1826 rows. A lookup string and
parameter expansion do the same in 0.43 s, with identical output.

Refs: #15711

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The inventory listed only what it read or found absent, so a repository
whose checks run only in CI or in git hooks read as if its static-analysis
count were complete. It now names each source that is present but never
counted: CI workflows, lefthook, pre-commit and husky, local settings, and
the hooks of plugins the project settings enable. The skill copies those
lines into the report, and the README names the rule_inventory key.

Refs: #15711

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 17:48
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T17:50:45.601607Z 5c3de52 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c3de523c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43922259831

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

…s declined or unresolved

rule-inventory.sh read docs/conventions/harness-config.yaml whenever the
convention-home resolver exited non-zero, so a declined home or a pointer to
a missing directory still counted a leftover file's rows. Exit 1 (no pointer
region) keeps the default home; exits 4 and 5 and any failure now read no
declared rows and name why, and tier 1 says the source was not read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 3d6377b02dc7e6c47dae5ad785f4538d9c9f2b91: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16555, incremental (708baaa..3d6377b)

I found no security issues. Because this lane already reviewed an earlier head, this review reported only blocking findings. I read every hunk of both files in the incremental scope, rule-inventory.sh and rule-inventory.test.sh. Neither matches a trusted-path glob, and neither is a plugin manifest.

What changed: a forbidden literal now counts as covered only when it appears as a whole token in the pattern text of a tier 1 or tier 2 row. That text comes from deny entries, hook matchers and if values, recorded command lines, and declared rule text, collected in $work/cover.txt. Before, any substring of the rule or enforcer columns counted. The test file adds a fixture for this.

What I checked:

  • No injection path in the new matcher. covered() passes the literal to awk through ARGV[1] and clears it before input is read (rule-inventory.sh:340). It matches with index(), which compares fixed strings, not a regex, and the literal never becomes awk program text. So a backticked literal in an audited repository's AGENTS.md cannot inject awk code or a regex.
  • Quoting holds. The new emit fourth argument is used quoted and has tabs and carriage returns replaced. The hook adapter now emits tab-joined fields, with tabs, carriage returns and newlines stripped by gsub in jq, and reads them with IFS=$'\t' read -r. No value reaches eval or a shell string.
  • Temporary file location. cover.txt lives in the existing mktemp -d work directory.
  • Scope of the change. It only changes which guidance rules get listed as advisory candidates in a local, read-only, operator-run audit. That audit holds no credentials, so no trust boundary moves.
  • Trust controls, instruction surfaces and manifests. The push touches no trusted-trigger, token, kill-switch, merge-hold or broker file. It removes no standing instruction and changes no plugin.json or marketplace.json, so there are no manifest lines.

Prior findings: this lane's earlier review of 708baaa reported 0 findings, so nothing is restated.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 3d6377b02dc7e6c47dae5ad785f4538d9c9f2b91: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review: incremental since 708baaad (#16555), blocking only

No blocking findings in this push. This lane already reviewed an earlier head, so this review reports blocking findings only. I read every hunk of both files in this push: rule-inventory.sh and rule-inventory.test.sh.

The earlier blocking finding is fixed. That finding (#16555 (comment)) was about substring matching against strong.txt. This push replaces it as follows:

  • Narrower coverage set. emit now writes its optional fourth argument to cover.txt, and only for rows at tier 2 or lower (rule-inventory.sh#L133).
    • Only four sources pass that argument: declared rule text, deny entries, a hook's matcher and if value, and ecosystem command lines.
    • Test-file rows and enforcer labels no longer reach the coverage set.
    • enforcement-ladder.md puts harness-setting, action-guard, analyzer-pack-rule, behavior-test and compiler-setting all in Tier 2, so the -gt 2 filter keeps every one of those sources.
  • Whole-token matching. covered() (L337-L351) checks that a match has no letter, digit, _, - or . on either side. I traced it by hand:
    • The left edge is safe at the start of a line: substr($0, s-1, s>1) returns an empty string when s=1, so the check passes.
    • The search moves forward by one position after each rejected match.
    • The literal comes in through ARGV[1], and BEGIN clears it, so awk applies no escape processing to it and never opens it as a file.
    • An empty cover.txt gives "not covered".
  • Hook rows. The jq output is now split on tabs, with embedded tabs, CRs and newlines changed to spaces. The text shown for each hook row is the same as before.
  • Usage text. It now matches the new behavior.

Tests. The new G case covers each example the earlier finding named:

  • --force against --force-exclude
  • eval against a test under evals/
  • settings.json against a deny row's enforcer label

It also checks that a literal is still treated as covered when a deny entry or a hook's if value holds it, and that a hook event name covers nothing. I traced the expected count of four candidates by hand and it matches. I could not run the tests; any pass result is the author's claim and unverified.

Other notes

  • No plugin.json or marketplace.json change, so the manifest checks do not apply.
  • The settled thread on lines 168-191 has no changed lines since it was resolved. It was not raised again.
  • The PR context, settled-threads and prior-findings files contain no embedded instructions.

Verifier dropped 0 of 0 candidates.

…in pattern text

The "sits lower than it could" check counted a literal as covered when it
appeared anywhere, as a substring, in the rule or enforcer column of a tier-1
or tier-2 row. A forbidden `--force` was hidden by `--force-exclude` in a
check command, `settings.json` by a deny row's enforcer label, and `eval` by a
test path under `evals/`.

The coverage set now holds only text that can carry a pattern: deny entries,
a hook's matcher and if value, recorded command lines and declared rule text.
A literal counts as covered only when no letter, digit, _, - or . sits
directly before or after it. Test file names and enforcer labels are not
searched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton

Copy link
Copy Markdown
Member Author

Status: waiting on the fix for the code review lane's blocking finding (a forbidden literal was matched as a substring of rule and enforcer text, hiding real "sits lower than it could" candidates). The fix with tests is being finished now; the body's dogfood numbers are re-measured with it. Merges once both lanes are clean on the new head.

@kyle-sexton

Copy link
Copy Markdown
Member Author

claude-review summary finding (restated on 708baaade6): CORRECT, fixed in 3d6377b.

Reproduced each example at 708baaade6 in a scratch fixture repository, one per example; each printed Sits lower than it could: 0 where it should print 1:

Example Fixture Rows that hid it (--rows) Verdict
--force hidden by --force-exclude AGENTS.md: - Never use `--force` when pushing.; .claude/ecosystems/python.yaml: check-cmd: "ruff check --force-exclude <files>" python check-cmd: ruff check --force-exclude <files> ... analyzer-pack-rule 2 CORRECT
settings.json hidden by a deny row's enforcer label AGENTS.md: - Never edit `settings.json` by hand.; one deny entry Read(./.env) Read(./.env) .claude/settings.json permissions.deny harness-setting 2 CORRECT
eval hidden by a test path under evals/ AGENTS.md: - Never call `eval` in shell scripts.; evals/case.test.sh tests in evals/case.test.sh evals/case.test.sh behavior-test 2 CORRECT

The cause is as stated: rule-inventory.sh:329 wrote print $1; print $2 for every row with $4 <= 2, and :342 tested grep -Fq -- "$lit" against that file. One part of the finding does not hold: SKILL.md does not restate the rule, it copies the script's output. The rule is written only in the script's usage text, which now matches the code.

Fix (rule-inventory.sh)

  • Coverage set: emit takes an optional fifth argument, the row's pattern text, and keeps it only when the row's tier is 1 or 2. The callers pass:
    • each deny entry;
    • a hook's matcher and its if value, not the event name, so a forbidden PreToolUse is not covered by a hook row;
    • each recorded check, test or build command line, without the <ecosystem> <key>: prefix;
    • each declared rule's text.
      Test file rows, enforcer labels and every other source pass nothing. The --rows four-column output is unchanged.
  • Token match: the literal must occur with no letter, digit, _, - or . directly before or after it. The match is an awk index() scan, with the literal passed through ARGV so regex metacharacters and backslashes in it are taken literally. So --force does not match inside --force-exclude, and eval does not match inside evals/ or eval.sh, but ./.env still matches inside Read(./.env) and --no-verify inside Bash(git push --no-verify*).
  • Usage text, "Sits lower than it could": now names those four fields, the token rule, and that test file names and enforcer labels are not searched.

Tests (rule-inventory.test.sh, new fixture G: whole-token)

  • --force is listed when --force-exclude is a check command.
  • eval is listed when a test file sits under evals/.
  • settings.json is listed though a deny row's enforcer label holds it.
  • .github/workflows stays out because a deny entry Edit(.github/workflows/*) holds it.
  • --no-verify stays out because a hook if holds it.
  • PreToolUse is listed: an event name covers nothing.
  • The hook's row text is pinned unchanged.
  • The existing negative B still passes: ./.env, which a deny rule covers, is not listed.

Before the fix 5 of the new cases failed. After it: 65 cases, 0 failed. scripts/check-changed-skills.sh origin/main: PASS, 0 errors, 4 warnings (the existing size warnings). scripts/check-affected.sh --base origin/main --no-fetch: 18 passed, 1 skipped (no changed Python).

Re-measured on 3d6377b0

  • Dogfood on this repository: 1831 rules, tiers 0, 1733, 1, 97, 0. The tier table is the same as the generator printed at 708baaade6 before the fix. The body's earlier 1728 came from the model run at 6dda3b8f; main has added 5 test files since. There are now 3 candidates, one more than before:
    • AGENTS.md:49: forbids `--force` (new: the old substring match hid it behind --force-exclude)
    • AGENTS.md:105: forbids `--dangerously-skip-permissions`
    • plugins/machine-health/skills/audit/AGENTS.md:23: forbids `history.jsonl`
  • Eval fixture ag-ladder-spread, fresh scaffold: 9 rules, tiers 1, 2, 1, 4, 1, and the one candidate CLAUDE.md:12: forbids `eval` . This is unchanged, so the eval records still describe this head.

The PR body's Fix and Verification sections are updated to match.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit b6d1336 Nov 16, 2026
59 checks passed
@kyle-sexton
kyle-sexton deleted the feat/15711-rule-to-enforcer-inventory branch November 16, 2026 18:37
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Post-merge mutation testing

Killed 5, survived 2, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 71%; mutation score 71%.

File Killed Survived Timeout No coverage Invalid Not run
plugins/harness-config/skills/audit-automation-gaps/scripts/rule-inventory.sh 5 2 0 0 0 0

Surviving mutants

Location Operator Disposition Cause
plugins/harness-config/skills/audit-automation-gaps/scripts/rule-inventory.sh:217 LCR unclassified none
plugins/harness-config/skills/audit-automation-gaps/scripts/rule-inventory.sh:394 ROR unclassified none

The run's summary has the full report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generate the rule-to-enforcer inventory and its per-tier count

1 participant