Repository navigation
feat(harness-config): print the ladder spread from a generated rule-to-enforcer inventory - #16555
Conversation
rule-inventory.sh writes one row per rule (rule, enforcer, rung, tier) from the repository's own sources, and rolls the rows up into a count per ladder tier and a list of guidance rules that sit lower than they could. Rung ids and tiers are read from the ladder copy beside the skill. A source it cannot find or read is named, never counted as zero. Rungs no file shows come from rule_inventory.declared in the harness-config convention file, which the schema now describes. Refs: #15711 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…report Present Results now runs rule-inventory.sh and copies its five tier counts and its "sits lower than it could" candidates into the report, read-only and with no target. Candidates go to /corrections:codify, never into this audit's own candidate list. Refs: #15711 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The instruction filter that skips fixtures, evals and tests directories also dropped every test file kept in tests/. Test files now skip only fixtures and testdata directories. Refs: #15711 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The quay fixture gains a ladder variant with one rule on each enforcement ladder tier and one guidance rule forbidding a literal no check covers. The case asks only about scheduling, so a report that carries the five tier counts (1, 2, 1, 4, 1) and names the eval rule as sitting lower than it could does so because Present Results says to; it also checks that no tier target is set and that no settings, hook or convention file is written. Refs: #15711 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
rule-inventory.sh ran two processes per row to find its tier and trim its text, which took 9.4 s on this repository's 1826 rows. A lookup string and parameter expansion do the same in 0.43 s, with identical output. Refs: #15711 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The inventory listed only what it read or found absent, so a repository whose checks run only in CI or in git hooks read as if its static-analysis count were complete. It now names each source that is present but never counted: CI workflows, lefthook, pre-commit and husky, local settings, and the hooks of plugins the project settings enable. The skill copies those lines into the report, and the README names the rule_inventory key. Refs: #15711 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c3de523c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Lane run report
Failed GitHub API calls
|
…s declined or unresolved rule-inventory.sh read docs/conventions/harness-config.yaml whenever the convention-home resolver exited non-zero, so a declined home or a pointer to a missing directory still counted a leftover file's rows. Exit 1 (no pointer region) keeps the default home; exits 4 and 5 and any failure now read no declared rows and name why, and tier 1 says the source was not read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
claude-security-review reviewed Security review: PR #16555, incremental (708baaa..3d6377b)I found no security issues. Because this lane already reviewed an earlier head, this review reported only blocking findings. I read every hunk of both files in the incremental scope, What changed: a forbidden literal now counts as covered only when it appears as a whole token in the pattern text of a tier 1 or tier 2 row. That text comes from deny entries, hook matchers and What I checked:
Prior findings: this lane's earlier review of 708baaa reported 0 findings, so nothing is restated. critical-findings: 0 |
|
claude-review reviewed Code review: incremental since
|
…in pattern text The "sits lower than it could" check counted a literal as covered when it appeared anywhere, as a substring, in the rule or enforcer column of a tier-1 or tier-2 row. A forbidden `--force` was hidden by `--force-exclude` in a check command, `settings.json` by a deny row's enforcer label, and `eval` by a test path under `evals/`. The coverage set now holds only text that can carry a pattern: deny entries, a hook's matcher and if value, recorded command lines and declared rule text. A literal counts as covered only when no letter, digit, _, - or . sits directly before or after it. Test file names and enforcer labels are not searched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Status: waiting on the fix for the code review lane's blocking finding (a forbidden literal was matched as a substring of rule and enforcer text, hiding real "sits lower than it could" candidates). The fix with tests is being finished now; the body's dogfood numbers are re-measured with it. Merges once both lanes are clean on the new head. |
|
claude-review summary finding (restated on Reproduced each example at
The cause is as stated: Fix (
Tests (
Before the fix 5 of the new cases failed. After it: 65 cases, 0 failed. Re-measured on
The PR body's Fix and Verification sections are updated to match. |
Post-merge mutation testingKilled 5, survived 2, timeout 0, no coverage 0, invalid 0, not run 0.
Surviving mutants
The run's summary has the full report. |
Closes #15711
Summary
/harness-config:audit-automation-gapsnow prints, read-only and with no target, how therepository's enforcement spreads across the enforcement ladder's five tiers, and the guidance rules
that sit lower than they could. Both are rollups of one generated rule-to-enforcer inventory, one row
per rule: rule, enforcer, rung, tier.
Fix
plugins/harness-config/skills/audit-automation-gaps/scripts/rule-inventory.shgenerates theinventory.
--rowsprints it as tab-separated rows; the default prints the per-tier table and the"Sits lower than it could" list. Rung ids and tiers are read from the ladder copy the skill
carries (
context/enforcement-ladder.md), so a ladder release changes the output with no edithere; the
semgrep-rulealias is read assyntax-pattern-rulewith a notice..claude/settings.jsondeny entries (
harness-setting) and PreToolUse or PermissionRequest handlers (action-guard);.claude/ecosystems/*.yamlcheck, test and build commands; test files by name, including filesunder a
testsdirectory;control-*skills (runtime-verification);REVIEW.mdstatements(
automated-review);CLAUDE.md,AGENTS.mdand.claude/rules/*.mdstatements(
instruction); other repository skills (skill);CONTRIBUTING.mdand the pull requesttemplate checklist (
human-review). A source it cannot find or read is named in the output, nevercounted as zero; every tier prints, an empty one with its reason.
pre-commit, husky),
.claude/settings.local.json, and the hooks and settings of plugins theproject settings enable. A check that runs only there is in no row, and the output says so.
repository declares under
rule_inventory.declaredin itsharness-config.yamlconvention file;the schema describes the key and the plugin README names it. An id the ladder does not list is
reported and skipped.
within six words of the same clause by a backticked literal, unless the statement carries
Enforced by:or the literal appears as a whole token in the pattern text of a codebase orstatic-analysis row: a deny entry, a hook's matcher and
ifvalue, a recorded command line, or adeclared rule. A whole token has no letter, digit,
_,-or.directly before or after it.Test file names and enforcer labels are not searched. Listed as candidates for a person to judge;
the skill sends one worth moving to
/corrections:codifyand adds none to its own candidate list.SKILL.mdPresent Results gains "Ladder spread": run the script on every run, whatever thecategory filter, copy its table, its list and its source lines as printed, and state no share,
ratio or count a tier should reach. The phase checklist names it. Changelog fragment:
minor, noplugin version edit.
ag-ladder-spreadand aladdervariant of the quay fixture.Decisions taken inside the item's contract:
harness-config.yamlconvention file, not a new file. Basis:docs/conventions/harness-config.yamlalready holds this plugin's team settings, read through theplugin's
lib/yaml-subset.awkandlib/resolve-convention-home.sh.judgment; nothing reads a committed copy.
payload is unchanged. Basis: the issue places the output in Present Results.
Verification
bash plugins/harness-config/skills/audit-automation-gaps/scripts/rule-inventory.test.sh: 46cases, 0 failed. A fixture with one rule per tier yields one row each and counts of 1 on all five
tiers; its guidance rule forbidding
evalis listed as sitting lower than it could. A secondfixture holds the negatives: a judgment-only rule, a literal a deny rule already covers, a rule
carrying
Enforced by:, and a forbidding word in another clause; none is listed. Further casescover a repository with no sources (every tier printed with its reason), ecosystem block scalars,
statement parsing, test files under
tests/, the alias, an unknown rung, the uncounted sourcesbeing named without changing any count, and ladder parity: every rung id and its tier must match
the ladder copy.
scripts/check-changed-skills.sh origin/main: PASS, 0 errors, 4 warnings (description length457 against the 419 warn setting, body 480 lines, about 8,046 tokens against the 5,000 router cap,
body grew 456 to 480 lines).
scripts/check-affected.sh --base origin/main --no-fetch: 17 passed,1 skipped (no changed Python).
scripts/check-changelog-fragments.sh --checkand--check-required origin/main: valid.shellcheckon the script, its test, the scaffold and thefixture: clean. markdownlint on the changed markdown: 0 issues.
validate-cases.pyon the suite:no FAIL; the new case's WARN lines are the
--allow-toolsreminders its siblings carry and one llmgrader needing judge calibration.
Dogfood (done-when item 3): the audit run against this repository by a model (Claude Opus 5.5)
following this branch's
SKILL.mdtext in the branch worktree,scheduledcategory, read-only,on 2026-11-16 at 17:42 UTC, head
6dda3b8f41671e3a89aba33bf3485d39432d2cec. It was not aclaude plugin evalrun and not the installed skill:${CLAUDE_PLUGIN_ROOT}was read as theworktree's
plugins/harness-config. Steps run: the pre-computedinventory.sh, a scan of the 17workflows with a
schedule:trigger, and Present Results' "Ladder spread". No candidate verdictwas derived for the
scheduledcategory, no finding was persisted, and nothing in the repositorywas written.
The block below is the generator's output re-run on head
3d6377b02dc7e6c47dae5ad785f4538d9c9f2b91, after the whole-token fix, not the model run's copy(the header line naming the checkout path left out):
Five tier counts at
3d6377b0: 0, 1733, 1, 97, 0 (1831 rules). The tier table is the same asthe generator printed at
708baaade6before the fix; the 5 extra behavior-test rows against themodel run's 1728 are test files main added since
6dda3b8f. The fix added one candidate,AGENTS.md:49forbidding--force, which the old substring match hid behind--force-excludein a recorded check command. The codebase tier reads zero because thisrepository declares no
rule_inventory.declaredrows.Eval record 1: the registered before/after of
ag-ladder-spreadran once (a 2-run sonnet smoke onthe AFTER side, then sonnet 6 runs a side and opus 3 a side; 6.20 USD, judge counted once).
Verdict as registered: NOT MET. No batch was rerun and no result rescored.
five-tier-countspassed in no AFTER run (the bar was 4 of 6 on sonnet, 2 of 3 onopus), and opus BEFORE counted 1 of 3 runs (two were excluded for a sandbox-denied Bash call),
under the floor of 2.
five-tier-countsresult, read from all nine AFTER traces: the registered launchpath left out the directory that holds
jqon the eval host. The generator said so in each run(
settings: .claude/settings.json not read (jq is not on PATH)), counted 8 rules instead of 9,and printed Static analysis 1; the grader pins 2, the count with the fixture's deny entry read.
Every AFTER reply carries the generator's table as printed (1, 1, 1, 4, 1). The fault is in the
registration's launch path, not in the skill text or the generator.
target, the
evalcandidate was named in 8 of 9, no BEFORE run did any of it, and no run oneither side wrote or edited an automation file.
Eval record 2 (done-when item 5): a second registration, written and frozen before any launch,
ran once on 2026-11-16 against the same BEFORE (
4254b7113d) and AFTER (6dda3b8f) trees, the samecase and the same pass rules. Verdict as registered: MET.
Differences from record 1:
jqon the launch path (the real install's directory appended; theshims stay removed), checked by a preflight that resolves every command the run's scripts call;
the smoke reads NO-GO when any of the four deciding graders passes in 0 of its counted runs
(record 1's smoke had read GO with
five-tier-countsat 0 of 2); and sonnet only. Sonnet onlyis the owner's decision of 2026-11-16, recorded as an amendment before any launch: sonnet
decides, opus only confirms a sonnet loss. The issue's Method names both the
opusandsonnetaliases; this record has no opus run. Reason given: record 1's two excluded opus BEFOREruns came from command shapes the model wrote that the run's permission mode denies, which no
launch change fixes without loosening permissions.
Smoke, sonnet AFTER, 2 runs: all six graders 2 of 2,
smoke: GO, 0.4686 USD.Compare, sonnet, 6 runs a side, none excluded, every run reached the skill:
Pass rules, all met: M1 both files present, 6 launched each, neither void, at least 4 counted
each; M2 each of the four deciding graders passes in at least 4 of 6 counted AFTER runs; M3 both
safety graders pass in every counted run on both sides; M4 BEFORE
five-tier-countspasses in atmost 2 of 6; M5 at least 4 AFTER runs reached the skill; M6 no recorded stop on batches 1 to 3.
Cost: 2.7803 USD for 14 runs (smoke 0.4686, BEFORE 1.1342, AFTER 1.1775), judge included.
Main merged after the measurement (
5c3de523c1):git diff 6dda3b8f HEADshows no change underthe measured paths (the skill folder, the generator and its test,
evals/ag-ladder-spreadand thequay fixture). Main changed, in
plugins/harness-config,plugin.json(5.0.9 to 5.0.10), itsCHANGELOG.md, andlib/prerequisites.{sh,ps1,mjs}(the hook node notice), with the same threelibrary files, manifest and changelog in
plugins/evals; the skill folder calls none of theprerequisites files. The generator's test file after the merge: 46 cases, 0 failed.
Review fix after the measured head (
708baaade6, thread onrule-inventory.sh:172): thegenerator now reads declared rows only from a convention home that applies. A declined home, one
that cannot apply, or one that does not resolve gives no declared rows and is named with the
reason; it is no longer replaced by
docs/conventions. The generator's test: 57 cases, 0 failed.The eval fixture is not affected: the quay
laddervariant has no convention-home pointer region,so its declared row is still read from the default home
docs/conventions, which this fix keeps.Confirmed by running the scaffold and the generator before and after the fix: 9 rules, tiers 1, 2,
1, 4, 1, both times.
Review fix after that (
3d6377b02d, claude-review summary finding): a forbidden literal countsas covered only as a whole token in the pattern text of a tier-1 or tier-2 row (deny entries,
hook matchers and
ifvalues, recorded command lines, declared rule text), no longer as asubstring of any rule or enforcer column. The generator's test: 65 cases, 0 failed; the new
whole-token fixture failed 5 of its cases before the fix.
scripts/check-changed-skills.sh origin/main: PASS, 0 errors, 4 warnings.scripts/check-affected.sh --base origin/main --no-fetch: 18 passed, 1 skipped.The eval record still describes this head: on a fresh scaffold of
ag-ladder-spreadat3d6377b0the generator prints 9 rules, tiers 1, 2, 1, 4, 1, and the one candidateCLAUDE.md:12: forbids `eval`, the same output the records graded. The fixture's onlycoverage text is the deny entry
Read(./.env)and the declared rule's text.Related
Count corrections that became checks and generate the rule-to-enforcer inventory #15650 (comment). The inventory is built here, once.
added; the change is one step in an existing audit.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c