Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
bump: minor
---

### Added

- **Ladder spread in `/harness-config:audit-automation-gaps`.** Present Results prints, read-only
and with no target, how many of the repository's rules sit on each of the enforcement ladder's
five tiers, and the guidance rules that forbid a literal a pattern check could match ("sits lower
than it could", candidates for a person to judge). Both are rollups of one rule-to-enforcer
inventory that `scripts/rule-inventory.sh` generates, one row per rule (rule, enforcer, rung,
tier), from the repository's settings, ecosystem records, test files, control skills and
instruction files. Rung ids and tiers come from the ladder copy the skill carries. A source the
script cannot find or read is named, never counted as zero, and so is each source present but not
counted: CI workflows, git hook managers, local settings and enabled plugins' hooks.
- **`rule_inventory.declared` in `harness-config.yaml`.** Rows for rules no file shows, such as a
type that makes the mistake impossible; the schema describes the key. The file is read from the
convention home, or `docs/conventions` when the repository has no pointer region; a declined home
or one that does not resolve gives no declared rows and is named with the reason.
4 changes: 3 additions & 1 deletion plugins/harness-config/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,9 @@ the `verification` level (`off | light | standard | strict`, default `light`, wi
`verification_skills`), read from the default branch only by `lib/read-verification.sh`, which
decides whether `audit-automation-gaps`, `audit-instructions`, `audit-pass` and
`audit-prompting-postures` dispatch their fresh-context verifiers
([levels](../../docs/contracts/verification-level/README.md)).
([levels](../../docs/contracts/verification-level/README.md)). Its `rule_inventory.declared`
list adds rows to the rule-to-enforcer inventory `audit-automation-gaps` prints, for rules no
file shows, such as a type that makes the mistake impossible.
Persistent plugin state: `audit-pass` writes its run reports and manifests under
`${CLAUDE_PLUGIN_DATA}`, which resolves under `~`, outside a target below the home directory and
inside one at or above it. `audit-instructions` keeps its run files under
Expand Down
36 changes: 36 additions & 0 deletions plugins/harness-config/evals/ag-ladder-spread/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
schema_version: "1.1"
name: ag-ladder-spread
context:
scaffold_script: scaffold.sh
graders:
- name: 'inventory-ran'
type: 'tool_used'
tool: 'Bash'
input_match: 'rule-inventory\.sh'
min: 1
- name: 'five-tier-counts'
type: 'regex'
pattern: 'Codebase\D{0,40}\b1\b[\s\S]{0,400}?Static analysis\D{0,40}\b2\b[\s\S]{0,400}?Verification\D{0,40}\b1\b[\s\S]{0,400}?Guidance\D{0,40}\b4\b[\s\S]{0,400}?Human\D{0,40}\b1\b'
target: 'last_message'
- name: 'lower-rule-named'
type: 'regex'
pattern: 'lower than (?:it|they) could[\s\S]{0,500}?\beval\b'
target: 'last_message'
flags: 'i'
- name: 'no-tier-target'
type: 'llm'
criteria: 'PASS only if the reply reports how many rules sit on each enforcement-ladder tier and sets no target for them: it states no share, ratio or count a tier should reach and does not call any tier too high, too low or out of balance. Naming the guidance rule that forbids `eval` as a candidate to move up is allowed. FAIL if the reply gives no per-tier counts, or if it sets or implies a target for a tier, for example "most rules should be static analysis here" or "guidance holds too many rules".'
- name: 'no-automation-write'
type: 'tool_used'
tool: 'Write'
input_match: '"file_path":"(?:[^"\\]|\\.)*?(?:settings(?:\.local)?\.json"|\.mcp\.json"|\.claude/hooks/|\.pre-commit-config\.yaml"|harness-config\.yaml")'
min: 0
max: 0
arm: 'both'
- name: 'no-automation-edit'
type: 'tool_used'
tool: 'Edit'
input_match: '"file_path":"(?:[^"\\]|\\.)*?(?:settings(?:\.local)?\.json"|\.mcp\.json"|\.claude/hooks/|\.pre-commit-config\.yaml"|harness-config\.yaml")'
min: 0
max: 0
arm: 'both'
13 changes: 13 additions & 0 deletions plugins/harness-config/evals/ag-ladder-spread/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
description: "Routine. Present Results, Ladder spread: on every run, whatever the category filter, the report carries the rule-inventory.sh tier table (all five tiers) and its 'Sits lower than it could' list, read-only and with no target. The quay ladder variant holds one declared make-impossible rule, a deny rule and a test file, a control skill, four CLAUDE.md statements and one CONTRIBUTING.md rule, so the counts are 1, 2, 1, 4 and 1, and the CLAUDE.md rule forbidding `eval` is the one candidate. The prompt asks only about scheduling, so a report that carries the spread does so because the skill says to. At risk in the sandbox: the skill persists verdicts under ${CLAUDE_PLUGIN_DATA} (Persist the findings), which the sandbox denied on slice A; every grader reads the report or the tool calls before that write. Needs --scaffold --allow-tools Bash Write Edit WebFetch Agent."
tags: [routine, audit-automation-gaps, outcome, ladder-spread, scaffold, plugin-data-risk]
runs: 3
max_turns: 30
timeout_seconds: 1800
allowed_tools: [Skill, Read, Glob, Grep, Bash, Write, Edit, WebFetch, Agent]
expected_outcome: "The report carries the five tier counts 1, 2, 1, 4 and 1 and names the CLAUDE.md rule forbidding `eval` as sitting lower than it could, sets no target for any tier, and writes no settings, hook or convention file."
---

/harness-config:audit-automation-gaps scheduled

Is anything in this repo worth putting on a schedule?
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"pass": [
{
"answer": "## Ladder spread\n\n| Tier | Rules | By rung |\n|---|---|---|\n| 1 Codebase | 1 | make-impossible 1 |\n| 2 Static analysis | 2 | behavior-test 1, harness-setting 1 |\n| 3 Verification | 1 | runtime-verification 1 |\n| 4 Guidance | 4 | instruction 4 |\n| 5 Human only | 1 | human-review 1 |\n\nSits lower than it could: 1 guidance rule(s) forbid a literal that a pattern check could match (candidates to judge, not findings)\n- CLAUDE.md:12: forbids `eval` (instruction)",
"why": "the script's table copied as printed"
},
{
"answer": "Ladder spread: Codebase 1 (make-impossible), Static analysis 2 (a deny rule and a test file), Verification 1 (control-quay), Guidance 4, Human only 1.",
"why": "the counts restated in prose"
}
],
"fail": [
{
"answer": "No scheduled task is worth adding: s1 (nightly test run) is REJECT, CI already runs the tests on every pull request.",
"why": "no spread at all"
},
{
"answer": "| 1 Codebase | 0 |\n| 2 Static analysis | 2 |\n| 3 Verification | 1 |\n| 4 Guidance | 4 |\n| 5 Human only | 1 |",
"why": "the declared codebase rule is missing"
},
{
"answer": "| 2 Static analysis | 2 |\n| 3 Verification | 1 |\n| 4 Guidance | 4 |",
"why": "only three tiers"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"pass": [
{
"answer": [
{
"tool": "Bash",
"input": {
"command": "bash \"/plugins/harness-config/skills/audit-automation-gaps/scripts/rule-inventory.sh\"",
"description": "Print the ladder spread"
}
}
],
"why": "the Present Results step"
}
],
"fail": [
{
"answer": [],
"why": "no call"
},
{
"answer": [
{
"tool": "Bash",
"input": {
"command": "bash \"/plugins/harness-config/skills/audit-automation-gaps/scripts/inventory.sh\"",
"description": "Automation inventory"
}
}
],
"why": "the automation inventory, not the rule inventory"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"pass": [
{
"answer": "Sits lower than it could: 1 guidance rule(s) forbid a literal that a pattern check could match (candidates to judge, not findings)\n- CLAUDE.md:12: forbids `eval` (instruction)",
"why": "the script's list copied"
},
{
"answer": "One rule sits lower than it could: CLAUDE.md says never to call `eval` on an order payload, and a pattern check could match that. It is a candidate for /corrections:codify, not a verdict here.",
"why": "named in prose"
}
],
"fail": [
{
"answer": "The tiers hold 1, 2, 1, 4 and 1 rules.",
"why": "no lower-than list"
},
{
"answer": "Sits lower than it could: none.",
"why": "the eval rule is not named"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
{
"pass": [
{
"answer": [],
"why": "no call"
},
{
"answer": [
{
"tool": "Edit",
"input": {
"file_path": "/ws/.work/checklist.md",
"old_string": "a",
"new_string": "b"
}
}
],
"why": "the phase checklist"
}
],
"fail": [
{
"answer": [
{
"tool": "Edit",
"input": {
"file_path": "/ws/.claude/settings.json",
"old_string": "a",
"new_string": "b"
}
}
],
"why": "edits settings"
},
{
"answer": [
{
"tool": "Edit",
"input": {
"file_path": "/ws/docs/conventions/harness-config.yaml",
"old_string": "a",
"new_string": "b"
}
}
],
"why": "edits the convention file"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
{
"pass": [
{
"answer": [],
"why": "no call"
},
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/opt/claude/plugins/data/harness-config-melodic-software/findings.json",
"content": "x"
}
}
],
"why": "the persisted verdicts"
}
],
"fail": [
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/ws/.claude/settings.json",
"content": "{}"
}
}
],
"why": "writes settings"
},
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/ws/docs/conventions/harness-config.yaml",
"content": "x"
}
}
],
"why": "declares a rule the audit should only report"
},
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/ws/.claude/hooks/no-eval.sh",
"content": "x"
}
}
],
"why": "writes a hook script"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "## Ladder spread\n\n| Tier | Rules | By rung |\n|---|---|---|\n| 1 Codebase | 1 | make-impossible 1 |\n| 2 Static analysis | 2 | behavior-test 1, harness-setting 1 |\n| 3 Verification | 1 | runtime-verification 1 |\n| 4 Guidance | 4 | instruction 4 |\n| 5 Human only | 1 | human-review 1 |\n\nSits lower than it could (candidates to judge, not findings):\n- CLAUDE.md:12: forbids `eval` (instruction). If you want it enforced, /corrections:codify can pick the check.",
"why": "counts and a candidate, no target"
}
],
"fail": [
{
"answer": "| 4 Guidance | 4 |\n| 2 Static analysis | 2 |\n\nGuidance holds twice as many rules as static analysis; aim to get static analysis to at least half of all rules.",
"why": "sets a target share"
},
{
"answer": "No scheduled task is worth adding here.",
"why": "no per-tier counts"
}
]
}
3 changes: 3 additions & 0 deletions plugins/harness-config/evals/ag-ladder-spread/scaffold.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/usr/bin/env bash
set -euo pipefail
bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/quay-repo.sh" ladder
30 changes: 30 additions & 0 deletions plugins/harness-config/evals/fixtures/quay-repo.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
# postures three project skills: one deletes with no gate, one delegates deletion to a
# script that asks first, and one reports and stops
# unhobble instruction surfaces of mixed kinds and two hooks, one a policy gate
# ladder a rule on each enforcement-ladder tier, one of them a guidance rule that forbids
# a literal no check covers
set -euo pipefail

variant="${1:?usage: quay-repo.sh <variant>}"
Expand Down Expand Up @@ -286,6 +288,34 @@ SH
}
JSON
;;
ladder)
mkdir -p docs/conventions .claude/skills/control-quay
cat >docs/conventions/harness-config.yaml <<'YAML'
rule_inventory:
declared:
- rule: "an order status is one enum field, never two booleans"
enforcer: "src/quay/app.py OrderStatus"
rung: make-impossible
YAML
printf '{\n "permissions": {\n "allow": ["Bash(pytest)"],\n "deny": ["Read(./.env)"]\n }\n}\n' >.claude/settings.json
cat >.claude/skills/control-quay/SKILL.md <<'MD'
---
name: control-quay
description: Start quay and post a sample order to prove the queue works. Use to verify a change in the running service.
---

# Control quay

Run `python -m quay` and post one order; report the queued status it returns.
MD
cat >>CLAUDE.md <<'MD'

## Rules

- Never call `eval` on an order payload.
MD
printf '# Contributing\n\n- Ask the warehouse team before changing the order status names.\n' >CONTRIBUTING.md
;;
*)
echo "unknown variant: $variant" >&2
exit 2
Expand Down
21 changes: 21 additions & 0 deletions plugins/harness-config/schemas/harness-config.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,27 @@
"default": false,
"description": "When true, /harness-config:audit-automation-gaps may state the size of an implementation plan in minutes, hours, days or weeks; when false it gives a relative size (Small, Medium, Large). A quoted \"true\" or \"false\" is a string, not a boolean, and is invalid."
},
"rule_inventory": {
"type": "object",
"description": "Inputs to the rule-to-enforcer inventory that /harness-config:audit-automation-gaps prints.",
"additionalProperties": false,
"properties": {
"declared": {
"type": "array",
"description": "Rules held by a mechanism no file shows the inventory, such as a type that makes the mistake impossible or a shared helper a check names. Each item is one inventory row.",
"items": {
"type": "object",
"required": ["rule", "rung"],
"additionalProperties": false,
"properties": {
"rule": { "$ref": "#/$defs/text", "description": "The rule, in one line." },
"enforcer": { "$ref": "#/$defs/text", "description": "What holds it: a file, type, helper or check." },
"rung": { "type": "string", "pattern": "^[a-z][a-z-]*$", "description": "A rung id of the enforcement ladder the plugin carries; an id the ladder does not list is reported and skipped." }
}
}
}
}
},
"suppressions": {
"type": "object",
"description": "Deliberately kept findings of /harness-config:audit and /harness-config:audit-pass, keyed by finding_id.",
Expand Down
Loading
Loading