You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Should a check's output have to name the fix, and be tested for it? #15653
Should a check's output format require the message to name the fix, and be tested for it?
From the talk
Chapter 6, slide 23: every correction becomes a lint whose message names the root-cause fix, so the agent fixes the cause instead of silencing the check.
Today and the gap
The ladder requires a fix-naming message on paper (docs/contracts/enforcement-ladder/README.md:117-120, criterion 4), but the check output format does not hold it: findings are <path>:<line>: <message> with no remedy required (docs/contracts/check-placement/README.md:167).
Our typos pilot showed a model allow-listing words instead of fixing them, because the per-edit message suggested only allow-listing (docs/contracts/check-placement/README.md:360-361).
Recommendation
Do it first (menu priority P0, proof-of-concept size): add "the message names the remedy" to the check output format, with a test that fails on a message that names none. Basis: the contract gap above; OpenAI's harness-engineering post on lint messages written for agents (2026-02).
Alternatives
Keep it a ladder criterion only and rely on review; or enforce it only in /corrections:codify proposals.
Done would add
A required field or rule in the check-placement output contract and a test of it. Dogfood: this repository's own scripts/check-* output meets it, enforced in CI.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: you want the check-placement output contract to require that a finding's message names the remedy, with a test that fails on a message that names none. You also want this repository's own scripts/check-* output to meet it, enforced in CI. The issue recommends this over the two alternatives (ladder criterion only, or /corrections:codify only), so I treated the design as chosen.
Open for whoever picks it up: how the test decides that a message "names a remedy", and which existing scripts/check-* outputs fail today. No duplicate or merged fix found.
Resolved: yes. Every finding a check prints names how to fix what it flags, and each covered check is tested for it. Decided by the owner in a four-round interview on 2026-11-12 (26 questions, 24 answered, two sent to design).
From the talk
Chapter 6, slide 23 (1:10:19): "every time we correct our agents, it turns into one of these deterministic checks", with a message "telling the agent to fix the root cause".
1:16:34, her order of work: lint first, clean up, strengthen verification, then automate.
Today and the gap
The ladder already asks for it on paper: "Its message names the owner or the API to use instead, so the person or agent who trips it knows the fix" (docs/contracts/enforcement-ladder/README.md:119-120).
The output format does not hold it: "Findings | <path>:<line>: <message> on stderr, exit 1" (docs/contracts/check-placement/README.md:167).
Our typos pilot showed a model allow-listing words because the message suggested only that (docs/contracts/check-placement/README.md:358-361).
The decision
The rule. Every finding names one or more remedies, root-cause remedies first. A suppression appears only after them, as a named exception with the condition under which it is legitimate.
Which checks. It applies outright to a check a plugin ships, a check a plugin emits into a consumer repository, and a check /corrections:codify proposes. This repository's 68 existing check scripts are held by a ratchet: a CI count of the scripts that do not yet meet the rule, which fails if it rises and is lowered by each change that fixes a script, ending at zero with no date.
The test. Each covered check's planted-violation proof also asserts that the finding carries a remedy. A shipped check with no such proof gets one. A script leaves the ratchet count when its message names a remedy and its proof asserts it.
Naming a tool. A consumer's own check, including one codify proposes, may name the consumer's own API or helper. A check a plugin ships or emits states the change needed and picks no tool or library.
A script whose only way out is a suppression. It does not qualify for a deterministic rung, advisory or blocking: all eight guardrail criteria bind both (docs/contracts/enforcement-ladder/README.md:106). It is given a real remedy, or it is deleted and, if the rule is still wanted, the rule moves to a guidance rung. A script that is a security control is never deleted or downgraded without the owner's decision.
Prose copies. When a check holds a rule, its prose copy is removed or cut to a pointer, so the finding is the one place the remedy is stated. In this repository that happens in the change that adds the check, or in the change that takes an existing script out of the ratchet count. In a consumer repository codify only proposes it, naming the prose line the check replaces, and the consumer decides.
What it commits to
The contract that defines a check's findings requires a remedy.
The ladder's planted-violation criterion gains the remedy assertion.
This repository's CI gains the ratchet.
A shipped check with no planted-violation proof gets one.
Codify's proposal names the prose line the check replaces.
Sent to design
Which contract holds the remedy rule and which one points at it, and the exact output shape (a marker, a field, a second line).
Which skill helps a consumer that has no check, tool or library yet pick one: the ecosystem-specifics Brief's resolution procedure covers it.
Not settled here
Codify's boundary stands: "It never installs a check unasked, and never installs an unproven one." (plugins/corrections/skills/codify/SKILL.md:181).
How many of the 68 scripts have a suppression as their only remedy has not been counted.
How it was checked
Three fresh-context sweeps looked for contradictions and undecided points; the third found that an "advisory" exit for suppression-only scripts contradicted the ladder's criterion, and the owner accepted the two-exit rule above in its place.
Activity
github-actions commentedon Nov 12, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: you want the check-placement output contract to require that a finding's message names the remedy, with a test that fails on a message that names none. You also want this repository's own
scripts/check-*output to meet it, enforced in CI. The issue recommends this over the two alternatives (ladder criterion only, or/corrections:codifyonly), so I treated the design as chosen.Open for whoever picks it up: how the test decides that a message "names a remedy", and which existing
scripts/check-*outputs fail today. No duplicate or merged fix found.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 12, 2026
The triage verdict left this issue at
needs-owner-decision:Reason:
intentkyle-sexton commentedon Nov 12, 2026
kyle-sexton commentedon Nov 12, 2026
Resolved: yes. Every finding a check prints names how to fix what it flags, and each covered check is tested for it. Decided by the owner in a four-round interview on 2026-11-12 (26 questions, 24 answered, two sent to design).
From the talk
Today and the gap
docs/contracts/enforcement-ladder/README.md:119-120).<path>:<line>: <message>on stderr, exit 1" (docs/contracts/check-placement/README.md:167).docs/contracts/check-placement/README.md:358-361).The decision
/corrections:codifyproposes. This repository's 68 existing check scripts are held by a ratchet: a CI count of the scripts that do not yet meet the rule, which fails if it rises and is lowered by each change that fixes a script, ending at zero with no date.docs/contracts/enforcement-ladder/README.md:106). It is given a real remedy, or it is deleted and, if the rule is still wanted, the rule moves to a guidance rung. A script that is a security control is never deleted or downgraded without the owner's decision.What it commits to
Sent to design
Not settled here
plugins/corrections/skills/codify/SKILL.md:181).How it was checked
Three fresh-context sweeps looked for contradictions and undecided points; the third found that an "advisory" exit for suppression-only scripts contradicted the ladder's criterion, and the owner accepted the two-exit rule above in its place.
kyle-sexton commentedon Nov 12, 2026
Resolved by the owner's decision in the resolution comment above; the pointer is on the map's Decisions-so-far (#15651).