Skip to content

Should a check's output have to name the fix, and be tested for it? #15653

Closed

Description

@kyle-sexton

Question

Should a check's output format require the message to name the fix, and be tested for it?

From the talk

Chapter 6, slide 23: every correction becomes a lint whose message names the root-cause fix, so the agent fixes the cause instead of silencing the check.

Today and the gap

  • The ladder requires a fix-naming message on paper (docs/contracts/enforcement-ladder/README.md:117-120, criterion 4), but the check output format does not hold it: findings are <path>:<line>: <message> with no remedy required (docs/contracts/check-placement/README.md:167).
  • Our typos pilot showed a model allow-listing words instead of fixing them, because the per-edit message suggested only allow-listing (docs/contracts/check-placement/README.md:360-361).

Recommendation

Do it first (menu priority P0, proof-of-concept size): add "the message names the remedy" to the check output format, with a test that fails on a message that names none. Basis: the contract gap above; OpenAI's harness-engineering post on lint messages written for agents (2026-02).

Alternatives

Keep it a ladder criterion only and rely on review; or enforce it only in /corrections:codify proposals.

Done would add

A required field or rule in the check-placement output contract and a test of it. Dogfood: this repository's own scripts/check-* output meets it, enforced in CI.

Activity

  1. added
    wayfind: interviewWayfind decision item: contract/requirements call the user must make; human in the loop.
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: you want the check-placement output contract to require that a finding's message names the remedy, with a test that fails on a message that names none. You also want this repository's own scripts/check-* output to meet it, enforced in CI. The issue recommends this over the two alternatives (ladder criterion only, or /corrections:codify only), so I treated the design as chosen.

    Open for whoever picks it up: how the test decides that a message "names a remedy", and which existing scripts/check-* outputs fail today. No duplicate or merged fix found.

    Automated triage. A maintainer reviews it.

  4. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    The triage verdict left this issue at needs-owner-decision:

    • held by needs-human
    • no lanes App verdict comment shows whether an earlier verdict was flagged

    Reason: intent

  5. self-assigned this
    on Nov 12, 2026
  6. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor
    No description provided.
  7. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Resolved: yes. Every finding a check prints names how to fix what it flags, and each covered check is tested for it. Decided by the owner in a four-round interview on 2026-11-12 (26 questions, 24 answered, two sent to design).

    From the talk

    • Chapter 6, slide 23 (1:10:19): "every time we correct our agents, it turns into one of these deterministic checks", with a message "telling the agent to fix the root cause".
    • 1:16:34, her order of work: lint first, clean up, strengthen verification, then automate.

    Today and the gap

    • The ladder already asks for it on paper: "Its message names the owner or the API to use instead, so the person or agent who trips it knows the fix" (docs/contracts/enforcement-ladder/README.md:119-120).
    • The output format does not hold it: "Findings | <path>:<line>: <message> on stderr, exit 1" (docs/contracts/check-placement/README.md:167).
    • Our typos pilot showed a model allow-listing words because the message suggested only that (docs/contracts/check-placement/README.md:358-361).

    The decision

    1. The rule. Every finding names one or more remedies, root-cause remedies first. A suppression appears only after them, as a named exception with the condition under which it is legitimate.
    2. Which checks. It applies outright to a check a plugin ships, a check a plugin emits into a consumer repository, and a check /corrections:codify proposes. This repository's 68 existing check scripts are held by a ratchet: a CI count of the scripts that do not yet meet the rule, which fails if it rises and is lowered by each change that fixes a script, ending at zero with no date.
    3. The test. Each covered check's planted-violation proof also asserts that the finding carries a remedy. A shipped check with no such proof gets one. A script leaves the ratchet count when its message names a remedy and its proof asserts it.
    4. Naming a tool. A consumer's own check, including one codify proposes, may name the consumer's own API or helper. A check a plugin ships or emits states the change needed and picks no tool or library.
    5. A script whose only way out is a suppression. It does not qualify for a deterministic rung, advisory or blocking: all eight guardrail criteria bind both (docs/contracts/enforcement-ladder/README.md:106). It is given a real remedy, or it is deleted and, if the rule is still wanted, the rule moves to a guidance rung. A script that is a security control is never deleted or downgraded without the owner's decision.
    6. Prose copies. When a check holds a rule, its prose copy is removed or cut to a pointer, so the finding is the one place the remedy is stated. In this repository that happens in the change that adds the check, or in the change that takes an existing script out of the ratchet count. In a consumer repository codify only proposes it, naming the prose line the check replaces, and the consumer decides.

    What it commits to

    • The contract that defines a check's findings requires a remedy.
    • The ladder's planted-violation criterion gains the remedy assertion.
    • This repository's CI gains the ratchet.
    • A shipped check with no planted-violation proof gets one.
    • Codify's proposal names the prose line the check replaces.

    Sent to design

    • Which contract holds the remedy rule and which one points at it, and the exact output shape (a marker, a field, a second line).
    • Which skill helps a consumer that has no check, tool or library yet pick one: the ecosystem-specifics Brief's resolution procedure covers it.

    Not settled here

    • Codify's boundary stands: "It never installs a check unasked, and never installs an unproven one." (plugins/corrections/skills/codify/SKILL.md:181).
    • How many of the 68 scripts have a suppression as their only remedy has not been counted.

    How it was checked

    Three fresh-context sweeps looked for contradictions and undecided points; the third found that an "advisory" exit for suppression-only scripts contradicted the ladder's criterion, and the owner accepted the two-exit rule above in its place.

  8. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Resolved by the owner's decision in the resolution comment above; the pointer is on the map's Decisions-so-far (#15651).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.wayfind: interviewWayfind decision item: contract/requirements call the user must make; human in the loop.

Type

Fields

Work status

needs-human

Priority

medium

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions