Skip to content

Count corrections that became checks and generate the rule-to-enforcer inventory #15650

Description

@kyle-sexton

Decision (owner, 2026-11-14; answers researched and recorded 2026-11-15)

Source: the talk "All Code Is Slop (Except Mine)" (Lauren Tan, AI Engineer Code Summit, 2026-11-11), slides 12 and 23: every correction becomes a check.

The owner's instruction of 2026-11-14 was to research each interview question and take the recommendation. The answers, recorded 2026-11-15:

  • What counts. A merged change that adds or raises a check, or removes the mistake by design (the make-impossible rung), and names the correction it answers. The rule-to-enforcer inventory is no longer this item's: Generate the rule-to-enforcer inventory and its per-tier count #15711 builds it.

  • The record. One line in the pull request body, starting at column 0 on a line of its own (not a list item, indented line or quote), at most 72 characters so the squash wrap cannot split it. The source is #<n> (the issue, or the pull request whose review raised the correction) or the literal session, never a URL:

    Codifies: <#issue-or-PR | session> rung=<rung id> <blocking|advisory>
    

    The whole line must match ^Codifies: (#[1-9][0-9]*|session) rung=<rung id> (blocking|advisory)$, so a prose mention or quoted example is not counted. It is not a git trailer: the squash commit appends its Co-authored-by: paragraph after the body, so a body line is never in the trailer block. It is read with git log --grep='^Codifies: '. The key is fixed. make-impossible records are blocking.

  • Rung ids. Counted rungs are the codebase, static-analysis and verification tiers of the ladder; guidance and human-review rungs are listed as not counted. semgrep-rule reads as syntax-pattern-rule.

  • Ladder ids (owner, 2026-11-15). The ladder rows "Compiler or type settings" and "Tests" get rung ids, as a minor ladder version (3.0.0 to 3.1.0). The owner copy is docs/contracts/enforcement-ladder/README.md, with a [3.1.0] entry in the CHANGELOG.md beside it; the five plugin copies listed in scripts/shared-copies.txt are regenerated by scripts/sync-shared-copies.sh, and each plugin carrying a copy takes a version bump. Proposed ids, for /planning:plan to confirm against the existing ids: compiler-setting and behavior-test.

  • Who writes it. /corrections:codify's proposal holds the line, and for work filed as an issue, a task-list item carrying it in a code span. The issue lane copies such a line from the issue into its pull request body, the same way it copies Refs: lines today (.github/scripts/work-resolve-issue/report-outcome.mjs, linkageLines), and never from the run's own text.

  • Who reads it. A new read-only action, /corrections:codify count-checks [--since] [--until] [--bucket week|month], from default-branch git history only, with no forge call.

  • The number. A cumulative count per week or month and the window total, like slide 23. Blocking checks are the headline; advisory checks get their own column; a column per rung that occurs. No ratio.

  • Advisory checks. The change that makes an advisory check blocking carries the line again with blocking, and the check moves columns; it is counted once. How that line names the earlier record is for /planning:plan to settle.

  • Start. The count starts when count-checks ships. No backfill, and no hand-kept list inside the action. The seven checks that landed on 2026-11-13 without a record (feat(guardrails): block foreground until loops and --watch waits #15955, feat(scripts): flag docs-page links with no section anchor in skill and agent files #15960, ci: run the rules-index check on every pull request #15965, feat(guardrails): make the unpinned force-with-lease a setting; this repository blocks it #15968, ci: check that an as-of date in a skill file carries a recheck trigger #15989, feat(ci): report a Follow-ups bullet without the advisory marker #16008, ci: fail a price or savings figure in skills, docs and prompts #16022) are named in the dogfood result posted on this issue as landed before the record.

  • Trust measures. They stay at three. Pull-request volume is covered by deployment frequency (/work-items:measure-delivery), app performance by /performance:protect. Release cadence is skipped because it tracks the release bot's schedule; contributors per month is skipped because it is a per-person count.

  • No-human-touch merge share. One new item builds it in /work-items:morning-brief, beside Owner touches. No combined trust view.

Done when

  • /corrections:codify count-checks is listed in the skill's Actions table, argument hint and action words; a bad --since, --until or --bucket value stops it before it reads history; it writes no file and calls no forge.
  • It counts each Codifies: line on the default branch that matches the grammar above in full; a malformed line, a line with other text on it, a list-item line, or one naming a guidance or human-review rung is listed as not counted, with its commit.
  • Its output has one row per period (new and cumulative) and a window row, with blocking, advisory and per-rung columns, and no ratio; the cumulative value counts from the first record in history whatever --since is; an empty history prints zeros.
  • A check recorded as advisory and later recorded as blocking is counted once, in the blocking column from the later change.
  • A record whose commit was later reverted on the default branch is not counted, and is listed as reverted.
  • Tests over a fixture repository cover the squash shape (line, then the Co-authored-by: paragraph), a list-item line, a line with trailing prose, a malformed and a placeholder line, the semgrep-rule alias, a record naming one of the two new rung ids, a promotion, a revert, and an empty history.
  • Codify's proposal for a checking rung or make-impossible holds the literal line, and the task-list item form for an issue; a make-impossible proposal names the type or API to change.
  • The issue lane copies a Codifies: line from an issue's task-list item into its pull request body in both of its body builders, and does not copy one from the run's own text; its tests cover both.
  • docs/contracts/enforcement-ladder/README.md is at enforcement-ladder: 3.1.0, its "Compiler or type settings" and "Tests" rows carry backticked rung ids, its CHANGELOG.md has a [3.1.0] entry naming both, and scripts/sync-shared-copies.sh --check and --check-bump pass on the pull request.
  • One issue is filed for the no-human-touch merge share in /work-items:morning-brief.
  • Map: integrate the 'All Code Is Slop (Except Mine)' talk into plugins and skills #15651 records that the trust measures stay at three, with the covered and skipped slides.

Out of scope

  • The recorded /app-control:drive run of count-checks and its posted output: moved to Let the pr-verify capture record a command-running skill's drive run #16383 (owner, 2026-11-15), because the capture cannot record a command-running skill yet.

  • The rule-to-enforcer inventory and its per-tier count: Generate the rule-to-enforcer inventory and its per-tier count #15711.

  • A ratio of converted corrections, or a store of every correction.

  • Backfilling checks that landed before the record.

  • Release cadence or contributors per month as measures; a combined trust view.

  • Building the no-human-touch share (this item only files it).

  • A configurable record key; reading the findings buffer.

  • Moving, renaming or removing any ladder rung.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: intent
    Blocked by: none

    Understood: count the corrections that became checks, and generate the rule-to-enforcer inventory (each written rule and the check enforcing it) instead of keeping it by hand. Dogfood on this repo's always-loaded rules. I found no duplicate open issue or merged fix.

    Open design questions the issue leaves unpicked:

    1. Which skill owns the inventory: /corrections:codify (group action) or /session-flow:audit-sessions? RECOMMENDED: /corrections:codify, because corrections are its subject and the count then sits beside the rule-to-check step. Basis: the issue's candidate list; no repo evidence checked beyond it.
    2. What counts as a correction, and where is the count read from (session transcripts, codified rule history, or the generated inventory)? RECOMMENDED: count rules in the inventory that name an enforcer, so the count is a rollup of the inventory and needs no separate source. Basis: this keeps one generated artifact, not two.
    3. Are the inventory and the ecosystem-specifics Brief's per-tier enforcement count one artifact? RECOMMENDED: yes, with the count as a rollup. Basis: avoids two counts that can drift.

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated triage. A maintainer reviews it.

  4. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    The triage verdict left this issue at needs-owner-decision:

    • flagged: intent
    • held by needs-owner-decision
    • no lanes App verdict comment shows whether an earlier verdict was flagged

    Reason: intent

  5. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: A generated rule-to-enforcer inventory, probably owned by /corrections:codify, with the count of rules that name an enforcer as a rollup of it, plus a dogfood run on this repository's always-loaded rules.

    Related work

    Open questions

    1. Which skill owns the inventory: /corrections:codify (a new action or the group action) or /session-flow:audit-sessions?
      RECOMMENDED: /corrections:codify, as a new read-only action rather than an extension of group, because group counts buffer rows by mistake and the inventory reads rule files. Basis: .base/plugins/corrections/skills/codify/SKILL.md:34
    2. What counts as a correction, and where is the count read from: buffer rows, recurring mistakes that reached a rung, or rules in the inventory that name an enforcer?
      RECOMMENDED: Count inventory rules that name an enforcer, so the count is a rollup of the inventory and needs no second source. The owner said /planning:interview should lock this. Basis: judgment
    3. Are the inventory and the ecosystem-specifics Brief's per-tier enforcement count one artifact? The issue defers this to design.
      RECOMMENDED: Yes, one artifact with the count as a rollup. The Brief was not found in the checkout, so confirm it exists before design. Basis: judgment
    4. What is an always-loaded rule, and what are the sources the generator reads: AGENTS.md, CLAUDE.md, or rule files in plugins?
      RECOMMENDED: Read AGENTS.md and the root rule files, and map each rule to a check by an explicit marker or reference in the rule text. Do not infer by guessing. Basis: .base/docs/upstream/cursor-pstack.md:101
    5. Does generating the inventory reopen the earlier decision against a rule-to-enforcer table? That decision rested on AGENTS.md listing only rules CI does not enforce, and on removing prose that repeats a check.
      RECOMMENDED: Yes, in generated form only: output goes to a report or generated file, never into AGENTS.md. Record this so the cursor-pstack row is updated to match. Basis: .base/docs/upstream/cursor-pstack.md:101

    Sources read

    • .base/docs/upstream/cursor-pstack.md:101: the 'correct' skill was Not taken. It declined a rule-to-enforcer table in the agent instructions because AGENTS.md lists only rules CI does not enforce and prose repeating a check is removed. corrections:codify already counts recurring mistakes and picks a ladder rung.
    • .base/plugins/corrections/skills/codify/SKILL.md:34: the group action is read-only. It groups buffer rows by field and mistake and shows each count against the threshold, with detail in actions/group.md.
    • .base/plugins/corrections/skills/codify/SKILL.md:187: group writes no file, buffer row or proposal and runs no proof.
    • Grep for 'per-tier enforcement' and 'ecosystem-specifics' in .base: no match, so the Brief the issue mentions is not in the checkout.

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  6. removed
    needs-owner-decisionAwaiting a human or maintainer judgment call.
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  7. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor
    No description provided.
  8. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Narrowing, as #15711 asks (plan decision D5): this item is the corrections-to-checks counter only. The rule-to-enforcer inventory, its per-tier count and the "sits lower than it could" list are built once, in #16555, as the "Ladder spread" step of /harness-config:audit-automation-gaps. The counter stays a separate measure and reads none of it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    needs-triage

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions