Skip to content

Route codify findings by capability and hand the build to implement #15728

Description

@kyle-sexton

Parent

Refs #15703
Source: PLAN Phase 7, topic ecosystem-specifics

Depends on

Outcome

/corrections:codify routes a finding by a language-neutral class, and its proposal names the rung, the check, the requirement that the check's message names the fix, and /implementation:implement as the builder. A pytest finding is no longer sent to a C#-specific tutorial.

Demo: give codify a pytest testing/audit/ finding; its proposal names a rung and a check to prove, points at no C# tutorial, and hands the build to /implementation:implement.

What changes:

  • Codify's finding-classes reference: class dotnet-invariant becomes semantic-invariant (its rows and the class table). The analyzer tutorial, pattern-tool, rule-creator and architecture-test tool pointers go. Each "owner or pointer" cell becomes one hand-off: the rung and the check to prove, to /implementation:implement if installed, otherwise the proposal stops there.
  • Codify's skill body: "hand over the rung and the check"; the proposal states that the check's message must name the fix.
  • The existing declines-to-implement case's tutorial outcome is retired by name and replaced with a new case; the tool clauses of the two graders that pin tool strings are dropped.

Method:

  • Loss cases first (rung selection, declines to implement, writes nothing, installs nothing), graders accepting both rung id sets.
  • Gain cases, reported as superiority: a pytest testing/audit/ finding not sent to a C# tutorial; a TypeScript invariant on custom-analyzer; a hand-off naming the tool the fixture already runs.
  • Run budget: 580 runs (43 codify cases plus 4 new, on the opus and sonnet aliases, three runs per side, after two measuring runs). Select cases with --case, use --runs 3 and a --max-cost-usd ceiling, and start no batch after a usage-limit error.
  • Eval batches run in the one local eval-runner session in the main checkout. Open the pull request with the do-not-merge label; the compare record releases it.

Decisions

  • D3: codify picks the rung and proves the check; /implementation:implement builds the rule in the consumer's tooling.
  • D13: the finding class for an invariant that needs meaning, not syntax, is semantic-invariant.
  • D18: every finding names its remedies, root-cause first; a check's message names the fix.
  • D19: a codify-proposed check is the consumer's own.
  • Constraint: a skill reaches another only by /plugin:skill, presence-gated with a stated fallback.

Key interfaces

  • "corrections (codify): the one rung picker. A skill that carries no ladder copy calls /corrections:codify, presence-gated, with a two-sentence strongest-first fallback. Codify's finding class for an invariant that needs meaning, not syntax, is semantic-invariant."
  • Two-way door: skills calling codify.

Done when

  • AC3: codify no longer routes a pytest testing/audit/ finding to a C#-specific tutorial, and its proposal names the fix-message requirement and hands the build to /implementation:implement.
  • The finding-classes reference names no dotnet-invariant class and no analyzer, architecture-test or dependency tool by product name, and uses semantic-invariant.
  • The loss cases are non-inferior and the gain cases pass; the compare record is in the pull request body, with this repository's dogfooding step.
  • AC7: each partial cut here carries a detection-control result.
  • AC10: no new enforcement-placing skill is added here.

Out of scope

  • Building any rule in a consumer's tooling.
  • The ladder contract itself.
  • Codify's carrier copy of the generic resolution text (a later item).

Activity

  1. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 12, 2026
  2. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: #15709, #15708

    I read this as a bounded change to /corrections:codify. It would rename the dotnet-invariant class to semantic-invariant and drop the C# and tool-specific pointers. Each finding would then get one hand-off: the rung and the check to prove, passed to /implementation:implement when it is installed. The proposal would also state that the check's message names the fix. Done-when, method and decisions are all stated, so I have no open questions.

    The issue says it depends on #15709 and #15708, both still open, so I listed them as blockers. I found no duplicate among the open issues and no merged pull request that already does this.

    Automated triage. A maintainer reviews it.

  3. melodic-automation-lanes commented on Nov 13, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane yet: blocker #15709 is not closed by a listed human or a merge; blocker #15708 is not closed by a listed human or a merge.

  4. kyle-sexton commented on Nov 14, 2026

    @kyle-sexton
    MemberAuthor

    Two findings for this item, from the review of #16078's eval cases

    Hub session (GitHub Support). Both sit in files this issue already plans to rewrite, so they are recorded here instead of being changed ahead of it. #15708 put codify's finding classes out of scope for the ladder 3.0.0 change and its done-when allows the old id inside the corrections plugin.

    1. plugins/corrections/skills/codify/reference/finding-classes.md still writes semgrep-rule as a current rung id in five places (lines 17, 23, 107, 141, 155 on main at 4f2dcaaff3) and says "the Semgrep rung" at line 154. The ladder 3.0.0 text says "Every skill that writes the id is updated in the same change"; feat(contracts)!: release enforcement ladder 3.0.0 with a verification tier and capability-named rungs #16087's commit body lists this file under "Not verified". The default codify path reads this file for its install step, so a run sees the old id as a current heading.
    2. Codify's SKILL.md gives the alias notice no place. The ladder says a skill that meets the old id "takes it as the new id and prints a notice naming the rename", but codify's Report section has no line for it, so whether a reply carries the notice is left to the model.

    A small change that does only these two things (the five ids renamed, line 154 reworded, one Report line pointing at the ladder's alias line; 3 files, 16 insertions, 8 deletions; check-changed-skills.sh passes) is built and pushed as branch fix/codify-current-rung-id-and-alias-notice (commit 83679f0418) with no pull request. It is there to take or discard when this item is built; nothing depends on it.

    Effect on #16078's compare: the new case codify-old-rung-id-read-as-alias is graded on the reply proposing the rung under its current id and stating the alias. Until one of the two points above lands, its result is reported with this confound beside it.

  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"711ea903537646e525001c5d840b4fdfed1ac580b54b98ffc54f6945556ce9e7","body_sha256":"47e2a467906567ca04d842c993101010fd9403cb34fb831b142bdcf920f9e41e"}

  6. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor
    No description provided.
  7. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  8. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
    It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    host-only: the codify text and eval-case changes are built in the working tree, but three done-when items need host runs this lane cannot make. (1) The loss/gain compare: 580 runs on the opus and sonnet aliases, which the issue says run in the one local eval-runner session in the main checkout. (2) AC7's detection-control result for each partial cut. (3) A pull request opened with the do-not-merge label, which the compare record releases. This lane opens PRs ready and cannot add that label, so a PR from here could merge before the compare. Resume from this branch on the host: run the compare and the detection control, put the compare record and this repository's dogfooding step in the PR body, and open the PR with do-not-merge.

    What the run says it did not do:

    • Loss/gain eval compare (580 runs, 47 codify cases on opus and sonnet, three runs per side after two measuring runs): host-only, run in the local eval-runner session per the issue. The compare record and this repository's dogfooding step are not in a PR body.
    • AC7 detection-control result for each partial cut (the rule-id rows and the declines case's tutorial outcome): needs eval runs on the host.
    • Opening the PR with the do-not-merge label: this lane opens PRs itself and cannot apply the label.
    • The new and changed regex graders' samples were checked by hand and by the review subagent, not by the eval-cases validator (validate-cases.py), which check-affected lists as CI only.
    • Tidy skipped: the change is 24 files and touches scripts/ (docs-anchor-baseline.txt), over the 10-file cutoff and on a listed shared path.
    • verify step skipped: the touched files are markdown, YAML, JSON and eval scaffold scripts, and no build or test suite covers them beyond the checks above. The scaffold scripts were not executed here.
    • Untracked files at the checkout root (.bash_profile, .bashrc, .gitconfig, .idea, .mcp.json, .profile, .ripgreprc, .vscode, .zprofile, .zshrc) come from the runner environment, not from this change; left untouched.

    Partial work is on the branch work-resolve-issue/issue-15728-43770035123-1. The next run continues from it while it is not behind the default branch, and otherwise starts fresh.

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43770035123

    To hand it back to the lane, remove host-run and set Work status to agent-ready again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions