Skip to content

feat(corrections): route codify findings by capability and hand the build to implement - #16603

Open
kyle-sexton wants to merge 5 commits into
mainfrom
work-resolve-issue/issue-15728-43770035123-1
Open

kyle-sexton wants to merge 5 commits into
mainfrom
work-resolve-issue/issue-15728-43770035123-1

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #15728

Summary

/corrections:codify now routes a finding by what its check has to decide, not by the language it
is written in. Its proposal names the rung, the check to prove and the requirement that the check's
message names the fix, and hands the build to /implementation:implement. A pytest testing/audit/
finding is no longer sent to a C# analyzer tutorial.

The before and after eval (sonnet) met on all four gain cases and the loss case read NO LOSS. The
detection control for the analyzer-tutorial cut missed its registered prediction: it read UNABLE TO
DETECT where the registration predicted DETECTS. That is why this links Refs: (see Remaining).

Fix

  • reference/finding-classes.md: the class dotnet-invariant becomes semantic-invariant (an
    invariant that needs types or symbols, in any language) in the class table, the rule-id rows and the
    testing/audit/ family row. The Microsoft Learn tutorial, the Semgrep plugin and docs, ArchUnitNET,
    dependency-cruiser and the named pack diagnostics are gone. Each "Owner or pointer" cell is now a
    hand-off: the rung and the check to prove, built in the repository's own tooling, with
    /implementation:implement named as the builder; without that skill the proposal stops there. The
    file writes the ladder's current id syntax-pattern-rule everywhere it wrote semgrep-rule.
  • SKILL.md: the proposal's install step for a checking rung is that hand-off, and the proposal says
    the check's message must name the fix. Asked to "just write the analyzer", codify declines and hands
    over the rung and the check. The Report section gives one line naming the old id and the current one
    when codify reads a rung through the ladder's alias. Main's /architecture:enforce-boundaries step
    for the architecture-test rung is kept.
  • actions/findings.md: the report's "owner" column is "hand-off".
  • Eval cases: codify-declines-to-implement-the-rung and its tutorial outcome are retired, replaced by
    codify-declines-to-implement-hands-off-the-build. The semgrep-pointer grader no longer accepts the
    Semgrep docs URL. New gain cases: codify-pytest-oracle-finding-gets-no-csharp-tutorial,
    codify-typescript-invariant-takes-custom-analyzer and codify-hand-off-names-the-tool-the-repo-runs.
    New loss case for the rule-id rows: codify-inert-assertion-names-the-pack-diagnostic (a pytest
    repository running ruff with F631 off; graders pack-rule-rung and names-the-diagnostic).
  • scripts/docs-anchor-baseline.txt: finding-classes.md's row goes, since the file has no anchorless
    docs-page link left.

Verification

Branch checks

  • validate-cases.py on plugins/corrections/evals: FAIL 0, with the JavaScript regex cross-check.
  • markdownlint-cli2 on the changed markdown: 0 issues.
  • scripts/check-changed-skills.sh origin/main: codify PASS, 0 errors.
  • scripts/check-affected.sh --base origin/main: 17 passed, 1 skipped.
  • scripts/check-changelog-fragments.sh, scripts/check-cost-figures.sh --check and
    node scripts/validate-plugin-contracts.mjs: pass.
  • The corrections plugin's test scripts (codify-settings, count-checks, group-findings,
    prove-check, findings-buffer and its conformance): pass.

Eval record (this repository's dogfooding step)

No repository document defines a "dogfooding step"; the targeted before and after eval run here, on
this repository's own codify skill, is that step. It was registered before any launch, with the
control's prediction written down first.

Policy. The owner's eval policy, given in his own words in the dispatching session on 2026-11-16
("Go with recommendations." to: "one targeted case per change, sonnet, 3 runs a side ... opus only to
confirm a sonnet loss"), supersedes the owner-decisions row Q21 on #15703 ("gate each change on one
targeted before and after case (sonnet at 6 runs a side, opus at 3)") for changes registered after
it. The same reply decided this change's run: "Q6. #15728: approve a 50.61 USD run (cap 119.78), with
a loss case for the rule-id rows and 'unable to detect' accepted for the rest". The 580-run paired
no-loss compare in the issue's Method is dropped under the
#15703 decision.

Shape. Sonnet only. BEFORE is this branch with the three codify skill files taken from the merge
base; AFTER is the measured head c71a818bf466145b434a1106c0e9da7e3c0066c4. Each full batch launched
6 runs and needed at least 3 counted. A smoke (FIT on all four batches) gated the full run.

Gain cases, per grader, BEFORE vs AFTER passes over counted runs: all four MET.

Case Grader BEFORE AFTER
pytest-oracle-finding-gets-no-csharp-tutorial names-no-csharp-tutorial 6/6 5/5
rung-and-builder 6/6 5/5
message-names-the-fix 0/6 5/5
typescript-invariant-takes-custom-analyzer custom-analyzer-rung 0/6 6/6
not-sent-to-dotnet 2/6 6/6
hand-off-names-the-tool-the-repo-runs names-the-repo-tool 3/6 4/6
names-no-other-ecosystem-tool 1/6 6/6
declines-to-implement-hands-off-the-build hands-off-to-implement 0/6 6/6
writes-no-analyzer 6/6 6/6

names-the-repo-tool moved only from 3/6 to 4/6; the hand-off case is MET on
names-no-other-ecosystem-tool. In an earlier smoke, that grader's one miss was a correct call: the
reply missed the [tool.importlinter] configuration in pyproject.toml because its file search never
covered that file. The two AFTER misses in the full run (runs 1 and 4) were not inspected.

Loss case (rule-id rows): NO LOSS. pack-rule-rung 5/5 vs 6/6, names-the-diagnostic 5/5 vs 6/6.
BEFORE named the diagnostic in every counted run, so the case could have shown a loss. It does not
show that the trimmed row is what makes AFTER name it: the model may know ruff's F631 unaided. It
covers one language and one pack.

Detection control (analyzer-tutorial cut): UNABLE TO DETECT; the registration predicted DETECTS.
The retired case codify-declines-to-implement-the-rung ran on BEFORE and on a DEGRADED copy (BEFORE
with only the tutorial pointer removed). BEFORE mean 0.600, DEGRADED mean 0.500: a fall of 0.100
against the 0.15 bar. Its deciding grader, hands-over-the-pointer, passed 1/5 on BEFORE and 0/6 on
DEGRADED (1/2 on BEFORE in the smoke). A grader already near zero on BEFORE has no room to show a
0.15 fall, so this control cannot show the cut harmless or harmful. The control is not a pass. Under
the owner's Q6 decision, "unable to detect" is accepted for the cuts no grader reads; whether it also
covers this cut, whose control was registered to detect it and did not, is open (see Remaining).

Exclusions (each excluded run was still paid for):

  • pytest-oracle AFTER, run 1, and loss case BEFORE, run 3: the model ran its own Grep at the plugin
    folder, which the eval sandbox denies.
  • control BEFORE, run 1: a denied Write of a .gitignore under the sandbox's .work/codify/.
  • The first registration's smoke excluded both runs of its pytest-oracle batch: one for the same
    plugin-folder Grep and one for a Read of a mistyped path. That registration was never launched past
    its smoke.

No opus run was made. The policy runs opus only to confirm a sonnet loss, and none occurred.

Spend. 15.0609 USD across three registrations: 1.2707 (first registration's 4-batch smoke), 0
(second, registered and never launched; it lacked the loss case) and 13.7902 (the measured run:
smoke 1.6963, full 12.0939). Approved: 50.61 USD, cap 119.78; the run cost less because it ran sonnet
only.

Main merged after the measurement. The branch took main by a merge commit after the eval. Main
changed plugins/corrections/lib/prerequisites.*, .claude-plugin/plugin.json and CHANGELOG.md;
the codify skill folder and plugins/corrections/evals/ are byte-identical to the measured head.

Done-when

  • AC3 (pytest finding not sent to a C# tutorial; fix-message requirement; hand-off to
    /implementation:implement): delivered by finding-classes.md and SKILL.md, and shown by the
    pytest-oracle gain case (MET).
  • No dotnet-invariant and no analyzer, architecture-test or dependency tool by product name; uses
    semantic-invariant: delivered by finding-classes.md (a grep for the retired ids and product
    names returns nothing).
  • Loss cases non-inferior, gain cases pass, compare record and dogfooding step in this body: the
    paired no-loss compare is replaced by the owner's decision above; the loss case reads NO LOSS and
    all four gain cases MET; the record is above.
  • AC7, a detection-control result for each partial cut: the rule-id rows carry the loss case (Q6);
    the uncontrolled cuts (Semgrep, ArchUnitNET and dependency-cruiser pointers, the
    defined-diagnostic examples, the .NET-only sentence) stand as "unable to detect" under Q6; the
    analyzer-tutorial cut's control read UNABLE TO DETECT against a DETECTS prediction. Not settled.
  • AC10, no new enforcement-placing skill: the diff adds no skill.

Remaining

  • The owner's call on the analyzer-tutorial cut (AC7): accept its control's UNABLE TO DETECT reading
    under Q6's "'unable to detect' accepted for the rest", after which the issue can be completed on
    this record; or register a control whose BEFORE base rate can show a fall. Until then the issue
    stays open.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c

melodic-automation-lanes Bot and others added 5 commits November 15, 2026 22:49
Lane-Activity: work-resolve-issue/resolve-issue
Two conflicts, both in codify:

- reference/finding-classes.md: main changed "installed" to "enabled"
  in the semgrep-rule cells and the semgrep-rule step, lines this
  branch replaces with the build hand-off. This branch's lines stand.
- SKILL.md, Boundary: main added that the architecture-test rung also
  names /architecture:enforce-boundaries; this branch hands the build
  to /implementation:implement. Both are kept.

Refs: #15728

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ve graders

The Report line for the ladder's alias covered only an old rung id read
from the repository's instructions or a stub. It now covers any read,
a findings file included, gives one line naming the old id and the
current one, and links the ladder's alias line, as the ladder asks of a
skill that meets the old id.

Three negative graders on the new cases failed a correct reply:
hands-off-to-implement on any mention of "tutorial" or of Roslyn (the
fixture's row 3 is C#, where a Roslyn analyzer is the repository's own
tooling), names-no-csharp-tutorial on "no C# tutorial applies", and
not-sent-to-dotnet on "custom-analyzer is not .NET-only". Each now
matches the tutorial link, a tutorial the reply points to, or the
claim itself, and each has those replies as pass samples.
validate-cases: exit 0, FAIL 0.

Refs: #15728

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The rule-id rows no longer name the pack diagnostics. This case gives
codify a pytest rule-inert-assertion finding in a repository that runs
ruff with the assert-tuple diagnostic off, and checks the row still
lands on analyzer-pack-rule and names ruff's F631 (assert-tuple) to
enable. Its samples pass and fail as listed under validate-cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:07
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:09:44.867190Z 91fa346 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 91fa34650dd14b9c38751cde8b104cdec9ec11c8: read 28 of 28 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16603, head 91fa346

No security issues found.

This was a first review: the prior-findings file lists no earlier heads, so the review was not limited to blocking findings. I read every hunk of all 28 listed files in .claude-lane/pr.diff. Two entries are pure renames with no hunks (writes-no-analyzer.json and the declines-case scaffold.sh), and I read their headers.

What changed. The PR changes the text of the /corrections:codify skill (SKILL.md, actions/findings.md, reference/finding-classes.md), its eval cases, a changelog fragment and one line removed from scripts/docs-anchor-baseline.txt.

  • No changed file matches a trusted-path glob.
  • No workflow, action, token request, trusted-actor list, kill switch, merge hold or plugin manifest changes.
  • No userConfig key is added or changed.

Instruction-surface lens. I checked the removed and reworded rules in the codify skill by what breaks without them.

  • The Boundary rules stay in force: "It never installs a check unasked, and never installs an unproven one" and "It never implements a rung". The "just write the analyzer" refusal still declines. Only what it hands over changes: the rung and the check, with /implementation:implement named as the builder, in place of a tutorial pointer.
  • The consent step for writing a proposal is unchanged.
  • The deleted text was a Microsoft Learn tutorial link, Semgrep docs and plugin pointers, ArchUnitNET and dependency-cruiser links, and named pack diagnostics. These were reference pointers, not guardrails. Removing them unblocks no write, install or outward action.
  • docs/contracts/instruction-exception-register/README.md was not consulted (this lane reads only the named lane files). I judged by consequence: no removed rule falls in a protected class.

Eval scaffolds. The four new scaffold.sh scripts only run git init/config/commit and write fixture files with quoted heredocs in the eval sandbox. They do not use the network, credentials or any outside input.

PR context. The description contains no text telling the lane to pass the PR, skip a check or post elsewhere.

Spend. This diff changes no lane, model, schedule or billing settings. The ~15 USD of prepaid eval spend in the description was already incurred and is not a hold.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 91fa34650dd14b9c38751cde8b104cdec9ec11c8; the summary is in this lane's pull request comment.

claude-lane: claude-review

- name: hands-off-to-implement
type: 'regex'
pattern: 'learn\.microsoft\.com[^\s)>]*roslyn-sdk'
pattern: '^(?![\s\S]*(?:learn\.microsoft\.com|roslyn-sdk|(?:follow|see|read|use|start with|work through)[^\n]{0,40}\btutorial))(?=[\s\S]*/implementation:implement)'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regex can fail correct answers. The verbs in the negative lookahead (?:follow|see|read|use|start with|work through)[^\n]{0,40}\btutorial have no word boundaries, so they also match inside other words:

  • read matches inside "already"
  • use matches inside "because"
  • see matches inside "foresee" and "seen"

"Already" is likely here: finding-classes.md now tells codify to build the check in "the analyzer, matcher or test library it already runs".

How to show it: this is a correct reply that should pass:

I won't write it. Hand the rung and the check to /implementation:implement, built in the analyzer setup the repository already runs; no tutorial applies.

The lookahead matches ready runs; no tutorial, so the grader fails it. The same happens with ...because there is no tutorial....

Same problem elsewhere: codify-pytest-oracle-finding-gets-no-csharp-tutorial/case.yaml:17 uses the same verb list in names-no-csharp-tutorial. That one needs a C# between the verb and "tutorial", so it misfires less often, but the cause is the same.

Fix: add word boundaries around the verbs in both patterns: \b(?:follow|see|read|use|start with|work through)\b. Then add a sample like the one above to each pass list so the fix stays covered.

Suggested change
pattern: '^(?![\s\S]*(?:learn\.microsoft\.com|roslyn-sdk|(?:follow|see|read|use|start with|work through)[^\n]{0,40}\btutorial))(?=[\s\S]*/implementation:implement)'
pattern: '^(?![\s\S]*(?:learn\.microsoft\.com|roslyn-sdk|\b(?:follow|see|read|use|start with|work through)\b[^\n]{0,40}\btutorial))(?=[\s\S]*/implementation:implement)'

claude-lane: claude-review

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 91fa34650dd14b9c38751cde8b104cdec9ec11c8: read 28 of 28 changed file(s); 1 finding(s), 1 as inline comments.

CI code review: PR #16603

I found one problem: an eval grader regex can fail correct answers. The rest of the change fits the PR description:

  • finding-classes.md drops dotnet-invariant for semantic-invariant and writes semgrep-rule as syntax-pattern-rule everywhere.
  • Every "Owner or pointer" cell is now a build hand-off naming /implementation:implement.
  • SKILL.md adds the requirement that the check's message names the fix, and the alias-notice line in the Report section.

Checks I ran with my own tools

  • Links: both new links resolve. reference/enforcement-ladder.md#tier-2-static-analysis matches the "Tier 2: Static analysis" heading. "Guardrail criterion" item 4 is at enforcement-ladder.md#L171. "Where boundary rules live" exists at L140.
  • Leftover ids: nothing under plugins/corrections/skills/codify/ still uses dotnet-invariant or "Owner or pointer". The only remaining semgrep-rule mentions in codify are the alias line and the alias handling in count-checks.md.
  • Docs-anchor baseline: finding-classes.md has no external docs URLs left, so dropping its row from scripts/docs-anchor-baseline.txt is consistent.

Not verified by me: the PR's eval numbers and its branch-check results (validate-cases, markdownlint, check-affected and the rest) are the author's claims. I had no shell to run them.

Closing issue: none. The PR uses Refs: #15728 and has no closing keyword, so there were no acceptance criteria to check. The PR itself says AC7 is still open for the owner to decide.

Manifests: no plugin.json or marketplace.json changes. The changelog fragment's bump: minor fits a behaviour change to the skill.

Spend: the eval runs described in the PR body (about 15 USD of sonnet runs) draw on prepaid API credit. That is not a hold.

Embedded text: none in the PR context.

I did not use a separate verifier; I checked the one candidate finding by hand against the regex. Verifier dropped 0 of 1 candidates.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant