Repository navigation
feat(corrections): route codify findings by capability and hand the build to implement - #16603
kyle-sexton wants to merge 5 commits into
Conversation
Lane-Activity: work-resolve-issue/resolve-issue
Two conflicts, both in codify: - reference/finding-classes.md: main changed "installed" to "enabled" in the semgrep-rule cells and the semgrep-rule step, lines this branch replaces with the build hand-off. This branch's lines stand. - SKILL.md, Boundary: main added that the architecture-test rung also names /architecture:enforce-boundaries; this branch hands the build to /implementation:implement. Both are kept. Refs: #15728 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ve graders The Report line for the ladder's alias covered only an old rung id read from the repository's instructions or a stub. It now covers any read, a findings file included, gives one line naming the old id and the current one, and links the ladder's alias line, as the ladder asks of a skill that meets the old id. Three negative graders on the new cases failed a correct reply: hands-off-to-implement on any mention of "tutorial" or of Roslyn (the fixture's row 3 is C#, where a Roslyn analyzer is the repository's own tooling), names-no-csharp-tutorial on "no C# tutorial applies", and not-sent-to-dotnet on "custom-analyzer is not .NET-only". Each now matches the tutorial link, a tutorial the reply points to, or the claim itself, and each has those replies as pass samples. validate-cases: exit 0, FAIL 0. Refs: #15728 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The rule-id rows no longer name the pack diagnostics. This case gives codify a pytest rule-inert-assertion finding in a repository that runs ruff with the assert-tuple diagnostic off, and checks the row still lands on analyzer-pack-rule and names ruff's F631 (assert-tuple) to enable. Its samples pass and fail as listed under validate-cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sue-15728-43770035123-1
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Security review: PR #16603, head 91fa346No security issues found. This was a first review: the prior-findings file lists no earlier heads, so the review was not limited to blocking findings. I read every hunk of all 28 listed files in What changed. The PR changes the text of the
Instruction-surface lens. I checked the removed and reworded rules in the codify skill by what breaks without them.
Eval scaffolds. The four new PR context. The description contains no text telling the lane to pass the PR, skip a check or post elsewhere. Spend. This diff changes no lane, model, schedule or billing settings. The ~15 USD of prepaid eval spend in the description was already incurred and is not a hold. critical-findings: 0 |
| - name: hands-off-to-implement | ||
| type: 'regex' | ||
| pattern: 'learn\.microsoft\.com[^\s)>]*roslyn-sdk' | ||
| pattern: '^(?![\s\S]*(?:learn\.microsoft\.com|roslyn-sdk|(?:follow|see|read|use|start with|work through)[^\n]{0,40}\btutorial))(?=[\s\S]*/implementation:implement)' |
There was a problem hiding this comment.
This regex can fail correct answers. The verbs in the negative lookahead (?:follow|see|read|use|start with|work through)[^\n]{0,40}\btutorial have no word boundaries, so they also match inside other words:
readmatches inside "already"usematches inside "because"seematches inside "foresee" and "seen"
"Already" is likely here: finding-classes.md now tells codify to build the check in "the analyzer, matcher or test library it already runs".
How to show it: this is a correct reply that should pass:
I won't write it. Hand the rung and the check to /implementation:implement, built in the analyzer setup the repository already runs; no tutorial applies.
The lookahead matches ready runs; no tutorial, so the grader fails it. The same happens with ...because there is no tutorial....
Same problem elsewhere: codify-pytest-oracle-finding-gets-no-csharp-tutorial/case.yaml:17 uses the same verb list in names-no-csharp-tutorial. That one needs a C# between the verb and "tutorial", so it misfires less often, but the cause is the same.
Fix: add word boundaries around the verbs in both patterns: \b(?:follow|see|read|use|start with|work through)\b. Then add a sample like the one above to each pass list so the fix stays covered.
| pattern: '^(?![\s\S]*(?:learn\.microsoft\.com|roslyn-sdk|(?:follow|see|read|use|start with|work through)[^\n]{0,40}\btutorial))(?=[\s\S]*/implementation:implement)' | |
| pattern: '^(?![\s\S]*(?:learn\.microsoft\.com|roslyn-sdk|\b(?:follow|see|read|use|start with|work through)\b[^\n]{0,40}\btutorial))(?=[\s\S]*/implementation:implement)' |
claude-lane: claude-review
|
claude-review reviewed CI code review: PR #16603I found one problem: an eval grader regex can fail correct answers. The rest of the change fits the PR description:
Checks I ran with my own tools
Not verified by me: the PR's eval numbers and its branch-check results (validate-cases, markdownlint, check-affected and the rest) are the author's claims. I had no shell to run them. Closing issue: none. The PR uses Manifests: no Spend: the eval runs described in the PR body (about 15 USD of sonnet runs) draw on prepaid API credit. That is not a hold. Embedded text: none in the PR context. I did not use a separate verifier; I checked the one candidate finding by hand against the regex. Verifier dropped 0 of 1 candidates. |
Refs: #15728
Summary
/corrections:codifynow routes a finding by what its check has to decide, not by the language itis written in. Its proposal names the rung, the check to prove and the requirement that the check's
message names the fix, and hands the build to
/implementation:implement. A pytesttesting/audit/finding is no longer sent to a C# analyzer tutorial.
The before and after eval (sonnet) met on all four gain cases and the loss case read NO LOSS. The
detection control for the analyzer-tutorial cut missed its registered prediction: it read UNABLE TO
DETECT where the registration predicted DETECTS. That is why this links
Refs:(see Remaining).Fix
reference/finding-classes.md: the classdotnet-invariantbecomessemantic-invariant(aninvariant that needs types or symbols, in any language) in the class table, the rule-id rows and the
testing/audit/family row. The Microsoft Learn tutorial, the Semgrep plugin and docs, ArchUnitNET,dependency-cruiser and the named pack diagnostics are gone. Each "Owner or pointer" cell is now a
hand-off: the rung and the check to prove, built in the repository's own tooling, with
/implementation:implementnamed as the builder; without that skill the proposal stops there. Thefile writes the ladder's current id
syntax-pattern-ruleeverywhere it wrotesemgrep-rule.SKILL.md: the proposal's install step for a checking rung is that hand-off, and the proposal saysthe check's message must name the fix. Asked to "just write the analyzer", codify declines and hands
over the rung and the check. The Report section gives one line naming the old id and the current one
when codify reads a rung through the ladder's alias. Main's
/architecture:enforce-boundariesstepfor the
architecture-testrung is kept.actions/findings.md: the report's "owner" column is "hand-off".codify-declines-to-implement-the-rungand its tutorial outcome are retired, replaced bycodify-declines-to-implement-hands-off-the-build. Thesemgrep-pointergrader no longer accepts theSemgrep docs URL. New gain cases:
codify-pytest-oracle-finding-gets-no-csharp-tutorial,codify-typescript-invariant-takes-custom-analyzerandcodify-hand-off-names-the-tool-the-repo-runs.New loss case for the rule-id rows:
codify-inert-assertion-names-the-pack-diagnostic(a pytestrepository running ruff with F631 off; graders
pack-rule-rungandnames-the-diagnostic).scripts/docs-anchor-baseline.txt: finding-classes.md's row goes, since the file has no anchorlessdocs-page link left.
Verification
Branch checks
validate-cases.pyonplugins/corrections/evals: FAIL 0, with the JavaScript regex cross-check.markdownlint-cli2on the changed markdown: 0 issues.scripts/check-changed-skills.sh origin/main: codify PASS, 0 errors.scripts/check-affected.sh --base origin/main: 17 passed, 1 skipped.scripts/check-changelog-fragments.sh,scripts/check-cost-figures.sh --checkandnode scripts/validate-plugin-contracts.mjs: pass.codify-settings,count-checks,group-findings,prove-check,findings-bufferand its conformance): pass.Eval record (this repository's dogfooding step)
No repository document defines a "dogfooding step"; the targeted before and after eval run here, on
this repository's own codify skill, is that step. It was registered before any launch, with the
control's prediction written down first.
Policy. The owner's eval policy, given in his own words in the dispatching session on 2026-11-16
("Go with recommendations." to: "one targeted case per change, sonnet, 3 runs a side ... opus only to
confirm a sonnet loss"), supersedes the owner-decisions row Q21 on #15703 ("gate each change on one
targeted before and after case (sonnet at 6 runs a side, opus at 3)") for changes registered after
it. The same reply decided this change's run: "Q6. #15728: approve a 50.61 USD run (cap 119.78), with
a loss case for the rule-id rows and 'unable to detect' accepted for the rest". The 580-run paired
no-loss compare in the issue's Method is dropped under the
#15703 decision.
Shape. Sonnet only. BEFORE is this branch with the three codify skill files taken from the merge
base; AFTER is the measured head
c71a818bf466145b434a1106c0e9da7e3c0066c4. Each full batch launched6 runs and needed at least 3 counted. A smoke (FIT on all four batches) gated the full run.
Gain cases, per grader, BEFORE vs AFTER passes over counted runs: all four MET.
names-the-repo-toolmoved only from 3/6 to 4/6; the hand-off case is MET onnames-no-other-ecosystem-tool. In an earlier smoke, that grader's one miss was a correct call: thereply missed the
[tool.importlinter]configuration inpyproject.tomlbecause its file search nevercovered that file. The two AFTER misses in the full run (runs 1 and 4) were not inspected.
Loss case (rule-id rows): NO LOSS.
pack-rule-rung5/5 vs 6/6,names-the-diagnostic5/5 vs 6/6.BEFORE named the diagnostic in every counted run, so the case could have shown a loss. It does not
show that the trimmed row is what makes AFTER name it: the model may know ruff's F631 unaided. It
covers one language and one pack.
Detection control (analyzer-tutorial cut): UNABLE TO DETECT; the registration predicted DETECTS.
The retired case
codify-declines-to-implement-the-rungran on BEFORE and on a DEGRADED copy (BEFOREwith only the tutorial pointer removed). BEFORE mean 0.600, DEGRADED mean 0.500: a fall of 0.100
against the 0.15 bar. Its deciding grader,
hands-over-the-pointer, passed 1/5 on BEFORE and 0/6 onDEGRADED (1/2 on BEFORE in the smoke). A grader already near zero on BEFORE has no room to show a
0.15 fall, so this control cannot show the cut harmless or harmful. The control is not a pass. Under
the owner's Q6 decision, "unable to detect" is accepted for the cuts no grader reads; whether it also
covers this cut, whose control was registered to detect it and did not, is open (see Remaining).
Exclusions (each excluded run was still paid for):
folder, which the eval sandbox denies.
.gitignoreunder the sandbox's.work/codify/.plugin-folder Grep and one for a Read of a mistyped path. That registration was never launched past
its smoke.
No opus run was made. The policy runs opus only to confirm a sonnet loss, and none occurred.
Spend. 15.0609 USD across three registrations: 1.2707 (first registration's 4-batch smoke), 0
(second, registered and never launched; it lacked the loss case) and 13.7902 (the measured run:
smoke 1.6963, full 12.0939). Approved: 50.61 USD, cap 119.78; the run cost less because it ran sonnet
only.
Main merged after the measurement. The branch took main by a merge commit after the eval. Main
changed
plugins/corrections/lib/prerequisites.*,.claude-plugin/plugin.jsonandCHANGELOG.md;the codify skill folder and
plugins/corrections/evals/are byte-identical to the measured head.Done-when
/implementation:implement): delivered byfinding-classes.mdandSKILL.md, and shown by thepytest-oracle gain case (MET).
dotnet-invariantand no analyzer, architecture-test or dependency tool by product name; usessemantic-invariant: delivered byfinding-classes.md(a grep for the retired ids and productnames returns nothing).
paired no-loss compare is replaced by the owner's decision above; the loss case reads NO LOSS and
all four gain cases MET; the record is above.
the uncontrolled cuts (Semgrep, ArchUnitNET and dependency-cruiser pointers, the
defined-diagnosticexamples, the.NET-onlysentence) stand as "unable to detect" under Q6; theanalyzer-tutorial cut's control read UNABLE TO DETECT against a DETECTS prediction. Not settled.
Remaining
under Q6's "'unable to detect' accepted for the rest", after which the issue can be completed on
this record; or register a control whose BEFORE base rate can show a fall. Until then the issue
stays open.
Related
580-run compare.
codify-old-rung-id-read-as-aliascase reads the alias notice this change adds.🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c