Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
bump: minor
---

### Changed

- `/corrections:codify` routes a finding by what its check must decide, not by language. The finding class `dotnet-invariant` is now `semantic-invariant`, an invariant that needs types or symbols in any language, and still takes `custom-analyzer`. The finding-classes reference writes the current id `syntax-pattern-rule` and no longer points at an analyzer tutorial, a pattern-matcher plugin or an architecture-test library by name.
- Each "Owner or pointer" cell is now a hand-off. A checking rung's proposal hands over the rung and the check to prove, names `/implementation:implement` as the builder in the repository's own tooling, and states that the check's message must name the fix. When that skill is not available, the proposal stops there.
- Codify's report adds the ladder's alias notice, one line naming the old id and the current one, when it reads a rung under an old id from the repository's instructions, a stub or a findings file.
- Eval cases: `codify-declines-to-implement-the-rung` and its Microsoft Learn tutorial outcome are retired. `codify-declines-to-implement-hands-off-the-build` replaces them. The `semgrep-pointer` grader no longer accepts the Semgrep docs URL. New gain cases cover a pytest `testing/audit/` finding, a TypeScript `semantic-invariant`, and a hand-off naming the tool the repository already runs.
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
schema_version: "1.1"
name: codify-declines-to-implement-hands-off-the-build
description: 'Asked to just write the analyzer, the skill declines and hands the build of the rung and its check to /implementation:implement, pointing at no language-specific tutorial. Replaces codify-declines-to-implement-the-rung, whose tutorial outcome is retired.'
tags: [codify, ported]
plugins: ["../.."]
expected_outcome: 'The answer declines, hands the rung and the check to /implementation:implement, links no analyzer tutorial, and writes no analyzer.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/corrections:codify findings findings.md Good, the third finding needs a custom analyzer. Just write the analyzer for me.'
allowed_tools: [Read, Glob, Grep, Skill, Write, Edit]
max_turns: 15
timeout_seconds: 420
graders:
- name: hands-off-to-implement
type: 'regex'
pattern: '^(?![\s\S]*(?:learn\.microsoft\.com|roslyn-sdk|\b(?:follow(?:s|ed|ing)?|see|read(?:s|ing)?|us(?:e|es|ed|ing)|start(?:s|ed|ing)? with|work(?:s|ed|ing)? through)\b[^\n]{0,40}\btutorial))(?=[\s\S]*/implementation:implement)'
flags: 'i'
arm: 'both'
- name: writes-no-analyzer
type: 'tool_used'
tool: 'Write'
input_match: '"file_path":"[^"]*\.cs"'
min: 0
max: 0
arm: 'both'
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"pass": [
{
"answer": "I won't write the analyzer: codify only proposes. Row 3 is semantic-invariant, rung custom-analyzer. Hand the rung and the check to /implementation:implement, and the analyzer's message must name the injected clock as the fix."
},
{
"answer": "I won't write it, and there is no tutorial to point you at. Row 3 needs a Roslyn analyzer in the project's own analyzer setup; hand the rung and the check to /implementation:implement."
},
{
"answer": "I won't write it. Hand the rung and the check to /implementation:implement, built in the analyzer setup the repository already runs; no tutorial applies."
}
],
"fail": [
{
"answer": ""
},
{
"answer": "Here is the analyzer."
},
{
"answer": "See https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/tutorials/how-to-write-csharp-analyzer-code-fix"
},
{
"answer": "Hand it to /implementation:implement, following the typescript-eslint custom rule tutorial."
},
{
"answer": "Hand it to /implementation:implement; the tutorial is at https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/tutorials/how-to-write-csharp-analyzer-code-fix"
}
]
}

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
schema_version: "1.1"
name: codify-hand-off-names-the-tool-the-repo-runs
description: 'Gain case: a layer-boundary row in a Python repository that already runs import-linter takes architecture-test, and the build hand-off names that tool and /implementation:implement rather than an architecture-test library for another ecosystem.'
tags: [codify]
plugins: ["../.."]
expected_outcome: 'Row 1 is architecture-test. The hand-off names import-linter (its pyproject contract or the lint-imports command) as the tooling the check is built in, names /implementation:implement as the builder, and names no .NET or JavaScript architecture-test library.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/corrections:codify findings findings.md What rung does the row get, and what is the next step? This run has no Bash tool: skip the proof and report the check as unproven, and present the proposal in the report, writing nothing.'
allowed_tools: [Read, Glob, Grep, Skill]
max_turns: 15
timeout_seconds: 420
graders:
- name: names-the-repo-tool
type: 'regex'
pattern: '^(?=[\s\S]*architecture-test)(?=[\s\S]*(?:import-linter|importlinter|lint-imports))(?=[\s\S]*/implementation:implement)'
flags: 'i'
arm: 'both'
- name: names-no-other-ecosystem-tool
type: 'regex'
pattern: 'ArchUnit|dependency-cruiser|depcruise'
flags: 'i'
match: 'not_contains'
arm: 'both'
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"pass": [
{
"answer": "Row 1: architecture-test, a new import-linter contract, built by /implementation:implement."
}
],
"fail": [
{
"answer": "Row 1: architecture-test. Pointer: ArchUnitNET for .NET; dependency-cruiser for JS/TS."
},
{
"answer": "Write the boundary test with dependency-cruiser."
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "Row 1: structure, rung architecture-test. Build hand-off: add a forbidden contract to [tool.importlinter] in pyproject.toml, built by /implementation:implement; the check is `lint-imports`."
}
],
"fail": [
{
"answer": ""
},
{
"answer": "Row 1: architecture-test, handed to /implementation:implement."
},
{
"answer": "Row 1: architecture-test. Add an import-linter contract."
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# A Python repository that already runs import-linter for its layer contracts, and a findings file
# whose one row breaks a layer boundary that no contract covers yet.
set -euo pipefail
git init -q -b main .
git config user.email eval@example.invalid
git config user.name eval
git config commit.gpgsign false
mkdir -p src/app/web src/app/domain src/app/persistence
cat >pyproject.toml <<'EOF'
[project]
name = "app"
version = "0.1.0"

[tool.importlinter]
root_package = "app"

[[tool.importlinter.contracts]]
name = "Domain does not import web"
type = "forbidden"
source_modules = ["app.domain"]
forbidden_modules = ["app.web"]
EOF
printf 'lint:\n\tlint-imports\n' >Makefile
cat >src/app/persistence/orders.py <<'EOF'
def load(order_id):
return {"id": order_id}
EOF
cat >src/app/web/orders.py <<'EOF'
from app.persistence.orders import load


def show(order_id):
return load(order_id)
EOF
touch src/app/__init__.py src/app/web/__init__.py src/app/domain/__init__.py src/app/persistence/__init__.py
git add .
git commit -q -m "feat: order page"
cat >findings.md <<'EOF'
---
type: review-findings
date: 2026-01-01T00:00:00Z
branch: fixture
tier: medium
---

## Findings

| Rank | Tier | Confidence | Location | Surface(s) | Finding | Action |
|------|------|------------|----------|------------|---------|--------|
| 1 | IMPORTANT | high | src/app/web/orders.py:1 | architecture-guardian | The web layer imports the persistence package directly, skipping the domain layer. | Assert the layer boundary and route through the domain layer. |

## By dimension

### architecture

| Rank | Tier | Confidence | Location | Surface(s) | Finding | Action |
|------|------|------------|----------|------------|---------|--------|
| 1 | IMPORTANT | high | src/app/web/orders.py:1 | architecture-guardian | The web layer imports the persistence package directly, skipping the domain layer. | Assert the layer boundary and route through the domain layer. |

## Unparsed

None.

## Surfaces

Ran: architecture-guardian.
EOF
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
schema_version: "1.1"
name: codify-inert-assertion-names-the-pack-diagnostic
description: 'Loss case for the rule-id rows: a testing:audit rule-inert-assertion finding on a pytest test in a repository that runs ruff stays on analyzer-pack-rule and names the ruff diagnostic to enable, now that the row no longer lists the diagnostic names.'
tags: [codify]
plugins: ["../.."]
expected_outcome: 'Row 1 is defined-diagnostic on analyzer-pack-rule from its rule id. The next step enables ruff''s assert-tuple diagnostic (F631) in the ruff configuration the repository already runs, rather than writing a new analyzer.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/corrections:codify findings findings.md What rung does the row get, and what exactly gets enabled? This run has no Bash tool: skip the proof and report the check as unproven, and present the proposal in the report, writing nothing.'
allowed_tools: [Read, Glob, Grep, Skill]
max_turns: 15
timeout_seconds: 420
graders:
- name: pack-rule-rung
type: 'regex'
pattern: 'analyzer-pack-rule'
flags: 'i'
arm: 'both'
- name: names-the-diagnostic
type: 'regex'
pattern: '\bF631\b|assert-tuple'
flags: 'i'
arm: 'both'
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "Rung: analyzer-pack-rule. Add F631 to [tool.ruff.lint] select in pyproject.toml."
},
{
"answer": "Enable ruff's assert-tuple rule in the existing ruff configuration."
}
],
"fail": [
{
"answer": ""
},
{
"answer": "Rung: analyzer-pack-rule. Enable the test-framework pack diagnostic for inert assertions in the project's linter configuration."
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"pass": [
{
"answer": "Rung: analyzer-pack-rule. The rule id maps to defined-diagnostic; enable the pack diagnostic in the project's ruff configuration."
}
],
"fail": [
{
"answer": ""
},
{
"answer": "Rung: custom-analyzer. Hand the analyzer build to /implementation:implement."
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# A Python repository tested with pytest and linted with ruff, whose ruff selection leaves the
# assert-on-tuple diagnostic off, and a findings file whose one row is a testing:audit
# rule-inert-assertion finding on that test.
set -euo pipefail
git init -q -b main .
git config user.email eval@example.invalid
git config user.name eval
git config commit.gpgsign false
mkdir -p src/shop tests
cat >pyproject.toml <<'EOF'
[project]
name = "shop"
version = "0.1.0"

[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.ruff.lint]
select = ["E", "W", "I"]
EOF
cat >src/shop/pricing.py <<'EOF'
def total(prices, rate):
return sum(prices) * (1 + rate)
EOF
cat >tests/test_pricing.py <<'EOF'
from shop.pricing import total


def test_total_applies_tax():
result = total([10], 0.2)
assert (result == 12, "total should include tax")
EOF
git add .
git commit -q -m "feat: pricing total"
cat >findings.md <<'EOF'
---
type: review-findings
date: 2026-01-01T00:00:00Z
branch: fixture
tier: medium
---

## Findings

| Rank | Tier | Confidence | Location | Surface(s) | Finding | Action |
|------|------|------------|----------|------------|---------|--------|
| 1 | IMPORTANT | high | tests/test_pricing.py:6 | testing:audit | testing/audit/rule-inert-assertion fired=1 excerpt=assert (result == 12, "total should include tax") | Assert the comparison itself, not a tuple that is always true. |

## Unparsed

None.

## Surfaces

Ran: testing:audit.
EOF
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
schema_version: "1.1"
name: codify-invalid-state-lands-on-make-impossible
description: 'An invalid-state row goes to make-impossible with the redesign handed off; a dangerous-call row gets the syntax-pattern rung or its Semgrep pointer; nothing installs a plugin.'
description: 'An invalid-state row goes to make-impossible with the redesign handed off; a dangerous-call row gets the syntax-pattern rung under its current id or its alias; nothing installs a plugin.'
tags: [codify, ported]
plugins: ["../.."]
expected_outcome: 'Row 1 is make-impossible, handed to /architecture:improve or /planning:design, implementing nothing. Row 2 is the syntax-pattern rung or names Semgrep''s documentation. The report prints no plugin install command.'
expected_outcome: 'Row 1 is make-impossible, handed to /architecture:improve or /planning:design, implementing nothing. Row 2 is the syntax-pattern rung, as syntax-pattern-rule or its alias semgrep-rule. The report prints no plugin install command.'
context:
scaffold_script: scaffold.sh
execution:
Expand All @@ -19,7 +19,7 @@ graders:
arm: 'both'
- name: semgrep-pointer
type: 'regex'
pattern: 'semgrep-rule|syntax-pattern-rule|docs\.semgrep\.dev'
pattern: 'semgrep-rule|syntax-pattern-rule'
flags: 'i'
arm: 'both'
- name: prints-no-plugin-install
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"pass": [
{
"answer": "Row 2: semgrep-rule, https://docs.semgrep.dev/writing-rules/overview"
"answer": "Row 2: semgrep-rule (alias of syntax-pattern-rule)."
},
{
"answer": "Row 2: syntax-pattern-rule."
Expand All @@ -13,6 +13,9 @@
},
{
"answer": "Row 2: instruction."
},
{
"answer": "Row 2: see https://docs.semgrep.dev/writing-rules/overview"
}
]
}
Loading
Loading