Skip to content

feat(performance): rank slow checks per lane and ratchet their growth - #16648

Merged
kyle-sexton merged 1 commit into
mainfrom
feat/check-speed-performance
Nov 16, 2026
Merged

kyle-sexton merged 1 commit into
mainfrom
feat/check-speed-performance

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

No related issue: owner decision to extend the existing performance and ratchet skills for check speed rather than add a skill.

Summary

Slow checks (git hooks, local check scripts, CI jobs) become a target type of the existing performance chain instead of a new skill. performance:target ranks them per check and lane, performance:goal records a p50 and p90 baseline per lane, and toolchain:ratchet holds a growing check count as a counter.

Fix

  • plugins/performance/skills/target/SKILL.md: a "Slow checks" input row and a "Slow checks, hooks and CI" section. It inventories each check and its lane from the repository's own hook config, scripts and workflows, reads the repository's placement record and budgets first (this repository's worked case is docs/contracts/check-placement/README.md, linked, no numbers copied), names the lane in the Candidate cell so the table's columns stay as they are, and ranks five moves: scope to the change, move to a later lane (never drop; CI stays the hard stop), parallel runs that report every job's failure, the tool's cache or incremental mode, and a tool swap. An upgrade or swap carries a dependency-collision check. Stack specifics resolve at run time through /discovery:research and the repository's conventions, with no per-stack examples or tool links. A zero-input pass is a finding, and a growing count goes to /toolchain:ratchet. The description gains one trigger phrase, and ## Next names the ratchet.
  • plugins/performance/skills/goal/SKILL.md: a check-lane goal records 50,90 through the existing percentile list (the 1/(1-p) floor already covers p90). The p50/p95 default is unchanged.
  • plugins/toolchain/skills/ratchet/SKILL.md: /performance:target on slow checks is a fifth counter source (checks per hook stage or CI job, files a check scans, test suites a change selects). Durations stay refused.
  • plugins/performance/evals/ta-check-lanes-per-stage/: a new eval case with a scaffold, three regex graders and pass/fail samples.
  • .changes/performance/…, .changes/toolchain/…: minor-bump fragments. docs/skill-cheat-sheet.md: regenerated for the new ## Next entry.

Verification

  • scripts/validate-plugins.sh --only performance and --only toolchain: both end "All plugin manifests and the catalog validated." The first run flagged the stale cheat sheet; node scripts/generate-cheatsheet.mjs fixed it.
  • plugins/evals/skills/validate/scripts/validate-cases.py plugins/performance/evals: exit 0, no FAIL. The new case's samples pass offline, and its only WARNs come from other cases.
  • scripts/check-vocabulary.py over the three edited skill bodies: no findings.
  • claude plugin eval on the new case (ta-check-lanes-per-stage, 1 run per arm, plugin copied outside git because the eval refuses a plugin inside a repository with many linked worktrees): with plugin 1.00 (3/3 graders), without 0.00, delta 1.00.
  • claude plugin eval on the existing ta-ranks-by-evidence-tier (1 run, no ablation) as a regression check: 1.00 (4/4).
  • The pre-commit and pre-push hooks passed on commit and push.

Related

Left out

  • plugins/toolchain/skills/ratchet/SKILL.md links ESLint, PHPStan and Oxlint docs in sections 2 and "Replacing a lint tool", which the no-per-stack-tool-links decision would remove; this PR leaves them unchanged.

🤖 Generated with Claude Code

performance:target takes slow git hooks, local check scripts and CI jobs
as a target type: one candidate per check and lane, a p50 and p90
baseline per lane, the moves to rank (scope to the change, a later lane,
parallel runs that report every job's failure, the tool's cache or
incremental mode, a tool swap), a dependency-collision check on any
upgrade or swap, and stack specifics resolved at run time. A growing
check count goes to toolchain:ratchet, which now names check counts as
a counter source. performance:goal records 50,90 for a check-lane goal.

A new eval case, ta-check-lanes-per-stage, covers the target path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 21:48
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T21:50:41.752127Z c5126d1 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43963931023

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed c5126d188f9e8a2190737ee7d25a89d00dca4443: read 11 of 11 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16648 (head c5126d1)

I found no security issues. I read every hunk of all 11 changed files in the full diff.

What changed

  • New prose in three skill bodies: performance:target, performance:goal and toolchain:ratchet.
  • Two changelog fragments.
  • A regenerated row in the cheat sheet.
  • A new eval case, ta-check-lanes-per-stage, made of case.yaml, three regex sample files and scaffold.sh.

What I checked

  • Trusted paths and trust controls. No changed file matches a trusted-path glob. The diff does not touch the trusted-trigger check, the lane token, the kill switch, the broker, the merge holds or any action under .github/actions/.
  • Instruction surface. The skill edits only add guidance. No standing rule is deleted, narrowed or softened. The new text tightens behaviour in places:
    • Never drop a check; CI stays the hard stop.
    • A parallel runner must report every job's failure.
    • A check that passes with zero input counts as a finding.
    • The ratchet still refuses durations.
  • Plugin manifests. No plugin.json or marketplace.json changed. The description frontmatter edits in two skills are not manifest fields.
  • Eval scaffold. scaffold.sh sets up a throwaway git repository in the eval's working directory. It writes only fixed heredoc content and takes no outside input, so nothing can be interpolated into it. The case's allowed_tools is read-only (Read, Grep, Glob, Skill).
  • PR context. The description contains no embedded instructions aimed at the lane.

critical-findings: 0

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed c5126d188f9e8a2190737ee7d25a89d00dca4443: read 11 of 11 changed file(s); 2 finding(s), 0 as inline comments.

Code review: PR #16648, rank slow checks per lane and ratchet their growth

I read all 11 changed files in full in .claude-lane/pr.diff. I also read the head plugins/performance/skills/target/SKILL.md, the surrounding section of goal/SKILL.md, an existing eval scaffold for comparison, and the path-scoped rules that match these files (skill-bodies-state-current-rules.md, eval-case-transcripts.md, cost-claims.md).

Closing issue: none. The PR body says this was the owner's decision, so there are no acceptance criteria to check.

Embedded instructions: none found in the PR context.

Rules:

  • The new ## Next bullet in target follows the <outcome>: /plugin:skill [args]. form, and the section still has two to four bullets.
  • The check-placement link points at a section anchor and copies no numbers from it.
  • The skill body leaves stack specifics to /discovery:research instead of restating them.
  • The eval fixtures are synthetic and hold no transcript.

Ratchet: "four places" became "five places", and the ratchet still refuses durations (ratchet/SKILL.md:317), so the new source row is consistent.

Goal: 50,90 meets the 1/(1-p) floor, and the p50/p95 default is unchanged.

Verification: the PR's validation and eval runs are claimed by the author. I could not run them here.

Findings (2, both advisory, inline):

  1. target/SKILL.md:106: the new check-speed text tells the run to put a counter name in the Growing state column. That column is already defined as yes — /performance:goal will require scaling arms / no, so the two sections give conflicting instructions for one cell.
  2. ta-check-lanes-per-stage/case.yaml:63: the stage-in-candidate-cell grader passes a table with a separate Stage column placed between Candidate and Tier, which is the column-order change the case is meant to catch.

Spend: the new eval case (max_turns: 15, timeout_seconds: 400, run with and without the plugin) uses prepaid model credit each time the performance evals run. This is not a hold.

Verifier dropped 0 of 2 candidates. No verifier subagent ran; I checked both candidates myself against the head files and the eval samples.

critical-findings: 2

Findings not posted inline (their lines are not in the diff):

  • plugins/performance/evals/ta-check-lanes-per-stage/case.yaml:63 (RIGHT): Advisory: the grader accepts a separate Stage column placed just before Tier. ^\|[^\n]*\b(?:pre-commit|pre-push)\b[^|\n]*\|[ \t]*(?:\*\*)?E1\b only checks that the cell right before the E1 cell contains a stage name. It doesn't check that this cell is Candidate. So | 1 | unit-tests | pre-push | E1 | ... | passes. That table has a Stage column inserted between Candidate and Tier, which breaks the column order the expected_outcome requires and the third fail sample in stage-in-candidate-cell.json is meant to reject (that sample only covers the column inserted before Candidate).

Fix: anchor the stage to the second cell:

    pattern: '^\|[^|\n]*\|[^|\n]*\b(?:pre-commit|pre-push)\b[^|\n]*\|[ \t]*(?:\*\*)?E1\b'

I traced this pattern by hand against the existing samples, without running it: both pass samples still match (| 1 | unit-tests, pre-push | E1 and | **1** | \shell-lint` (pre-commit) | **E1), and all three failsamples still fail. It also rejects| 1 | unit-tests | pre-push | E1 |. A fail` sample for that layout would lock the fix in.

Findings not posted inline (their path, line or side is malformed):

  • plugins/performance/skills/target/SKILL.md:106: Advisory: conflicting instructions for the Growing state column. This line says to fill the Growing state column "with the counter that grows" and send it to /toolchain:ratchet. The Output section (line 132-134) defines that same column as yes — /performance:goal will require scaling arms or no, and the Gotchas section (line 176) says a yes means the goal needs scaling arms.

How it goes wrong: on a check candidate like shell-lint, pre-commit, a run following line 106 writes files scanned in the column. A run following line 132 writes yes — /performance:goal will require scaling arms, which also tells /performance:goal to plan scaling arms that this section never asks for. Downstream readers can't tell which meaning the column has.

Fix: pick one cell format for check candidates and say how it relates to the scaling-arm meaning. For example: "Fill Growing state with yes and name the counter that grows (files a check scans, ...); route that counter to /toolchain:ratchet". Or extend the column definition at line 132 to cover check-count growth.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit f73bc9b Nov 16, 2026
104 of 106 checks passed
@kyle-sexton
kyle-sexton deleted the feat/check-speed-performance branch November 16, 2026 22:15
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant