From c5126d188f9e8a2190737ee7d25a89d00dca4443 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Mon, 16 Nov 2026 16:45:34 -0500 Subject: [PATCH] feat(performance): rank slow checks per lane and ratchet their growth performance:target takes slow git hooks, local check scripts and CI jobs as a target type: one candidate per check and lane, a p50 and p90 baseline per lane, the moves to rank (scope to the change, a later lane, parallel runs that report every job's failure, the tool's cache or incremental mode, a tool swap), a dependency-collision check on any upgrade or swap, and stack specifics resolved at run time. A growing check count goes to toolchain:ratchet, which now names check counts as a counter source. performance:goal records 50,90 for a check-lane goal. A new eval case, ta-check-lanes-per-stage, covers the target path. Co-Authored-By: Claude Opus 5.5 --- .../feat-check-speed-target-3c7a91e2.md | 8 ++++ .../feat-ratchet-check-counts-5d1e0b74.md | 7 +++ docs/skill-cheat-sheet.md | 2 +- .../evals/ta-check-lanes-per-stage/case.yaml | 29 +++++++++++ .../samples/bare-wait-flagged.json | 22 +++++++++ .../samples/growth-routed-to-ratchet.json | 26 ++++++++++ .../samples/stage-in-candidate-cell.json | 26 ++++++++++ .../ta-check-lanes-per-stage/scaffold.sh | 48 +++++++++++++++++++ plugins/performance/skills/goal/SKILL.md | 3 +- plugins/performance/skills/target/SKILL.md | 42 +++++++++++++++- plugins/toolchain/skills/ratchet/SKILL.md | 8 +++- 11 files changed, 216 insertions(+), 5 deletions(-) create mode 100644 .changes/performance/feat-check-speed-target-3c7a91e2.md create mode 100644 .changes/toolchain/feat-ratchet-check-counts-5d1e0b74.md create mode 100644 plugins/performance/evals/ta-check-lanes-per-stage/case.yaml create mode 100644 plugins/performance/evals/ta-check-lanes-per-stage/samples/bare-wait-flagged.json create mode 100644 plugins/performance/evals/ta-check-lanes-per-stage/samples/growth-routed-to-ratchet.json create mode 100644 plugins/performance/evals/ta-check-lanes-per-stage/samples/stage-in-candidate-cell.json create mode 100755 plugins/performance/evals/ta-check-lanes-per-stage/scaffold.sh diff --git a/.changes/performance/feat-check-speed-target-3c7a91e2.md b/.changes/performance/feat-check-speed-target-3c7a91e2.md new file mode 100644 index 00000000000..da518abd249 --- /dev/null +++ b/.changes/performance/feat-check-speed-target-3c7a91e2.md @@ -0,0 +1,8 @@ +--- +bump: minor +--- + +### Added + +- `target` ranks slow checks (git hooks, local check scripts, CI jobs): one candidate per check and lane, a p50 and p90 baseline per lane, the moves to rank (scope to the change, move to a later lane, run in parallel with every job's failure reported, the tool's cache or incremental mode, a tool swap), a dependency-collision check on any upgrade or swap, stack specifics resolved at run time, and growing check counts routed to `/toolchain:ratchet`. A new eval case, `ta-check-lanes-per-stage`, covers it. +- `goal` records `50,90` as the percentile list for a check-lane goal. diff --git a/.changes/toolchain/feat-ratchet-check-counts-5d1e0b74.md b/.changes/toolchain/feat-ratchet-check-counts-5d1e0b74.md new file mode 100644 index 00000000000..da216d4241e --- /dev/null +++ b/.changes/toolchain/feat-ratchet-check-counts-5d1e0b74.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Added + +- `ratchet` takes check counts from `/performance:target` as a counter source: checks per hook stage or CI job, files a check scans, or test suites a change selects, so a lane's growth fails the change that adds to it while its duration stays unratcheted. diff --git a/docs/skill-cheat-sheet.md b/docs/skill-cheat-sheet.md index 7a5e4207883..849d50292e5 100644 --- a/docs/skill-cheat-sheet.md +++ b/docs/skill-cheat-sheet.md @@ -80,7 +80,7 @@ From the stage definitions: A conditional stage runs only when its trigger holds | [`/discovery:explore`](../plugins/discovery/skills/explore/SKILL.md) | `discovery` | Explore code, history, tests, and config before changing anything | | `[--output change-prep\|explain] [scope]` | [pstack `how`](upstream/cursor-pstack.md#attribution-table) | `/discovery:research`, `/planning:interview`, `/planning:design`, `/session-flow:workflow` | | [`/discovery:trace-intent`](../plugins/discovery/skills/trace-intent/SKILL.md) | `discovery` | Reconstruct why a thing was built this way, from evidence outside the code | | `` | [pstack `why`](upstream/cursor-pstack.md#attribution-table) | `/planning:design`, `/planning:plan` | | [`/performance:go-faster`](../plugins/performance/skills/go-faster/SKILL.md) | `performance` | Whole-process speed sweep with evidenced, ranked findings and adopt-now speedups | | `[--unattended]` | | `/performance:goal`, `/overengineering:audit`, `/harness-ops:audit-performance`, `/performance:target` | -| [`/performance:target`](../plugins/performance/skills/target/SKILL.md) | `performance` | Rank optimization candidates by evidence quality, not suspicion | | `[]` | [pstack `perf-issue`](upstream/cursor-pstack.md#attribution-table) | `/performance:goal`, `/performance:go-faster` | +| [`/performance:target`](../plugins/performance/skills/target/SKILL.md) | `performance` | Rank optimization candidates by evidence quality, not suspicion | | `[]` | [pstack `perf-issue`](upstream/cursor-pstack.md#attribution-table) | `/performance:goal`, `/performance:go-faster`, `/toolchain:ratchet` | ## 3. Research diff --git a/plugins/performance/evals/ta-check-lanes-per-stage/case.yaml b/plugins/performance/evals/ta-check-lanes-per-stage/case.yaml new file mode 100644 index 00000000000..368a22508db --- /dev/null +++ b/plugins/performance/evals/ta-check-lanes-per-stage/case.yaml @@ -0,0 +1,29 @@ +schema_version: "1.1" +name: ta-check-lanes-per-stage +description: 'Target on slow git-hook checks: a hook config puts two checks in pre-commit and two in pre-push, per-check p50 and p90 timings exist, the parallel runner ends on a bare wait, and the scanned file count grows each release. The run ranks one candidate per check with its stage named in the Candidate cell, flags that the runner can pass a failed shard, and routes the growing file count to /toolchain:ratchet as a counter. A run needs --scaffold.' +tags: [target, check-speed] +plugins: ["../.."] +expected_outcome: 'A ranked table with the existing Rank, Candidate, Tier, What is known, Counter that would settle it and Cheapest next instrument columns, each check candidate naming its stage (pre-commit or pre-push) in the Candidate cell and ranked at E1 from the per-check timings. The report says scripts/run-checks.sh ends on a bare wait that drops each shard''s exit status, and hands the growing count of files a check scans to /toolchain:ratchet as a counter, not a duration.' +context: + scaffold_script: scaffold.sh +execution: + prompt: '/performance:target our git hooks are slow. The hook config is hooks.yml and per-check timings are in measurements/hook-timings.txt. Rank what to speed up.' + allowed_tools: [Read, Grep, Glob, Skill] + max_turns: 15 + timeout_seconds: 400 +graders: + - name: stage-in-candidate-cell + type: regex + pattern: '^\|[^\n]*\b(?:pre-commit|pre-push)\b[^|\n]*\|[ \t]*(?:\*\*)?E1\b' + flags: 'mi' + arm: both + - name: bare-wait-flagged + type: regex + pattern: '^(?=[^\n]*\bwait\b)(?=[^\n]*\b(?:exit|status|fail\w*)\b)[^\n]*$' + flags: 'mi' + arm: both + - name: growth-routed-to-ratchet + type: regex + pattern: '^(?=[^\n]*toolchain:ratchet)(?=[^\n]*\b(?:files?|count|counter)\b)[^\n]*$' + flags: 'mi' + arm: both diff --git a/plugins/performance/evals/ta-check-lanes-per-stage/samples/bare-wait-flagged.json b/plugins/performance/evals/ta-check-lanes-per-stage/samples/bare-wait-flagged.json new file mode 100644 index 00000000000..9c7aeee9cd5 --- /dev/null +++ b/plugins/performance/evals/ta-check-lanes-per-stage/samples/bare-wait-flagged.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "`scripts/run-checks.sh` ends on a bare `wait`, which drops each shard's exit status, so a failed shard can pass the hook.", + "why": "names wait and the lost exit status" + }, + { + "answer": "- The runner's final wait does not track each job, so a shard that fails is reported as a pass.", + "why": "names wait and a failure that passes" + } + ], + "fail": [ + { + "answer": "The runner shards the check four ways in parallel, which is already a good use of the cores.", + "why": "no mention of wait" + }, + { + "answer": "Waiting on the pre-push hook is the slowest part of the loop.", + "why": "waiting is not the wait builtin and no exit status is named" + } + ] +} diff --git a/plugins/performance/evals/ta-check-lanes-per-stage/samples/growth-routed-to-ratchet.json b/plugins/performance/evals/ta-check-lanes-per-stage/samples/growth-routed-to-ratchet.json new file mode 100644 index 00000000000..a883516a039 --- /dev/null +++ b/plugins/performance/evals/ta-check-lanes-per-stage/samples/growth-routed-to-ratchet.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "Next: /toolchain:ratchet the count of files shell-lint scans, which grew from 610 to 1840 over three releases.", + "why": "file count routed to the ratchet" + }, + { + "answer": "- A check count growing with the repository: `/toolchain:ratchet` on the counter `files_scanned`.", + "why": "names the counter" + }, + { + "answer": "Route the growing file count to `/toolchain:ratchet`, which holds a counter, never a duration.", + "why": "echoes the rule that the ratchet takes no duration" + } + ], + "fail": [ + { + "answer": "Hold the pre-commit stage under its budget with /toolchain:ratchet on p90 seconds.", + "why": "a duration handed to the ratchet" + }, + { + "answer": "Next: /performance:goal shell-lint, pre-commit.", + "why": "growth never routed to the ratchet" + } + ] +} diff --git a/plugins/performance/evals/ta-check-lanes-per-stage/samples/stage-in-candidate-cell.json b/plugins/performance/evals/ta-check-lanes-per-stage/samples/stage-in-candidate-cell.json new file mode 100644 index 00000000000..fb5ef176f38 --- /dev/null +++ b/plugins/performance/evals/ta-check-lanes-per-stage/samples/stage-in-candidate-cell.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "| Rank | Candidate | Tier | What is known | Counter that would settle it | Cheapest next instrument |\n|---|---|---|---|---|---|\n| 1 | unit-tests, pre-push | E1 | p50 52.0 s, p90 61.5 s, all 31 suites | suites selected per push | none: timed per check |\n| 2 | shell-lint, pre-commit | E1 | p50 7.8 s, scans 1840 files | files scanned | none |\n", + "why": "stage named in the Candidate cell, E1 tier next to it" + }, + { + "answer": "| Rank | Candidate | Tier | What is known | Counter that would settle it | Cheapest next instrument |\n|---|---|---|---|---|---|\n| **1** | `shell-lint` (**pre-commit**) | **E1, attributed measurement** | scans every tracked file | files scanned | none |\n", + "why": "bold cells" + } + ], + "fail": [ + { + "answer": "| Rank | Candidate | Tier | What is known | Counter | Next |\n|---|---|---|---|---|---|\n| 1 | unit-tests | E1 | 52 s | suites | none |\n| 2 | shell-lint | E1 | 7.8 s | files | none |\n", + "why": "no stage named in any Candidate cell" + }, + { + "answer": "| Rank | Candidate | Tier | What is known | Counter | Next |\n|---|---|---|---|---|---|\n| 1 | pre-push hook total | E2 | 66 s | spawns | time each check |\n", + "why": "only a lane total at E2" + }, + { + "answer": "| Rank | Stage | Candidate | Tier | What is known |\n|---|---|---|---|---|\n| 1 | pre-push | unit-tests | E1 | 52 s |\n", + "why": "a Stage column inserted before Candidate breaks the existing column order" + } + ] +} diff --git a/plugins/performance/evals/ta-check-lanes-per-stage/scaffold.sh b/plugins/performance/evals/ta-check-lanes-per-stage/scaffold.sh new file mode 100755 index 00000000000..5a8831ac944 --- /dev/null +++ b/plugins/performance/evals/ta-check-lanes-per-stage/scaffold.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +set -euo pipefail + +git init -q -b main . +git config user.email eval@example.invalid +git config user.name eval + +mkdir -p scripts measurements src +cat >hooks.yml <<'EOF' +# Hook manager config: which checks run in which git hook stage. +pre-commit: + commands: + format-check: { run: scripts/run-checks.sh format } + shell-lint: { run: scripts/run-checks.sh lint } +pre-push: + commands: + type-check: { run: scripts/run-checks.sh types } + unit-tests: { run: scripts/run-checks.sh tests } +EOF + +cat >scripts/run-checks.sh <<'EOF' +#!/usr/bin/env bash +# Runs the named check over every tracked source file, four shards in parallel. +check="$1" +for shard in 0 1 2 3; do + "scripts/check-$check.sh" "$shard" & +done +wait +EOF +chmod +x scripts/run-checks.sh + +cat >measurements/hook-timings.txt <<'EOF' +Per-check timings, 10 runs each on a typical 12-file change, seconds. + +| Check | Stage | p50 | p90 | Files inspected | +|---|---|---|---|---| +| format-check | pre-commit | 0.4 | 0.6 | 12 | +| shell-lint | pre-commit | 7.8 | 9.1 | 1840 (every tracked file) | +| type-check | pre-push | 14.2 | 15.0 | whole project | +| unit-tests | pre-push | 52.0 | 61.5 | all 31 suites | + +Tracked source files by release: 1.0 = 610, 2.0 = 1120, 3.0 = 1840. +EOF + +for i in $(seq 1 20); do printf 'echo %d\n' "$i" >"src/f$i.sh"; done +printf '.work/\n' >.gitignore +git add . +git commit -q -m "hooks, check runner and per-check timings" diff --git a/plugins/performance/skills/goal/SKILL.md b/plugins/performance/skills/goal/SKILL.md index 51190eea9a1..803a399bf12 100644 --- a/plugins/performance/skills/goal/SKILL.md +++ b/plugins/performance/skills/goal/SKILL.md @@ -133,7 +133,8 @@ can reach. Default: **p50 and p95 over at least 20 samples**, alongside the counter. The goal records its percentile list; `/performance:snapshot` passes that list to `ab.sh` as `--percentiles`, and the -`1/(1-p)` floor below applies to every entry. +`1/(1-p)` floor below applies to every entry. A goal for a check lane (a git hook stage or a CI +job, from `/performance:target`) records `50,90` for each lane. State plainly that this is a **house convention, not field consensus**: diff --git a/plugins/performance/skills/target/SKILL.md b/plugins/performance/skills/target/SKILL.md index c8afe8aa803..7098bb88c13 100644 --- a/plugins/performance/skills/target/SKILL.md +++ b/plugins/performance/skills/target/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Ranking of optimization targets by evidence quality, from session pain, a named path or telemetry, so an unmeasured system yields 'instrument this first' instead of a guess. Use when: 'what should we optimize', 'find the bottleneck', 'this feels slow', 'is X worth optimizing'. Skip: a whole-process sweep is /performance:go-faster. Skip when the target is already chosen and measured, or when a failure needs debugging rather than a candidate ranking." +description: "Ranking of optimization targets by evidence quality, from session pain, a named path or telemetry, so an unmeasured system yields 'instrument this first' instead of a guess. Use when: 'what should we optimize', 'find the bottleneck', 'this feels slow', 'is X worth optimizing', 'our git hooks or CI checks are slow'. Skip: a whole-process sweep is /performance:go-faster. Skip when the target is already chosen and measured, or when a failure needs debugging rather than a candidate ranking." user-invocable: true argument-hint: "[]" disable-model-invocation: false @@ -29,6 +29,7 @@ Rank by evidence, and say so when there is none. | The current session's own pain | Name the operation that felt slow and what was observed. A screenshot or recording counts. Anecdote is a valid *candidate source* and an invalid *ranking basis*. | | A named path or component | Enumerate the layers it spans before choosing one (see "Measure the layers first"). | | A telemetry store | `/harness-ops:observability` for Claude Code's own; otherwise the project's. Prefer it over every other source, after checking its accuracy and coverage. | +| Slow checks: git hooks, a local check script or CI jobs | Rank per check and lane (see "Slow checks, hooks and CI"). Claude Code's own hooks are the go-faster `hooks` area, not this row. | | Open-ended "what is slow here" | The open-ended sweep belongs to `/performance:go-faster`; this skill ranks what that sweep, or the user, brings. With nothing brought, expect "instrument this first". | ## Evidence tiers @@ -68,6 +69,44 @@ To name a candidate's mechanism, find the row of the profile-to-family table in the evidence matches; a mechanism with no matching row is not named. A deletion candidate is the exception: it comes from reading the path's callers, so it ranks E3 until a measurement covers it. +## Slow checks, hooks and CI + +A repository's checks run in lanes: a commit hook, a push hook and CI. Treat each lane as a layer +and attribute before choosing. + +1. **Inventory each check and its lane** from the repository's hook-manager config, its check + scripts and its CI workflows. Read the repository's own placement record and budgets first (rules, + conventions, agent instructions); they override this skill. This repository's worked case is the + [check-placement contract, Duration budgets](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/check-placement/README.md#duration-budgets); + borrow no number from it. With no recorded budget, setting one is `/performance:goal`'s job. +2. **One candidate per check and lane**, the lane named in the Candidate cell + (`shell lint, pre-commit`). A lane total is E2; one check timed alone in its lane is E1. +3. **The baseline is p50 and p90 per lane**, on a named sample of changed files (one file, a + typical change, a large change), never a single run. `/performance:goal` records that list. +4. **Rank these moves per candidate**, each at the tier its evidence earns: + - **Scope to the change**: the check reads only the staged or pushed files it applies to. + - **Move to a later lane**: a slow check goes from commit to push, or from push to CI. Never drop + it: CI stays the hard stop, whatever a local hook runs. + - **Run in parallel**: the runner must report every job's failure and exit non-zero on any; a + runner that returns only its last job's status passes a failed check. Confirm the shell's or + hook manager's exit semantics at run time. + - **Turn on the tool's cache or incremental mode**, keyed so a cold CI runner still checks + everything. + - **Swap the tool** for a faster one that covers the same rules. +5. **An upgrade or swap carries a dependency-collision check**: every other tool that loads the one + being replaced or upgraded still resolves to a version that supports it, read from the package + manager's dependency tree before the change is proposed. +6. **Resolve stack specifics at run time**: which faster tool exists, its cache and incremental + flags and the runner's parallel semantics come from `/discovery:research` and the repository's + own conventions, never from this skill. + +**A check that got faster because it checked nothing is broken, not fast.** Before and after a +change, confirm each check reports the files it inspected; a zero-input pass is a finding. + +**Checks slow as the repository grows.** Fill the Growing state column with the counter that grows +(files a check scans, test suites a change selects, checks per lane) and route it to +`/toolchain:ratchet`, which holds a counter, never a duration. + ## Name the counter, not just the duration For each ranked candidate, name the **drift-immune counter** that would settle it: process spawns, @@ -121,6 +160,7 @@ On a re-scan after a MET result, look first for the next slow spot in the same j on an E3 or E4 candidate, or one this ranking says to instrument, until the named instrument has run. - No candidates yet and a whole-process sweep wanted: /performance:go-faster. +- A check count growing with the repository: /toolchain:ratchet . ## Gotchas diff --git a/plugins/toolchain/skills/ratchet/SKILL.md b/plugins/toolchain/skills/ratchet/SKILL.md index bc3942e9e3c..8b10d838e6a 100644 --- a/plugins/toolchain/skills/ratchet/SKILL.md +++ b/plugins/toolchain/skills/ratchet/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Count ratchet: freezes a lint violation, suppression, performance or telemetry count at today's value as a checked-in CI ceiling that fails on a rise and is lowered as the count falls. Counters only, never durations; proposes files for a human to merge. Use when asked to 'ratchet' a lint rule or suppressions, freeze a violation count, or 'stop this count from growing' in CI." +description: "Count ratchet: freezes a lint violation, suppression, check, performance or telemetry count at today's value as a checked-in CI ceiling that fails on a rise and is lowered as the count falls. Counters only, never durations; proposes files for a human to merge. Use when asked to 'ratchet' a lint rule or suppressions, freeze a violation count, or 'stop this count from growing' in CI." user-invocable: true argument-hint: "[]" disable-model-invocation: false @@ -25,7 +25,7 @@ This skill writes the proposed files into the working tree and commits nothing. ## 1. Take the counter -The counter arrives from one of four places: +The counter arrives from one of five places: | Source | What it hands over | |---|---| @@ -33,6 +33,10 @@ The counter arrives from one of four places: | the ratchet offer on a counting rung's proposal from `/corrections:codify` | a lint rule, once it exists, whose violation count is above zero | | `/code-metrics:audit-suppressions`, when that skill is available | a command that prints the suppression count | | the user, from a telemetry store | a command that queries one run's count and prints it | +| `/performance:target` on slow checks | a check count that grows with the repository: checks per hook stage or CI job, files a check scans, or test suites a change selects | + +A check count holds a lane's growth where its duration cannot be held: a ceiling on files scanned +or checks per stage fails the change that adds one, and the lane's speed is measured separately. Each needs a command that prints a `=` token on stdout, run through the shell from the repository root.