Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changes/performance/feat-check-speed-target-3c7a91e2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
bump: minor
---

### Added

- `target` ranks slow checks (git hooks, local check scripts, CI jobs): one candidate per check and lane, a p50 and p90 baseline per lane, the moves to rank (scope to the change, move to a later lane, run in parallel with every job's failure reported, the tool's cache or incremental mode, a tool swap), a dependency-collision check on any upgrade or swap, stack specifics resolved at run time, and growing check counts routed to `/toolchain:ratchet`. A new eval case, `ta-check-lanes-per-stage`, covers it.
- `goal` records `50,90` as the percentile list for a check-lane goal.
7 changes: 7 additions & 0 deletions .changes/toolchain/feat-ratchet-check-counts-5d1e0b74.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: minor
---

### Added

- `ratchet` takes check counts from `/performance:target` as a counter source: checks per hook stage or CI job, files a check scans, or test suites a change selects, so a lane's growth fails the change that adds to it while its duration stays unratcheted.
2 changes: 1 addition & 1 deletion docs/skill-cheat-sheet.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ From the stage definitions: A conditional stage runs only when its trigger holds
| [`/discovery:explore`](../plugins/discovery/skills/explore/SKILL.md) | `discovery` | Explore code, history, tests, and config before changing anything | | `[--output change-prep\|explain] [scope]` | [pstack `how`](upstream/cursor-pstack.md#attribution-table) | `/discovery:research`, `/planning:interview`, `/planning:design`, `/session-flow:workflow` |
| [`/discovery:trace-intent`](../plugins/discovery/skills/trace-intent/SKILL.md) | `discovery` | Reconstruct why a thing was built this way, from evidence outside the code | | `<target>` | [pstack `why`](upstream/cursor-pstack.md#attribution-table) | `/planning:design`, `/planning:plan` |
| [`/performance:go-faster`](../plugins/performance/skills/go-faster/SKILL.md) | `performance` | Whole-process speed sweep with evidenced, ranked findings and adopt-now speedups | | `[--unattended]` | | `/performance:goal`, `/overengineering:audit`, `/harness-ops:audit-performance`, `/performance:target` |
| [`/performance:target`](../plugins/performance/skills/target/SKILL.md) | `performance` | Rank optimization candidates by evidence quality, not suspicion | | `[<path\|component\|'session'\|'telemetry'>]` | [pstack `perf-issue`](upstream/cursor-pstack.md#attribution-table) | `/performance:goal`, `/performance:go-faster` |
| [`/performance:target`](../plugins/performance/skills/target/SKILL.md) | `performance` | Rank optimization candidates by evidence quality, not suspicion | | `[<path\|component\|'session'\|'telemetry'>]` | [pstack `perf-issue`](upstream/cursor-pstack.md#attribution-table) | `/performance:goal`, `/performance:go-faster`, `/toolchain:ratchet` |

## 3. Research

Expand Down
29 changes: 29 additions & 0 deletions plugins/performance/evals/ta-check-lanes-per-stage/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
schema_version: "1.1"
name: ta-check-lanes-per-stage
description: 'Target on slow git-hook checks: a hook config puts two checks in pre-commit and two in pre-push, per-check p50 and p90 timings exist, the parallel runner ends on a bare wait, and the scanned file count grows each release. The run ranks one candidate per check with its stage named in the Candidate cell, flags that the runner can pass a failed shard, and routes the growing file count to /toolchain:ratchet as a counter. A run needs --scaffold.'
tags: [target, check-speed]
plugins: ["../.."]
expected_outcome: 'A ranked table with the existing Rank, Candidate, Tier, What is known, Counter that would settle it and Cheapest next instrument columns, each check candidate naming its stage (pre-commit or pre-push) in the Candidate cell and ranked at E1 from the per-check timings. The report says scripts/run-checks.sh ends on a bare wait that drops each shard''s exit status, and hands the growing count of files a check scans to /toolchain:ratchet as a counter, not a duration.'
context:
scaffold_script: scaffold.sh
execution:
prompt: '/performance:target our git hooks are slow. The hook config is hooks.yml and per-check timings are in measurements/hook-timings.txt. Rank what to speed up.'
allowed_tools: [Read, Grep, Glob, Skill]
max_turns: 15
timeout_seconds: 400
graders:
- name: stage-in-candidate-cell
type: regex
pattern: '^\|[^\n]*\b(?:pre-commit|pre-push)\b[^|\n]*\|[ \t]*(?:\*\*)?E1\b'
flags: 'mi'
arm: both
- name: bare-wait-flagged
type: regex
pattern: '^(?=[^\n]*\bwait\b)(?=[^\n]*\b(?:exit|status|fail\w*)\b)[^\n]*$'
flags: 'mi'
arm: both
- name: growth-routed-to-ratchet
type: regex
pattern: '^(?=[^\n]*toolchain:ratchet)(?=[^\n]*\b(?:files?|count|counter)\b)[^\n]*$'
flags: 'mi'
arm: both
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"pass": [
{
"answer": "`scripts/run-checks.sh` ends on a bare `wait`, which drops each shard's exit status, so a failed shard can pass the hook.",
"why": "names wait and the lost exit status"
},
{
"answer": "- The runner's final wait does not track each job, so a shard that fails is reported as a pass.",
"why": "names wait and a failure that passes"
}
],
"fail": [
{
"answer": "The runner shards the check four ways in parallel, which is already a good use of the cores.",
"why": "no mention of wait"
},
{
"answer": "Waiting on the pre-push hook is the slowest part of the loop.",
"why": "waiting is not the wait builtin and no exit status is named"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"pass": [
{
"answer": "Next: /toolchain:ratchet the count of files shell-lint scans, which grew from 610 to 1840 over three releases.",
"why": "file count routed to the ratchet"
},
{
"answer": "- A check count growing with the repository: `/toolchain:ratchet` on the counter `files_scanned`.",
"why": "names the counter"
},
{
"answer": "Route the growing file count to `/toolchain:ratchet`, which holds a counter, never a duration.",
"why": "echoes the rule that the ratchet takes no duration"
}
],
"fail": [
{
"answer": "Hold the pre-commit stage under its budget with /toolchain:ratchet on p90 seconds.",
"why": "a duration handed to the ratchet"
},
{
"answer": "Next: /performance:goal shell-lint, pre-commit.",
"why": "growth never routed to the ratchet"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"pass": [
{
"answer": "| Rank | Candidate | Tier | What is known | Counter that would settle it | Cheapest next instrument |\n|---|---|---|---|---|---|\n| 1 | unit-tests, pre-push | E1 | p50 52.0 s, p90 61.5 s, all 31 suites | suites selected per push | none: timed per check |\n| 2 | shell-lint, pre-commit | E1 | p50 7.8 s, scans 1840 files | files scanned | none |\n",
"why": "stage named in the Candidate cell, E1 tier next to it"
},
{
"answer": "| Rank | Candidate | Tier | What is known | Counter that would settle it | Cheapest next instrument |\n|---|---|---|---|---|---|\n| **1** | `shell-lint` (**pre-commit**) | **E1, attributed measurement** | scans every tracked file | files scanned | none |\n",
"why": "bold cells"
}
],
"fail": [
{
"answer": "| Rank | Candidate | Tier | What is known | Counter | Next |\n|---|---|---|---|---|---|\n| 1 | unit-tests | E1 | 52 s | suites | none |\n| 2 | shell-lint | E1 | 7.8 s | files | none |\n",
"why": "no stage named in any Candidate cell"
},
{
"answer": "| Rank | Candidate | Tier | What is known | Counter | Next |\n|---|---|---|---|---|---|\n| 1 | pre-push hook total | E2 | 66 s | spawns | time each check |\n",
"why": "only a lane total at E2"
},
{
"answer": "| Rank | Stage | Candidate | Tier | What is known |\n|---|---|---|---|---|\n| 1 | pre-push | unit-tests | E1 | 52 s |\n",
"why": "a Stage column inserted before Candidate breaks the existing column order"
}
]
}
48 changes: 48 additions & 0 deletions plugins/performance/evals/ta-check-lanes-per-stage/scaffold.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
set -euo pipefail

git init -q -b main .
git config user.email eval@example.invalid
git config user.name eval

mkdir -p scripts measurements src
cat >hooks.yml <<'EOF'
# Hook manager config: which checks run in which git hook stage.
pre-commit:
commands:
format-check: { run: scripts/run-checks.sh format }
shell-lint: { run: scripts/run-checks.sh lint }
pre-push:
commands:
type-check: { run: scripts/run-checks.sh types }
unit-tests: { run: scripts/run-checks.sh tests }
EOF

cat >scripts/run-checks.sh <<'EOF'
#!/usr/bin/env bash
# Runs the named check over every tracked source file, four shards in parallel.
check="$1"
for shard in 0 1 2 3; do
"scripts/check-$check.sh" "$shard" &
done
wait
EOF
chmod +x scripts/run-checks.sh

cat >measurements/hook-timings.txt <<'EOF'
Per-check timings, 10 runs each on a typical 12-file change, seconds.

| Check | Stage | p50 | p90 | Files inspected |
|---|---|---|---|---|
| format-check | pre-commit | 0.4 | 0.6 | 12 |
| shell-lint | pre-commit | 7.8 | 9.1 | 1840 (every tracked file) |
| type-check | pre-push | 14.2 | 15.0 | whole project |
| unit-tests | pre-push | 52.0 | 61.5 | all 31 suites |

Tracked source files by release: 1.0 = 610, 2.0 = 1120, 3.0 = 1840.
EOF

for i in $(seq 1 20); do printf 'echo %d\n' "$i" >"src/f$i.sh"; done
printf '.work/\n' >.gitignore
git add .
git commit -q -m "hooks, check runner and per-check timings"
3 changes: 2 additions & 1 deletion plugins/performance/skills/goal/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,8 @@ can reach.

Default: **p50 and p95 over at least 20 samples**, alongside the counter. The goal records its
percentile list; `/performance:snapshot` passes that list to `ab.sh` as `--percentiles`, and the
`1/(1-p)` floor below applies to every entry.
`1/(1-p)` floor below applies to every entry. A goal for a check lane (a git hook stage or a CI
job, from `/performance:target`) records `50,90` for each lane.

State plainly that this is a **house convention, not field consensus**:

Expand Down
42 changes: 41 additions & 1 deletion plugins/performance/skills/target/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: "Ranking of optimization targets by evidence quality, from session pain, a named path or telemetry, so an unmeasured system yields 'instrument this first' instead of a guess. Use when: 'what should we optimize', 'find the bottleneck', 'this feels slow', 'is X worth optimizing'. Skip: a whole-process sweep is /performance:go-faster. Skip when the target is already chosen and measured, or when a failure needs debugging rather than a candidate ranking."
description: "Ranking of optimization targets by evidence quality, from session pain, a named path or telemetry, so an unmeasured system yields 'instrument this first' instead of a guess. Use when: 'what should we optimize', 'find the bottleneck', 'this feels slow', 'is X worth optimizing', 'our git hooks or CI checks are slow'. Skip: a whole-process sweep is /performance:go-faster. Skip when the target is already chosen and measured, or when a failure needs debugging rather than a candidate ranking."
user-invocable: true
argument-hint: "[<path|component|'session'|'telemetry'>]"
disable-model-invocation: false
Expand Down Expand Up @@ -29,6 +29,7 @@ Rank by evidence, and say so when there is none.
| The current session's own pain | Name the operation that felt slow and what was observed. A screenshot or recording counts. Anecdote is a valid *candidate source* and an invalid *ranking basis*. |
| A named path or component | Enumerate the layers it spans before choosing one (see "Measure the layers first"). |
| A telemetry store | `/harness-ops:observability` for Claude Code's own; otherwise the project's. Prefer it over every other source, after checking its accuracy and coverage. |
| Slow checks: git hooks, a local check script or CI jobs | Rank per check and lane (see "Slow checks, hooks and CI"). Claude Code's own hooks are the go-faster `hooks` area, not this row. |
| Open-ended "what is slow here" | The open-ended sweep belongs to `/performance:go-faster`; this skill ranks what that sweep, or the user, brings. With nothing brought, expect "instrument this first". |

## Evidence tiers
Expand Down Expand Up @@ -68,6 +69,44 @@ To name a candidate's mechanism, find the row of the profile-to-family table in
the evidence matches; a mechanism with no matching row is not named. A deletion candidate is the
exception: it comes from reading the path's callers, so it ranks E3 until a measurement covers it.

## Slow checks, hooks and CI

A repository's checks run in lanes: a commit hook, a push hook and CI. Treat each lane as a layer
and attribute before choosing.

1. **Inventory each check and its lane** from the repository's hook-manager config, its check
scripts and its CI workflows. Read the repository's own placement record and budgets first (rules,
conventions, agent instructions); they override this skill. This repository's worked case is the
[check-placement contract, Duration budgets](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/check-placement/README.md#duration-budgets);
borrow no number from it. With no recorded budget, setting one is `/performance:goal`'s job.
2. **One candidate per check and lane**, the lane named in the Candidate cell
(`shell lint, pre-commit`). A lane total is E2; one check timed alone in its lane is E1.
3. **The baseline is p50 and p90 per lane**, on a named sample of changed files (one file, a
typical change, a large change), never a single run. `/performance:goal` records that list.
4. **Rank these moves per candidate**, each at the tier its evidence earns:
- **Scope to the change**: the check reads only the staged or pushed files it applies to.
- **Move to a later lane**: a slow check goes from commit to push, or from push to CI. Never drop
it: CI stays the hard stop, whatever a local hook runs.
- **Run in parallel**: the runner must report every job's failure and exit non-zero on any; a
runner that returns only its last job's status passes a failed check. Confirm the shell's or
hook manager's exit semantics at run time.
- **Turn on the tool's cache or incremental mode**, keyed so a cold CI runner still checks
everything.
- **Swap the tool** for a faster one that covers the same rules.
5. **An upgrade or swap carries a dependency-collision check**: every other tool that loads the one
being replaced or upgraded still resolves to a version that supports it, read from the package
manager's dependency tree before the change is proposed.
6. **Resolve stack specifics at run time**: which faster tool exists, its cache and incremental
flags and the runner's parallel semantics come from `/discovery:research` and the repository's
own conventions, never from this skill.

**A check that got faster because it checked nothing is broken, not fast.** Before and after a
change, confirm each check reports the files it inspected; a zero-input pass is a finding.

**Checks slow as the repository grows.** Fill the Growing state column with the counter that grows
(files a check scans, test suites a change selects, checks per lane) and route it to
`/toolchain:ratchet`, which holds a counter, never a duration.

## Name the counter, not just the duration

For each ranked candidate, name the **drift-immune counter** that would settle it: process spawns,
Expand Down Expand Up @@ -121,6 +160,7 @@ On a re-scan after a MET result, look first for the next slow spot in the same j
on an E3 or E4 candidate, or one this ranking says to instrument, until the named instrument has
run.
- No candidates yet and a whole-process sweep wanted: /performance:go-faster.
- A check count growing with the repository: /toolchain:ratchet <the counter>.

## Gotchas

Expand Down
8 changes: 6 additions & 2 deletions plugins/toolchain/skills/ratchet/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: "Count ratchet: freezes a lint violation, suppression, performance or telemetry count at today's value as a checked-in CI ceiling that fails on a rise and is lowered as the count falls. Counters only, never durations; proposes files for a human to merge. Use when asked to 'ratchet' a lint rule or suppressions, freeze a violation count, or 'stop this count from growing' in CI."
description: "Count ratchet: freezes a lint violation, suppression, check, performance or telemetry count at today's value as a checked-in CI ceiling that fails on a rise and is lowered as the count falls. Counters only, never durations; proposes files for a human to merge. Use when asked to 'ratchet' a lint rule or suppressions, freeze a violation count, or 'stop this count from growing' in CI."
user-invocable: true
argument-hint: "[<counter, rule or claim>]"
disable-model-invocation: false
Expand All @@ -25,14 +25,18 @@ This skill writes the proposed files into the working tree and commits nothing.

## 1. Take the counter

The counter arrives from one of four places:
The counter arrives from one of five places:

| Source | What it hands over |
|---|---|
| `/performance:protect` | a verified counter: name, command, field, and the goal with its Correlation value |
| the ratchet offer on a counting rung's proposal from `/corrections:codify` | a lint rule, once it exists, whose violation count is above zero |
| `/code-metrics:audit-suppressions`, when that skill is available | a command that prints the suppression count |
| the user, from a telemetry store | a command that queries one run's count and prints it |
| `/performance:target` on slow checks | a check count that grows with the repository: checks per hook stage or CI job, files a check scans, or test suites a change selects |

A check count holds a lane's growth where its duration cannot be held: a ceiling on files scanned
or checks per stage fails the change that adds one, and the lane's speed is measured separately.

Each needs a command that prints a `<field>=<number>` token on stdout, run through the shell from
the repository root.
Expand Down
Loading