Repository navigation
perf(claude-ops): plugins sync in one script call, cache-content-check batched (7 min -> ~1 min) - #3958
Conversation
…ON digest sync-run.sh executes the sync/audit algorithm's Steps 1 through 5b for every target marketplace in one process and prints one digest the model reports from, instead of one model turn per step. Every invariant context/sync.md fixes holds in the script: per-step fleet-state.sh re-reads, the run journal, the PIPESTATUS[0] capture after each tee-journaled mutating call, the downgrade guard and its --allow-downgrade opt-in, the exit-2-is-not-empty projection check, the per-marketplace failure rule, and Step 4/5 gating on the fresh pre-install read. The cache-content checker is called ONCE per marketplace: the stale ids are extracted CR-safe from the JSON that call already wrote, and the checker's --ids form is a fallback for a JSON that could not be produced rather than a second pass beside one that was. install_new stays the model's decision because the user_config render happens only in SKILL.md: --install-new carries the rendered word, and on ask with a non-empty install gap the script reports the gap and stops before Step 4. --only-install <ids> --run-dir <dir> is the narrow re-entry that then performs Steps 4 and 5 against the same run journal and re-emits the digest without re-running the checker. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
Every sibling script in this directory is 100755; a 100644 helper is not runnable from a checkout that honours the mode bit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
Every external call in `cache-content-check.sh` was per install record: 77
installs on this fleet cost 400 PATH-visible process creations (git 238, find
77, sort 77, jq 8) and ~100 s, process-creation-bound rather than CPU-bound.
The compare itself is set arithmetic and never needed a process per install.
Now batched per MARKETPLACE, so the process count is a property of the
marketplace rather than of how many installs it holds:
- one `jq` over installed_plugins.json: the shape gate and every record for
every marketplace, read once, filtered by suffix in the shell;
- one `jq` over known_marketplaces.json: shape gate, membership, and the
installLocation in the same pass;
- one `git cat-file --batch-check` over every DISTINCT sha, replacing 77
`cat-file -e` calls (`missing` -> `sha-not-local`, positional by contract);
- per distinct sha, one `git show` + one `jq` for the source map (unchanged);
- per distinct sha, ONE `git ls-tree -r -z <sha> -- <every source dir at that
sha>`. A multi-pathspec ls-tree cannot report "this pathspec matched
nothing", so that signal is recovered in the join: an install whose source
directory contributed zero blobs is `no-source-at-sha`, never `match` and
never `stale-content`;
- ONE `find` over every install root into one `sort -z` (chunked on a byte
budget so an argv over the OS limit cannot silently read as an empty cache);
- ONE `git hash-object --stdin-paths` over the CACHE-INTERSECT-TREE files
only (3,402 here, not all 9,547) — a cache-only file's hash answers no
question, only its existence does;
- ONE `git check-ignore -z --no-index --stdin` for the whole marketplace,
keyed by `<source dir>/<relative path>` so records sharing a source
directory read one answer.
Counter, `--marketplace melodic-software`, 77 user installs:
before 400 spawns (git 238, find 77, sort 77, jq 8) ~100 s p50
after 17 spawns (git 10, find 1, sort 1, jq 5) 15.6 s p50
Three independent censuses of the new script agreed at 17 (two runs of the
performance plugin's spawn-census.sh, one of the baseline's own first-arg
argcensus.sh), and the shimmed stdout is byte-identical to the unshimmed
reference run — the shims are not a variable in the subject.
Equivalence. Output contract byte-for-byte unchanged:
- live fleet, `--marketplace melodic-software`: OLD and two NEW runs all
24,492 bytes, `cmp` identical, and identical to the baseline's recorded
reference run. checked 77 / match 77 / stale 0 / unverifiable 0. `--ids`
identical (empty, correct on a clean fleet);
- a synthetic fixture exercises the paths[] ordering the clean fleet cannot:
stale-content with two `differs`, one `missing-from-cache` and three
`extra-in-cache` (one of them gitignored and correctly excluded), plus
`match`, `no-source-at-sha`, `sha-not-local`, `no-git-commit-sha`,
`install-path-missing` and a second marketplace with no installLocation.
OLD vs NEW `cmp` identical across `--marketplace`, `--ids`, `--all` and
`--scope all`;
- cache-content-check.test.sh: 24 cases, 0 failed, unchanged. The bundled
process-budget probe measures 12 creations against its ceiling of 26.
Two error-path divergences, both untested edges rather than output-contract
changes. Under `--all`, an installed_plugins.json whose record read fails
mid-pass now exits 2 fleet-wide instead of emitting an inline `{"error": …}`
for one marketplace — the read is now one pass for the whole file, and failing
loud on state-file drift is what the rest of this script already does. Also
under `--all`, a marketplace whose value is not an object now verdicts
`no-install-location` instead of an inline error block.
One deliberate degradation, noted in the code: a single hash batch means a
`hash-object` contract break lands on the install that owns the gap and on
every install after it, where a per-install batch confined it to one. Both
report the same "refused to compare" verdict.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
…its executor The step bodies keep every rule and its reason and drop the inline bash a reader would otherwise retype: the redirect-project-loop shape, the per-step fleet-state.sh invocations, the catalog-regression jq, the run-directory and scratch-directory creation, and the marketplace enumeration loop are all sync-run.sh's, and the file now says what each step does and why rather than spelling out its shell twice. Step 5b is one call. The checker's JSON already carries the stale ids, so a second --ids pass recomputed the run's most expensive read; --ids is now the fallback for a JSON that could not be produced. A new "Execution" section carries the script's usage, what the model still owns (the install_new render, the journal root, and Step 6), and the ask-policy stop before Step 4 with its --only-install re-entry. Every section anchor is unchanged; the file loses 88 lines. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
The action router now spells the execution shape out: one bundled sync-run.sh
call with the substituted journal root and the rendered install_new word, read
the JSON digest, resolve an ask-policy install gap through the batched prompt
and the --only-install re-entry, then emit the Report. context/sync.md loads
when a digest carries errors, a gap, a catalog regression, or a cache finding,
and when the user asks why a step behaved as it did.
The policy is passed as a WORD, never as the ${user_config.install_new} token:
an unset key leaves that token literal, and a literal token inside a command is
a shell substitution error rather than a policy. The render itself, the Report
template, and every scope, downgrade, stale-record, cache-content and reload
guarantee are unchanged.
Step 5b's contract line now says the checker runs once per marketplace and that
--ids is the fallback for a JSON that could not be produced.
The two evals that named fleet-state.sh as the thing the response calls now name
sync-run.sh as the executor that takes those reads; no eval asserted the
step-per-turn shape, so nothing else changed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
…carry the state rows the report needs Three defects the first pass left in sync-run.sh, each one a report the operator would have had to distrust. An id the CLI answers with "already at the latest version" changed nothing, and a post-sweep snapshot cannot tell that apart from an update whose state write has not landed. The CLI line is now classified at capture time, so a no-op is dropped instead of rendering as `<id>: <old> -> unknown` under Updated: — on an already-current fleet that was most of the sweep. The move ledger writes an empty version as `-`. Tab is IFS whitespace, so two adjacent tabs collapse into one delimiter and an empty interior field shifted every later column left by one. The digest now carries the Report rows whose only source is a fleet-state read: the marketplace's autoUpdate, the stale project records with their distinct paths, the user-scope orphans, and the share of actionable divergences that belongs to the repo the run stands in. Step 4 also inspects the project-scope enabledPlugins map with --report-project after an install, which is a report row and never a write. The --only-install re-entry keeps the first pass's sweep rows, project_root and withheld downgrades, so the digest that supersedes the first one is still a whole-run report. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
… the run it re-enters The documented re-entry carries no marketplace argument, so with no target the run fell through to default-marketplace resolution: after a `--marketplace foo` or an `--all` first pass it would have performed Steps 4 and 5 against the default marketplace and emitted a one-block digest for a run that covered several. That is the silent partial sweep the marketplace-scoping rule exists to prevent. The re-entry now derives its marketplaces from the run directory's own pre-refresh snapshots, and a marketplace the run never swept is exit 2 with the reason rather than a run started under the wrong name. The suite exercises the documented form (no marketplace argument) and the refusal. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
…t path Steps 1-5b of the plugins skill's sync and audit now run through one sync-run.sh invocation with a JSON digest (2 model turns on a current fleet, was 14); cache-content-check.sh batches fleet-wide (400 spawns -> 17, ~100s -> ~15s on the authoring host); Step 5b calls the checker once instead of twice; a journal-creation failure now exits 2 before any mutation instead of degrading. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
…rce dir `tree_hash` was keyed `<source-dir>␟<rel>` while `owner_idx` and `g_seen` are keyed by sha too. Two records sharing one source directory at different `gitCommitSha` — the same plugin id at user and project scope, same version, one `installPath` — merged their trees: a file present only at the newer sha read as in-tree for the older record, never reached `extras_idx`, and a stale cache verdicted `match`. Reachable at `--scope project` and `--scope all`. The lookup in the cache-split pass now spells the same key. Every other map in the batch join was audited: `owner_idx`, `g_seen`, `uniq_seen` and `tree_rc_bad` already carry the sha; `cache_hash` and `extras_idx` are keyed per install record; `extra_seen`/`ignored` are sha-free on purpose, because a `.gitignore` answer does not depend on the commit. The new case builds a third commit whose only delta is one added file, so the older record's verdict rests on the extra-in-cache direction alone — the one the merged map suppressed. It fails on the unfixed script and passes on the fixed one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
…ent rows a digest source
Three Report rows could not be rendered from the digest:
- `In-repo:` chose between its three variants from `project_root` and the
intersection of `currentProject` records with the divergences, so a current
fleet inside a repo that HAS project/local installs rendered as
"0 — <root> has no project/local installs", which is false. The digest now
carries `in_repo_records` (and `in_repo_ids`), the count of records belonging
to the root regardless of whether any moved, and SKILL.md keys the three
variants off `project_root` null / `in_repo_records == 0` / `> 0` with
`in_repo.updated | length` as the count.
- `Stale project records` needs a count per path: `stale_project_records` is now
`{total, by_path:[{path,count}]}` instead of `{records, paths}`.
- `Cache content` needs a per-id `<version> — <n> file(s) differ`:
`cache_content.stale[]` is derived from the checker JSON already on disk,
summing all three directions of disagreement. `stale_ids` stays alongside it.
Also:
- `self_updated` scanned only the user sweep and the downgrades. A project/local
record for this plugin that Step 2 moved is the same self-update, so
`IR_UPDATED` joins the scan.
- `--allow-downgrade` is now persisted in the run directory and honoured on an
`--only-install` re-entry, which otherwise rebuilt the withheld list from the
run's own downgrade projection and reported a rollback the first pass had
already applied. A run directory without the file keeps the previous
behaviour.
- `proj_err` bounds its own output through `trunc` rather than leaving each of
its six call sites to embed unbounded stderr in a digest string.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fc813d32c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Resolves plugin.json (0.45.0 stays above main's 0.44.6) and CHANGELOG.md (the 0.45.0 entry sits above main's 0.44.6). sync.md auto-merged: main's Step 1 issue-status paragraph lands unchanged beside the restructure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
|
PR body contract — issue linkage This PR body does not yet satisfy the issue-linkage contract:
Edit the body and this comment updates itself on the next run. |
… for shellcheck CI lints at info severity: SC2016 on eleven single-quoted jq programs and test literals (every $var there is jq's or the unrendered placeholder under test), and SC2249 on the self-update case. Same line-level directive the sibling scripts use, plus an explicit default arm. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg
An ask-policy re-entry reset IR_FAILED, US_FAILED, and MP_ERRORS, so the replacement digest could report a successful sync while a plugin stayed stale. Persist those rows (and the refresh digest fields) in the run directory and restore them on re-entry. The batched cache-content join now attaches each blob to every matching source directory, not only the longest, so nested marketplace sources keep the same expected trees the per-source ls-tree produced. Co-authored-by: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com>
Summary
/claude-ops:plugins syncon an already-current fleet took 7:00 wall-clock. Measured on the authoring host: 47% was Step 5b callingcache-content-check.shtwice (~100 s each), 53% was 14 model turns executingsync.mdone bash call at a time.sync-run.sh(new, tested): executes Steps 1–5b in one invocation and prints a JSON digest. SKILL.md routessync/auditthrough it: 14 turns → 2 on a current fleet (+2 only on aninstall_new: askgap, via--only-installre-entry). Every guarantee insync.mdis preserved: per-step re-reads, journaledtee+PIPESTATUS[0], downgrade guard /--allow-downgrade, audit zero-mutation with scratch dir, per-marketplaceallloop, Step 4/5 gating on the fresh pre-install read, project-scope enable gaps reported not written, catalog regression check, pre/mid/post divergence attribution, self-update note. One guarantee strengthened: no run directory → exit 2 before any mutation (was "best-effort").cache-content-check.shbatched fleet-wide: 400 external spawns → 17; ~100 s → ~15 s here. Output byte-identical to the previous script on the live fleet and on a synthetic stale-content fixture. Tree map keyed by sha (regression test added; fails on the previous batching draft).sync.md799 → 715 lines; rationale kept, retyped inline bash removed.claude-ops0.44.3 → 0.45.0.Evidence
Goal, floor and baseline:
.work/plugins-sync-under-2min/(gitignored, machine-bound). Counters (drift-immune, primary): checker spawns 400 → 17 (three independent censuses agree); checker calls/marketplace 2 → 1 (test-asserted); model turns 14 → 2 (by construction, to be confirmed post-merge from a transcript). Durations are host-state-bound and reported separately in the verify.Test plan
cache-content-check.test.sh25/25,sync-run.test.shall pass,fleet-state.test.sh90/90check-shell-portability.sh,check-skill-portability.sh, markdownlint, changelog parity/bump checks--auditsmoke: zero mutations, digest completeclaude plugin update claude-ops,/reload-plugins, one converging sync, one measured sync → turn count and wall-clock from the transcript🤖 Generated with Claude Code
https://claude.ai/code/session_01Pg4g2e7wXamNq2h4JeU8yg