From 0aa54ad9ad357be7fd5ee4b7e0a9018ba637c6c2 Mon Sep 17 00:00:00 2001 From: Poytr1 Date: Wed, 12 Aug 2026 00:30:39 +0800 Subject: [PATCH] docs: attach rollout provenance; flag nullcheck as single-rollout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit prompted by a collaborator voiding one of their own datasets: the nullcheck -354 figure was measured at --rollouts 1, which this same document elsewhere calls unreliable for A/B (prompt-cache carryover between the two sequential episodes) — the identical defect that led to retracting the aksops +14%. It was sitting in two summary tables with no rollout count attached. - Add a `rollouts` column to both summary tables (nullcheck 1, shellseq 3, migrate 3, aksops 6). - Mark nullcheck indicative-only with an explicit "do not quote the figure". - Show the actual --rollouts 1 flag in its result block + caveat. The headline conclusion is unchanged and does not depend on it: shellseq independently gives ~0% at 3 rollouts, aksops ~neutral at 6. Co-Authored-By: Claude Opus 4.8 --- .../specs/2026-07-06-benchmark-findings.md | 35 ++++++++++++------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/docs/superpowers/specs/2026-07-06-benchmark-findings.md b/docs/superpowers/specs/2026-07-06-benchmark-findings.md index 7d8aa8c..4af8d12 100644 --- a/docs/superpowers/specs/2026-07-06-benchmark-findings.md +++ b/docs/superpowers/specs/2026-07-06-benchmark-findings.md @@ -11,12 +11,18 @@ **Across four repetition workflows, no compiled skill produced a *valid* token win at iso-accuracy.** -| shape | kind | skill source | verified result | -|---|---|---|---| -| `nullcheck` | code-edit | hand-written | ~0% (−354 on ~236k), 100%/100% | -| `shellseq` | trivial shell | real `aj compile` | ~0% (−19..35 on ~93k), 100%/100% | -| `migrate` | code-edit (exploration) | hand-written | **−25% (JIT ~48k WORSE)**, 100%/100% | -| `aksops` | SRE / tool-use | real `aj compile` | ~neutral (−5% mean / +7% median), 100%/100% — noisy, no clear win | +| shape | kind | skill source | rollouts | verified result | +|---|---|---|---|---| +| `nullcheck` | code-edit | hand-written | **1 ⚠️** | ~0% (−354 on ~236k), 100%/100% — **single rollout, indicative only; do not cite the figure** | +| `shellseq` | trivial shell | real `aj compile` | 3 | ~0% (−19..35 on ~93k), 100%/100% | +| `migrate` | code-edit (exploration) | hand-written | 3 | **−25% (JIT ~48k WORSE)**, 100%/100% | +| `aksops` | SRE / tool-use | real `aj compile` | **6** | ~neutral (−5% mean / +7% median), 100%/100% — noisy, no clear win | + +⚠️ **Provenance caveat (added 2026-08-12).** The `nullcheck` figure was measured at `--rollouts 1`, which +this document elsewhere states is unreliable for A/B comparison (prompt-cache state carries across the two +sequential episodes) — the same defect that led to retracting the `aksops` +14%. It is retained as an +*indicative* data point only; **do not quote −354 as a measurement.** The headline conclusion does not depend +on it: `shellseq` independently gives ~0% at 3 rollouts and `aksops` ~neutral at 6. **On `aksops` the apparent +14% was a small-sample artifact and is retracted.** At `--rollouts 3` the JIT arm happened to pass 3/3 and looked like a ~14% saving; at `--rollouts 6` it succeeded only **2/6** @@ -76,11 +82,16 @@ positive claim (this one) before it becomes folklore. ## Result — `nullcheck` (n = 2) ``` -aj bench --gen nullcheck --n 2 --compare +aj bench --gen nullcheck --n 2 --compare --rollouts 1 # ⚠️ ONE rollout — indicative only nullcheck-2: baseline 100% / jit 100% (iso-accuracy) T2S baseline 235,695 vs jit 235,341 → saving 354 tokens/use ``` +⚠️ **This was a single rollout.** See the provenance caveat in the TL;DR: at `--rollouts 1` the A/B is +confounded by prompt-cache carryover between the two sequential episodes, so treat the direction (~no +saving) as indicative and **do not quote the 354 figure as a measurement.** `shellseq` (3 rollouts) +independently reproduces the ~0% result. + Both arms verified (2 nil-guards added, package builds). Earlier single-arm baseline sweep: ``` @@ -158,11 +169,11 @@ project-local skill can be **net-negative even on an exploration-heavy task**. **Cross-shape summary (all at iso-accuracy):** -| shape | skill source | per-use token delta | -|---|---|---| -| `nullcheck` | hand-written | ~0% (−354 tok on ~236k) | -| `shellseq` | real `aj compile` | ~0% (−19..35 tok on ~93k) | -| `migrate` | hand-written | **+25% (JIT ~48k MORE)** | +| shape | skill source | rollouts | per-use token delta | +|---|---|---|---| +| `nullcheck` | hand-written | **1 ⚠️** | ~0% (−354 tok on ~236k) — indicative only, don't quote | +| `shellseq` | real `aj compile` | 3 | ~0% (−19..35 tok on ~93k) | +| `migrate` | hand-written | 3 | **+25% (JIT ~48k MORE)** | **Takeaway:** injecting a skill is not free, and for these Claude-Code-shaped tasks the skill's context cost tends to swamp what it saves — sometimes badly. "Repetitive workflow → compile a skill" is *not*