docs: attach rollout provenance; flag nullcheck figure as single-rollout - #19
Merged
Merged
Conversation
Audit prompted by a collaborator voiding one of their own datasets: the nullcheck -354 figure was measured at --rollouts 1, which this same document elsewhere calls unreliable for A/B (prompt-cache carryover between the two sequential episodes) — the identical defect that led to retracting the aksops +14%. It was sitting in two summary tables with no rollout count attached. - Add a `rollouts` column to both summary tables (nullcheck 1, shellseq 3, migrate 3, aksops 6). - Mark nullcheck indicative-only with an explicit "do not quote the figure". - Show the actual --rollouts 1 flag in its result block + caveat. The headline conclusion is unchanged and does not depend on it: shellseq independently gives ~0% at 3 rollouts, aksops ~neutral at 6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Self-audit prompted by a collaborator voiding one of their own datasets. The
nullcheck−354 tokens figure was measured at--rollouts 1— which this same document elsewhere states is unreliable for A/B comparison (prompt-cache state carries across the two sequential episodes). That is the identical defect that led to retracting theaksops+14% in #16. It was sitting in two summary tables with no rollout count attached, and was about to be cited externally.Changes
rolloutscolumn added to both summary tables:nullcheck1shellseq3,migrate3,aksops6.nullcheckmarked indicative only — do not quote the figure, with a provenance caveat in the TL;DR.--rollouts 1flag plus the caveat inline.Impact on conclusions
None. "No JIT token win on any shape" does not rest on
nullcheck:shellseqindependently reproduces ~0% at 3 rollouts, andaksopsis ~neutral at 6. The confound-isolation argument is also unaffected — it rests on traced bypass behaviour inshellseq/aksops, not on any token figure.Docs-only.
🤖 Generated with Claude Code