[ryanduguid]

Summary

Makes the head-to-head benchmark safe to pay for. These fixes follow #20, which merged before they were pushed. No rule text changes, and no paid session has run.

run.py      pins every source; free preflight; retries; stops after 5 failed cells; --resume;
            keeps a paid session's workspace if the harness fails; scores each cell in a child process
judge.py    blinds arm markers; judges the diff on fixture tasks; records the cell
compare.py  fixed primary metrics with Bonferroni; size counts only where the job was done;
            --strict refuses missing, duplicate or unscored cells
README.md   the exact pilot and full-run commands, and the analysis plan fixed in advance

run.py now records its sources in sources.json, which replaces rivals.json.

Evidence

• Before: a harness exception after a session left its workspace in scratch, so a resume paid for it again.
  After: with a fake CLI, an exception right after the first session kept both cells, and resume finished 2 of 2 with 0 new sessions.
• Before: harness files counted as agent-added, so abstain-fixed failed every arm.
  After: a selftest failed before the fix and passes after it.
• Before: --strict rejected complete results merged from separate runs, because each run numbers its cells from zero.
  After: a selftest failed before the fix and passes after it; passing one run twice still fails.
• Fake CLI: a usage limit stopped after 5 failed cells and resume finished 8 of 8 with 6 new sessions. Fewer cells and changed sources were refused. A hard kill at scoring kept 3 cells and resume ran 5 more.
• 40 reference solutions score the same in the child process as in-process.
• The real CLI passed the free preflight for baseline, puritanical, yagni and concise.
• run.py --selftest and complete.py --selftest-offline pass at this head. node --test tests/*.test.js gives 154 passed, 2 failed, 4 skipped; it was last run before the final compare.py commit, which touches no JS.

Merge danger

Door: two-way. Blast radius: benchmarks

Only benchmarks/agentic changes. Runs started under the old harness cannot resume under this one.

Unverified

• No paid session has run, so a successful stream and a real rate limit are unproven. The pilot is the next step and needs the owner's approval.
• The ponytail, superpowers and karpathy arms have not run under preflight, because their hooks would execute. Whether Superpowers' brainstorming stops to ask is untested.
• 2 correctness.test.js cases (email bare unfenced code, csv timeout) fail locally, as they do on main; cause not established.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues
0 New dependency risks

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud