Skip to content

Measure Puritanical against its alternatives and probe the rules it added - #20

Merged
ryanduguid merged 5 commits into
mainfrom
ryan/best-on-github
Oct 30, 2026
Merged

ryanduguid merged 5 commits into
mainfrom
ryan/best-on-github

Conversation

@ryanduguid

Copy link
Copy Markdown
Collaborator

Summary

Lets Puritanical be measured against its popular alternatives, and tests whether the rules this fork added change what the agent does. Also closes licence and trust gaps. No rule text changes.

benchmarks/agentic/run.py   ponytail, superpowers and karpathy arms from pinned checkouts; rivals.json records each commit
benchmarks/behavior.*       probes: ambiguity, deadcode, notrun
NOTICE                      Caveman's MIT notice for the vendored skill
SECURITY.md                 private reporting and scope
README.md                   How it compares
publish.yml                 manual only, so a v* tag cannot trigger a failing npm publish

Evidence

  • Before: 'I have not run it, but I can confirm all tests pass' and 'legacy_format is unused. I deleted it.' had no probe to fail them.
    After: tests/behavior.test.js has 31 passed, 0 failed, with both as expected failures.
  • A rival arm that is unset or not a git checkout stops the run before any spend; run.py --selftest covers this and passes from a checkout and from a source archive.
  • sync --check, check-versions and pi-extension (30 of 30) pass.

Merge danger

Door: two-way. Blast radius: benchmarks, docs

A v* tag no longer publishes to npm; run the publish workflow by hand once trusted publishing is set up. Nothing is published to npm today.

Unverified

  • No arm has run against a live model, so the probes and rival arms are unproven end to end, and no results exist. Plugin activation under the Bash-free setup needs one transcript per arm.
  • The probes are regex heuristics with known misses: 'I have not run and verified the fix' fails notrun wrongly, and 'created before 1 January 2020' fails ambiguity wrongly.
  • SECURITY.md points to private vulnerability reporting, which is off until the owner enables it.
  • Locally, 2 correctness.test.js cases (email bare unfenced code, csv timeout) fail, as they do on unmodified main; cause not established. CI decides.

Run Ponytail, Superpowers and the Karpathy guidelines from pinned checkouts
alongside Puritanical, with nothing vendored. Probe the rules this fork added:
stated reading, dead-code reporting and not-run reporting.

Add Caveman's MIT notice and SECURITY.md, compare the alternatives in the
README, and stop a v* tag from triggering an npm publish that cannot succeed.
Fail replies that report dead code and delete it, hedge and then claim a
result, or state an assumption unrelated to the ambiguous request. Record each
rival's remote and commit, and stop before any spend on a bad checkout.
Drop a negated clause whole so 'cannot verify that the tests pass' claims
nothing, narrow what counts as a reading of 'old', and give the rival selftest
its own git fixture so a source archive still runs it.
'Sorted before pruning' no longer counts as a reading of 'old', and a clause
after 'but' or 'and' is no longer dropped with the negation before it.
Same 23 cases, without the repeated test blocks.
@sonarqubecloud

Copy link
Copy Markdown

@ryanduguid
ryanduguid merged commit 56d4252 into main Oct 30, 2026
9 checks passed
@ryanduguid
ryanduguid deleted the ryan/best-on-github branch October 30, 2026 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant