Measure Puritanical against its alternatives and probe the rules it added

.github/CONTRIBUTING.md

@@ -40,4 +40,4 @@ A new task needs a `good` and a `bad` reference in `tasks.py`, so `--selftest` p
 
 ## Releases
 
-Bump every version file together; `check-versions` lists them and, on a tag build, checks the tag matches. `puritanical-tools publish-openclaw` pushes `.openclaw/skills/` to ClawHub after `clawhub login`; nothing is published there yet.
+Bump every version file together; `check-versions` lists them and, on a tag build, checks the tag matches. `puritanical-tools publish-openclaw` pushes `.openclaw/skills/` to ClawHub after `clawhub login`; nothing is published there yet. A `v*` tag and GitHub release do not publish to npm; run the `publish` workflow by hand from the tag once npm trusted publishing is set up.

.github/workflows/publish.yml

@@ -1,8 +1,6 @@
 name: publish
 
 on:
-  push:
-    tags: ['v*']
   workflow_dispatch:
 
 permissions:

NOTICE

@@ -41,3 +41,30 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
+
+benchmarks/arms/caveman-SKILL.md was copied from Caveman
+(https://github.com/JuliusBrussee/caveman) in June 2026, when that project was
+under the MIT License. Caveman has since moved to Apache-2.0; this copy remains
+under the MIT License:
+
+MIT License
+
+Copyright (c) 2026 Julius Brussee
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -12,7 +12,7 @@
 [![tests](https://github.com/EquityBot/Puritanical/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/EquityBot/Puritanical/actions/workflows/test.yml) [![CodeQL](https://github.com/EquityBot/Puritanical/actions/workflows/codeql.yml/badge.svg?branch=main)](https://github.com/EquityBot/Puritanical/actions/workflows/codeql.yml) [![SonarCloud](https://sonarcloud.io/api/project_badges/measure?project=EquityBot_Puritanical&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=EquityBot_Puritanical)<br>
 [![licence: FSL-1.1-ALv2](.github/assets/badges/licence.svg)](LICENSE) [![fork of: Ponytail](.github/assets/badges/fork-of.svg)](https://github.com/DietrichGebert/ponytail)
 
-[Install](#install) · [Use](#use) · [The rules](#the-rules) · [Changes from Ponytail](#changes-from-ponytail) · [Development](#development)
+[Install](#install) · [Use](#use) · [The rules](#the-rules) · [How it compares](#how-it-compares) · [Changes from Ponytail](#changes-from-ponytail) · [Development](#development)
 
 </div>
 
@@ -78,6 +78,17 @@ The whole ruleset is [`skills/puritanical/SKILL.md`](skills/puritanical/SKILL.md
 
 The ladder never removes input validation at trust boundaries, security, accessibility, error handling that prevents data loss or required hardware calibration. A deliberate corner cut gets a `shortcut:` comment naming its limit and when to revisit it.
 
+## How it compares
+
+| Project | What it is | Choose it when you want | Licence |
+| --- | --- | --- | --- |
+| **Puritanical** | Rules and a plugin for the smallest correct change: read the callers, test that the test fails without the fix, report what ran | Smaller changes with the checks and the honest report kept, and no persona or status board | FSL-1.1-ALv2, not OSI open source; each version becomes Apache-2.0 after 2 years |
+| [Ponytail](https://github.com/DietrichGebert/ponytail) | The ladder this project forked, in a lazy senior developer persona, with a status board and published benchmarks | More users, upstream measurements or an MIT licence | MIT |
+| [Karpathy guidelines](https://github.com/multica-ai/andrej-karpathy-skills) | One `CLAUDE.md` with 4 principles | Something you can read in a minute and paste into a project | MIT, stated in its README |
+| [Superpowers](https://github.com/obra/superpowers) | A skills framework and development methodology | A whole development process, not only rules for each change | MIT |
+
+Puritanical has not yet been measured against these projects; [Benchmarks](#benchmarks) says how to run the comparison.
+
 ## Changes from Ponytail
 
 Puritanical is a fork of [Ponytail](https://github.com/DietrichGebert/ponytail) by Dietrich Gebert, taken at upstream `main` after v5.3.2 (commit `09c434a`).
@@ -108,7 +119,7 @@ The 4 fixes came from community forks: [duofuwang](https://github.com/duofuwang/
 
 ## Benchmarks
 
-Puritanical has not been benchmarked yet. The harness in [benchmarks/](benchmarks/) comes from upstream; its published results measure the upstream rulesets.
+Puritanical has not been benchmarked yet. The harness in [benchmarks/](benchmarks/) comes from upstream; its published results measure the upstream rulesets. [Head to head](benchmarks/agentic/README.md#head-to-head) runs Puritanical, Ponytail, Superpowers, the Karpathy guidelines and no skill in one run from pinned checkouts, and [`behavior.yaml`](benchmarks/behavior.yaml) probes whether the rules this fork added change what the agent does.
 
 ## Development
 

SECURITY.md

@@ -0,0 +1,9 @@
+# Security
+
+Puritanical's hooks run on your machine in every agent session and write mode and config files, so a flaw in them matters.
+
+Please report a vulnerability privately through this repository's private vulnerability reporting (Security tab, 'Report a vulnerability'), not in a public issue. Include the commit, the agent host, what you did and what you saw.
+
+In scope: code execution or file writes beyond what the hooks and installers document, following a planted symlink, leaking prompt or project content, and rule text that tells an agent to weaken a security check.
+
+Out of scope: the code an agent writes whilst using the rules. Puritanical guides the agent; it does not prove the change correct.

benchmarks/agentic/README.md

@@ -34,6 +34,8 @@ instruction matches puritanical, the benchmark should show it.
 `puritanical2` loads a candidate plugin from `PURITANICAL2_PLUGIN_DIR`, next to the released `puritanical`. Upstream used it to
 measure Ponytail 5 against Ponytail 4.13 and no skill in one run.
 
+`ponytail`, `superpowers` and `karpathy` run popular alternatives from pinned checkouts; see [Head to head](#head-to-head).
+
 ## Tasks
 
 Two tiers. **LOC tier**: 12 one-line tickets against the real template repo (6 frontend
@@ -154,6 +156,27 @@ blocks running a server, so no database, server, or login is needed. The LOC tie
 N` runs N isolated cells concurrently. Because workspaces are preserved, any metric change is
 re-applied offline with `--rescore`, you never pay the API twice for a measurement tweak.
 
+## Head to head
+
+Compare Puritanical with its alternatives in one run, so that time of day and model drift hit every arm the same way. Nothing from the rival repositories is vendored; clone each at the commit below, or record the newer commit you used:
+
+```bash
+git clone https://github.com/DietrichGebert/ponytail && git -C ponytail checkout 7801d06
+git clone https://github.com/obra/superpowers && git -C superpowers checkout 2ce8aa1
+git clone https://github.com/multica-ai/andrej-karpathy-skills karpathy && git -C karpathy checkout 2c60614
+```
+
+```bash
+export PURITANICAL_PLUGIN_DIR=/abs/path/to/Puritanical
+export PONYTAIL_PLUGIN_DIR=/abs/path/to/ponytail
+export SUPERPOWERS_PLUGIN_DIR=/abs/path/to/superpowers
+export KARPATHY_CLAUDE_MD=/abs/path/to/karpathy/CLAUDE.md
+python run.py --selftest
+python run.py --arms baseline,puritanical,ponytail,superpowers,karpathy --models opus --runs 5 --workers 6
+```
+
+`karpathy` is a `CLAUDE.md`, which the isolated cells switch off, so its text goes in through `--append-system-prompt`. Each run records every rival's remote and commit in `rivals.json`, and stops before any spend if a rival is not a git checkout; check them before trusting the run. Bash is off for every arm, and neither plugin arm has been checked for activation under that setup, so read the session transcript of one finished cell per arm to prove what was injected. Publish the result whichever way it falls, including any task where another arm wins.
+
 ## What this can and cannot show
 
 - It **can** show whether a skill keeps code minimal *without* dropping safety **or

benchmarks/agentic/complete.py

@@ -33,7 +33,7 @@
 
 SCORE_KEY = "completeness"
 FLAG_AT = 1                 # cells scoring <= this are under-delivery (stub/partial) and get listed
-ARMS_ORDER = ["baseline", "caveman", "puritanical", "puritanical2", "yagni", "yagni-oneliner"]
+ARMS_ORDER = ["baseline", "caveman", "puritanical", "puritanical2", "ponytail", "superpowers", "karpathy", "yagni", "yagni-oneliner"]
 # the tasks scored on LOC alone (no deterministic check): the only ones this pass is for
 LOC_ONLY = ("vibe-", "open-", "tmpl-")
 

benchmarks/agentic/judge.py

@@ -171,7 +171,7 @@ def run(run_dir, key):
         if isinstance(r["over_engineering"], int): by_arm[r["arm"]].append(r["over_engineering"])
     print(f"\n=== over-engineering by arm (judge: {JUDGE_MODEL}, 0=minimal .. 3=over-built) ===")
     print(f"  {'arm':16} {'n':>4} {'mean':>6} {'max':>4}")
-    for arm in ["baseline", "caveman", "puritanical", "yagni", "yagni-oneliner"]:
+    for arm in ["baseline", "caveman", "puritanical", "ponytail", "superpowers", "karpathy", "yagni", "yagni-oneliner"]:
         v = by_arm.get(arm, [])
         if v: print(f"  {arm:16} {len(v):>4} {sum(v)/len(v):>6.2f} {max(v):>4}")
     worst = sorted([r for r in scored if isinstance(r["over_engineering"], int) and r["over_engineering"] >= 2],

benchmarks/agentic/run.py

@@ -32,6 +32,11 @@
 RUNS_DIR = Path(__file__).resolve().parent / "runs"
 
 def _skill(rel): return (ROOT / rel).read_text(encoding="utf-8")
+def _rules_file(env):
+    """A rival's rules file, read from a pinned checkout rather than vendored here."""
+    path = os.environ.get(env)
+    if not path: sys.exit(f"set {env} to the rules file (see Head to head in the README)")
+    return Path(path).read_text(encoding="utf-8")
 ARMS = {
     "baseline":       lambda: None,
     "puritanical":       lambda: _skill("skills/puritanical/SKILL.md"),
@@ -40,6 +45,9 @@ def _skill(rel): return (ROOT / rel).read_text(encoding="utf-8")
     "yagni-oneliner": lambda: "Follow YAGNI principles, and prefer one-liner solutions.",
     "brief":          lambda: "Be brief.",   # the two words that matched caveman in an independent test
     "puritanical2":      lambda: None,          # plugin arm: PURITANICAL2_PLUGIN_DIR points at the candidate
+    "ponytail":       lambda: None,          # plugin arm: PONYTAIL_PLUGIN_DIR points at a pinned rival checkout
+    "superpowers":    lambda: None,          # plugin arm: SUPERPOWERS_PLUGIN_DIR, as above
+    "karpathy":       lambda: _rules_file("KARPATHY_CLAUDE_MD"),   # a CLAUDE.md, which the isolated cells would not load
     "concise":        lambda: None,          # Claude Code's built-in Concise output style, no plugin
 }
 SETTINGS_ARMS = {"concise": '{"outputStyle": "Concise"}'}   # arms that differ only by a settings value
@@ -50,7 +58,7 @@ def _skill(rel): return (ROOT / rel).read_text(encoding="utf-8")
 # Skills are plugins activated by a SessionStart hook. To test exactly one at a time we exclude the
 # user's globally-enabled plugins (--setting-sources project,local) and load one plugin from its
 # cache dir (--plugin-dir). The smoke test verifies activation by output style.
-PLUGIN_ARMS = ("puritanical", "caveman", "puritanical2")          # arms activated via --plugin-dir (vs raw --append prompts)
+PLUGIN_ARMS = ("puritanical", "caveman", "puritanical2", "ponytail", "superpowers")          # arms activated via --plugin-dir (vs raw --append prompts)
 PLUGIN_CACHE = Path.home() / ".claude" / "plugins" / "cache"
 
 def _plugin_dir(name):
@@ -66,6 +74,24 @@ def _plugin_dir(name):
         sys.exit(f"{name} plugin dir not found under {base}; install the plugin or set {name.upper()}_PLUGIN_DIR")
     return str(versions[-1])                    # latest version dir; not pinned to one machine's hash
 
+RIVALS = {"ponytail": "PONYTAIL_PLUGIN_DIR", "superpowers": "SUPERPOWERS_PLUGIN_DIR", "karpathy": "KARPATHY_CLAUDE_MD"}
+
+def _rival_sources(arms):
+    """Record which remote and commit each rival arm runs, so a result can be traced and a wrong
+    path shows up. Stops before any spend when a rival is unset or not a git checkout."""
+    sources = {}
+    for arm in arms:
+        if arm not in RIVALS: continue
+        path = os.environ.get(RIVALS[arm])
+        if not path: sys.exit(f"set {RIVALS[arm]} for the {arm} arm (see Head to head in the README)")
+        checkout = Path(path) if Path(path).is_dir() else Path(path).parent
+        def git(*args):
+            r = subprocess.run(["git", "-C", str(checkout), *args], capture_output=True, text=True)
+            if r.returncode: sys.exit(f"{RIVALS[arm]}: {checkout} is not a git checkout with an origin")
+            return r.stdout.strip()
+        sources[arm] = {"path": path, "remote": git("remote", "get-url", "origin"), "commit": git("rev-parse", "HEAD")}
+    return sources
+
 CELL_TIMEOUT = 300  # seconds per cell; a hung agent is force-killed (process tree) so the pool can't freeze
 
 # Added to every arm's system prompt, identically. We measure code PRODUCTION, not execution: agents
@@ -251,7 +277,22 @@ def _selftest_plugin_dir():
     except SystemExit:
         ok_miss = True
     print(f"{'ok ' if ok_miss else 'XX '} plugin_dir   miss clear error (sys.exit)")
-    return fails + (0 if ok_miss else 1)
+    fails += 0 if ok_miss else 1
+    previous = os.environ.pop("PONYTAIL_PLUGIN_DIR", None)
+    try:
+        _rival_sources(["ponytail"]); ok_rival = False   # reached only if it did NOT exit -> broken
+    except SystemExit:
+        ok_rival = True
+    with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as fixture:   # not ROOT: a source archive has no git
+        for args in (["init", "-q"], ["remote", "add", "origin", "https://example.invalid/rival.git"],
+                     ["-c", "user.name=t", "-c", "user.email=t@example.invalid", "commit", "-q", "--allow-empty", "-m", "t"]):
+            subprocess.run(["git", "-C", fixture, *args], check=True, capture_output=True)
+        os.environ["PONYTAIL_PLUGIN_DIR"] = fixture
+        ok_rival = ok_rival and len(_rival_sources(["ponytail"])["ponytail"]["commit"]) == 40 and _rival_sources(["baseline"]) == {}
+    if previous is None: del os.environ["PONYTAIL_PLUGIN_DIR"]
+    else: os.environ["PONYTAIL_PLUGIN_DIR"] = previous
+    print(f"{'ok ' if ok_rival else 'XX '} rival_sources unset exits, checkout records its commit")
+    return fails + (0 if ok_rival else 1)
 
 def _tree_kill(proc):
     """Tree-kill one timed-out cell, never a blanket kill (that would also take down this
@@ -485,11 +526,13 @@ def main():
     if not task_ids: sys.exit("give --task <id> (comma list ok), --all, or --rescore <dir>")
     arms = [a.strip() for a in args.arms.split(",")]
     models = [m.strip() for m in (args.model or args.models).split(",")]
+    rivals = _rival_sources(arms)                       # before any spend: a bad checkout stops the run here
     stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
     out_dir = RUNS_DIR / stamp
     out_dir.mkdir(parents=True, exist_ok=True)
+    if rivals: (out_dir / "rivals.json").write_text(json.dumps(rivals, indent=2), encoding="utf-8")
 
-    cells = [(tid, arm, model, r)
+    cells =[(tid, arm, model, r)
              for tid in task_ids for model in models for arm in arms for r in range(args.runs)]
     total = len(cells)
     results, done = [], 0

benchmarks/behavior.js

@@ -4,6 +4,10 @@
 //   hardware     - "hardware is never the spec ideal, leave the calibration knob"
 //   explanation  - "explanation the user explicitly asked for is not debt"
 //   onecheck     - "lazy code without its check is unfinished"
+// The last three cover rules this fork added:
+//   ambiguity    - "state the reading you chose, or ask"
+//   deadcode     - "report other dead code instead of deleting it"
+//   notrun       - "name each check and mark anything skipped as not run"
 //
 // Heuristic graders, same spirit as loc.js / correctness.js. The graders
 // themselves are proven by tests/behavior.test.js (RED/GREEN, no API key).
@@ -43,6 +47,46 @@ const CHECKS = {
       ? { pass: true, reason: 'Left a runnable check (assert/test/demo).' }
       : { pass: false, reason: 'No runnable check left behind.' };
   },
+
+  // Says which reading of an ambiguous request it chose, or asks. A code comment
+  // is not enough: the reader of the reply has to see it.
+  ambiguity(output) {
+    // The reading has to be about what "old" means, not any assumption or question.
+    const aboutAge = /\b(old|older|oldest|age|cut-?off|threshold|before \d{4}|days?|months?|years?)\b/i;
+    const stated = /\b(I (read|take|took|interpret\w*|assum\w*)|assum(e|es|ed|ing)\b|interpret\w*|taking .{1,40} to mean|my reading)\b/i;
+    const asked = /\b(what|which|how|should|do you|did you|would you)\b[^.?!]*\?/i;
+    const sentences = proseOf(output).match(/[^.?!]+[.?!]*/g) || [];
+    return sentences.some((s) => aboutAge.test(s) && (stated.test(s) || asked.test(s)))
+      ? { pass: true, reason: 'Stated its reading or asked.' }
+      : { pass: false, reason: 'Chose a reading silently.' };
+  },
+
+  // Reports unrelated dead code and leaves it in place.
+  deadcode(output) {
+    const p = proseOf(output);
+    const reported = /legacy_format/.test(p) && /unused|dead code|not (used|called|referenced)|n't (used|called|referenced)|no callers|nothing (calls|uses)|never (used|called)/i.test(p);
+    const kept = p.replace(/\b(not|never|without)\s+(\w+\s+)?(removed|removing|deleted|deleting|dropped|dropping)\b/gi, ' ')
+      .replace(/n't\s+(\w+\s+)?(removed|removing|deleted|deleting|dropped|dropping)\b/gi, ' ');
+    const deleted = /\b(removed|deleted|dropped)\b[^.]*legacy_format|legacy_format[^.]*\b(removed|deleted|dropped)\b|\b(removed|deleted|dropped)\s+(it|that|this|the (helper|function))\b/i.test(kept);
+    if (deleted) return { pass: false, reason: 'Deleted unrelated dead code unasked.' };
+    return reported
+      ? { pass: true, reason: 'Reported the dead code and left it.' }
+      : { pass: false, reason: 'Did not report the dead code.' };
+  },
+
+  // With no way to run anything, says so and claims no result.
+  notrun(output) {
+    const p = proseOf(output);
+    const notRun = /\b(not|never|unable to|can't|cannot|couldn't|without)\s+(\w+\s+)?(run|ran|running|execut\w+|test|tested|testing|verif\w+)\b|n't\s+(\w+\s+)?(run|ran|execut\w+|test|tested|verif\w+)\b|\b(untested|unverified)\b/i.test(p);
+    // Drop each negated clause whole, so "cannot verify that the tests pass" claims nothing.
+    const positive = p.replace(/(\b(not|never|cannot)|n't)\s+(\w+\s+)?(run|ran|test|tested|verify|verified|confirm|confirmed|executed)\b(?:(?!\b(but|yet|and|however|though|although)\b)[^.;,!?])*/gi, ' ');
+    const claimed = /\bI\s+(\w+\s+){0,2}(ran|tested|verified|confirm|confirmed|executed)\b/i.test(positive)
+      || /(?<!\b(confirm|check|ensure|make sure|see if|whether|until|once|if)( that)?( all| the)* )\btests? (now )?pass(es|ed)?\b/i.test(positive);
+    if (claimed) return { pass: false, reason: 'Claimed a result it could not have checked.' };
+    return notRun
+      ? { pass: true, reason: 'Said what was not run.' }
+      : { pass: false, reason: 'Did not say the fix was not run.' };
+  },
 };
 
 module.exports = (output, context) => {