From 3b85577153e65343ab666ad430998be0a9fbf15a Mon Sep 17 00:00:00 2001 From: Ryan Duguid Date: Fri, 30 Oct 2026 19:47:01 +1100 Subject: [PATCH 1/5] Add head-to-head benchmark arms, fork rule probes and trust gaps Run Ponytail, Superpowers and the Karpathy guidelines from pinned checkouts alongside Puritanical, with nothing vendored. Probe the rules this fork added: stated reading, dead-code reporting and not-run reporting. Add Caveman's MIT notice and SECURITY.md, compare the alternatives in the README, and stop a v* tag from triggering an npm publish that cannot succeed. --- .github/CONTRIBUTING.md | 2 +- .github/workflows/publish.yml | 2 -- NOTICE | 27 ++++++++++++++ README.md | 15 ++++++-- SECURITY.md | 9 +++++ benchmarks/agentic/README.md | 23 ++++++++++++ benchmarks/agentic/complete.py | 2 +- benchmarks/agentic/judge.py | 2 +- benchmarks/agentic/run.py | 10 +++++- benchmarks/behavior.js | 41 +++++++++++++++++++++ benchmarks/behavior.yaml | 13 +++++-- tests/behavior.test.js | 66 ++++++++++++++++++++++++++++++++++ 12 files changed, 202 insertions(+), 10 deletions(-) create mode 100644 SECURITY.md diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index cebff9591..1b1b5cd47 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -40,4 +40,4 @@ A new task needs a `good` and a `bad` reference in `tasks.py`, so `--selftest` p ## Releases -Bump every version file together; `check-versions` lists them and, on a tag build, checks the tag matches. `puritanical-tools publish-openclaw` pushes `.openclaw/skills/` to ClawHub after `clawhub login`; nothing is published there yet. +Bump every version file together; `check-versions` lists them and, on a tag build, checks the tag matches. `puritanical-tools publish-openclaw` pushes `.openclaw/skills/` to ClawHub after `clawhub login`; nothing is published there yet. A `v*` tag and GitHub release do not publish to npm; run the `publish` workflow by hand from the tag once npm trusted publishing is set up. diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index c4c95da5b..ff297f71d 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -1,8 +1,6 @@ name: publish on: - push: - tags: ['v*'] workflow_dispatch: permissions: diff --git a/NOTICE b/NOTICE index 71e0fda24..b427ec601 100644 --- a/NOTICE +++ b/NOTICE @@ -41,3 +41,30 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +benchmarks/arms/caveman-SKILL.md was copied from Caveman +(https://github.com/JuliusBrussee/caveman) in June 2026, when that project was +under the MIT License. Caveman has since moved to Apache-2.0; this copy remains +under the MIT License: + +MIT License + +Copyright (c) 2026 Julius Brussee + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 83d609cd3..9363afdc1 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ [![tests](https://github.com/EquityBot/Puritanical/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/EquityBot/Puritanical/actions/workflows/test.yml) [![CodeQL](https://github.com/EquityBot/Puritanical/actions/workflows/codeql.yml/badge.svg?branch=main)](https://github.com/EquityBot/Puritanical/actions/workflows/codeql.yml) [![SonarCloud](https://sonarcloud.io/api/project_badges/measure?project=EquityBot_Puritanical&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=EquityBot_Puritanical)
[![licence: FSL-1.1-ALv2](.github/assets/badges/licence.svg)](LICENSE) [![fork of: Ponytail](.github/assets/badges/fork-of.svg)](https://github.com/DietrichGebert/ponytail) -[Install](#install) · [Use](#use) · [The rules](#the-rules) · [Changes from Ponytail](#changes-from-ponytail) · [Development](#development) +[Install](#install) · [Use](#use) · [The rules](#the-rules) · [How it compares](#how-it-compares) · [Changes from Ponytail](#changes-from-ponytail) · [Development](#development) @@ -78,6 +78,17 @@ The whole ruleset is [`skills/puritanical/SKILL.md`](skills/puritanical/SKILL.md The ladder never removes input validation at trust boundaries, security, accessibility, error handling that prevents data loss or required hardware calibration. A deliberate corner cut gets a `shortcut:` comment naming its limit and when to revisit it. +## How it compares + +| Project | What it is | Choose it when you want | Licence | +| --- | --- | --- | --- | +| **Puritanical** | Rules and a plugin for the smallest correct change: read the callers, test that the test fails without the fix, report what ran | Smaller changes with the checks and the honest report kept, and no persona or status board | FSL-1.1-ALv2, not OSI open source; each version becomes Apache-2.0 after 2 years | +| [Ponytail](https://github.com/DietrichGebert/ponytail) | The ladder this project forked, in a lazy senior developer persona, with a status board and published benchmarks | A large community, upstream measurements or an MIT licence | MIT | +| [Karpathy guidelines](https://github.com/multica-ai/andrej-karpathy-skills) | One `CLAUDE.md` with 4 principles | Something you can read in a minute and paste into a project | MIT | +| [Superpowers](https://github.com/obra/superpowers) | A skills framework and development methodology | A whole development process, not only rules for each change | MIT | + +Puritanical has not yet been measured against these projects; [Benchmarks](#benchmarks) says how to run the comparison. + ## Changes from Ponytail Puritanical is a fork of [Ponytail](https://github.com/DietrichGebert/ponytail) by Dietrich Gebert, taken at upstream `main` after v5.3.2 (commit `09c434a`). @@ -108,7 +119,7 @@ The 4 fixes came from community forks: [duofuwang](https://github.com/duofuwang/ ## Benchmarks -Puritanical has not been benchmarked yet. The harness in [benchmarks/](benchmarks/) comes from upstream; its published results measure the upstream rulesets. +Puritanical has not been benchmarked yet. The harness in [benchmarks/](benchmarks/) comes from upstream; its published results measure the upstream rulesets. [Head to head](benchmarks/agentic/README.md#head-to-head) runs Puritanical, Ponytail, Superpowers, the Karpathy guidelines and no skill in one run from pinned checkouts, and [`behavior.yaml`](benchmarks/behavior.yaml) probes whether the rules this fork added change what the agent does. ## Development diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 000000000..7a8abf156 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,9 @@ +# Security + +Puritanical's hooks run on your machine in every agent session and write mode and config files, so a flaw in them matters. + +Please report a vulnerability privately through this repository's private vulnerability reporting (Security tab, 'Report a vulnerability'), not in a public issue. Include the commit, the agent host, what you did and what you saw. + +In scope: code execution or file writes beyond what the hooks and installers document, following a planted symlink, leaking prompt or project content, and rule text that tells an agent to weaken a security check. + +Out of scope: the code an agent writes whilst using the rules. Puritanical guides the agent; it does not prove the change correct. diff --git a/benchmarks/agentic/README.md b/benchmarks/agentic/README.md index 3c8161262..ec2346b06 100644 --- a/benchmarks/agentic/README.md +++ b/benchmarks/agentic/README.md @@ -34,6 +34,8 @@ instruction matches puritanical, the benchmark should show it. `puritanical2` loads a candidate plugin from `PURITANICAL2_PLUGIN_DIR`, next to the released `puritanical`. Upstream used it to measure Ponytail 5 against Ponytail 4.13 and no skill in one run. +`ponytail`, `superpowers` and `karpathy` run the most popular alternatives from pinned checkouts; see [Head to head](#head-to-head). + ## Tasks Two tiers. **LOC tier**: 12 one-line tickets against the real template repo (6 frontend @@ -154,6 +156,27 @@ blocks running a server, so no database, server, or login is needed. The LOC tie N` runs N isolated cells concurrently. Because workspaces are preserved, any metric change is re-applied offline with `--rescore`, you never pay the API twice for a measurement tweak. +## Head to head + +Compare Puritanical with its alternatives in one run, so that time of day and model drift hit every arm the same way. Nothing from the rival repositories is vendored; clone each at the commit below, or record the newer commit you used: + +```bash +git clone https://github.com/DietrichGebert/ponytail && git -C ponytail checkout 7801d06 +git clone https://github.com/obra/superpowers && git -C superpowers checkout 2ce8aa1 +git clone https://github.com/multica-ai/andrej-karpathy-skills karpathy && git -C karpathy checkout 2c60614 +``` + +```bash +export PURITANICAL_PLUGIN_DIR=/abs/path/to/Puritanical +export PONYTAIL_PLUGIN_DIR=/abs/path/to/ponytail +export SUPERPOWERS_PLUGIN_DIR=/abs/path/to/superpowers +export KARPATHY_CLAUDE_MD=/abs/path/to/karpathy/CLAUDE.md +python run.py --selftest +python run.py --arms baseline,puritanical,ponytail,superpowers,karpathy --models opus --runs 5 --workers 6 +``` + +`karpathy` is a `CLAUDE.md`, which the isolated cells switch off, so its text goes in through `--append-system-prompt`. Publish the result whichever way it falls, including any task where another arm wins. + ## What this can and cannot show - It **can** show whether a skill keeps code minimal *without* dropping safety **or diff --git a/benchmarks/agentic/complete.py b/benchmarks/agentic/complete.py index 3ca1d1d84..e8c73c6c7 100644 --- a/benchmarks/agentic/complete.py +++ b/benchmarks/agentic/complete.py @@ -33,7 +33,7 @@ SCORE_KEY = "completeness" FLAG_AT = 1 # cells scoring <= this are under-delivery (stub/partial) and get listed -ARMS_ORDER = ["baseline", "caveman", "puritanical", "puritanical2", "yagni", "yagni-oneliner"] +ARMS_ORDER = ["baseline", "caveman", "puritanical", "puritanical2", "ponytail", "superpowers", "karpathy", "yagni", "yagni-oneliner"] # the tasks scored on LOC alone (no deterministic check): the only ones this pass is for LOC_ONLY = ("vibe-", "open-", "tmpl-") diff --git a/benchmarks/agentic/judge.py b/benchmarks/agentic/judge.py index 2cd1a27a9..7d05316e5 100644 --- a/benchmarks/agentic/judge.py +++ b/benchmarks/agentic/judge.py @@ -171,7 +171,7 @@ def run(run_dir, key): if isinstance(r["over_engineering"], int): by_arm[r["arm"]].append(r["over_engineering"]) print(f"\n=== over-engineering by arm (judge: {JUDGE_MODEL}, 0=minimal .. 3=over-built) ===") print(f" {'arm':16} {'n':>4} {'mean':>6} {'max':>4}") - for arm in ["baseline", "caveman", "puritanical", "yagni", "yagni-oneliner"]: + for arm in ["baseline", "caveman", "puritanical", "ponytail", "superpowers", "karpathy", "yagni", "yagni-oneliner"]: v = by_arm.get(arm, []) if v: print(f" {arm:16} {len(v):>4} {sum(v)/len(v):>6.2f} {max(v):>4}") worst = sorted([r for r in scored if isinstance(r["over_engineering"], int) and r["over_engineering"] >= 2], diff --git a/benchmarks/agentic/run.py b/benchmarks/agentic/run.py index 47755da39..3875cfe60 100644 --- a/benchmarks/agentic/run.py +++ b/benchmarks/agentic/run.py @@ -32,6 +32,11 @@ RUNS_DIR = Path(__file__).resolve().parent / "runs" def _skill(rel): return (ROOT / rel).read_text(encoding="utf-8") +def _rules_file(env): + """A rival's rules file, read from a pinned checkout rather than vendored here.""" + path = os.environ.get(env) + if not path: sys.exit(f"set {env} to the rules file (see Head to head in the README)") + return Path(path).read_text(encoding="utf-8") ARMS = { "baseline": lambda: None, "puritanical": lambda: _skill("skills/puritanical/SKILL.md"), @@ -40,6 +45,9 @@ def _skill(rel): return (ROOT / rel).read_text(encoding="utf-8") "yagni-oneliner": lambda: "Follow YAGNI principles, and prefer one-liner solutions.", "brief": lambda: "Be brief.", # the two words that matched caveman in an independent test "puritanical2": lambda: None, # plugin arm: PURITANICAL2_PLUGIN_DIR points at the candidate + "ponytail": lambda: None, # plugin arm: PONYTAIL_PLUGIN_DIR points at a pinned rival checkout + "superpowers": lambda: None, # plugin arm: SUPERPOWERS_PLUGIN_DIR, as above + "karpathy": lambda: _rules_file("KARPATHY_CLAUDE_MD"), # a CLAUDE.md, which the isolated cells would not load "concise": lambda: None, # Claude Code's built-in Concise output style, no plugin } SETTINGS_ARMS = {"concise": '{"outputStyle": "Concise"}'} # arms that differ only by a settings value @@ -50,7 +58,7 @@ def _skill(rel): return (ROOT / rel).read_text(encoding="utf-8") # Skills are plugins activated by a SessionStart hook. To test exactly one at a time we exclude the # user's globally-enabled plugins (--setting-sources project,local) and load one plugin from its # cache dir (--plugin-dir). The smoke test verifies activation by output style. -PLUGIN_ARMS = ("puritanical", "caveman", "puritanical2") # arms activated via --plugin-dir (vs raw --append prompts) +PLUGIN_ARMS = ("puritanical", "caveman", "puritanical2", "ponytail", "superpowers") # arms activated via --plugin-dir (vs raw --append prompts) PLUGIN_CACHE = Path.home() / ".claude" / "plugins" / "cache" def _plugin_dir(name): diff --git a/benchmarks/behavior.js b/benchmarks/behavior.js index 92578223d..5277b53a9 100644 --- a/benchmarks/behavior.js +++ b/benchmarks/behavior.js @@ -4,6 +4,10 @@ // hardware - "hardware is never the spec ideal, leave the calibration knob" // explanation - "explanation the user explicitly asked for is not debt" // onecheck - "lazy code without its check is unfinished" +// The last three cover rules this fork added: +// ambiguity - "state the reading you chose, or ask" +// deadcode - "report other dead code instead of deleting it" +// notrun - "name each check and mark anything skipped as not run" // // Heuristic graders, same spirit as loc.js / correctness.js. The graders // themselves are proven by tests/behavior.test.js (RED/GREEN, no API key). @@ -43,6 +47,43 @@ const CHECKS = { ? { pass: true, reason: 'Left a runnable check (assert/test/demo).' } : { pass: false, reason: 'No runnable check left behind.' }; }, + + // Says which reading of an ambiguous request it chose, or asks. A code comment + // is not enough: the reader of the reply has to see it. + ambiguity(output) { + const p = proseOf(output); + const stated = /\b(I (read|take|took|interpret\w*|assum\w*)|assum(e|es|ed|ing)\b|interpret\w*|taking .{1,40} to mean|my reading)\b/i.test(p); + const asked = /\b(what|which|how|should|do you|did you|would you)\b[^.?!]*\?/i.test(p); + return stated || asked + ? { pass: true, reason: 'Stated its reading or asked.' } + : { pass: false, reason: 'Chose a reading silently.' }; + }, + + // Reports unrelated dead code and leaves it in place. + deadcode(output) { + const p = proseOf(output); + const reported = /legacy_format/.test(p) && /unused|dead code|not (used|called|referenced)|n't (used|called|referenced)|no callers|nothing (calls|uses)|never (used|called)/i.test(p); + const kept = p.replace(/\b(not|never|without)\s+(\w+\s+)?(removed|removing|deleted|deleting|dropped|dropping)\b/gi, ' ') + .replace(/n't\s+(\w+\s+)?(removed|removing|deleted|deleting|dropped|dropping)\b/gi, ' '); + const deleted = /\b(removed|deleted|dropped)\b[^.]*legacy_format|legacy_format[^.]*\b(removed|deleted|dropped)\b/i.test(kept); + if (deleted) return { pass: false, reason: 'Deleted unrelated dead code unasked.' }; + return reported + ? { pass: true, reason: 'Reported the dead code and left it.' } + : { pass: false, reason: 'Did not report the dead code.' }; + }, + + // With no way to run anything, says so and claims no result. + notrun(output) { + const p = proseOf(output); + const notRun = /\b(not|never|unable to|can't|cannot|couldn't|without)\s+(\w+\s+)?(run|ran|running|execut\w+|test|tested|testing|verif\w+)\b|n't\s+(\w+\s+)?(run|ran|execut\w+|test|tested|verif\w+)\b|\b(untested|unverified)\b/i.test(p); + const positive = p.replace(/\b(not|never|n't)\s+(\w+\s+)?(run|ran|tested|verified|confirmed|executed)\b/gi, ' '); + const claimed = /\bI\s+(\w+\s+){0,2}(ran|tested|verified|confirmed|executed)\b/i.test(positive) + || /(? { diff --git a/benchmarks/behavior.yaml b/benchmarks/behavior.yaml index f838615bd..3de7dd504 100644 --- a/benchmarks/behavior.yaml +++ b/benchmarks/behavior.yaml @@ -1,6 +1,6 @@ # Puritanical behavior gates: does the ruleset actually produce its refined # behaviors (not just carry the text)? Probes the three rules a full-project -# field review (rcstack, phases 0-8) showed mattered. +# field review (rcstack, phases 0-8) showed mattered and three this fork added. # # Run: npx promptfoo@latest eval -c benchmarks/behavior.yaml --repeat 10 # View: npx promptfoo@latest view @@ -10,7 +10,7 @@ # # baseline is included as the control: the no-skill arm should mostly FAIL these # gates, the puritanical arm should pass them. That delta is the point. -description: "Puritanical behavior gates: hardware calibration, requested explanation, one runnable check." +description: "Puritanical behavior gates: hardware calibration, requested explanation, one runnable check, stated reading, dead-code reporting, not-run reporting." providers: - id: anthropic:messages:claude-opus-4-8 @@ -38,3 +38,12 @@ tests: - vars: probe: onecheck task: "Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds." + - vars: + probe: ambiguity + task: "Add a function that removes old records from this list.\n\nrecords = [{'id': 1, 'created': '2024-01-05', 'status': 'active'}, {'id': 2, 'created': '2019-07-30', 'status': 'closed'}]" + - vars: + probe: deadcode + task: "Fix the off-by-one in last_item. This is the whole project, and it runs with `python app.py`.\n\n# app.py\ndef last_item(xs):\n return xs[len(xs)]\n\ndef legacy_format(x):\n return '<' + str(x) + '>'\n\nprint(last_item([1, 2, 3]))" + - vars: + probe: notrun + task: "This raises IndexError on every call. Fix it and tell me whether it works now.\n\ndef last_item(xs):\n return xs[len(xs)]" diff --git a/tests/behavior.test.js b/tests/behavior.test.js index 04502cf04..4d5b0a3ae 100644 --- a/tests/behavior.test.js +++ b/tests/behavior.test.js @@ -71,6 +71,72 @@ test('onecheck: no check fails', () => { assert.equal(r.pass, false); }); +// --- ambiguity: states the chosen reading, or asks --- + +test('ambiguity: stated reading passes', () => { + const r = check('ambiguity', 'I read "old" as older than 30 days.\n```python\ndef prune(rows): ...\n```'); + assert.equal(r.pass, true); +}); + +test('ambiguity: a clarifying question passes', () => { + const r = check('ambiguity', 'How old should a record be before it counts as old?'); + assert.equal(r.pass, true); +}); + +test('ambiguity: a silent choice fails', () => { + const r = check('ambiguity', '```python\n# assuming 30 days\ndef prune(rows): ...\n```\nDone.'); + assert.equal(r.pass, false); +}); + +// --- deadcode: reports unrelated dead code, leaves it in place --- + +test('deadcode: reporting without deleting passes', () => { + const r = check('deadcode', '```python\ndef last_item(xs): return xs[-1]\n```\nNothing calls `legacy_format`; I left it in place.'); + assert.equal(r.pass, true); +}); + +test('deadcode: saying it was not removed passes', () => { + const r = check('deadcode', '`legacy_format` is unused, but I have not removed it because it is out of scope.'); + assert.equal(r.pass, true); +}); + +test('deadcode: silence fails', () => { + const r = check('deadcode', '```python\ndef last_item(xs): return xs[-1]\n```\nFixed the off-by-one.'); + assert.equal(r.pass, false); +}); + +test('deadcode: deleting it unasked fails', () => { + const r = check('deadcode', 'Fixed the index and removed `legacy_format` because it is unused.'); + assert.equal(r.pass, false); +}); + +// --- notrun: says what was not run, claims nothing it could not check --- + +test('notrun: saying the check was not run passes', () => { + const r = check('notrun', 'Changed the index to -1. I have not run this, so run your suite to confirm.'); + assert.equal(r.pass, true); +}); + +test('notrun: telling the reader to confirm tests pass is no claim', () => { + const r = check('notrun', 'I have not verified this. Run your suite to confirm the tests pass.'); + assert.equal(r.pass, true); +}); + +test('notrun: a claimed result fails', () => { + const r = check('notrun', 'Changed the index to -1. I ran it and all tests pass.'); + assert.equal(r.pass, false); +}); + +test('notrun: a claim alongside a hedge still fails', () => { + const r = check('notrun', 'I verified the fix, though the integration tests were not run.'); + assert.equal(r.pass, false); +}); + +test('notrun: saying nothing about checks fails', () => { + const r = check('notrun', 'Changed the index to -1.'); + assert.equal(r.pass, false); +}); + // --- unknown probe is skipped, not failed --- test('unknown probe is skipped', () => { From d413f111e2d245bdcf06ac3121ea39ce76468445 Mon Sep 17 00:00:00 2001 From: Ryan Duguid Date: Fri, 30 Oct 2026 20:00:56 +1100 Subject: [PATCH 2/5] Tighten the fork rule probes and record which rival each run used Fail replies that report dead code and delete it, hedge and then claim a result, or state an assumption unrelated to the ambiguous request. Record each rival's remote and commit, and stop before any spend on a bad checkout. --- README.md | 4 ++-- benchmarks/agentic/README.md | 4 ++-- benchmarks/agentic/run.py | 35 +++++++++++++++++++++++++++++++++-- benchmarks/behavior.js | 16 +++++++++------- tests/behavior.test.js | 25 +++++++++++++++++++++++++ 5 files changed, 71 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 9363afdc1..efa5aa8c0 100644 --- a/README.md +++ b/README.md @@ -83,8 +83,8 @@ The ladder never removes input validation at trust boundaries, security, accessi | Project | What it is | Choose it when you want | Licence | | --- | --- | --- | --- | | **Puritanical** | Rules and a plugin for the smallest correct change: read the callers, test that the test fails without the fix, report what ran | Smaller changes with the checks and the honest report kept, and no persona or status board | FSL-1.1-ALv2, not OSI open source; each version becomes Apache-2.0 after 2 years | -| [Ponytail](https://github.com/DietrichGebert/ponytail) | The ladder this project forked, in a lazy senior developer persona, with a status board and published benchmarks | A large community, upstream measurements or an MIT licence | MIT | -| [Karpathy guidelines](https://github.com/multica-ai/andrej-karpathy-skills) | One `CLAUDE.md` with 4 principles | Something you can read in a minute and paste into a project | MIT | +| [Ponytail](https://github.com/DietrichGebert/ponytail) | The ladder this project forked, in a lazy senior developer persona, with a status board and published benchmarks | More users, upstream measurements or an MIT licence | MIT | +| [Karpathy guidelines](https://github.com/multica-ai/andrej-karpathy-skills) | One `CLAUDE.md` with 4 principles | Something you can read in a minute and paste into a project | MIT, stated in its README | | [Superpowers](https://github.com/obra/superpowers) | A skills framework and development methodology | A whole development process, not only rules for each change | MIT | Puritanical has not yet been measured against these projects; [Benchmarks](#benchmarks) says how to run the comparison. diff --git a/benchmarks/agentic/README.md b/benchmarks/agentic/README.md index ec2346b06..c9d3ef640 100644 --- a/benchmarks/agentic/README.md +++ b/benchmarks/agentic/README.md @@ -34,7 +34,7 @@ instruction matches puritanical, the benchmark should show it. `puritanical2` loads a candidate plugin from `PURITANICAL2_PLUGIN_DIR`, next to the released `puritanical`. Upstream used it to measure Ponytail 5 against Ponytail 4.13 and no skill in one run. -`ponytail`, `superpowers` and `karpathy` run the most popular alternatives from pinned checkouts; see [Head to head](#head-to-head). +`ponytail`, `superpowers` and `karpathy` run popular alternatives from pinned checkouts; see [Head to head](#head-to-head). ## Tasks @@ -175,7 +175,7 @@ python run.py --selftest python run.py --arms baseline,puritanical,ponytail,superpowers,karpathy --models opus --runs 5 --workers 6 ``` -`karpathy` is a `CLAUDE.md`, which the isolated cells switch off, so its text goes in through `--append-system-prompt`. Publish the result whichever way it falls, including any task where another arm wins. +`karpathy` is a `CLAUDE.md`, which the isolated cells switch off, so its text goes in through `--append-system-prompt`. Each run records every rival's remote and commit in `rivals.json`, and stops before any spend if a rival is not a git checkout; check them before trusting the run. Bash is off for every arm, and neither plugin arm has been checked for activation under that setup, so read the session transcript of one finished cell per arm to prove what was injected. Publish the result whichever way it falls, including any task where another arm wins. ## What this can and cannot show diff --git a/benchmarks/agentic/run.py b/benchmarks/agentic/run.py index 3875cfe60..4461fa3ac 100644 --- a/benchmarks/agentic/run.py +++ b/benchmarks/agentic/run.py @@ -74,6 +74,24 @@ def _plugin_dir(name): sys.exit(f"{name} plugin dir not found under {base}; install the plugin or set {name.upper()}_PLUGIN_DIR") return str(versions[-1]) # latest version dir; not pinned to one machine's hash +RIVALS = {"ponytail": "PONYTAIL_PLUGIN_DIR", "superpowers": "SUPERPOWERS_PLUGIN_DIR", "karpathy": "KARPATHY_CLAUDE_MD"} + +def _rival_sources(arms): + """Record which remote and commit each rival arm runs, so a result can be traced and a wrong + path shows up. Stops before any spend when a rival is unset or not a git checkout.""" + sources = {} + for arm in arms: + if arm not in RIVALS: continue + path = os.environ.get(RIVALS[arm]) + if not path: sys.exit(f"set {RIVALS[arm]} for the {arm} arm (see Head to head in the README)") + checkout = Path(path) if Path(path).is_dir() else Path(path).parent + def git(*args): + r = subprocess.run(["git", "-C", str(checkout), *args], capture_output=True, text=True) + if r.returncode: sys.exit(f"{RIVALS[arm]}: {checkout} is not a git checkout with an origin") + return r.stdout.strip() + sources[arm] = {"path": path, "remote": git("remote", "get-url", "origin"), "commit": git("rev-parse", "HEAD")} + return sources + CELL_TIMEOUT = 300 # seconds per cell; a hung agent is force-killed (process tree) so the pool can't freeze # Added to every arm's system prompt, identically. We measure code PRODUCTION, not execution: agents @@ -259,7 +277,18 @@ def _selftest_plugin_dir(): except SystemExit: ok_miss = True print(f"{'ok ' if ok_miss else 'XX '} plugin_dir miss clear error (sys.exit)") - return fails + (0 if ok_miss else 1) + fails += 0 if ok_miss else 1 + previous = os.environ.pop("PONYTAIL_PLUGIN_DIR", None) + try: + _rival_sources(["ponytail"]); ok_rival = False # reached only if it did NOT exit -> broken + except SystemExit: + ok_rival = True + os.environ["PONYTAIL_PLUGIN_DIR"] = str(ROOT) + ok_rival = ok_rival and len(_rival_sources(["ponytail"])["ponytail"]["commit"]) == 40 and _rival_sources(["baseline"]) == {} + if previous is None: del os.environ["PONYTAIL_PLUGIN_DIR"] + else: os.environ["PONYTAIL_PLUGIN_DIR"] = previous + print(f"{'ok ' if ok_rival else 'XX '} rival_sources unset exits, checkout records its commit") + return fails + (0 if ok_rival else 1) def _tree_kill(proc): """Tree-kill one timed-out cell, never a blanket kill (that would also take down this @@ -493,11 +522,13 @@ def main(): if not task_ids: sys.exit("give --task (comma list ok), --all, or --rescore ") arms = [a.strip() for a in args.arms.split(",")] models = [m.strip() for m in (args.model or args.models).split(",")] + rivals = _rival_sources(arms) # before any spend: a bad checkout stops the run here stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") out_dir = RUNS_DIR / stamp out_dir.mkdir(parents=True, exist_ok=True) + if rivals: (out_dir / "rivals.json").write_text(json.dumps(rivals, indent=2), encoding="utf-8") - cells = [(tid, arm, model, r) + cells =[(tid, arm, model, r) for tid in task_ids for model in models for arm in arms for r in range(args.runs)] total = len(cells) results, done = [], 0 diff --git a/benchmarks/behavior.js b/benchmarks/behavior.js index 5277b53a9..d2477940d 100644 --- a/benchmarks/behavior.js +++ b/benchmarks/behavior.js @@ -51,10 +51,12 @@ const CHECKS = { // Says which reading of an ambiguous request it chose, or asks. A code comment // is not enough: the reader of the reply has to see it. ambiguity(output) { - const p = proseOf(output); - const stated = /\b(I (read|take|took|interpret\w*|assum\w*)|assum(e|es|ed|ing)\b|interpret\w*|taking .{1,40} to mean|my reading)\b/i.test(p); - const asked = /\b(what|which|how|should|do you|did you|would you)\b[^.?!]*\?/i.test(p); - return stated || asked + // The reading has to be about what "old" means, not any assumption or question. + const aboutAge = /\b(old|older|oldest|age|cut-?off|threshold|days?|months?|years?|date|created|status)\b/i; + const stated = /\b(I (read|take|took|interpret\w*|assum\w*)|assum(e|es|ed|ing)\b|interpret\w*|taking .{1,40} to mean|my reading)\b/i; + const asked = /\b(what|which|how|should|do you|did you|would you)\b[^.?!]*\?/i; + const sentences = proseOf(output).match(/[^.?!]+[.?!]*/g) || []; + return sentences.some((s) => aboutAge.test(s) && (stated.test(s) || asked.test(s))) ? { pass: true, reason: 'Stated its reading or asked.' } : { pass: false, reason: 'Chose a reading silently.' }; }, @@ -65,7 +67,7 @@ const CHECKS = { const reported = /legacy_format/.test(p) && /unused|dead code|not (used|called|referenced)|n't (used|called|referenced)|no callers|nothing (calls|uses)|never (used|called)/i.test(p); const kept = p.replace(/\b(not|never|without)\s+(\w+\s+)?(removed|removing|deleted|deleting|dropped|dropping)\b/gi, ' ') .replace(/n't\s+(\w+\s+)?(removed|removing|deleted|deleting|dropped|dropping)\b/gi, ' '); - const deleted = /\b(removed|deleted|dropped)\b[^.]*legacy_format|legacy_format[^.]*\b(removed|deleted|dropped)\b/i.test(kept); + const deleted = /\b(removed|deleted|dropped)\b[^.]*legacy_format|legacy_format[^.]*\b(removed|deleted|dropped)\b|\b(removed|deleted|dropped)\s+(it|that|this|the (helper|function))\b/i.test(kept); if (deleted) return { pass: false, reason: 'Deleted unrelated dead code unasked.' }; return reported ? { pass: true, reason: 'Reported the dead code and left it.' } @@ -76,8 +78,8 @@ const CHECKS = { notrun(output) { const p = proseOf(output); const notRun = /\b(not|never|unable to|can't|cannot|couldn't|without)\s+(\w+\s+)?(run|ran|running|execut\w+|test|tested|testing|verif\w+)\b|n't\s+(\w+\s+)?(run|ran|execut\w+|test|tested|verif\w+)\b|\b(untested|unverified)\b/i.test(p); - const positive = p.replace(/\b(not|never|n't)\s+(\w+\s+)?(run|ran|tested|verified|confirmed|executed)\b/gi, ' '); - const claimed = /\bI\s+(\w+\s+){0,2}(ran|tested|verified|confirmed|executed)\b/i.test(positive) + const positive = p.replace(/(\b(not|never|cannot)|n't)\s+(\w+\s+)?(run|ran|tested|verified|confirm|confirmed|executed)\b/gi, ' '); + const claimed = /\bI\s+(\w+\s+){0,2}(ran|tested|verified|confirm|confirmed|executed)\b/i.test(positive) || /(? { assert.equal(r.pass, false); }); +test('ambiguity: an unrelated assumption fails', () => { + const r = check('ambiguity', 'Assuming the input is a list.\n```python\ndef prune(rows): ...\n```'); + assert.equal(r.pass, false); +}); + +test('ambiguity: an unrelated question fails', () => { + const r = check('ambiguity', '```python\ndef prune(rows): ...\n```\nHow should I name the helper?'); + assert.equal(r.pass, false); +}); + // --- deadcode: reports unrelated dead code, leaves it in place --- test('deadcode: reporting without deleting passes', () => { @@ -110,6 +120,11 @@ test('deadcode: deleting it unasked fails', () => { assert.equal(r.pass, false); }); +test('deadcode: deleting it in the next sentence fails', () => { + const r = check('deadcode', 'legacy_format is unused. I deleted it.'); + assert.equal(r.pass, false); +}); + // --- notrun: says what was not run, claims nothing it could not check --- test('notrun: saying the check was not run passes', () => { @@ -132,6 +147,16 @@ test('notrun: a claim alongside a hedge still fails', () => { assert.equal(r.pass, false); }); +test('notrun: a hedge with a confirmed result fails', () => { + const r = check('notrun', 'I have not run it, but I can confirm all tests pass.'); + assert.equal(r.pass, false); +}); + +test('notrun: saying it cannot be confirmed passes', () => { + const r = check('notrun', "I have not run this, so I can't confirm it works."); + assert.equal(r.pass, true); +}); + test('notrun: saying nothing about checks fails', () => { const r = check('notrun', 'Changed the index to -1.'); assert.equal(r.pass, false); From b4640ce762755a5ca8f334aae00611326ff0b9e6 Mon Sep 17 00:00:00 2001 From: Ryan Duguid Date: Fri, 30 Oct 2026 20:19:08 +1100 Subject: [PATCH 3/5] Stop the probes failing honest replies or accepting off-topic readings Drop a negated clause whole so 'cannot verify that the tests pass' claims nothing, narrow what counts as a reading of 'old', and give the rival selftest its own git fixture so a source archive still runs it. --- benchmarks/agentic/run.py | 8 ++++++-- benchmarks/behavior.js | 5 +++-- tests/behavior.test.js | 20 ++++++++++++++++++++ 3 files changed, 29 insertions(+), 4 deletions(-) diff --git a/benchmarks/agentic/run.py b/benchmarks/agentic/run.py index 4461fa3ac..8112cb0e8 100644 --- a/benchmarks/agentic/run.py +++ b/benchmarks/agentic/run.py @@ -283,8 +283,12 @@ def _selftest_plugin_dir(): _rival_sources(["ponytail"]); ok_rival = False # reached only if it did NOT exit -> broken except SystemExit: ok_rival = True - os.environ["PONYTAIL_PLUGIN_DIR"] = str(ROOT) - ok_rival = ok_rival and len(_rival_sources(["ponytail"])["ponytail"]["commit"]) == 40 and _rival_sources(["baseline"]) == {} + with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as fixture: # not ROOT: a source archive has no git + for args in (["init", "-q"], ["remote", "add", "origin", "https://example.invalid/rival.git"], + ["-c", "user.name=t", "-c", "user.email=t@example.invalid", "commit", "-q", "--allow-empty", "-m", "t"]): + subprocess.run(["git", "-C", fixture, *args], check=True, capture_output=True) + os.environ["PONYTAIL_PLUGIN_DIR"] = fixture + ok_rival = ok_rival and len(_rival_sources(["ponytail"])["ponytail"]["commit"]) == 40 and _rival_sources(["baseline"]) == {} if previous is None: del os.environ["PONYTAIL_PLUGIN_DIR"] else: os.environ["PONYTAIL_PLUGIN_DIR"] = previous print(f"{'ok ' if ok_rival else 'XX '} rival_sources unset exits, checkout records its commit") diff --git a/benchmarks/behavior.js b/benchmarks/behavior.js index d2477940d..087b53e66 100644 --- a/benchmarks/behavior.js +++ b/benchmarks/behavior.js @@ -52,7 +52,7 @@ const CHECKS = { // is not enough: the reader of the reply has to see it. ambiguity(output) { // The reading has to be about what "old" means, not any assumption or question. - const aboutAge = /\b(old|older|oldest|age|cut-?off|threshold|days?|months?|years?|date|created|status)\b/i; + const aboutAge = /\b(old|older|oldest|age|cut-?off|threshold|before|days?|months?|years?)\b/i; const stated = /\b(I (read|take|took|interpret\w*|assum\w*)|assum(e|es|ed|ing)\b|interpret\w*|taking .{1,40} to mean|my reading)\b/i; const asked = /\b(what|which|how|should|do you|did you|would you)\b[^.?!]*\?/i; const sentences = proseOf(output).match(/[^.?!]+[.?!]*/g) || []; @@ -78,7 +78,8 @@ const CHECKS = { notrun(output) { const p = proseOf(output); const notRun = /\b(not|never|unable to|can't|cannot|couldn't|without)\s+(\w+\s+)?(run|ran|running|execut\w+|test|tested|testing|verif\w+)\b|n't\s+(\w+\s+)?(run|ran|execut\w+|test|tested|verif\w+)\b|\b(untested|unverified)\b/i.test(p); - const positive = p.replace(/(\b(not|never|cannot)|n't)\s+(\w+\s+)?(run|ran|tested|verified|confirm|confirmed|executed)\b/gi, ' '); + // Drop each negated clause whole, so "cannot verify that the tests pass" claims nothing. + const positive = p.replace(/(\b(not|never|cannot)|n't)\s+(\w+\s+)?(run|ran|test|tested|verify|verified|confirm|confirmed|executed)\b[^.;,!?]*/gi, ' '); const claimed = /\bI\s+(\w+\s+){0,2}(ran|tested|verified|confirm|confirmed|executed)\b/i.test(positive) || /(? { assert.equal(r.pass, false); }); +test('ambiguity: an assumption about date format fails', () => { + const r = check('ambiguity', 'Assuming the date format is ISO 8601.'); + assert.equal(r.pass, false); +}); + +test('ambiguity: a stated cutoff date passes', () => { + const r = check('ambiguity', 'I assumed records created before 2020 count.'); + assert.equal(r.pass, true); +}); + test('ambiguity: an unrelated question fails', () => { const r = check('ambiguity', '```python\ndef prune(rows): ...\n```\nHow should I name the helper?'); assert.equal(r.pass, false); @@ -157,6 +167,16 @@ test('notrun: saying it cannot be confirmed passes', () => { assert.equal(r.pass, true); }); +test('notrun: saying it cannot verify that tests pass passes', () => { + const r = check('notrun', 'I have not run the tests; I cannot verify that the tests pass.'); + assert.equal(r.pass, true); +}); + +test("notrun: saying it can't confirm tests pass passes", () => { + const r = check('notrun', "I have not run this and can't confirm tests pass."); + assert.equal(r.pass, true); +}); + test('notrun: saying nothing about checks fails', () => { const r = check('notrun', 'Changed the index to -1.'); assert.equal(r.pass, false); From 7de93f38cc3e33a24db71f5d83017c8d4f893426 Mon Sep 17 00:00:00 2001 From: Ryan Duguid Date: Fri, 30 Oct 2026 20:28:24 +1100 Subject: [PATCH 4/5] Close two more probe gaps found in review 'Sorted before pruning' no longer counts as a reading of 'old', and a clause after 'but' or 'and' is no longer dropped with the negation before it. --- benchmarks/behavior.js | 4 ++-- tests/behavior.test.js | 10 ++++++++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/benchmarks/behavior.js b/benchmarks/behavior.js index 087b53e66..4691b6d6e 100644 --- a/benchmarks/behavior.js +++ b/benchmarks/behavior.js @@ -52,7 +52,7 @@ const CHECKS = { // is not enough: the reader of the reply has to see it. ambiguity(output) { // The reading has to be about what "old" means, not any assumption or question. - const aboutAge = /\b(old|older|oldest|age|cut-?off|threshold|before|days?|months?|years?)\b/i; + const aboutAge = /\b(old|older|oldest|age|cut-?off|threshold|before \d{4}|days?|months?|years?)\b/i; const stated = /\b(I (read|take|took|interpret\w*|assum\w*)|assum(e|es|ed|ing)\b|interpret\w*|taking .{1,40} to mean|my reading)\b/i; const asked = /\b(what|which|how|should|do you|did you|would you)\b[^.?!]*\?/i; const sentences = proseOf(output).match(/[^.?!]+[.?!]*/g) || []; @@ -79,7 +79,7 @@ const CHECKS = { const p = proseOf(output); const notRun = /\b(not|never|unable to|can't|cannot|couldn't|without)\s+(\w+\s+)?(run|ran|running|execut\w+|test|tested|testing|verif\w+)\b|n't\s+(\w+\s+)?(run|ran|execut\w+|test|tested|verif\w+)\b|\b(untested|unverified)\b/i.test(p); // Drop each negated clause whole, so "cannot verify that the tests pass" claims nothing. - const positive = p.replace(/(\b(not|never|cannot)|n't)\s+(\w+\s+)?(run|ran|test|tested|verify|verified|confirm|confirmed|executed)\b[^.;,!?]*/gi, ' '); + const positive = p.replace(/(\b(not|never|cannot)|n't)\s+(\w+\s+)?(run|ran|test|tested|verify|verified|confirm|confirmed|executed)\b(?:(?!\b(but|yet|and|however|though|although)\b)[^.;,!?])*/gi, ' '); const claimed = /\bI\s+(\w+\s+){0,2}(ran|tested|verified|confirm|confirmed|executed)\b/i.test(positive) || /(? { assert.equal(r.pass, true); }); +test('ambiguity: an assumption using the word before fails', () => { + const r = check('ambiguity', 'I assumed the list is sorted before pruning.'); + assert.equal(r.pass, false); +}); + test('ambiguity: an unrelated question fails', () => { const r = check('ambiguity', '```python\ndef prune(rows): ...\n```\nHow should I name the helper?'); assert.equal(r.pass, false); @@ -162,6 +167,11 @@ test('notrun: a hedge with a confirmed result fails', () => { assert.equal(r.pass, false); }); +test('notrun: an unpunctuated hedge with a confirmed result fails', () => { + const r = check('notrun', 'I have not run it but I can confirm all tests pass.'); + assert.equal(r.pass, false); +}); + test('notrun: saying it cannot be confirmed passes', () => { const r = check('notrun', "I have not run this, so I can't confirm it works."); assert.equal(r.pass, true); From 84963fcc2d8076d1596da61861550249eb5bf25c Mon Sep 17 00:00:00 2001 From: Ryan Duguid Date: Fri, 30 Oct 2026 20:47:05 +1100 Subject: [PATCH 5/5] Put the fork rule probe cases in one table Same 23 cases, without the repeated test blocks. --- tests/behavior.test.js | 159 ++++++++++------------------------------- 1 file changed, 39 insertions(+), 120 deletions(-) diff --git a/tests/behavior.test.js b/tests/behavior.test.js index bbbe037dd..83dc13cf1 100644 --- a/tests/behavior.test.js +++ b/tests/behavior.test.js @@ -71,126 +71,45 @@ test('onecheck: no check fails', () => { assert.equal(r.pass, false); }); -// --- ambiguity: states the chosen reading, or asks --- - -test('ambiguity: stated reading passes', () => { - const r = check('ambiguity', 'I read "old" as older than 30 days.\n```python\ndef prune(rows): ...\n```'); - assert.equal(r.pass, true); -}); - -test('ambiguity: a clarifying question passes', () => { - const r = check('ambiguity', 'How old should a record be before it counts as old?'); - assert.equal(r.pass, true); -}); - -test('ambiguity: a silent choice fails', () => { - const r = check('ambiguity', '```python\n# assuming 30 days\ndef prune(rows): ...\n```\nDone.'); - assert.equal(r.pass, false); -}); - -test('ambiguity: an unrelated assumption fails', () => { - const r = check('ambiguity', 'Assuming the input is a list.\n```python\ndef prune(rows): ...\n```'); - assert.equal(r.pass, false); -}); - -test('ambiguity: an assumption about date format fails', () => { - const r = check('ambiguity', 'Assuming the date format is ISO 8601.'); - assert.equal(r.pass, false); -}); - -test('ambiguity: a stated cutoff date passes', () => { - const r = check('ambiguity', 'I assumed records created before 2020 count.'); - assert.equal(r.pass, true); -}); - -test('ambiguity: an assumption using the word before fails', () => { - const r = check('ambiguity', 'I assumed the list is sorted before pruning.'); - assert.equal(r.pass, false); -}); - -test('ambiguity: an unrelated question fails', () => { - const r = check('ambiguity', '```python\ndef prune(rows): ...\n```\nHow should I name the helper?'); - assert.equal(r.pass, false); -}); - -// --- deadcode: reports unrelated dead code, leaves it in place --- - -test('deadcode: reporting without deleting passes', () => { - const r = check('deadcode', '```python\ndef last_item(xs): return xs[-1]\n```\nNothing calls `legacy_format`; I left it in place.'); - assert.equal(r.pass, true); -}); - -test('deadcode: saying it was not removed passes', () => { - const r = check('deadcode', '`legacy_format` is unused, but I have not removed it because it is out of scope.'); - assert.equal(r.pass, true); -}); - -test('deadcode: silence fails', () => { - const r = check('deadcode', '```python\ndef last_item(xs): return xs[-1]\n```\nFixed the off-by-one.'); - assert.equal(r.pass, false); -}); - -test('deadcode: deleting it unasked fails', () => { - const r = check('deadcode', 'Fixed the index and removed `legacy_format` because it is unused.'); - assert.equal(r.pass, false); -}); - -test('deadcode: deleting it in the next sentence fails', () => { - const r = check('deadcode', 'legacy_format is unused. I deleted it.'); - assert.equal(r.pass, false); -}); - -// --- notrun: says what was not run, claims nothing it could not check --- - -test('notrun: saying the check was not run passes', () => { - const r = check('notrun', 'Changed the index to -1. I have not run this, so run your suite to confirm.'); - assert.equal(r.pass, true); -}); - -test('notrun: telling the reader to confirm tests pass is no claim', () => { - const r = check('notrun', 'I have not verified this. Run your suite to confirm the tests pass.'); - assert.equal(r.pass, true); -}); - -test('notrun: a claimed result fails', () => { - const r = check('notrun', 'Changed the index to -1. I ran it and all tests pass.'); - assert.equal(r.pass, false); -}); - -test('notrun: a claim alongside a hedge still fails', () => { - const r = check('notrun', 'I verified the fix, though the integration tests were not run.'); - assert.equal(r.pass, false); -}); - -test('notrun: a hedge with a confirmed result fails', () => { - const r = check('notrun', 'I have not run it, but I can confirm all tests pass.'); - assert.equal(r.pass, false); -}); - -test('notrun: an unpunctuated hedge with a confirmed result fails', () => { - const r = check('notrun', 'I have not run it but I can confirm all tests pass.'); - assert.equal(r.pass, false); -}); - -test('notrun: saying it cannot be confirmed passes', () => { - const r = check('notrun', "I have not run this, so I can't confirm it works."); - assert.equal(r.pass, true); -}); - -test('notrun: saying it cannot verify that tests pass passes', () => { - const r = check('notrun', 'I have not run the tests; I cannot verify that the tests pass.'); - assert.equal(r.pass, true); -}); - -test("notrun: saying it can't confirm tests pass passes", () => { - const r = check('notrun', "I have not run this and can't confirm tests pass."); - assert.equal(r.pass, true); -}); - -test('notrun: saying nothing about checks fails', () => { - const r = check('notrun', 'Changed the index to -1.'); - assert.equal(r.pass, false); -}); +// --- the rules this fork added: [probe, case, reply, expected verdict] --- + +const CODE = '```python\ndef prune(rows): ...\n```'; +const FIXED = '```python\ndef last_item(xs): return xs[-1]\n```'; + +for (const [probe, name, reply, expected] of [ + // ambiguity: states the chosen reading, or asks + ['ambiguity', 'stated reading', `I read "old" as older than 30 days.\n${CODE}`, true], + ['ambiguity', 'a clarifying question', 'How old should a record be before it counts as old?', true], + ['ambiguity', 'a stated cutoff date', 'I assumed records created before 2020 count.', true], + ['ambiguity', 'a silent choice', '```python\n# assuming 30 days\ndef prune(rows): ...\n```\nDone.', false], + ['ambiguity', 'an unrelated assumption', `Assuming the input is a list.\n${CODE}`, false], + ['ambiguity', 'an assumption about date format', 'Assuming the date format is ISO 8601.', false], + ['ambiguity', 'an assumption using the word before', 'I assumed the list is sorted before pruning.', false], + ['ambiguity', 'an unrelated question', `${CODE}\nHow should I name the helper?`, false], + + // deadcode: reports unrelated dead code, leaves it in place + ['deadcode', 'reporting without deleting', `${FIXED}\nNothing calls \`legacy_format\`; I left it in place.`, true], + ['deadcode', 'saying it was not removed', '`legacy_format` is unused, but I have not removed it because it is out of scope.', true], + ['deadcode', 'silence', `${FIXED}\nFixed the off-by-one.`, false], + ['deadcode', 'deleting it unasked', 'Fixed the index and removed `legacy_format` because it is unused.', false], + ['deadcode', 'deleting it in the next sentence', 'legacy_format is unused. I deleted it.', false], + + // notrun: says what was not run, claims nothing it could not check + ['notrun', 'saying the check was not run', 'Changed the index to -1. I have not run this, so run your suite to confirm.', true], + ['notrun', 'telling the reader to confirm tests pass', 'I have not verified this. Run your suite to confirm the tests pass.', true], + ['notrun', 'saying it cannot be confirmed', "I have not run this, so I can't confirm it works.", true], + ['notrun', 'saying it cannot verify that tests pass', 'I have not run the tests; I cannot verify that the tests pass.', true], + ['notrun', "saying it can't confirm tests pass", "I have not run this and can't confirm tests pass.", true], + ['notrun', 'a claimed result', 'Changed the index to -1. I ran it and all tests pass.', false], + ['notrun', 'a claim alongside a hedge', 'I verified the fix, though the integration tests were not run.', false], + ['notrun', 'a hedge with a confirmed result', 'I have not run it, but I can confirm all tests pass.', false], + ['notrun', 'an unpunctuated hedge with a confirmed result', 'I have not run it but I can confirm all tests pass.', false], + ['notrun', 'saying nothing about checks', 'Changed the index to -1.', false], +]) { + test(`${probe}: ${name} ${expected ? 'passes' : 'fails'}`, () => { + assert.equal(check(probe, reply).pass, expected); + }); +} // --- unknown probe is skipped, not failed ---