Repository navigation
Say so and make no fix when a reported bug no longer reproduces - #18
Merged
Merged
Conversation
Add a fix-or-abstain sentence to the bug paragraph of the core skill and regenerate every rule copy with `puritanical-tools sync`. Pin the sentence as a rule invariant so `sync --check` fails if it is dropped. Gate it in the agentic benchmark with a paired fixture that sends one bug report twice: abstain-fixed (the bug is already fixed; any change to the workspace fails) and abstain-wrongfix (an earlier fix misses one reported case; abstaining fails). The selftest proves both scorers. Based on the ETH SRI fixed-code study: https://eth-sri.github.io/blog/fixedcode
Reword the rule to "If the reported failure no longer reproduces and the contract it concerns holds, say so and make no fix", so it requires the contract to hold and no longer overrides a requested regression test. Pin "say so and make no fix" and regenerate the rule copies. abstain-wrongfix now compares page_count with ceiling division over totals 0 to 25 at page sizes 10 and 3, so a patch for the reported numbers alone fails. abstain-fixed ignores only the root pager.py, the files run.py writes (_claude.json, _claude.stderr.txt, .git/) and __pycache__, so an added _test_pager.py or tests/pager.py counts as a change. Note that the reply's acknowledgement is unscored.
ryanduguid
enabled auto-merge (squash)
October 29, 2026 23:57
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a reported bug no longer reproduces, the skill now says so and makes no fix, and a paired benchmark checks both sides of that call: the same bug report against already-fixed code and against a wrong earlier fix.
The wording asks for the contract to hold, not only the reported examples, because the ETH SRI study behind this (one lab, older models) found that an abstain prompt also made agents skip fixes that were still needed. An arm that always patches fails
abstain-fixed; one that abstains whenever it sees a fix failsabstain-wrongfix. The 12 mirrored rule files are regenerated bypuritanical-tools sync.Evidence
sync --checkhas no "say so and make no fix" invariant and the benchmark has no abstain tasks.After: removing the new sentence makes
sync --checkfail withmissing rule invariants: ["say so and make no fix"].run.py --selftestreports all instruments valid, with both new tasks passing their correct reference and catching the wrong one. Extra scorer cases: a selective patch for the reported 5-item case, and added_test_pager.py,tests/pager.pyor.hidden.py, all fail; an untouched seed, CRLF endings and the harness's own files pass.cargo fmt --check,clippy -D warnings,cargo test(16 passed),check-versionsandnpm test(133 pass, 4 platform skips as on main, plus 30 inpi-extension) pass.Merge danger
Door: two-way. Blast radius: skill behaviour.
Agents using the skill may now report an already-fixed bug instead of rewriting working code. The benchmark task count is 47 (the README's 39 was already out of date at 45).
Unverified