docs(process): require correctness tests to be demonstrated to fail

[jamesburton]

Closes #417.

What

Adds a Prove the Test Can Fail section to CLAUDE.md, directly after the cross-backend rules.

CLAUDE.md:252 already requires regression tests that discriminate between broken and fixed forms. It gives no way to prove the requirement was met — so in practice discrimination is asserted, and the assertion has been wrong three times:

Case	How it went wrong
#410 pre-tokenizer parity	Passed CI, author testing and an advisor review. Forcibly swapping the pipeline on real GGUFs produced byte-identical token ids — the test could not fail when qwen35 was misrouted to qwen2. Neither Qwen vocab contains a single ASCII multi-digit token, so \p{N} vs \p{N}{1,3} was unobservable. The assertion was fine; the fixture data was the weak link.
#411 Jinja loop vars	A reviewer applied 7 mutants; 6 died, 1 survived — behaviour correct in code but guarded by nothing. Became a one-line test.
#372 NemotronH RoPE	Survived months behind "logits are finite and non-constant".

The rule

Demonstrate the failure, and record it in the PR body. Two methods:

• Mutation — deliberate wrong implementation, re-run, confirm failure. Best for logic.
• Substitution — swap the component for a plausible wrong alternative on real data, confirm output differs. Best for data-dependent paths.

Substitution is called out as not optional for data-dependent tests, because #410 is precisely the case a mutation-only rule would have missed: the code was right, the fixture couldn't tell.

Scope and deliberate non-goals

Correctness gates only — tests whose name or issue claims a guarantee. Smoke tests are explicitly unaffected.

Not proposed, on purpose:

• No coverage target. This is assertion sensitivity, not line coverage.
• No mutation framework or CI gate. Manual targeted mutants are what worked; automated whole-suite mutation would be slow, noisy, and would not have caught fix(tokenizers): add qwen2 + qwen35 pre-tokenizer pipelines; complete the #373/#387 pre-tokenizer work #410 at all, since that defect lived in fixture data rather than code.

Cross-references #344's existing per-type negative-control standard so the two do not drift apart.

Verification

Documentation-only; no code paths touched. The rule's own evidence is the three cases above, each with a linked issue or PR.

[Copilot]

Pull request overview

Adds a new process rule to CLAUDE.md requiring authors of correctness-invariant tests to demonstrate that the test can fail (via mutation or substitution) and to record that evidence in the PR body. This strengthens the repo’s existing “tests must discriminate” mandate by adding an explicit verification method, helping prevent non-discriminating tests from being merged.

Changes:

• Adds a “Prove the Test Can Fail” section with two accepted demonstration methods (mutation/substitution).
• Explicitly requires substitution for data-dependent tests to guard against weak fixtures.
• Calls out common weak assertions to be skeptical of for correctness gates.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.