[napetrov]

Summary

This PR adds a mutation-testing pilot for the measurement core: the paired statistics, the gates and the response-cache key. In this code a surviving mutant means a wrong number would ship and no test would catch it. The pilot uses mutmut 3.8.0, with an exact pin, hash-locked in its own requirements-mutation.lock.txt that only mutation.yml installs (and pip-audit.yml audits). A weekly or on-demand workflow writes the score and the survivors to the job summary. It never fails a build.

 .github/workflows/
+└── mutation.yml          # weekly + workflow_dispatch, contents: read, non-blocking
 dokimi/
 ├── pyproject.toml        # + [tool.mutmut]: only_mutate = the 7 pilot modules,
 │                         #   test selection = their 14 test files
 ├── requirements-mutation.txt       # mutmut==3.8.0
├── requirements-mutation.lock.txt  # hash-locked closure, constrained to requirements.lock.txt
 ├── CONTRIBUTING.md       # + "Mutation pilot" paragraph
 └── tests/                # + kills for the survivors that were real gaps

Mutant counts below are from the first local run; later review rounds added tests that kill further survivors.

Module	Killed before	Killed after
eval.stats	77.2%	87.7%
eval.binary_stats	92.0%	95.8%
gate.regression	65.6%	100.0%
gate.score_gate	88.0%	96.0%
gate.critical_bands	72.4%	95.5%
gate.trust_gate	64.1%	97.5%
caching	75.5%	87.1%
all 2087 mutants	75.3%	92.0%

The new tests pin behaviour that no existing test checked:

• Regression gate: every default threshold at its exact value. Spec thresholds replace the defaults.
• Critical bands: each condition reads its own summary key, and "below" is strict.
• Trust gate:
  • a hand-computed value, threshold and severity for every check
  • each threshold is inclusive
  • the rendered block, whole
• Paired test: the Wilcoxon veto on a significant t-test, and MIN_PAIRS_FOR_TEST.
• Bootstrap CI: the two-sided alpha and the n_resamples=1 edge.
• Arm significance: confidence passes through, and holm and bh give different results.
• McNemar (binary_stats): pseudo_replicated.
• Cache key: it separates role, sampling parameters and image blocks, and a pinned hash keeps stored entries findable.

What is still left is mostly equivalent mutants:

• scipy lowercases "ZSPLIT".
• zip(strict=) after a length check.
• caps that never bind, such as min(2.0, p) with p ≤ 1.
• <= vs < that differs only at p == 0.05 exactly.
• message text and display-rounding digits.

gate/novelty_gate.py is out of scope. mutmut 3 copies the whole 440-line evaluate_novelty once per mutant, and generation alone grew past 26 MB. A diff-only mode for PRs is a follow-up.

Evidence

• Before: on main, the mutant dokimi.gate.regression.x__classify_regression__mutmut_7 survives:

  -    if drop >= critical_threshold:
  +    if drop > critical_threshold:

  tests/test_regression.py from main against that mutant gives 6 passed. A drop of exactly the 0.08 default is reported as WARN instead of CRITICAL, and nothing notices.

  After: the same mutant against this branch's tests:

  FAILED test_default_thresholds_apply_at_their_exact_values[score--0.08-CRITICAL]
  FAILED test_default_thresholds_apply_at_their_exact_values[coverage--0.12-CRITICAL]
  

• Whole pilot: the local mutmut run --max-children 16 takes about 1 minute after the first mutant generation. Before: 1566 killed, 514 survived. After: 1913 killed, 167 survived.

• Workflow summary: the step-summary script was run on the local results and produced the table above.

• Checks:

  • pytest -n 16 --outcomes: 6986 passed, 2 skipped
  • ruff check .: clean
  • mypy: no issues in 222 files
  • zizmor --offline --min-severity medium .github/workflows: no findings
  • scripts/audit_test_quality.py: no new flags
  • tests/test_dependency_locks.py: passes
  • scripts/check_docs_review_triggers.py: no page to review

Merge Danger

Door: two-way. The workflow is new and never runs on PRs or pushes. Removing it, together with the [tool.mutmut] table, undoes the change.

Blast Radius: tests

• No package code changes.
• mutmut and its dependencies (libcst, textual, setproctitle and a few more) live only in requirements-mutation.lock.txt; the tools lock other jobs install is unchanged.
• The step summary flags a run cut short (mutants not checked, or a module with nothing scored).
• test_the_key_of_a_known_request_does_not_change pins the response-cache key format. A future change to the key has to update that hash on purpose, because a new key format orphans every stored entry.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.69%. Comparing base (5064dc2) to head (55b32d8).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #178      +/-   ##
==========================================
+ Coverage   91.64%   91.69%   +0.05%     
==========================================
  Files         222      223       +1     
  Lines       27409    27505      +96     
  Branches     4508     4525      +17     
==========================================
+ Hits        25118    25222     +104     
+ Misses       1620     1617       -3     
+ Partials      671      666       -5     

see 9 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

The tests are real and they kill what the PR says they kill. I'd fix the gaps below before merge, because the PR's point is that a surviving mutant is a wrong number that ships, and several mutants of the same kind survive the new tests.

What I ran, on a Linux review host at 432f037:

• make ci-local: 27 passed, 1 failed, 7 skipped; 7,175 tests passed. The one failure, test_repo_environment.py::test_unbalanced_quote_in_hook_completes_quickly (1.7 s against a 1 s limit), passes on its own (513 passed in that file). Python 3.13, so the 3.11 leg is not covered here.
• Hand-made mutants of the seven pilot modules (about 310), run against the new test files and against the 14-file selection in [tool.mutmut]. The drop >= critical_threshold -> > mutant from the description is killed. The survivors are in the inline comments.
• The step-summary script from mutation.yml, run on made-up mutmut results lines (function and class-method names, no tests, timeout, suspicious): the module grouping and the table are right. zizmor --offline on the workflow: no findings.
• I did not run mutmut run itself, so the 75.3% -> 92.0% figures are not checked.

Findings that cannot be anchored inline:

1. The mutation-only tool now lands in the shared tools lock. requirements-ci.lock.txt gains 352 lines for mutmut: libcst, textual, linkify-it-py, mdit-py-plugins, pyyaml-ft, setproctitle and others. ci.yml, release.yml, pip-audit.yml and the eval image install that whole lock, so each now installs and audits packages only the weekly workflow uses. An advisory on textual or libcst would fail the daily pip-audit. Suggestion: a separate requirements-mutation.lock.txt that only mutation.yml installs. If the shared lock is intended, say so in the PR description.

Checked, no issues

• Gate modules: all 22 mutants of critical_bands.py and all 11 of score_gate.py I tried are killed (per-condition keys, strict <, defaults, guards).
• Cache key: the pinned-hash test hashes an explicit JSON literal, not a stored digest, so it is stable across Python versions and platforms. Dropping the role, model, extra or sort_keys, or changing ensure_ascii, fails it.
• Config: all 14 files in the test selection exist, the only_mutate entries end in .py as mutmut 3 requires, the lock entry for mutmut==3.8.0 is hashed, and tests/test_dependency_locks.py passes.
• Tests: all four gate test files pass in other orders and with different pytest-randomly seeds; they use no I/O.

[antonwolfy]

I pushed a commit per nit, up to d57ed0e, which fix the nit from my review:

• d57ed0e: dokimi/CONTRIBUTING.md:147: now names the score, regression, critical-band and trust gates and says novelty_gate.py is not mutated.

make ci-local at d57ed0e: 28 passed, 0 failed, 7 skipped.

Not approving yet. Still open: the seven risks from my review.

[napetrov]

Review-body item 1 (mutmut in the shared tools lock): fixed in c5c6715. requirements-ci.txt and requirements-ci.lock.txt are back to their pre-PR content. mutmut==3.8.0 now sits in requirements-mutation.txt, locked into requirements-mutation.lock.txt (uv pip compile with the same options as the tools lock: --constraint=requirements.lock.txt, --universal, --python-version=3.11, --exclude-newer=P7D, --generate-hashes). Only mutation.yml installs it (requirements.lock.txt plus the new lock). tests/test_dependency_locks.py now covers the new lock (hashes, header, cooldown, stacked pins agree with the main lock, mutmut not in the tools lock); Renovate's pip-compile pattern and .licenserc.yaml include it. pip-audit does not audit the new lock; that keeps its closure out of the daily audit, as you suggested. Say so if you want it audited.

[napetrov]

Checked the new lock: regenerating requirements-mutation.lock.txt with the command in its header (uv 0.11.9, same options as the tools lock: --constraint=requirements.lock.txt, --universal, --python-version=3.11, --exclude-newer=P7D, --generate-hashes) gives the committed file byte for byte; the header, hashes and stacked pins pass tests/test_dependency_locks.py, and mutation.yml installs only requirements.lock.txt and this lock. One gap found and fixed in c6f1bfe: moving mutmut out of the tools lock had dropped its closure from pip-audit, so pip-audit.yml now audits requirements-mutation.lock.txt too (test_pip_audit_audits_every_lock fails without it), and CONTRIBUTING no longer says it is unaudited.

[antonwolfy]

Review at 91d9ae6. The six earlier threads are fixed: the default thresholds, the score flags, the both-arms filter, the coverage floor, MIN_PAIRS_FOR_TEST and the step timeout all hold. The new tests still leave mutants alive in the same modules. The six inline comments and the first numbered item below are worth fixing before merge.

What I ran, on a Linux review host at 91d9ae6:

• make ci-local: 28 passed, 1 failed, 7 skipped; 7,315 tests; changed-line coverage 100%. The failure is pip-audit: gitpython 3.2.0 (CVE-2026-114145, fix 3.2.1), pinned in requirements.lock.txt on the PR's base as well, so it is not caused by this PR.
• About 350 hand-made mutants of the seven pilot modules, run against the new test files and the 14-file selection from [tool.mutmut]. The survivors I re-ran are in the comments and below. I did not run mutmut run itself, so the 75.3% -> 92.0% figures are not checked.
• zizmor --offline --min-severity medium on the workflows: no findings. tests/test_dependency_locks.py: 28 passed. The summary script from mutation.yml on made-up mutmut results lines: the table is right.

Findings that cannot be anchored inline:

1. Length guards and the no-SciPy seed are pinned in one direction only. Each of these passes the 14-file selection (881 passed):
   • paired_test: len(treatment) != len(control) -> len(treatment) > len(control) (stats.py:479). A shorter treatment vector raises from zip(strict=True) instead of returning None.
   • mean_ci: bootstrap_ci(vals, confidence=confidence, seed=seed) -> without seed=seed (stats.py:154). Without SciPy the seed is ignored.
     Suggestion: assert stats.paired_test([1, 2, 3], [1, 2, 3, 4]) is None and the shorter-treatment mirror; with HAS_SCIPY monkeypatched to False, assert mean_ci(vals, seed=7) differs from the default seed on a sample whose bounds depend on it.
2. Nit: dokimi/SECURITY.md (outside this diff) lists the locks pip-audit runs against and does not name requirements-mutation.lock.txt. Suggestion: add it to that sentence.

Checked, no issues

• Fixes from my review: the four default-lowering mutants, has_critical/has_warnings dropping the score and == "WARN" -> != "OK", the dropped context half of the both-arms filter, min_retrieval_coverage 0.80 -> 0.75, MIN_PAIRS_FOR_TEST 6 -> 5, and the alpha, lo_idx rounding and seed + 1 mutants are all killed now.
• Locks: requirements-ci.lock.txt is back to the base content, mutmut sits only in requirements-mutation.lock.txt, installed by mutation.yml alone, and pip-audit.yml audits it.
• Workflow: Run mutmut has timeout-minutes: 75 under the job's 90, with continue-on-error and an if: always() summary step. Actions are SHA-pinned, permissions: contents: read.
• Determinism: the four gate test files and test_stats.py, test_binary_stats.py, test_caching.py pass under three different pytest-randomly seeds.
• Cache key: the pinned hash is of a literal payload with sorted keys, so it is platform-stable.

[antonwolfy]

The upper tail index is still unpinned: three hi_idx/lo_idx mutants pass the suite. This case (confidence 0.8, 100 resamples) is one where the variants below land on the same order statistic as the original.

Mutation check, each passes test_stats.py, test_binary_stats.py and the 14-file selection from [tool.mutmut] (881 passed):

• hi_idx: int(round((1.0 - alpha) * (n_resamples - 1))) -> int((1.0 - alpha) * (n_resamples - 1)) (stats.py:127)
• hi_idx: (n_resamples - 1) -> n_resamples (stats.py:127)
• lo_idx: (n_resamples - 1) -> n_resamples (stats.py:126)

Each moves a bound by one order statistic, so the interval stops being symmetric.

Suggestion: add cases where those variants differ, with literals, on the same sample [0, 1, 2, 3, 10, 4.5, 7.5] and seed=7: confidence=0.95, n_resamples=100 -> (1.2143, 6.5714) (all three mutants fail), and confidence=0.5, n_resamples=50 -> (3.0, 4.9286).

[napetrov]

Fixed in 705e015: test_bootstrap_interval_tails_are_symmetric_order_statistics pins your literals ((1.2143, 6.5714) at 0.95/100 and (3.0, 4.9286) at 0.5/50). I ran the three mutants (no round on hi_idx, n_resamples for n_resamples - 1 on hi_idx and on lo_idx): each fails it.

[antonwolfy]

The seed assertion is vacuous: it holds whether or not mcnemar_test forwards the seed. These deltas (6 gained, 0 lost) give (0.0, 0.6) for every seed, so seeded == boot is true for any value.

Mutation check: bootstrap_ci(deltas, confidence=confidence, seed=seed) -> bootstrap_ci(deltas, confidence=confidence) (binary_stats.py:215) passes the full file and the 14-file selection (881 passed). I checked seeds 0 to 11 on this data: all return 0.0 0.6.

Suggestion: use data whose interval depends on the seed, for example 6 gained, 6 lost and 3 both-pass pairs (seed 0 gives (-0.4, 0.4667), seed 7 gives (-0.4667, 0.4667)), and assert that the seed-7 result differs from the default one as well as equals bootstrap_ci(..., seed=7).

[napetrov]

Fixed in 705e015: test_the_seed_reaches_an_interval_that_depends_on_it uses 6 gained, 6 lost, 3 both-pass pairs, asserts the default (-0.4, 0.4667) and seed 7 (-0.4667, 0.4667). Dropping seed=seed from the bootstrap_ci call in mcnemar_test now fails it.

[antonwolfy]

has_warnings and has_critical are pinned only through the score regression, so a dropped or truncated metric list passes. test_a_score_drop_alone_sets_the_summary_flags has no metric regression, and test_every_tracked_metric_is_classified has exactly one metric, so it is always metric_regressions[0] and only checks has_critical.

Mutation check, all pass the file and the 14-file selection:

• has_warnings: [self.score_regression] + self.metric_regressions -> [self.score_regression] (regression.py:38)
• has_critical: + self.metric_regressions -> + self.metric_regressions[:1] (regression.py:46)

A coverage or readability drop at WARN level would stop raising has_warnings, and a CRITICAL on the second to fourth tracked metric would stop raising has_critical.

Suggestion: add detect_regressions({"score": 0.0, "coverage": -0.06}, {}) expecting has_warnings and not has_critical, and detect_regressions({"score": 0.0, "example_pass_rate": -0.5, "coverage": 0.0, "freshness_lite": 0.0, "readability": 0.0}, {}) expecting has_critical.

[napetrov]

Fixed in 705e015: test_a_metric_regression_alone_sets_the_summary_flags (coverage -0.06: has_warnings, not has_critical) and test_a_critical_on_a_later_tracked_metric_sets_has_critical. Both mutants (has_warnings without metric_regressions, has_critical with metric_regressions[:1]) fail them.

[antonwolfy]

The agreement check is only tested with ten rows that all carry agreement_score, so a single-row panel run can lose it. Mutation check: if agreement_scores: -> if len(agreement_scores) > 1: (trust_gate.py:186) passes the file and the 14-file selection (881 passed). A run where only one question has a panel score would then skip inter_rater_agreement instead of judging it.

Suggestion: add ten _row(70, 50) rows plus one _row(70, 50, agree=0.1) and assert inter_rater_agreement is present with passed false and value == 0.1.

[napetrov]

Fixed in 705e015: TestAtTheThreshold.test_one_agreement_score_among_many_rows_is_still_judged (ten rows without a score plus one with agree=0.1: inter_rater_agreement present, passed False, value 0.1). The len(agreement_scores) > 1 mutant fails it.

[antonwolfy]

The rendered-block test cannot tell the Warnings section from the Blocking section: the only block with warnings also has failures. Mutation check: if verdict.warnings: -> if verdict.failures: (trust_gate.py:288) passes the file and the 14-file selection (881 passed). A warn-only run would render without its **Warnings:** list. The mirror mutant at line 284 is killed.

Suggestion: render a warn-only verdict (ten rows with a high score CV, no failing check) and assert the output contains **Warnings:** and not **Blocking issues:**.

[napetrov]

Fixed in 705e015: test_a_warn_only_verdict_renders_warnings_and_no_blocking_section renders a verdict with warnings and no failures and asserts the Warnings list is there and Blocking issues is not. The verdict.failures mutant at line 288 fails it.

[antonwolfy]

I pushed a commit per nit, up to b5785d6, which fix the nit from my review:

• b5785d6: dokimi/SECURITY.md:47: the list of locks pip-audit runs against now names requirements-mutation.lock.txt.

make ci-local at b5785d6: 28 passed, 1 failed (pip-audit, same as at the reviewed head), 7 skipped.

Not approving yet. Still open: the seven risks from my review.

[xaleryb]

• Ran: the full suite, mutmut 3.8.0 as mutation.yml installs it, the summary script verbatim, and mutations of the pip-audit step and of the bootstrap defaults.
• Summary on a partial run: a step-timeout SIGINT leaves 1228 mutants not checked, yet eval.stats prints 94.7% with no incomplete marker. A failing baseline prints 0|0|n/a rows instead of "No results". Expected: a banner when any mutant is not checked or killed+survived is 0.
• Test selection: is None -> is not None in first_replayed passes the 14 files but fails test_judge_response_cache.py, and arm_significance_31 is killed only by test_arm_runner.py. Expected: survivors are mutants no test kills. Either drop the selection or widen it.
• The PR body still places mutmut in requirements-ci.lock.txt and quotes pre-merge totals.
• Pushed:
  • 60a99b6: the pip-audit lock test passed with locks dropped from the audited list; it now parses the step.
  • 55b32d8: pins bootstrap_ci's default seed and n_resamples, which grounding uses.
• Left: the summary banner, the selection, the PR body, and a rebase for pip-audit.

[antonwolfy]

Review at 55b32d8. The only change since b5785d6 is two commits by @xaleryb: the pip-audit lock test now parses the step, and a new test pins bootstrap_ci's defaults. Both are right. Five of my six threads still stand, and one item from my earlier review body is still open.

What I ran, on a Linux review host at 55b32d8:

• make ci-local: 28 passed, 1 failed, 7 skipped; 7,316 tests passed. The failure is pip-audit: gitpython 3.2.0, pinned in requirements.lock.txt on the PR's base as well, so this PR does not cause it.
• The mutants from my threads and from my earlier body, re-run against test_stats.py, test_binary_stats.py, test_regression.py, test_trust_gate.py, test_critical_bands.py, test_score_gate.py and test_caching.py (267 passed on each): 10 of 10 survive, listed below and in the threads.
• Ten mutants of the cache key (role, sampling parameters, image blocks, sort_keys, ensure_ascii, default=str, block joining): all killed. The two bootstrap_ci default mutants (seed 0 -> 1, n_resamples 10,000 -> 5,000): both killed by the new test.
• The five test files above under two pytest-randomly seeds and with sockets disabled: 221 passed each time. I did not run mutmut itself.

Findings that cannot be anchored inline:

1. Length guard and the no-SciPy seed are still pinned in one direction only. This is the first item of my earlier review body, not yet addressed. Both survive the seven test files:
   • paired_test: len(treatment) != len(control) -> len(treatment) > len(control) (stats.py:479). A control longer than the treatment is no longer rejected as degenerate.
   • mean_ci: bootstrap_ci(vals, confidence=confidence, seed=seed) -> without seed=seed (stats.py:154). The seed is ignored on the bootstrap fallback.
     Suggestion: assert stats.paired_test([1, 2, 3], [1, 2, 3, 4]) is None and the shorter-treatment mirror; with HAS_SCIPY monkeypatched to False, assert mean_ci(vals, seed=7) differs from the default seed on a sample whose bounds depend on it.

I agree with @xaleryb that the weekly summary does not mark a partial run. New evidence: made-up mutmut results lines with 1 killed, 1 survived and 3 not checked print | dokimi.eval.stats | 1 | 1 | 3 | 50.0% |, and a module with nothing checked prints n/a; nothing says the run was cut short.

Still open from my earlier review:

• dokimi/tests/test_stats.py:598: the hi_idx/lo_idx mutants (int(...) without round, n_resamples for n_resamples - 1) all still pass; the new defaults test does not reach them.
• dokimi/tests/test_binary_stats.py:211: removing seed=seed from mcnemar_test's bootstrap_ci call still passes.
• dokimi/tests/test_regression.py:197: has_warnings without metric_regressions, and has_critical with only the first metric, both still pass.
• dokimi/tests/test_trust_gate.py:318: if agreement_scores: -> if len(agreement_scores) > 1: still passes.
• dokimi/tests/test_trust_gate.py:404: if verdict.warnings: -> if verdict.failures: still passes.

Checked, no issues

• pip-audit test: test_pip_audit_audits_every_lock now reads the pip-audit step's run text. A lock name that survives only in a YAML comment no longer satisfies it: removing it from the sed command fails the test.
• Cache key: the new tests kill every key mutant I tried; the pinned hash is of a literal payload.
• Determinism: the pinned bootstrap_ci interval uses only random.Random, so it does not depend on SciPy or NumPy.
• Locks and workflow: mutmut sits only in requirements-mutation.lock.txt, installed by mutation.yml alone and audited by pip-audit.yml; Run mutmut has a 75-minute step timeout.

[napetrov]

Review-body items, by sha (fork/ci/mutation-pilot at d27b2e3):

1. Length guard and no-SciPy seed: 705e015. test_paired_test_rejects_a_length_mismatch_in_both_directions (the len(treatment) > len(control) mutant fails it) and test_mean_ci_forwards_the_seed_to_the_bootstrap_fallback (monkeypatches HAS_SCIPY and bootstrap_ci and asserts seed=7 arrives; the dropped-seed mutant fails it). I checked that a seed-differs literal does not exist for mean_ci at 10,000 resamples, hence the forwarding check.
2. The five inline mutants: replied on each thread, same sha.
3. Summary banner on a partial run (xaleryb): d27b2e3. The step summary prints an 'Incomplete run' note when any mutant is not checked or a module has nothing scored. Run against made-up results lines: banner shown for not-checked and for a 0/0 module, absent for a clean run.
4. PR body: edited. mutmut is described in requirements-mutation.lock.txt, not the tools lock; the counts are labelled as first-run numbers.
5. Merge: intel/main merged in 60eb92b (one conflict in CONTRIBUTING.md, both sides kept).
6. Not done, needs a decision: the test selection in [tool.mutmut] (drop it or widen it to the tests that kill first_replayed and arm_significance mutants). Left to the author.