Rust unit mutation is vacuous again — 0 mutants tested on a 146-line db.rs diff (same class as #659/#672)

> Rewritten. The first version of this issue cited the wrong CI run and presented this as a new discovery. Both corrected below; the conclusion (the gate is vacuous) survives, the diagnosis does not.

Problem

rust / Unit mutation — changed lines (rust) reports success without testing anything, on a PR that changed 146 lines of packages/rust/src/db.rs (#966).

This is the third occurrence of the same class: #659 (local false green) and #672 (CI false green) were fixed upstream in testing-conventions #467/#468 (0.0.88) and #469/#471 (0.0.89), closed 2026-07-27 with a green proof of 3 mutant(s) tested. Bespoke PR #685 was closed as obsolete at that point. It is vacuous again.

Evidence

CI. Job rust / Unit mutation — changed lines (rust) on head commit 63b83d3 — whose diff against base ddb7208 contains the whole db.rs change — logs:

unit mutation: no surviving mutants — every mutation was caught (0 mutant(s) tested)

Duration 60s, versus ~7 minutes for a real 22-mutant run. (My first filing cited run 32397912886, which is the tests-only RED commit — 0 is correct there. The head-commit job above is the real evidence.)

Reproduced locally, deterministically:

$ npx -y testing-conventions unit mutation --language rust \
    --base ddb7208765d75c67008696b98242d713f86aefd9 \
    --config testing-conventions.toml packages/rust
unit mutation: no surviving mutants — every mutation was caught (0 mutant(s) tested)

The diff is fine, and cargo-mutants is fine. mutation.rs::write_base_diff builds git diff --relative {base}...HEAD -- packages/rust from the cargo workspace root (confirmed /home/user/dirsql/Cargo.toml, so prefix = packages/rust). Running that exact command yields a correct 287-line diff whose paths are workspace-relative and which contains packages/rust/src/db.rs. Handing that same diff straight to the engine:

$ cargo mutants --features cli --in-diff /tmp/bigbase.diff --list
... 22 mutants, e.g.
packages/rust/src/db.rs:94:5:  replace is_path_char -&gt; bool with true
packages/rust/src/db.rs:842:9: replace Db::unquoted_path_hint_for -&gt; Option with None

and a real run kills them: 22 mutants tested in 7m: 19 caught, 3 unviable (scoped -- --lib; see #977 for why the full-suite baseline can't run here).

So the mutants exist, the diff is correct, the engine finds them from that diff, and the unit tests kill them — but the gate reports 0.

Not a version regression. I initially guessed 0.0.91 had regressed 0.0.89's fix. That is wrong — pinned runs of 0.0.89, 0.0.90 and 0.0.91 behave identically on the same inputs (132s / 125s / 132s, all reaching the engine's baseline stage, which cargo-mutants only does after --in-diff filtering).

It is base-sensitive. Same HEAD (2e0a272), same tree, same tool, same config — only --base differs:

`--base`	result
`3ec31c6` (immediate first parent)	engine finds the 22 mutants, reaches baseline
`ddb7208` (further back)	`0 mutant(s) tested` in ~80s, no baseline attempted

Both diffs contain the identical db.rs change, and both yield 22 mutants when passed to cargo mutants --in-diff by hand. The only content difference is one extra unrelated file (packages/rust/MIGRATIONS.md). So the divergence is inside the tool's own path between generating the diff and reporting the count — not in the diff and not in the engine. Narrowing further wants tracing inside measure_rust / run_cargo_mutants / rebase_report_paths, which is upstream's to do.

The separable defect that makes this silent

mutation.rs already distinguishes two outcomes — Measurement::EngineNotRun ("unit mutation: no mutatable changed lines — engine not run") from Measurement::Tested { count, survivors }. Yet Tested { count: 0 } prints the all-caught success line. That state is a contradiction: nothing was tested, so nothing was caught.

Upstream already states the right rule for the sibling case, in its own changelog for #469:

> A baseline that still cannot build remains a hard failure naming the engine output, never a 0 mutant(s) tested pass.

The same should hold here. If Tested { count: 0 } were fatal (or routed to EngineNotRun), all three occurrences of this bug would have been loud on first contact instead of silently green for weeks.

Enforcement — the actual ask

This has now silently broken three times, so a fix without a test that fails when it breaks again is not a fix.

Upstream (testing-conventions): packages/rust/tests/mutation_base_rust.rs builds its primary fixture with [workspace] in the crate's own Cargo.toml — a standalone workspace root, the one layout where the prefix logic is trivially None. The vacuous-pass assertion needs to be on the workspace-member layout, which is the shape #467 existed to fix and the shape dirsql actually has. Assert a nonzero tested count there, not merely the absence of survivors.

Here (dirsql): #672 rejected a repo-side guard as fragile because "a genuinely 0-mutant diff would false-positive". That objection only applies to a guard keyed on the PR's own diff. A canary keyed on a fixed fixture has no such failure mode:

a small fixture crate laid out as a workspace member, plus a committed synthetic diff with known mutants;
a dirsql-checks mutation-canary subcommand (per AGENTS.md's "CI logic lives in scripts") that runs the gate's exact invocation against it and fails unless the reported count is > 0;
wired into dirsql-rust-ci.yml on every PR — which matters because @v0 + npx resolves latest, so this dependency can change under us on any run with no commit to blame.

That guard is independent of whatever the current root cause turns out to be, and would have caught all three occurrences.

Tasks

Report upstream with the base-sensitivity repro above; get the root cause fixed — testing-conventions#481
Upstream: make Tested { count: 0 } impossible or fatal — written at upstream HEAD, not yet published (see the comment below)
Upstream: assert a nonzero tested count on a workspace-member fixture
Here: add the fixture-based mutation-canary and wire it into dirsql-rust-ci.yml — split out to Add a mutation canary so a vacuous unit mutation gate fails loudly instead of passing green #980
Confirm whether python-sdk / typescript-sdk mutation are affected — no, structurally impossible (see the comment below)
Re-run mutation over Rust source that merged while the gate was blind (at least Hint at quoting when an unquoted path is used as a table name #966), and fix any survivors — Audit the window the vacuous unit mutation gate left unmutated, and kill what escaped #1013; the four that remain are Four mutants in effectful core survive every test tier (found by the #975 blind-window audit) #1012
Nit, same neighborhood: .gitignore has mutants.out/ but not mutants.out.old/ — already fixed in eb9443f

Note for whoever picks this up

cargo mutants cannot establish a baseline in the hosted sandbox — the uid-0 vtab permission test (#977) fails, and cargo-mutants aborts with cargo test failed in an unmutated tree. Scope with -- --lib, which is also the semantically right scope: the gate is about unit tests killing mutants. (#977 has since been fixed by #992, so the unscoped baseline is green under uid 0 again — but see #1013 for the -C --lib argument clash that produces an identical-looking error for an entirely different reason.)

Found while working #951 / #966.

Split the two follow-ups out of this issue's task list so they can land independently:

Upstream root cause → thekevinscott/testing-conventions#481, carrying the base-sensitivity repro, the Tested { count: 0 } hole in ci: probe the packaging gate for rust in conventions.yml (#413) #459's vacuous-pass fix, and the standalone-vs-workspace-member fixture gap in mutation_base_rust.rs.
In-repo canary → Add a mutation canary so a vacuous unit mutation gate fails loudly instead of passing green #980. Independent of upstream's root cause and unblocked, so it can land first.

This issue stays as the defect report and the record of what was verified here: #966's 22 mutants are confirmed killed (19 caught, 3 unviable, scoped -- --lib), so no survivors were merged behind the blind gate. Still open here: confirming whether the python/typescript arms share the affected path, re-checking Rust source that merged while the gate was blind, and the mutants.out.old/ gitignore nit.

Generated by Claude Code

docs: record the #975 blind-window mutation audit

Closing out the two investigation items. PR: #1013.

The python and typescript arms are not affected, and cannot be

Read at upstream 9077875. measure_rust is the only arm that materializes a diff file and delegates the filtering to its engine: write_base_diff runs git diff --relative <base>...HEAD -- <prefix> at the cargo workspace root so the paths reach cargo mutants --in-diff workspace-relative, and rebase_report_paths strips the prefix back off the report. That workspace-prefix round-trip is exactly where #659, #672 and this issue have lived, and it exists nowhere else:

measure_typescript calls mutate_ranges → patch_coverage::changed_lines and turns the result into explicit --mutate <file>:<start>-<end> specs for Stryker, prefixed against the package root (package.json), with no workspace concept.
measure_python calls patch_coverage::changed_lines directly, passes the changed .py files to cosmic-ray as --module, and post-filters the mutants to the changed lines.

changed_lines(root, base) runs git diff --unified=0 --relative with current_dir set to the scan dir, so its paths come out scan-relative already — no rebase step to get wrong. It is also the same helper the changed-lines unit coverage check uses on every PR, which is independent evidence that path is sound.

What all three arms do share is the reporting hole: each can return Measurement::Tested { count: 0 }, and the published CLI prints that as the all-caught success line. That is the part worth fixing generically.

Upstream's fix exists but is unpublished — CI is still blind today

At upstream 9077875, measure_rust has gained zero_mutant_verdict: when cargo-mutants writes no outcomes.json, the crate's mutants are re-listed and the run bails if any of them sits on a line the diff inserted. The count == 0 branch also now prints the engine found no mutants to test instead of the all-caught line. Both are exactly what this issue asked for.

Neither has shipped. strings on the published binary finds only the two old messages, at 0.0.86 and 0.0.91 alike:

npm pack @testing-conventions/x86_64-unknown-linux-gnu@0.0.91
tar xzf testing-conventions-x86_64-unknown-linux-gnu-0.0.91.tgz
strings -n 10 package/bin/testing-conventions | grep -c 'no mutants to test'   # 0

0.0.91 is the npm latest that @v0 + npx resolves, so the gate stays vacuous until upstream cuts a release. That makes #980's fixture canary the only in-repo signal that does not depend on someone remembering to check. (Pin the version when probing — a bare npm pack testing-conventions served a stale 0.0.86 packument through this sandbox's registry proxy even though the latest dist-tag says 0.0.91.)

Nothing escaped behind the blind gate except test debt

Every packages/rust/src line that merged while the gate could have been blind was re-mutated: base 6bc91e2 (the last main commit on or before 2026-07-27, the day #672 closed with a green proof) through 6f9e34a — 22 files, +7413/-2411, 470 mutants.

Two scopings, because the repo's docs and its CI disagree about which one the rust gate uses. -C --lib is the documented rule ("fails on any mutant no unit test kills"); CI's actual invocation is mutants_argv's cargo mutants --output <out> --in-diff <diff> --features cli with no target scoping, so the whole cargo test suite judges each mutant.

	result
unit scope	330 caught, 61 survivors, 75 unviable, 4 timeouts
CI scope, over those 61	40 killed by the integration tier, 3 timeout, 18 escape every tier

The 18 split three ways:

7 real gaps, closed in Audit the window the vacuous unit mutation gate left unmutated, and kill what escaped #1013. Six operator mutations in matcher.rs's placeholder_spans — its own documented grammar counter-examples ({}, {1a}, {a-b}, an unclosed {) had no assertions at all — and lib.rs's prepare_persist, where meta_current: compatible && lengths match survives && → ||, which would call a stale glob hash current and skip the meta rewrite. Confirmed fixed by re-running cargo-mutants: the five new tests kill 14 mutants at those sites.
7 equivalent mutants, proved not argued. | → ^ over disjoint bit fields is a no-op — function_flags' SQLITE_UTF8|SQLITE_DETERMINISTIC, base64_encode's (b0<<16)|(b1<<8)|b2 (exhaustive, 16,777,216 inputs), base64_decode's (triple<<6)|value_of(b) (67,108,864). And in placeholder_spans, j += 1 → j *= 1 is re-consumed by the following while, while i = j + 1 → i = j * 1 / i = j - 1 land on the closing brace or a name byte, never a { — zero differing results over all 335,923 strings up to length 7 on { } a 1 _ -. No exemptions added; the gate is diff-scoped, so these surface only if someone edits those lines.
4 in effectful code no tier reaches — RootedFs::hash (two), cli/run.rs's load_state guard and wait_for_shutdown. Filed as Four mutants in effectful core survive every test tier (found by the #975 blind-window audit) #1012 rather than re-litigating testing-conventions: integrate unit lint (isolation) #233 here.

So the answer to "did anything bad merge while the gate was blind" is no — no silent behavior regression. What the blind gate hid was test debt in two pure functions.

Method, the two scopings, and two sandbox traps that cost real time — -j4 unscoped exhausts the disk allowance and cargo-mutants records the resulting No space left on device as Unviable, a wrong verdict that looks real; and -- --lib on top of -C --lib makes cargo reject the duplicated flag so the run dies with cargo test failed in an unmutated tree, indistinguishable at a glance from #977 — are written up in agents/reference/testing-gates.md in #1013.

Generated by Claude Code

Navigation Menu

Repository navigation

Rust `unit mutation` is vacuous again — 0 mutants tested on a 146-line `db.rs` diff (same class as #659/#672) #975

Problem

Evidence

The separable defect that makes this silent

Enforcement — the actual ask

Tasks

Note for whoever picks this up

The python and typescript arms are not affected, and cannot be

Upstream's fix exists but is unpublished — CI is still blind today

Nothing escaped behind the blind gate except test debt

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Rust unit mutation is vacuous again — 0 mutants tested on a 146-line db.rs diff (same class as #659/#672) #975

Description

Problem

Evidence

The separable defect that makes this silent

Enforcement — the actual ask

Tasks

Note for whoever picks this up

Activity

thekevinbot commented on Aug 20, 2026

thekevinbot commented on Aug 21, 2026

The python and typescript arms are not affected, and cannot be

Upstream's fix exists but is unpublished — CI is still blind today

Nothing escaped behind the blind gate except test debt

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Rust `unit mutation` is vacuous again — 0 mutants tested on a 146-line `db.rs` diff (same class as #659/#672) #975