Repository navigation
Description
Why
The Rust unit mutation gate has now silently degraded to a no-op three times — #659, #672, and #975 — each time reporting no surviving mutants — every mutation was caught (0 mutant(s) tested) and going green. Between #672 closing on 2026-07-27 and #975 being found on 2026-08-20, nothing noticed.
The root cause is upstream each time (testing-conventions#467, #469, now #481). Fixing it upstream is necessary but has already proven insufficient twice: this repo consumes the gate through @v0 + npx, which resolves latest, so the dependency can change under us on any run with no commit to blame and no signal that anything changed.
This issue is the durable in-repo protection, and it is independent of whatever upstream's current root cause turns out to be.
Why the previously-rejected guard was rejected, and why this one is different
#672 considered and rejected a repo-side guard:
A repo-side guard that fails when a rust-source PR yields
0 mutant(s) testedis fragile once upstream is fixed (a genuinely 0-mutant diff would false-positive).
That objection is correct — but it only applies to a guard keyed on the PR's own diff. A canary keyed on a fixed fixture has no such failure mode: the expected count is a property of the committed fixture, not of whatever the contributor happened to change. The rejected design and this one are not the same design.
Proposal
A dirsql-checks mutation-canary subcommand (per AGENTS.md, "CI logic lives in scripts, not workflow YAML"):
- A small fixture crate laid out as a cargo workspace member — the layout this repo actually has, and the one where every occurrence of this bug has lived. A standalone crate would not exercise the broken path.
- A committed synthetic diff over that fixture with known mutants.
- Run the same invocation
dirsql-rust-ci.ymluses against the fixture, parse the reported count out of the success line, and fail unless it is> 0. The failure message should name the likely cause and link this issue, per the repo's actionable-error rule. - Wire it into
dirsql-rust-ci.ymlon every PR — not nightly. The tool version can roll on any run, so a daily cadence would still ship blind PRs. - Colocated unit tests for the parsing/decision logic; the real invocation belongs in the package's
tests/integration/tier, matching how the other checks ininternals/checksare structured.
Cheap alternative worth a decision either way: pin the testing-conventions version instead of tracking latest, converting a silent regression into an explicit bump. That contradicts the deliberate moving-tag choice documented in agents/reference/testing-conventions-ci.md, so it is a maintainer call rather than an obvious win — but it addresses the exposure that the canary only detects.
Worth considering: the same hole in other diff-scoped gates
Mutation is where this was caught, but it is not structurally unique. Every diff-scoped gate can degrade to measuring nothing and still report success — unit coverage --base (changed-lines) has the same shape. If the canary is built as a general "assert this gate actually did work on a known fixture" harness rather than a mutation-only script, it covers those too. Worth deciding at design time; scoping it to mutation first is fine if that is the smaller PR.
Blocked-by-nothing: this can land before upstream #481 is fixed, and should — the canary going red is exactly the signal we currently lack.
Activity
thekevinbot commentedon Aug 21, 2026
Superseded upstream: the vacuous-gate root cause was fixed in testing-conventions#481 (cargo-mutants' three "nothing to do" outcomes no longer read as a green 0-mutant run), so a repo-side canary would duplicate the engine's own loud failure. The window the vacuous gate left unmutated was audited in #1013; the four escapees it found are tracked in #1012.