Skip to content

Add a mutation canary so a vacuous unit mutation gate fails loudly instead of passing green #980

Closed

Description

@thekevinbot

Why

The Rust unit mutation gate has now silently degraded to a no-op three times — #659, #672, and #975 — each time reporting no surviving mutants — every mutation was caught (0 mutant(s) tested) and going green. Between #672 closing on 2026-07-27 and #975 being found on 2026-08-20, nothing noticed.

The root cause is upstream each time (testing-conventions#467, #469, now #481). Fixing it upstream is necessary but has already proven insufficient twice: this repo consumes the gate through @v0 + npx, which resolves latest, so the dependency can change under us on any run with no commit to blame and no signal that anything changed.

This issue is the durable in-repo protection, and it is independent of whatever upstream's current root cause turns out to be.

Why the previously-rejected guard was rejected, and why this one is different

#672 considered and rejected a repo-side guard:

A repo-side guard that fails when a rust-source PR yields 0 mutant(s) tested is fragile once upstream is fixed (a genuinely 0-mutant diff would false-positive).

That objection is correct — but it only applies to a guard keyed on the PR's own diff. A canary keyed on a fixed fixture has no such failure mode: the expected count is a property of the committed fixture, not of whatever the contributor happened to change. The rejected design and this one are not the same design.

Proposal

A dirsql-checks mutation-canary subcommand (per AGENTS.md, "CI logic lives in scripts, not workflow YAML"):

  1. A small fixture crate laid out as a cargo workspace member — the layout this repo actually has, and the one where every occurrence of this bug has lived. A standalone crate would not exercise the broken path.
  2. A committed synthetic diff over that fixture with known mutants.
  3. Run the same invocation dirsql-rust-ci.yml uses against the fixture, parse the reported count out of the success line, and fail unless it is > 0. The failure message should name the likely cause and link this issue, per the repo's actionable-error rule.
  4. Wire it into dirsql-rust-ci.yml on every PR — not nightly. The tool version can roll on any run, so a daily cadence would still ship blind PRs.
  5. Colocated unit tests for the parsing/decision logic; the real invocation belongs in the package's tests/integration/ tier, matching how the other checks in internals/checks are structured.

Cheap alternative worth a decision either way: pin the testing-conventions version instead of tracking latest, converting a silent regression into an explicit bump. That contradicts the deliberate moving-tag choice documented in agents/reference/testing-conventions-ci.md, so it is a maintainer call rather than an obvious win — but it addresses the exposure that the canary only detects.

Worth considering: the same hole in other diff-scoped gates

Mutation is where this was caught, but it is not structurally unique. Every diff-scoped gate can degrade to measuring nothing and still report success — unit coverage --base (changed-lines) has the same shape. If the canary is built as a general "assert this gate actually did work on a known fixture" harness rather than a mutation-only script, it covers those too. Worth deciding at design time; scoping it to mutation first is fine if that is the smaller PR.

Blocked-by-nothing: this can land before upstream #481 is fixed, and should — the canary going red is exactly the signal we currently lack.

Activity

  1. thekevinbot commented on Aug 21, 2026

    @thekevinbot
    CollaboratorAuthor

    Superseded upstream: the vacuous-gate root cause was fixed in testing-conventions#481 (cargo-mutants' three "nothing to do" outcomes no longer read as a green 0-mutant run), so a repo-side canary would duplicate the engine's own loud failure. The window the vacuous gate left unmutated was audited in #1013; the four escapees it found are tracked in #1012.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions