Skip to content

Boundary-value coverage of the salt derivations is fuzz-seed luck: max-salt mutants survive the whole suite #70

Description

@thedavidmeister

Unit

LibICloneableFactoryV4.effectiveOpenSalt(bytes32,bytes) (src/lib/LibICloneableFactoryV4.sol:87-89) and LibICloneableFactoryV4.predictCloneAddress(address,address,bytes32) (src/lib/LibICloneableFactoryV4.sol:111-125), and the fuzz configuration that covers them (foundry.toml [fuzz] runs = 2048, no seed).

Intent oracle

ICloneableFactoryV4:34-48 pins both derivations to exact bytes over the whole input domain — salt is a bytes32 with no excluded values, and ICloneableFactoryV4:201-203 says only that "distinct (salt, data) pairs yield distinct clones". Nothing carves out a boundary value, so the derivation must be the same formula at bytes32(0), at bytes32(type(uint256).max) and everywhere between.

Property that appears to be violated

Not the library's behaviour — the derivations are correct at both boundaries. What is violated is the suite's claim to cover them. With an unpinned fuzz seed, whether a boundary value is ever drawn varies run to run, so which behaviours the suite actually pins varies run to run.

Measured directly by mutation probing at c1c2afd, before any new test was written. Four mutants that differ from correct code only at a boundary value:

mutant function boundary special-cased verdict
salt == bytes32(0) ? bytes32(1) : salt effectiveSalt zero salt KILLED
salt == bytes32(0) ? bytes32(1) : salt effectiveOpenSalt zero salt KILLED
salt == bytes32(type(uint256).max) ? bytes32(0) : salt effectiveSalt max salt KILLED
salt == bytes32(type(uint256).max) ? bytes32(0) : salt effectiveOpenSalt max salt SURVIVED
derivedSalt == bytes32(0) ? bytes32(1) : derivedSalt predictCloneAddress zero salt KILLED
derivedSalt == bytes32(type(uint256).max) ? bytes32(0) : derivedSalt predictCloneAddress max salt SURVIVED

The same mutation survives on effectiveOpenSalt and dies on effectiveSalt — not because the two are tested differently in intent, but because effectiveSalt is reached from more fuzz tests and therefore gets more draws. The kills in that table are luck, not coverage: nothing in the suite asks for either boundary.

The instability is directly observable in the tallies too. testEffectiveOpenSaltEmptyData completed all 2048 runs green in one probe and, under a max-salt mutant in a later run, failed at runs: 904 — the same test, the same tree, a different draw.

Verified repro

nix run github:rainlanguage/adversarial-mutation-test#mutation-probe -- mutants.toml, baseline 44 passed after unflaking testCheckImplementationCodeEtched (see the sibling issue). Full-suite pass, no new tests present:

R08 effectiveOpenSalt extreme: max salt special-cased: SURVIVED
S04 predictCloneAddress extreme: max derived salt special-cased: SURVIVED

Both are killed deterministically (runs: 0, i.e. the first case) once the boundaries are asserted explicitly, which PR 2026-08-24-amt-g1-libicloneablefactoryv4-effec does for these two functions.

Triage framing

Two questions for the maintainers, neither adjudicated here:

  1. Whether foundry.toml should pin [fuzz] seed, so that "the suite is green" and "the suite covers X" mean the same thing on every run and a mutation matrix is reproducible. The trade is the usual one — a pinned seed stops finding new counterexamples over time.
  2. Whether boundary values (bytes32(0), bytes32(type(uint256).max), empty and long data, zero and max addresses) should be asserted deliberately across the derivations rather than left to the fuzzer. testEffectiveOpenSaltEmptyData already does this for one boundary, so the convention exists; the linked PR extends it to the two boundaries whose mutants survived, and deliberately does not extend it to the ones that happened to die, so the ledger stays honest about what is coverage and what is luck.

Found during adversarial mutation testing of LibICloneableFactoryV4 at c1c2afd3d88405d3228cb3b21e55c9d63ba8f5be.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    adversarialFound by adversarial reviewauditAudit finding

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions