diff --git a/AGENTS.md b/AGENTS.md index 508fe9a83..787033812 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -125,7 +125,7 @@ The colocation rule is a blocking CI gate ([`testing-conventions`](https://githu ### Mutation (testing-conventions) -The `unit mutation` gate mutates PR-changed source lines and fails on any mutant no unit test kills (engines: cosmic-ray / Stryker / cargo-mutants, all self-provisioned; PR-only, diff-scoped via `--base`). All three SDKs run it inside their own CI workflow. Fix a survivor with a **new assertion**, never by weakening a test; only a genuinely equivalent mutant gets a reason-required `mutation` exemption (none today). Local run commands and per-language details: `agents/reference/testing-gates.md`. +The `unit mutation` gate mutates PR-changed source lines and fails on any mutant no unit test kills (engines: cosmic-ray / Stryker / cargo-mutants, all self-provisioned; PR-only, diff-scoped via `--base`). All three SDKs run it inside their own CI workflow. The rust arm is the exception to *unit*: the CLI hands cargo-mutants no target scoping, so the whole `cargo test` suite judges each mutant and an integration test counts as killing one (#975). Fix a survivor with a **new assertion**, never by weakening a test; only a genuinely equivalent mutant gets a reason-required `mutation` exemption (none today). Local run commands and per-language details: `agents/reference/testing-gates.md`. ### Test Boundaries -- What to Mock, What Not To diff --git a/agents/reference/testing-gates.md b/agents/reference/testing-gates.md index 69d51bf2b..00c95cc12 100644 --- a/agents/reference/testing-gates.md +++ b/agents/reference/testing-gates.md @@ -69,6 +69,33 @@ The message names no cause because cosmic-ray hands back an empty `output` for t The Rust arm handles `packages/rust` being a member of the repo-root cargo workspace: testing-conventions ≥ 0.0.88 rebases the diff it feeds cargo-mutants onto the workspace root (dirsql#659, fixed upstream in thekevinscott/testing-conventions#467), and `[rust].features` in `testing-conventions.toml` supplies `--features cli` to the engine's build. A passing run states its evidence — `(N mutant(s) tested)` with N > 0 whenever the diff touched mutatable Rust lines; treat a `0 mutant(s) tested` pass on a diff that changed Rust source as a bug, not a pass. +**Third vacuous pass (#975), and why only the Rust arm can go vacuous.** The gate reported `no surviving mutants — every mutation was caught (0 mutant(s) tested)` over a 146-line `db.rs` change (#966) -- the same class as #659 and #672, both already "fixed" upstream. The failure is **base-sensitive**: same HEAD, same tree, same tool, 22 mutants against the immediate parent and 0 against a base four merges further back, though *both* diffs yield the same 22 when handed to `cargo mutants --in-diff` by hand. Root cause is upstream ([testing-conventions#481](https://github.com/thekevinscott/testing-conventions/issues/481)); the in-repo fixture canary that would make it loud is #980. + +**It cannot reach the python or typescript arms** (read at upstream `9077875`). `measure_rust` is the only arm that materializes a diff *file* and delegates the filtering to its engine: `write_base_diff` runs `git diff --relative ...HEAD -- ` **at the cargo workspace root** so the paths reach `cargo mutants --in-diff` workspace-relative, and `rebase_report_paths` strips the prefix back off the report. That workspace-prefix round-trip is where all three occurrences have lived, and it exists nowhere else: `measure_typescript` and `measure_python` both call `patch_coverage::changed_lines(, base)` -- a `git diff --unified=0 --relative` run **inside the scan dir**, so its paths come out scan-relative already -- and hand the engine explicit scoping (Stryker `--mutate :-`; cosmic-ray `--module` plus a changed-line post-filter). No diff file, no workspace root, no rebase, and it is the same helper the changed-lines `unit coverage` check exercises on every PR. What the three arms *do* share is the reporting: each can return `Measurement::Tested { count: 0 }`, which the published CLI prints as the all-caught success line. + +**Upstream's fix is written but unpublished, so CI is still blind.** At upstream `9077875` `measure_rust` gained `zero_mutant_verdict` -- when cargo-mutants writes no `outcomes.json`, the crate's mutants are re-listed and the run **bails** if any sits on a line the diff inserted -- and the `count == 0` branch prints `the engine found no mutants to test` instead of the all-caught line. Neither string is in the shipped binary: `strings` on `@testing-conventions/x86_64-unknown-linux-gnu` at **0.0.86 and 0.0.91** finds only the two old messages, and 0.0.91 is the npm `latest` that `@v0` + `npx` resolves. Re-check after any upstream release with `npm pack @testing-conventions/x86_64-unknown-linux-gnu@` (pin the version -- a bare `npm pack` can serve a stale packument), then `strings -n 10 package/bin/testing-conventions | grep 'no mutants to test'`. + +**Two scopings, because the docs and CI disagree about which one the gate uses.** `-C --lib` scopes **both** the per-mutant build and its test run to the lib target: that is the *documented* rule ("fails on any mutant no **unit** test kills"), and it also cuts each rebuild from ~56s to ~7-25s by dropping the integration-test binaries. CI's actual invocation is `mutants_argv`'s `cargo mutants --output --in-diff --features cli` with **no** target scoping, so cargo-mutants runs the whole `cargo test` suite and an integration test counts as killing a mutant. Run both when auditing: the `--lib` pass finds unit-tier gaps, and re-running just those survivors unscoped says which ones CI would actually have failed on. Do **not** pass `-- --lib` on top of `-C --lib` -- cargo rejects the duplicated flag and the run dies with `cargo test failed in an unmutated tree`, which reads exactly like #977's uid-0 vtab baseline failure but is an argument error (#977 itself is fixed by #992, so the unscoped baseline is green under uid 0 again). + +**The blind-window audit (#975).** Every `packages/rust/src` line that merged while the gate could have been blind was re-mutated by hand -- base `6bc91e2` (the last `main` commit on or before 2026-07-27, the day upstream closed #672 with a green proof) through `6f9e34a`, 22 files, +7413/-2411, **470 mutants**: + +```bash +cargo install cargo-mutants --locked # not preinstalled in the hosted sandbox +git fetch --unshallow origin main # the session clone is shallow; July is not in it +git diff --relative ...origin/main -- packages/rust/src > /tmp/window.diff +cargo mutants --features cli --in-diff /tmp/window.diff -j4 -C --lib --output /tmp/mutants --timeout 300 +``` + +Keep the job count low enough that the build dirs fit: each *unscoped* job dir reaches ~6 GB, so `-j4` exhausts the sandbox's disk allowance mid-run and cargo-mutants records the resulting `No space left on device` as **`Unviable`** -- a wrong verdict that looks like a real one. `-j2` is the safe ceiling for an unscoped run; `-C --lib` dirs are far smaller. + +**What the audit found.** Unit scope: 330 caught, **61 survivors**, 75 unviable, 4 timeouts. Re-running those 61 under CI's scoping, **40 are killed by the integration tier** -- the #233 split (effectful filesystem / subprocess / HTTP / signal code lives in the integration tier by design) showing up as mutation survivors rather than gate escapes -- 3 time out, and **18 escape every tier**. Those 18 divide three ways, and the split is what matters for anyone repeating this: + +- **7 were real gaps and are now closed** -- six operator mutations in `matcher.rs`'s `placeholder_spans` (its own grammar counter-examples were untested) and `lib.rs`'s `prepare_persist` reading `meta_current` as `compatible && lengths match`, where `&&` -> `||` survived. The tests added for them kill 14 mutants at those sites. +- **7 are equivalent mutants, proved rather than argued.** `|` -> `^` over disjoint bit fields is always a no-op: `function_flags`' `SQLITE_UTF8|SQLITE_DETERMINISTIC` (bits `0x1` and `0x800`), `base64_encode`'s `(b0<<16)|(b1<<8)|b2` (exhaustive over all 16,777,216 inputs), and `base64_decode`'s `(triple<<6)|value_of(b)` (all 67,108,864). In `placeholder_spans`, `j += 1` -> `j *= 1` is re-consumed by the following `while`, and `i = j + 1` -> `i = j * 1` / `i = j - 1` land on the closing brace or a name byte, never a `{` -- zero differing results over all 335,923 strings up to length 7 on `{ } a 1 _ -`. **No exemptions were added**: the gate is diff-scoped, so these only surface if someone edits those lines, and the exemption count stays zero. +- **4 sit in effectful code no tier reaches** -- `RootedFs::hash` (two), `cli/run.rs`'s `load_state` guard and `wait_for_shutdown`. Tracked separately rather than re-litigating #233 here. + +The headline: nothing merged behind the blind gate turned out to be a silent behavior regression. What it hid was **test debt** -- two pure functions whose own documented edge cases had no assertions. + **Survivors.** The fix is almost always a **new assertion** that kills the mutant. Only a genuinely *equivalent* or intentionally-defensive mutant is lifted, via a `[[.exempt]]` entry whose `rules` includes `"mutation"` -- in the config that lane actually reads (`packages/python/testing-conventions.toml` / `packages/ts/testing-conventions.toml` since #933; the root file for the rust core) (with a real `path` and a `reason`) -- never weaken a test to make a survivor pass. There are none today. ## Coverage Floor diff --git a/packages/python/e2e-attestations/claude-tackle-975-n8mc2h.json b/packages/python/e2e-attestations/claude-tackle-975-n8mc2h.json new file mode 100644 index 000000000..f2432db5b --- /dev/null +++ b/packages/python/e2e-attestations/claude-tackle-975-n8mc2h.json @@ -0,0 +1,7 @@ +{ + "command": "uv run python -m pytest tests/e2e/ -x -q", + "ran_at": 1787287690, + "exit_code": 0, + "commit": "8e3939201856c8bcaf2b7c2c951371711c753dbb", + "branch": "claude/tackle-975-n8mc2h" +} diff --git a/packages/rust/src/lib.rs b/packages/rust/src/lib.rs index 25b1df0e0..ffb8da749 100644 --- a/packages/rust/src/lib.rs +++ b/packages/rust/src/lib.rs @@ -3338,6 +3338,41 @@ mod internal_tests { assert!(second.query("SELECT * FROM t").is_ok()); } + /// A cache whose stored meta has the same key count as the expected one + /// but a different glob hash is incompatible, so the meta is not current + /// and the rewrite cannot be skipped. Equal length alone must not stand + /// in for compatibility. + #[test] + fn prepare_persist_reports_an_incompatible_meta_of_equal_length_as_stale() { + let dir = TempDir::new().unwrap(); + let cache = dir.path().join("cache.db"); + drop( + DirSQL::builder() + .root(dir.path()) + .tables(vec![Table::new( + "t", + "CREATE TABLE t (x TEXT)", + "*.txt", + |_| vec![], + )]) + .persist(Some(&cache)) + .build() + .unwrap(), + ); + + // `build_meta` always mints the same keys, so the stored and expected + // blocks are the same length; only the glob hash differs. + let tables = vec![Table::new( + "t", + "CREATE TABLE t (x TEXT)", + "*.csv", + |_| vec![], + )]; + let ctx = prepare_persist(dir.path(), &tables, &[], Some(&cache)).unwrap(); + + assert!(!ctx.meta_current, "an incompatible meta is never current"); + } + #[test] fn prepare_persist_cold_start_reports_rebuild() { let dir = TempDir::new().unwrap(); diff --git a/packages/rust/src/matcher.rs b/packages/rust/src/matcher.rs index fcb68f912..28071a70e 100644 --- a/packages/rust/src/matcher.rs +++ b/packages/rust/src/matcher.rs @@ -332,4 +332,52 @@ mod tests { fn placeholder_names_empty_when_none() { assert!(placeholder_names("data/*/metadata.json").is_empty()); } + + #[test] + fn placeholder_names_rejects_everything_outside_the_grammar() { + // The grammar is exactly `{` `[a-zA-Z_][a-zA-Z0-9_]*` `}`. Each of these + // fails it at a different point -- empty name, a leading digit, a + // disallowed interior byte, and a brace that runs off the end of the + // pattern -- and all are left alone rather than read as a placeholder. + for pattern in ["{}", "{1a}", "{a-b}", "{a b}", "{", "a{", "{a", "{_"] { + assert!( + placeholder_names(pattern).is_empty(), + "`{pattern}` is outside the grammar, so it is not a placeholder" + ); + assert_eq!( + glob_with_placeholders_as_star(pattern), + pattern, + "`{pattern}` is not a placeholder, so the glob is unrewritten" + ); + } + } + + #[test] + fn placeholder_names_accepts_the_whole_grammar() { + // A name starts with a letter or `_` and continues with those plus + // digits. + assert_eq!(placeholder_names("{a}"), vec!["a".to_string()]); + assert_eq!(placeholder_names("{_}"), vec!["_".to_string()]); + assert_eq!(placeholder_names("{_a1}"), vec!["_a1".to_string()]); + assert_eq!(placeholder_names("{A_1z}"), vec!["A_1z".to_string()]); + } + + #[test] + fn the_scan_resumes_at_the_byte_after_a_closing_brace() { + // Leftmost-first and non-overlapping: back-to-back placeholders both + // register, and every byte outside a span survives the rewrite. + assert_eq!( + placeholder_names("{a}{b}"), + vec!["a".to_string(), "b".to_string()] + ); + assert_eq!(glob_with_placeholders_as_star("x{a}y{b}z"), "x*y*z"); + } + + #[test] + fn an_unclosed_brace_leaves_a_later_placeholder_intact() { + // The `{` at 0 fails the grammar, so the scan advances one byte at a + // time and still finds the real placeholder behind it. + assert_eq!(placeholder_names("{a{b}"), vec!["b".to_string()]); + assert_eq!(glob_with_placeholders_as_star("{a{b}"), "{a*"); + } } diff --git a/packages/ts/e2e-attestations/claude-tackle-975-n8mc2h.json b/packages/ts/e2e-attestations/claude-tackle-975-n8mc2h.json new file mode 100644 index 000000000..3ffe3db3b --- /dev/null +++ b/packages/ts/e2e-attestations/claude-tackle-975-n8mc2h.json @@ -0,0 +1,7 @@ +{ + "command": "pnpm test:e2e", + "ran_at": 1787288002, + "exit_code": 0, + "commit": "a728714d316828367e7927e8d82bc47a31f7cf55", + "branch": "claude/tackle-975-n8mc2h" +}