Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ The colocation rule is a blocking CI gate ([`testing-conventions`](https://githu

### Mutation (testing-conventions)

The `unit mutation` gate mutates PR-changed source lines and fails on any mutant no unit test kills (engines: cosmic-ray / Stryker / cargo-mutants, all self-provisioned; PR-only, diff-scoped via `--base`). All three SDKs run it inside their own CI workflow. Fix a survivor with a **new assertion**, never by weakening a test; only a genuinely equivalent mutant gets a reason-required `mutation` exemption (none today). Local run commands and per-language details: `agents/reference/testing-gates.md`.
The `unit mutation` gate mutates PR-changed source lines and fails on any mutant no unit test kills (engines: cosmic-ray / Stryker / cargo-mutants, all self-provisioned; PR-only, diff-scoped via `--base`). All three SDKs run it inside their own CI workflow. The rust arm is the exception to *unit*: the CLI hands cargo-mutants no target scoping, so the whole `cargo test` suite judges each mutant and an integration test counts as killing one (#975). Fix a survivor with a **new assertion**, never by weakening a test; only a genuinely equivalent mutant gets a reason-required `mutation` exemption (none today). Local run commands and per-language details: `agents/reference/testing-gates.md`.

### Test Boundaries -- What to Mock, What Not To

Expand Down
27 changes: 27 additions & 0 deletions agents/reference/testing-gates.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,33 @@ The message names no cause because cosmic-ray hands back an empty `output` for t

The Rust arm handles `packages/rust` being a member of the repo-root cargo workspace: testing-conventions ≥ 0.0.88 rebases the diff it feeds cargo-mutants onto the workspace root (dirsql#659, fixed upstream in thekevinscott/testing-conventions#467), and `[rust].features` in `testing-conventions.toml` supplies `--features cli` to the engine's build. A passing run states its evidence — `(N mutant(s) tested)` with N > 0 whenever the diff touched mutatable Rust lines; treat a `0 mutant(s) tested` pass on a diff that changed Rust source as a bug, not a pass.

**Third vacuous pass (#975), and why only the Rust arm can go vacuous.** The gate reported `no surviving mutants — every mutation was caught (0 mutant(s) tested)` over a 146-line `db.rs` change (#966) -- the same class as #659 and #672, both already "fixed" upstream. The failure is **base-sensitive**: same HEAD, same tree, same tool, 22 mutants against the immediate parent and 0 against a base four merges further back, though *both* diffs yield the same 22 when handed to `cargo mutants --in-diff` by hand. Root cause is upstream ([testing-conventions#481](https://github.com/thekevinscott/testing-conventions/issues/481)); the in-repo fixture canary that would make it loud is #980.

**It cannot reach the python or typescript arms** (read at upstream `9077875`). `measure_rust` is the only arm that materializes a diff *file* and delegates the filtering to its engine: `write_base_diff` runs `git diff --relative <base>...HEAD -- <prefix>` **at the cargo workspace root** so the paths reach `cargo mutants --in-diff` workspace-relative, and `rebase_report_paths` strips the prefix back off the report. That workspace-prefix round-trip is where all three occurrences have lived, and it exists nowhere else: `measure_typescript` and `measure_python` both call `patch_coverage::changed_lines(<scan root>, base)` -- a `git diff --unified=0 --relative` run **inside the scan dir**, so its paths come out scan-relative already -- and hand the engine explicit scoping (Stryker `--mutate <file>:<a>-<b>`; cosmic-ray `--module` plus a changed-line post-filter). No diff file, no workspace root, no rebase, and it is the same helper the changed-lines `unit coverage` check exercises on every PR. What the three arms *do* share is the reporting: each can return `Measurement::Tested { count: 0 }`, which the published CLI prints as the all-caught success line.

**Upstream's fix is written but unpublished, so CI is still blind.** At upstream `9077875` `measure_rust` gained `zero_mutant_verdict` -- when cargo-mutants writes no `outcomes.json`, the crate's mutants are re-listed and the run **bails** if any sits on a line the diff inserted -- and the `count == 0` branch prints `the engine found no mutants to test` instead of the all-caught line. Neither string is in the shipped binary: `strings` on `@testing-conventions/x86_64-unknown-linux-gnu` at **0.0.86 and 0.0.91** finds only the two old messages, and 0.0.91 is the npm `latest` that `@v0` + `npx` resolves. Re-check after any upstream release with `npm pack @testing-conventions/x86_64-unknown-linux-gnu@<version>` (pin the version -- a bare `npm pack` can serve a stale packument), then `strings -n 10 package/bin/testing-conventions | grep 'no mutants to test'`.

**Two scopings, because the docs and CI disagree about which one the gate uses.** `-C --lib` scopes **both** the per-mutant build and its test run to the lib target: that is the *documented* rule ("fails on any mutant no **unit** test kills"), and it also cuts each rebuild from ~56s to ~7-25s by dropping the integration-test binaries. CI's actual invocation is `mutants_argv`'s `cargo mutants --output <out> --in-diff <diff> --features cli` with **no** target scoping, so cargo-mutants runs the whole `cargo test` suite and an integration test counts as killing a mutant. Run both when auditing: the `--lib` pass finds unit-tier gaps, and re-running just those survivors unscoped says which ones CI would actually have failed on. Do **not** pass `-- --lib` on top of `-C --lib` -- cargo rejects the duplicated flag and the run dies with `cargo test failed in an unmutated tree`, which reads exactly like #977's uid-0 vtab baseline failure but is an argument error (#977 itself is fixed by #992, so the unscoped baseline is green under uid 0 again).

**The blind-window audit (#975).** Every `packages/rust/src` line that merged while the gate could have been blind was re-mutated by hand -- base `6bc91e2` (the last `main` commit on or before 2026-07-27, the day upstream closed #672 with a green proof) through `6f9e34a`, 22 files, +7413/-2411, **470 mutants**:

```bash
cargo install cargo-mutants --locked # not preinstalled in the hosted sandbox
git fetch --unshallow origin main # the session clone is shallow; July is not in it
git diff --relative <base>...origin/main -- packages/rust/src > /tmp/window.diff
cargo mutants --features cli --in-diff /tmp/window.diff -j4 -C --lib --output /tmp/mutants --timeout 300
```

Keep the job count low enough that the build dirs fit: each *unscoped* job dir reaches ~6 GB, so `-j4` exhausts the sandbox's disk allowance mid-run and cargo-mutants records the resulting `No space left on device` as **`Unviable`** -- a wrong verdict that looks like a real one. `-j2` is the safe ceiling for an unscoped run; `-C --lib` dirs are far smaller.

**What the audit found.** Unit scope: 330 caught, **61 survivors**, 75 unviable, 4 timeouts. Re-running those 61 under CI's scoping, **40 are killed by the integration tier** -- the #233 split (effectful filesystem / subprocess / HTTP / signal code lives in the integration tier by design) showing up as mutation survivors rather than gate escapes -- 3 time out, and **18 escape every tier**. Those 18 divide three ways, and the split is what matters for anyone repeating this:

- **7 were real gaps and are now closed** -- six operator mutations in `matcher.rs`'s `placeholder_spans` (its own grammar counter-examples were untested) and `lib.rs`'s `prepare_persist` reading `meta_current` as `compatible && lengths match`, where `&&` -> `||` survived. The tests added for them kill 14 mutants at those sites.
- **7 are equivalent mutants, proved rather than argued.** `|` -> `^` over disjoint bit fields is always a no-op: `function_flags`' `SQLITE_UTF8|SQLITE_DETERMINISTIC` (bits `0x1` and `0x800`), `base64_encode`'s `(b0<<16)|(b1<<8)|b2` (exhaustive over all 16,777,216 inputs), and `base64_decode`'s `(triple<<6)|value_of(b)` (all 67,108,864). In `placeholder_spans`, `j += 1` -> `j *= 1` is re-consumed by the following `while`, and `i = j + 1` -> `i = j * 1` / `i = j - 1` land on the closing brace or a name byte, never a `{` -- zero differing results over all 335,923 strings up to length 7 on `{ } a 1 _ -`. **No exemptions were added**: the gate is diff-scoped, so these only surface if someone edits those lines, and the exemption count stays zero.
- **4 sit in effectful code no tier reaches** -- `RootedFs::hash` (two), `cli/run.rs`'s `load_state` guard and `wait_for_shutdown`. Tracked separately rather than re-litigating #233 here.

The headline: nothing merged behind the blind gate turned out to be a silent behavior regression. What it hid was **test debt** -- two pure functions whose own documented edge cases had no assertions.

**Survivors.** The fix is almost always a **new assertion** that kills the mutant. Only a genuinely *equivalent* or intentionally-defensive mutant is lifted, via a `[[<language>.exempt]]` entry whose `rules` includes `"mutation"` -- in the config that lane actually reads (`packages/python/testing-conventions.toml` / `packages/ts/testing-conventions.toml` since #933; the root file for the rust core) (with a real `path` and a `reason`) -- never weaken a test to make a survivor pass. There are none today.
## Coverage Floor

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "uv run python -m pytest tests/e2e/ -x -q",
"ran_at": 1787287690,
"exit_code": 0,
"commit": "8e3939201856c8bcaf2b7c2c951371711c753dbb",
"branch": "claude/tackle-975-n8mc2h"
}
35 changes: 35 additions & 0 deletions packages/rust/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3338,6 +3338,41 @@ mod internal_tests {
assert!(second.query("SELECT * FROM t").is_ok());
}

/// A cache whose stored meta has the same key count as the expected one
/// but a different glob hash is incompatible, so the meta is not current
/// and the rewrite cannot be skipped. Equal length alone must not stand
/// in for compatibility.
#[test]
fn prepare_persist_reports_an_incompatible_meta_of_equal_length_as_stale() {
let dir = TempDir::new().unwrap();
let cache = dir.path().join("cache.db");
drop(
DirSQL::builder()
.root(dir.path())
.tables(vec![Table::new(
"t",
"CREATE TABLE t (x TEXT)",
"*.txt",
|_| vec![],
)])
.persist(Some(&cache))
.build()
.unwrap(),
);

// `build_meta` always mints the same keys, so the stored and expected
// blocks are the same length; only the glob hash differs.
let tables = vec![Table::new(
"t",
"CREATE TABLE t (x TEXT)",
"*.csv",
|_| vec![],
)];
let ctx = prepare_persist(dir.path(), &tables, &[], Some(&cache)).unwrap();

assert!(!ctx.meta_current, "an incompatible meta is never current");
}

#[test]
fn prepare_persist_cold_start_reports_rebuild() {
let dir = TempDir::new().unwrap();
Expand Down
48 changes: 48 additions & 0 deletions packages/rust/src/matcher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -332,4 +332,52 @@ mod tests {
fn placeholder_names_empty_when_none() {
assert!(placeholder_names("data/*/metadata.json").is_empty());
}

#[test]
fn placeholder_names_rejects_everything_outside_the_grammar() {
// The grammar is exactly `{` `[a-zA-Z_][a-zA-Z0-9_]*` `}`. Each of these
// fails it at a different point -- empty name, a leading digit, a
// disallowed interior byte, and a brace that runs off the end of the
// pattern -- and all are left alone rather than read as a placeholder.
for pattern in ["{}", "{1a}", "{a-b}", "{a b}", "{", "a{", "{a", "{_"] {
assert!(
placeholder_names(pattern).is_empty(),
"`{pattern}` is outside the grammar, so it is not a placeholder"
);
assert_eq!(
glob_with_placeholders_as_star(pattern),
pattern,
"`{pattern}` is not a placeholder, so the glob is unrewritten"
);
}
}

#[test]
fn placeholder_names_accepts_the_whole_grammar() {
// A name starts with a letter or `_` and continues with those plus
// digits.
assert_eq!(placeholder_names("{a}"), vec!["a".to_string()]);
assert_eq!(placeholder_names("{_}"), vec!["_".to_string()]);
assert_eq!(placeholder_names("{_a1}"), vec!["_a1".to_string()]);
assert_eq!(placeholder_names("{A_1z}"), vec!["A_1z".to_string()]);
}

#[test]
fn the_scan_resumes_at_the_byte_after_a_closing_brace() {
// Leftmost-first and non-overlapping: back-to-back placeholders both
// register, and every byte outside a span survives the rewrite.
assert_eq!(
placeholder_names("{a}{b}"),
vec!["a".to_string(), "b".to_string()]
);
assert_eq!(glob_with_placeholders_as_star("x{a}y{b}z"), "x*y*z");
}

#[test]
fn an_unclosed_brace_leaves_a_later_placeholder_intact() {
// The `{` at 0 fails the grammar, so the scan advances one byte at a
// time and still finds the real placeholder behind it.
assert_eq!(placeholder_names("{a{b}"), vec!["b".to_string()]);
assert_eq!(glob_with_placeholders_as_star("{a{b}"), "{a*");
}
}
7 changes: 7 additions & 0 deletions packages/ts/e2e-attestations/claude-tackle-975-n8mc2h.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "pnpm test:e2e",
"ran_at": 1787288002,
"exit_code": 0,
"commit": "a728714d316828367e7927e8d82bc47a31f7cf55",
"branch": "claude/tackle-975-n8mc2h"
}
Loading