Use this route when the task changes code whose current behavior must be kept: a replace, rewrite, migrate or port, or a refactor of legacy code with thin tests. The old implementation, or its recorded output, is the oracle for the new one.
These tests are pins, not correctness claims: a recorded baseline pins a bug in the old code
along with everything else. Name them as pins (in the test name or a one-line comment) so nobody
reads them as a specification, and follow testing:test-value for where a pin's expected value may
come from.
| Shape | What it compares | Fits |
|---|---|---|
| Characterization | Current output, captured into an ordinary assertion | Legacy code before a refactor; a handful of behaviors |
| Approval / snapshot (golden master) | Current output, recorded to a file and reviewed on change | Large or structured outputs: reports, rendered documents, serialized objects |
| Differential | Old and new implementations, run on the same inputs | A replace or migrate where both can still run |
Feathers' recipe, per behavior you are about to touch:
- Call the code with a real input and assert a value you know is wrong.
- Run it. The failure message shows the actual value.
- Put the actual value in the assertion. The test now documents what the code does.
- Sabotage check: temporarily break the code under test (flip a condition, change a constant) and confirm each pin fails, then restore it. A pin that survives sabotage pins nothing.
- Commit the pins on their own, before the change, so the history shows the baseline.
After the change, delete each pin or rewrite it as a specification test whose expected value comes from intent; a pin left behind turns every later intended change into a re-approval.
- Recorded corpus. Real inputs (captured requests, production-shaped fixtures, the existing test data) fed to both sides, persisted with the tests so a run can be reproduced. This is the base. Random or property-generated inputs (see property.md) are an optional extra, never a substitute: the claim that they catch bugs a recorded corpus misses rests on LOW evidence in the research.
- Comparator. One function that decides equal or not, strict by default. Compare outputs and observable effects, not internal state.
- Scrubbers for nondeterminism. Normalize timestamps, generated ids, ordering of unordered collections, and machine paths before comparing, rather than dropping the comparison. Approval tools ship scrubbers or matchers for this. Where a value cannot be normalized, running old against old shows how much the output varies on its own; treat that as a suggestion, since its backing in the research is thin.
- Intentional-differences ledger. Every difference the change means to make gets an entry: an id, a predicate that matches exactly that difference, the reason, and the user's sign-off. The comparator skips a mismatch only through a ledger entry. Never widen the comparator, add a blanket scrubber, or re-approve a snapshot to turn a run green.
- Failure report that names the input. On a mismatch, print the input (or its corpus id), the old output, the new output, the diff, and whether any ledger entry was close, never only "outputs differ".
- Counts, not "tests pass". Report inputs run, mismatches, mismatches covered by each ledger id, and unexplained mismatches. Any unexplained mismatch means the replacement is not done.
Compare only read-only paths against live traffic or in production. A path with side effects (a write, a send, a charge) runs the new side against an isolated sink.
A snapshot your own change broke is a finding. Read the diff, decide whether it is an intended difference (add a ledger entry or say why in the commit) or a regression (fix the code), and only then re-approve. Never re-approve in bulk: that records whatever bug produced the diff.
Feathers, Characterization Testing (2016); Seemann, Empirical characterization testing (the sabotage check, 2025-11-03); GitHub Scientist README (control and candidate, ignored mismatches, the side-effect warning); tool docs for scrubbers and review: Verify scrubbers, Jest snapshot testing, ApprovalTests.
Basis: the agent-self-check research slice (2026-10-06). The practices come from human-led rewrites; no accepted source shows an agent validating a rewrite this way, so the agent application is extrapolated. Recheck trigger: a measured study of agents validating rewrites by differential comparison.