Repository navigation
run_gap_tests.sh is red on pristine main and its snapshot has gone stale — the gate reports 9 false regressions #9273
Description
Summary
scripts/run_gap_tests.sh reports 591 PASS / 14 FAIL on pristine b3f14e9cde, with 9 of those flagged as "regressions". They are not regressions, and the gate is currently non-functional as a signal for everyone.
Proof that the 9 are false
All 9 were compiled with a base compiler and a patched compiler in the same directory, and their outputs compared:
9/9 byte-identical output, identical exit codes.
Identical bytes cannot be one pass and one fail. So the failures are not produced by any change under test — the committed gap_snapshot.json has gone stale against the current expected outputs, and the suite is reporting the drift as regressions.
Why this matters more than 14 red tests
A gate that is red on arrival is CLAUDE.md hazard 2. Worse, this one is red in a way that manufactures attribution: it names 9 specific tests as "regressions" of whatever branch happens to be running, so the natural reading is that your change broke them. Anyone who trusts it wastes an afternoon; anyone who learns not to trust it stops reading the whole 603-test suite, which is the real cost — this is the suite that covers the compiled-output parity fixtures.
There is precedent for exactly this failure shape in this repo: #797's parity skip-list, where entries that had stopped being true cost nothing to keep and quietly went stale, and #9108, where an abort masked ~1,000 runtime tests. The pattern is the same — a signal that is wrong in a direction nobody audits.
What is wanted
- Re-derive
gap_snapshot.jsonagainst currentmainso the suite is green on arrival. - Make the drift self-invalidating, the way
SUITE_EXCLUSIONSinscripts/ci_e2e_scope.pyalready is: an entry that stops being true should fail the job with instructions to delete it, rather than sitting there silently. A snapshot that can only rot in one direction will rot again. - If any of the 14 are genuine expected-failures, record each with its issue number rather than as an unexplained snapshot entry.
Found by
A registry-probe performance change (#9272) that ran the suite as a pre-merge check, saw 9 "regressions", and disproved every one by byte-comparing base against patched output rather than trusting the label.
Activity
proggeramlug commentedon Aug 31, 2026
Five of the nine "false regressions" now have a concrete root cause, found during unrelated verification (#9315): missing npm dependencies, not snapshot drift.
backoff_options,cron_cronjob,dayjs_factory_arg,moment_methods,ratelimiter_memoryare gap tests comparing perry's native package emulation against the real package under node. The packages are declared inpackage.json, but neitherrun_gap_tests.shnorrun_parity_tests.shever provisionsnode_modules— they silently assume a priornpm ci. On a fresh checkout the node oracle exits with anERR_MODULE_NOT_FOUNDstack trace, and the harness scores the mismatch against the branch under test.Verified both directions: all five were byte-identical between base and patched binaries (so no branch caused them), and after
npm ciall five pass inside the gate with the same binaries.Two fixes wanted, the second mattering more:
npm ci --prefer-offlinein the harness, or a preflight check).ERR_MODULE_NOT_FOUND(or that the oracle produced any expected-shape output) converts a silent false-blame into an actionable setup error.The remaining 4 of the 9 may still be genuine snapshot drift — the npm finding covers exactly five.