test(gate): run every *.test.mjs suite β 529 node tests no gate had ever run - #309
Conversation
β¦ver run
`scripts/run-node-tests.sh` hard-coded its collection to one directory:
FILES=(scripts/browser-bridge/tests/*.test.mjs)
so every other `.test.mjs` suite in the repo was invisible to the only check
that runs node. Measured at origin/main (13bc8bd):
* scripts/dl-router/tests 508 tests, never gated
* scripts/collector/browser-ext/tests 21 tests, never gated
529 tests, ungated since each suite was written. The gate reported
`RESULT: PASS` with a 468-test total the whole time β a hard-coded list cannot
know what it is not looking at. Fourth instance of the same shape (#276: 913
pytest tests behind one list entry, #298: 166, #306: 989).
Adding two lines would leave the NEXT suite ungated identically, so collection
is now DISCOVERY (bash globstar over scripts/**/*.test.mjs) plus a TWO-WAY PIN:
* a discovered directory absent from SUITES -> FATAL (forces an accounting
entry with a measured floor, rather than being swept in under the total)
* a pinned suite discovery does not find -> FATAL (the suite vanished)
Discovery alone would reintroduce the silent-collapse hole the old hard-coded
glob at least did not have: an emptied dl-router/tests would just collect fewer
files and still pass over a global floor.
Each suite also runs in its OWN `node --test` invocation with its own TAP
summary and its own floor. A single global floor of 970 is fully satisfied by
browser-bridge (468) + dl-router (508) with browser-ext's 21 tests entirely
gone; per-suite floors make that loud.
A PORTABILITY DEFECT found while building this, worth recording because the
first draft shipped it and the harness hid it: discovery used
`find -printf '%h\n' 2>/dev/null`. `-printf` is a GNU extension, and THREE
different `find`s are reachable from this repo β busybox under bash
(~/.nix-profile/bin/find, rejects it), bfs 4.1.1 under the interactive zsh, GNU
findutils in the nix sandbox. Paired with `2>/dev/null` the rejection became an
EMPTY discovery list with no error β a false "no suites found". Collection is
now a bash builtin (globstar), which depends on no external binary and behaves
identically in every tier; `test_runner_does_not_use_find_printf` pins it.
MEASURED, both tiers:
nix build .#checks.x86_64-linux.nodetests 997 tests / 997 pass / 0 fail
bash scripts/run-node-tests.sh (dev host) 997 tests / 997 pass / 0 fail
468 (baseline, browser-bridge) + 508 + 21 = 997. No unexplained delta.
scripts/tests/test_run_node_tests_suites.py guards the pin, with its regression
and invariant guards labelled honestly in the module docstring, mutation proofs
that the guard goes red naming the offending directory, and a positive control
on both parsers (the reassuring answer here is an empty set, which an unwired
harness also produces).
The 529 newly-gated tests are NOT regression coverage for this change β they
are pre-existing tests whose value is that they now run at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gated on the MERGED tree, not just this branchPer RULES.md β a PR green on its own branch proves nothing about the tree its merge creates. Built
Combined exit Delta accounted exactly: 5792 (baseline) + 10 (#309's
|
β¦ 0 on failure (#312) #308 asserted that scripts/run-tests.sh "can exit 0 while printing RESULT: FAIL / collected=0". That does not reproduce and is structurally impossible on this revision: `RESULT: FAIL` prints only inside `if [ "$fail" -ne 0 ]`, and the very next statement is `exit "$fail"`. Measured on 13bc8bd, a run without pytest importable prints 17 x "could not parse pytest's summary", RESULT: FAIL, and exits 1 (2 in a second probe). The original report most likely read a status through a pipeline -- the rc=$? trap RULES.md already names. The real defect, fixed in #310, is narrower: REQUIRED_TOOLS checks BINARIES via `command -v`, but pytest is a MODULE, so a missing pytest yielded a diagnostic blaming pytest's output format instead of a clean precondition failure. It also asserted `python3` while the runner calls `python`. Bad diagnostics, not a false green. This was relayed from a subagent report and repeated to the operator twice without being tested -- the same class of error as the four corrections the doc itself records, except this one reached a merged document before anyone checked it. Left visible as a strikethrough rather than deleted, because the doc's purpose is recording what was believed and then disproved. Also updates three items now closed: - #309 gated the node suites: 529 ungated tests, not the 508 first reported (scripts/collector/browser-ext/tests was ungated too). Node gate 468 -> 997. - #311 shipped the telemetry deadman, including the verified property that "cannot tell" cannot render as "all healthy" -- deadman returns count=0 for both ok and unreachable, so a consumer reading count alone would be wrong. - session-create's cause is now known: `session.created` is a bus EVENT TYPE, not a plugin hook name, on opencode 1.18.4. Consequence: 2,736 of 2,799 tool-call rows carry session=''. Not fixed -- that file's last edit killed 11 hours of telemetry. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The defect
scripts/run-node-tests.shhard-coded its collection to one directory:FILES=(scripts/browser-bridge/tests/*.test.mjs)so every other
.test.mjssuite in the repo was invisible to the only check that runs node. Measured atorigin/main(13bc8bd):scripts/browser-bridge/testsscripts/dl-router/testsscripts/collector/browser-ext/tests529 tests, ungated since each suite was written. The gate reported
RESULT: PASSand a 468-test total the whole time β a hard-coded list cannot know what it is not looking at. This is the fourth instance of the shape: #276 (913 pytest tests behind one list entry), #298 (166), #306 (989).scripts/collector/browser-ext/testswas not in the brief β it surfaced from auditing every*.test.mjsin the repo rather than only the directory named. Itstest_receiver.pywas already gated byrun-tests.sh; only its two.mjsfiles were orphaned.Why not just add two lines
That leaves the next suite ungated identically. Collection is now discovery (bash globstar over
scripts/**/*.test.mjs) plus a two-way pin (SUITES), which fails in both directions:SUITESβ FATAL (forces an accounting entry with a measured floor, instead of being swept in under a global total)Discovery alone would reintroduce a hole the hard-coded glob did not have: an emptied
dl-router/testswould simply collect fewer files and still pass over a global floor.Each suite also runs in its own
node --testinvocation with its own TAP summary and its own floor. A single global floor of 970 is fully satisfied by browser-bridge (468) + dl-router (508) with browser-ext's 21 tests entirely gone β per-suite floors make that loud.A portability defect found while building this
The first draft discovered suites with
find -printf '%h\n' 2>/dev/null.-printfis a GNU extension, and three differentfinds are reachable from this repo:find-printf~/.nix-profile/bin/find)bfs4.1.1Paired with
2>/dev/null, the rejection became an empty discovery list with no error β a false "no suites found", the "an empty result cannot distinguish two mechanisms" trap. Collection is now a bash builtin (globstar), which depends on no external binary and behaves identically in every tier.test_runner_does_not_use_find_printfpins it (and strips comment lines first, because the header describes the trap and a naive substring search matched that prose).Measured β both tiers
nix build .#checks.x86_64-linux.nodetestsRESULT: PASS, exit 0bash scripts/run-node-tests.shRESULT: PASS, exit 0Delta accounted exactly: 468 (baseline) + 508 + 21 = 997. No unexplained gap.
Baseline at
origin/main(13bc8bd), sandbox: 468 tests / 14 files /RESULT: PASS.Red/green matrix
scripts/tests/test_run_node_tests_suites.py, base reforigin/main(13bc8bd), measured by swappingorigin/main's runner into place:origin/mainrunnerLabelled honestly in the module docstring:
test_every_test_mjs_directory_is_pinned(the defect in one assertion),test_check_suites_accepts_the_real_repo,test_no_pinned_suite_has_vanishedtest_dl_router_and_browser_ext_are_pinned(pins that the entries cannot be deleted to turn a future red green; the bug never violated it because the entries did not exist)The 529 newly-gated tests are NOT regression coverage for this change. They are pre-existing tests whose value is that they now run at all.
Control pair
The reassuring answer for
test_every_test_mjs_directory_is_pinnedis an empty set β which an unwired harness also produces.test_parsers_are_positive_controlledis the positive control: it asserts both independently-implemented discovery paths (the runner's bash globstar, the test'spathlib.rglob) return non-empty, plausibly-sized sets with real floors, so a silently-broken parser fails loudly instead of passing vacuously.Reported as a pair: 3 suites discovered / 3 pinned / 0 unpinned, not the zero alone.
Files touched
scripts/run-node-tests.shscripts/tests/test_run_node_tests_suites.py(new)flake.nix(thenodetestsheader comment claimed the check covered browser-bridge only β a comment is a claim too)No overlap with
nix/home.nix,scripts/bar-status-poll, orscripts/collector/**source (the browser-ext suite is referenced by path, not edited).π€ Generated with Claude Code