Skip to content

test(gate): run every *.test.mjs suite β€” 529 node tests no gate had ever run - #309

Merged
ZacxDev merged 1 commit into
mainfrom
gate/node-suite-discovery
Aug 3, 2026
Merged

test(gate): run every *.test.mjs suite β€” 529 node tests no gate had ever run#309
ZacxDev merged 1 commit into
mainfrom
gate/node-suite-discovery

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

The defect

scripts/run-node-tests.sh hard-coded its collection to one directory:

FILES=(scripts/browser-bridge/tests/*.test.mjs)

so every other .test.mjs suite in the repo was invisible to the only check that runs node. Measured at origin/main (13bc8bd):

suite tests gated before this PR
scripts/browser-bridge/tests 468 yes
scripts/dl-router/tests 508 no
scripts/collector/browser-ext/tests 21 no

529 tests, ungated since each suite was written. The gate reported RESULT: PASS and a 468-test total the whole time β€” a hard-coded list cannot know what it is not looking at. This is the fourth instance of the shape: #276 (913 pytest tests behind one list entry), #298 (166), #306 (989).

scripts/collector/browser-ext/tests was not in the brief β€” it surfaced from auditing every *.test.mjs in the repo rather than only the directory named. Its test_receiver.py was already gated by run-tests.sh; only its two .mjs files were orphaned.

Why not just add two lines

That leaves the next suite ungated identically. Collection is now discovery (bash globstar over scripts/**/*.test.mjs) plus a two-way pin (SUITES), which fails in both directions:

  • a discovered directory absent from SUITES β†’ FATAL (forces an accounting entry with a measured floor, instead of being swept in under a global total)
  • a pinned suite discovery does not find β†’ FATAL (the suite vanished)

Discovery alone would reintroduce a hole the hard-coded glob did not have: an emptied dl-router/tests would simply collect fewer files and still pass over a global floor.

Each suite also runs in its own node --test invocation with its own TAP summary and its own floor. A single global floor of 970 is fully satisfied by browser-bridge (468) + dl-router (508) with browser-ext's 21 tests entirely gone β€” per-suite floors make that loud.

A portability defect found while building this

The first draft discovered suites with find -printf '%h\n' 2>/dev/null. -printf is a GNU extension, and three different finds are reachable from this repo:

tier find -printf
bash on the dev host busybox (~/.nix-profile/bin/find) rejected
interactive zsh bfs 4.1.1 ok
nix build sandbox GNU findutils ok

Paired with 2>/dev/null, the rejection became an empty discovery list with no error β€” a false "no suites found", the "an empty result cannot distinguish two mechanisms" trap. Collection is now a bash builtin (globstar), which depends on no external binary and behaves identically in every tier. test_runner_does_not_use_find_printf pins it (and strips comment lines first, because the header describes the trap and a naive substring search matched that prose).

Measured β€” both tiers

tier command result
authoritative nix build .#checks.x86_64-linux.nodetests 997 tests / 997 pass / 0 fail, RESULT: PASS, exit 0
dev host bash scripts/run-node-tests.sh 997 tests / 997 pass / 0 fail, RESULT: PASS, exit 0

Delta accounted exactly: 468 (baseline) + 508 + 21 = 997. No unexplained gap.

Baseline at origin/main (13bc8bd), sandbox: 468 tests / 14 files / RESULT: PASS.

Red/green matrix

scripts/tests/test_run_node_tests_suites.py, base ref origin/main (13bc8bd), measured by swapping origin/main's runner into place:

revision result
origin/main runner 10 failed / 0 passed
this branch 0 failed / 10 passed

Labelled honestly in the module docstring:

  • regression coverage β€” test_every_test_mjs_directory_is_pinned (the defect in one assertion), test_check_suites_accepts_the_real_repo, test_no_pinned_suite_has_vanished
  • invariant guard, not regression coverage β€” test_dl_router_and_browser_ext_are_pinned (pins that the entries cannot be deleted to turn a future red green; the bug never violated it because the entries did not exist)
  • reachability/mutation proofs β€” dropping each pinned suite, and pinning a nonexistent one, each asserting the guard goes red naming the directory and its specific reason, so the test cannot be satisfied by a different guard tripping first

The 529 newly-gated tests are NOT regression coverage for this change. They are pre-existing tests whose value is that they now run at all.

Control pair

The reassuring answer for test_every_test_mjs_directory_is_pinned is an empty set β€” which an unwired harness also produces. test_parsers_are_positive_controlled is the positive control: it asserts both independently-implemented discovery paths (the runner's bash globstar, the test's pathlib.rglob) return non-empty, plausibly-sized sets with real floors, so a silently-broken parser fails loudly instead of passing vacuously.

Reported as a pair: 3 suites discovered / 3 pinned / 0 unpinned, not the zero alone.

Files touched

  • scripts/run-node-tests.sh
  • scripts/tests/test_run_node_tests_suites.py (new)
  • flake.nix (the nodetests header comment claimed the check covered browser-bridge only β€” a comment is a claim too)

No overlap with nix/home.nix, scripts/bar-status-poll, or scripts/collector/** source (the browser-ext suite is referenced by path, not edited).

πŸ€– Generated with Claude Code

…ver run

`scripts/run-node-tests.sh` hard-coded its collection to one directory:

    FILES=(scripts/browser-bridge/tests/*.test.mjs)

so every other `.test.mjs` suite in the repo was invisible to the only check
that runs node. Measured at origin/main (13bc8bd):

  * scripts/dl-router/tests              508 tests, never gated
  * scripts/collector/browser-ext/tests   21 tests, never gated

529 tests, ungated since each suite was written. The gate reported
`RESULT: PASS` with a 468-test total the whole time β€” a hard-coded list cannot
know what it is not looking at. Fourth instance of the same shape (#276: 913
pytest tests behind one list entry, #298: 166, #306: 989).

Adding two lines would leave the NEXT suite ungated identically, so collection
is now DISCOVERY (bash globstar over scripts/**/*.test.mjs) plus a TWO-WAY PIN:

  * a discovered directory absent from SUITES  -> FATAL (forces an accounting
    entry with a measured floor, rather than being swept in under the total)
  * a pinned suite discovery does not find     -> FATAL (the suite vanished)

Discovery alone would reintroduce the silent-collapse hole the old hard-coded
glob at least did not have: an emptied dl-router/tests would just collect fewer
files and still pass over a global floor.

Each suite also runs in its OWN `node --test` invocation with its own TAP
summary and its own floor. A single global floor of 970 is fully satisfied by
browser-bridge (468) + dl-router (508) with browser-ext's 21 tests entirely
gone; per-suite floors make that loud.

A PORTABILITY DEFECT found while building this, worth recording because the
first draft shipped it and the harness hid it: discovery used
`find -printf '%h\n' 2>/dev/null`. `-printf` is a GNU extension, and THREE
different `find`s are reachable from this repo β€” busybox under bash
(~/.nix-profile/bin/find, rejects it), bfs 4.1.1 under the interactive zsh, GNU
findutils in the nix sandbox. Paired with `2>/dev/null` the rejection became an
EMPTY discovery list with no error β€” a false "no suites found". Collection is
now a bash builtin (globstar), which depends on no external binary and behaves
identically in every tier; `test_runner_does_not_use_find_printf` pins it.

MEASURED, both tiers:

  nix build .#checks.x86_64-linux.nodetests   997 tests / 997 pass / 0 fail
  bash scripts/run-node-tests.sh (dev host)   997 tests / 997 pass / 0 fail

  468 (baseline, browser-bridge) + 508 + 21 = 997. No unexplained delta.

scripts/tests/test_run_node_tests_suites.py guards the pin, with its regression
and invariant guards labelled honestly in the module docstring, mutation proofs
that the guard goes red naming the offending directory, and a positive control
on both parsers (the reassuring answer here is an empty set, which an unwired
harness also produces).

The 529 newly-gated tests are NOT regression coverage for this change β€” they
are pre-existing tests whose value is that they now run at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ZacxDev

ZacxDev commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Gated on the MERGED tree, not just this branch

Per RULES.md β€” a PR green on its own branch proves nothing about the tree its merge creates. Built integration/309-310 off origin/main (13bc8bd), merged both #309 and #310, and ran the full authoritative gate there:

check result
nix build .#checks.x86_64-linux.pytests 5810 collected / 5809 passed / 1 skipped / 0 failed β€” RESULT: PASS
nix build .#checks.x86_64-linux.nodetests 997 tests / 997 pass / 0 fail (3 suites, 29 files) β€” RESULT: PASS

Combined exit 0.

Delta accounted exactly: 5792 (baseline) + 10 (#309's test_run_node_tests_suites.py) + 8 (#310's test_run_tests_preconditions.py) = 5810. Node: 468 (baseline) + 508 + 21 = 997. No unexplained gap on either.

git merge-tree --write-tree between the two branches exits 0 (branched on the exit code, not a marker grep β€” merge-tree prints only a tree OID on success and emits no conflict markers either way). The branches also share no files: #309 touches scripts/run-node-tests.sh, flake.nix, scripts/tests/test_run_node_tests_suites.py; #310 touches scripts/run-tests.sh, scripts/tests/test_run_tests_preconditions.py. The only semantic coupling is that both add a file under scripts/tests/, which is a HERMETIC_TARGETS entry β€” and that is what the 5810 total confirms.

@ZacxDev
ZacxDev merged commit 08fd649 into main Aug 3, 2026
ZacxDev added a commit that referenced this pull request Aug 3, 2026
… 0 on failure (#312)

#308 asserted that scripts/run-tests.sh "can exit 0 while printing RESULT: FAIL
/ collected=0". That does not reproduce and is structurally impossible on this
revision: `RESULT: FAIL` prints only inside `if [ "$fail" -ne 0 ]`, and the very
next statement is `exit "$fail"`. Measured on 13bc8bd, a run without pytest
importable prints 17 x "could not parse pytest's summary", RESULT: FAIL, and
exits 1 (2 in a second probe). The original report most likely read a status
through a pipeline -- the rc=$? trap RULES.md already names.

The real defect, fixed in #310, is narrower: REQUIRED_TOOLS checks BINARIES via
`command -v`, but pytest is a MODULE, so a missing pytest yielded a diagnostic
blaming pytest's output format instead of a clean precondition failure. It also
asserted `python3` while the runner calls `python`. Bad diagnostics, not a false
green.

This was relayed from a subagent report and repeated to the operator twice
without being tested -- the same class of error as the four corrections the doc
itself records, except this one reached a merged document before anyone checked
it. Left visible as a strikethrough rather than deleted, because the doc's
purpose is recording what was believed and then disproved.

Also updates three items now closed:
  - #309 gated the node suites: 529 ungated tests, not the 508 first reported
    (scripts/collector/browser-ext/tests was ungated too). Node gate 468 -> 997.
  - #311 shipped the telemetry deadman, including the verified property that
    "cannot tell" cannot render as "all healthy" -- deadman returns count=0 for
    both ok and unreachable, so a consumer reading count alone would be wrong.
  - session-create's cause is now known: `session.created` is a bus EVENT TYPE,
    not a plugin hook name, on opencode 1.18.4. Consequence: 2,736 of 2,799
    tool-call rows carry session=''. Not fixed -- that file's last edit killed
    11 hours of telemetry.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant