test(gate): run scripts/dl-router/tests — 989 tests no gate had ever run by ZacxDev · Pull Request #306 · innovation-upstream/devrc

ZacxDev · 2026-08-03T04:13:30Z

scripts/dl-router/tests was absent from scripts/run-tests.sh's HERMETIC_TARGETS
since the suite was written. 989 tests that no gate has ever run. Same
declarations-vs-instances shape as #298 (166 opencode tests) and #276 (913
guard-core tests): one missing line in a target list gating hundreds of instances.

Numbers — BOTH tiers, base ref `4eb5798` (#302)

Baseline re-measured tonight at 4eb5798 itself, not carried over from an earlier ref.

tier	`4eb5798` (base)	`4eb5798` + this PR
sandbox — `nix build .#checks.x86_64-linux.pytests` (authoritative)	4662 collected / 4661 passed / 1 skipped / 0 failed	5658 collected / 5657 passed / 1 skipped / 0 failed
dev host — `scripts/run-tests.sh --set all` under nix-shell	4662 / 4661 / 1 / 0	5658 / 5657 / 1 / 0

The two tiers report the same numbers before and after — deliberately checked,
because the defects this PR fixes are exactly the kind that make them disagree.

+996 accounted for exactly, no silent uncollection:

+991  scripts/dl-router/tests   (989 pre-existing + 2 new harness self-checks)
  +7  scripts/tests/test_runtime_shebangs.py   (new repo-wide guard)
  -2  scripts/collector/opencode/tests         (the directory-scoped scan it replaces)

dl-router collects 991 / skips 0 in both tiers. Nothing is gated, nothing is
skipped, no assertion was weakened. The single skip in the run is the pre-existing
pinned repo-cos live-store drift check.

MIN_TESTS 2850 → 5600. The old floor had drifted to less than half the real
total — a floor under which an entire 989-test suite could have vanished undetected.

What adding the suite surfaced — two SANDBOX-ONLY portability defects

Both are the two-tier hazard from RULES.md, and they point in opposite directions.

1. test_setup_script.py:134 wrote a stub with #!/usr/bin/env bash.
Probed directly: the nix build sandbox has no /usr/bin/env; every NixOS dev
host does. patchShebangs fixes the source tree and cannot touch a file a test
writes while running. Now goes through the helper #298 landed.

🔴 The finding that matters more than the fix: with the bad shebang restored,
all 989 tests still PASSED in the sandbox. The stub could not exec at all, and
the exit 0 cases are precisely the ones asserting that "a brave binary is running"
must not decide anything — so a pgrep that cannot run is indistinguishable from
one answering "no". The harness was inert and nothing in the suite could tell.
So this PR adds test_the_fake_pgrep_actually_execs[0,1] (the +2), which asserts the
stub's exit code directly and makes the harness's own breakage loud in this suite
rather than visible only to the structural scan.

2. test_cli.py / test_server_wiring.py hard-coded port 8799 for their
"sidecar is DOWN / unreachable" assertions. That is a claim about the whole machine,
not about the test. MEASURED on the workbench: an orphaned python3 (pid 2994086,
started Aug 1 13:40, ppid 1, in a stale tmux-spawn scope) was listening on
127.0.0.1:8799, so test_an_unreachable_sidecar_gives_an_actionable_message
reached a real dl-router and failed with sidecar HTTP 409: not_owned_tab.
Ports now come from a closed_port fixture (bind :0, read, release). Failed on the
dev-host tier only — the mirror image of defect 1.

One rule, one place

scripts/collector/opencode/tests/_mockbin.py → scripts/testlib/mockbin.py
(git mv, content unchanged but for the docstring), importable by any suite.
Reused, not reinvented — it still owns the shebang and still raises if a call site
supplies its own.
fix(opencode): tool-call telemetry captured NOTHING — 2,699/2,699 rows read text='unknown' #298's runtime-shebang scan was scoped to one directory, which is exactly why
it could not see the dl-router defect sitting in the repo the whole time. The
scanner moved to scripts/testlib/shebang_scan.py and the guard is now
repo-wide (scripts/tests/test_runtime_shebangs.py), covering every
test_*.py and conftest.py under scripts/. Needles are still assembled from
char codes so the scan cannot match its own source.
The allowlist fails both ways: an unpinned offender fails, and a pin that
matches nothing fails (a stale pin otherwise pre-approves whatever lands at that
path next). 8 entries, each naming how that site avoids /usr/bin/env
(/bin/sh, a resolved which result, or sys.executable). No entry is
/usr/bin/env — the allowlist is not an escape hatch for this hazard.

Verification — red/green matrix

Honest labelling: 989 of the 991 dl-router tests are pre-existing and are NOT
regression coverage for this change. Their value is that they now run at all.

test	at `4eb5798`	at `c76a4b1`	proves
`scripts/dl-router/tests` (989)	never ran — not in the target list	989 pass, both tiers	the suite is gated at all
`test_no_test_writes_a_usr_bin_env_shebang_at_runtime`	RED in the sandbox — reported `test_setup_script.py:134` on real code	green	genuine new coverage; reachable, not hypothetical
`test_the_fake_pgrep_actually_execs[0,1]`	RED in the sandbox (dl-router → 991 collected / 989 passed / 2 failed) with the old shebang restored	green	genuine new coverage
`test_every_allowlist_entry_still_matches_something`	n/a (new)	green — and observed RED during development on a too-narrow pin	the accounting is wired to something
the other 5 guard tests	n/a (new)	green	invariant guards, labelled as such — not regression coverage

Harness validation (RULES.md — validate against a known-bad state; pair every
reassuring zero with a positive control):

Negative control, at the authoritative gate. Reintroduced the exact defect and
re-ran nix build .#checks.x86_64-linux.pytests: RESULT: FAIL, 5658 collected / 5656 passed / 2 failed, failing in both places — scripts/tests (the scan) and
scripts/dl-router/tests (the self-check). Restored → green. The gate can go red.
Positive control for the sandbox claim. Did not assume the sandbox lacks
/usr/bin/env — probed it: a throwaway runCommandLocal printed NO-USR-BIN-ENV.
Positive controls for the zeros. not unpinned and not stale are both empty
lists, indistinguishable from a scan wired to nothing. Two tests feed the same
scan_tree / _matches a case that MUST produce a non-zero count and assert the
number moves (1 on the positive control, 0 under test). A third pins that
SCAN_ROOT actually resolves to >50 files, so the scan cannot pass vacuously on an
empty file set.
Positive control for the port fix. The orphan on :8799 was still listening
at the time of the final run (re-checked, not remembered) — so the dev-host green
is measured against the state that produced the original failure, not after
quietly clearing it. The orphan was left alone; it is not mine to kill.
The two _write_exec allowlist needles were caught being wrong by the stale-pin
test before this was committed.

Merged-tree check

Rebased onto current origin/main (4eb5798, which landed after this work started
and touches test_plugin.py — a file this PR also edits). git merge-tree --write-tree exited 0; the rebase applied clean; both tiers were then re-run on
the merged tree, not on the pre-rebase branch. The final numbers above are from
commit c76a4b1 itself, re-measured after the last amend.

Reported, NOT fixed here

🔴 scripts/dl-router/tests/*.test.mjs — 508 tests, also completely ungated.
scripts/run-node-tests.sh hard-codes FILES=(scripts/browser-bridge/tests/*.test.mjs),
so checks.nodetests has never seen dl-router's 13 .mjs files. Measured: 508
tests, 508 pass, 0 fail. Identical shape to this PR, other language, and it is
the larger remaining hole — the sidecar's routing logic and the extension's cached
fallback are asserted against shared fixture tables, and only half of that pair is
gated. Separate PR; note that node's spec reporter emits ℹ tests 508, not the
# tests TAP line the runner greps for.
No real pre-existing dl-router bugs found. Both defects were test-portability,
not product logic. The 989 tests pass on the code as it stands.
rebase.autoStash appears to be enabled for this repo — git rebase printed
Created autostash, i.e. git reached for the repo-global stash on my behalf.
It was transient and the stack was verified intact afterwards (9 pre-existing
entries, none mine, stash@{0} untouched), but per RULES.md this is the exact
mechanism that let two parallel agents steal each other's work, and the guard: enforce git stash + git clean -f for Claude Code, not just opencode #295 guard
cannot see it because git does it internally. Worth turning off.

Privacy

scripts/dl-router handles a private library and this repo is public. The suite's
fixtures are entirely synthetic by construction (Jane Doe, acme-studio,
example-site.test, tmp_path roots) — no real path, directory name, filename or
route-log content is committed, and none appears in this PR. The only literal path
added anywhere is /bin/sh.

🤖 Generated with Claude Code

`scripts/dl-router/tests` was absent from `scripts/run-tests.sh`'s target list since the suite was written: 989 tests, zero of them gated. Same declarations-vs-instances shape as #298 (166 opencode tests) and #276 (913 guard-core tests) — one missing line in a target list. Adding it surfaced two SANDBOX-ONLY portability defects, both of the kind that are structurally invisible on a dev host: * test_setup_script.py's write_pgrep() wrote a stub with `#!/usr/bin/env bash`. The nix build sandbox has no /usr/bin/env; every NixOS dev host does. Now goes through the helper #298 landed. * test_cli.py / test_server_wiring.py hard-coded port 8799 for their "sidecar is DOWN/unreachable" assertions. That is a claim about the whole machine, not the test — an orphaned python3 (pid 2994086, started Aug 1 13:40, ppid 1) was listening on it and the test reached a REAL dl-router, failing with `sidecar HTTP 409: not_owned_tab`. Ports now come from a `closed_port` fixture. Shared instead of re-derived (RULES.md "One rule, one place"): * `scripts/collector/opencode/tests/_mockbin.py` -> `scripts/testlib/ mockbin.py`, importable by any suite. * #298's runtime-shebang scan was scoped to ONE directory, which is exactly why it could not see the dl-router defect. The scanner moved to `scripts/testlib/shebang_scan.py` and the guard is now REPO-WIDE (`scripts/tests/test_runtime_shebangs.py`) with a pinned allowlist that fails BOTH ways — an unpinned offender and a pin that matches nothing. MIN_TESTS 2850 -> 5600. The old floor had drifted below HALF the real total, so a whole 989-test suite could have vanished underneath it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ed shebang The repo-wide runtime-shebang scanner (#306, scripts/tests/ test_runtime_shebangs.py) failed this file IN THE SANDBOX ONLY. The dev-host run was green because I ran this ONE file, and the scanner lives in another — a per-file run structurally cannot see a repo-wide guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ver run (#309) `scripts/run-node-tests.sh` hard-coded its collection to one directory: FILES=(scripts/browser-bridge/tests/*.test.mjs) so every other `.test.mjs` suite in the repo was invisible to the only check that runs node. Measured at origin/main (13bc8bd): * scripts/dl-router/tests 508 tests, never gated * scripts/collector/browser-ext/tests 21 tests, never gated 529 tests, ungated since each suite was written. The gate reported `RESULT: PASS` with a 468-test total the whole time — a hard-coded list cannot know what it is not looking at. Fourth instance of the same shape (#276: 913 pytest tests behind one list entry, #298: 166, #306: 989). Adding two lines would leave the NEXT suite ungated identically, so collection is now DISCOVERY (bash globstar over scripts/**/*.test.mjs) plus a TWO-WAY PIN: * a discovered directory absent from SUITES -> FATAL (forces an accounting entry with a measured floor, rather than being swept in under the total) * a pinned suite discovery does not find -> FATAL (the suite vanished) Discovery alone would reintroduce the silent-collapse hole the old hard-coded glob at least did not have: an emptied dl-router/tests would just collect fewer files and still pass over a global floor. Each suite also runs in its OWN `node --test` invocation with its own TAP summary and its own floor. A single global floor of 970 is fully satisfied by browser-bridge (468) + dl-router (508) with browser-ext's 21 tests entirely gone; per-suite floors make that loud. A PORTABILITY DEFECT found while building this, worth recording because the first draft shipped it and the harness hid it: discovery used `find -printf '%h\n' 2>/dev/null`. `-printf` is a GNU extension, and THREE different `find`s are reachable from this repo — busybox under bash (~/.nix-profile/bin/find, rejects it), bfs 4.1.1 under the interactive zsh, GNU findutils in the nix sandbox. Paired with `2>/dev/null` the rejection became an EMPTY discovery list with no error — a false "no suites found". Collection is now a bash builtin (globstar), which depends on no external binary and behaves identically in every tier; `test_runner_does_not_use_find_printf` pins it. MEASURED, both tiers: nix build .#checks.x86_64-linux.nodetests 997 tests / 997 pass / 0 fail bash scripts/run-node-tests.sh (dev host) 997 tests / 997 pass / 0 fail 468 (baseline, browser-bridge) + 508 + 21 = 997. No unexplained delta. scripts/tests/test_run_node_tests_suites.py guards the pin, with its regression and invariant guards labelled honestly in the module docstring, mutation proofs that the guard goes red naming the offending directory, and a positive control on both parsers (the reassuring answer here is an empty set, which an unwired harness also produces). The 529 newly-gated tests are NOT regression coverage for this change — they are pre-existing tests whose value is that they now run at all. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…dition (#310) * fix(gate): run-tests.sh could not check its own most important precondition Two measured holes, plus an honest correction to a third. 1. REQUIRED_TOOLS could not express the precondition that matters most. It is a list of BINARIES checked with `command -v`. pytest is not a binary this runner calls — it is a MODULE (`python -m pytest`) — so the one guard whose entire job is "the thing that runs the tests is present" was structurally unable to check pytest. It also asserted `python3` while the runner actually invokes `python`. MEASURED on the dev host with every REQUIRED_TOOLS binary present but no pytest importable: all 17 targets printed run-tests: ERROR — could not parse pytest's summary for <dir>. and the run ended `TOTAL collected=0 … RESULT: FAIL`, exit 1. CORRECTION, stated because it changes the severity: the gate did NOT go green, and the briefed "reports per-target PASS with collected=0" does not exist on this revision — GUARD 4 (unparseable summary) and GUARD 3 (the collected floor) both fire. The defect is DIAGNOSTIC: seventeen copies of a message blaming pytest's OUTPUT FORMAT for a missing dependency, pointing at the wrong subsystem. That is the #276 shape — a real finding that reads like an environment fault. Now one named FATAL, exit 2, before any suite runs. 2. `declare -a RESULTS` / `declare -a SKIP_LINES` leave the arrays DECLARED BUT UNSET. Under `set -u` the first `${#arr[@]}` on a still-empty array aborts the command with "unbound variable" (measured, bash 5.3.15). With zero skips this printed a raw scripts/run-tests.sh: line 479: SKIP_LINES: unbound variable where GUARD 2's skip list belonged, and the unpinned-skip loop below it never executed. No `set -e`, so the script continued and the skip-TOTAL accounting still fired — the damage was confined to the DIAGNOSTIC path, at exactly the moment someone is reading why the gate is red. Fixed with `NAME=()`. 3. "printed RESULT: FAIL and exited 0" did NOT reproduce. Measured: exit 1. The structure forbids it — `RESULT: FAIL` is printed only when `fail != 0` and the next statement is `exit "$fail"`, whose only non-zero value is 1. The likely origin is reading the status through a pipeline (`| tail`), which yields the last command's status rather than the runner's. Pinned end-to-end anyway by `test_a_failing_run_never_exits_zero`, which forces a red run and asserts the two can never disagree — and pytest.fail()s if it could not force one, so it cannot pass vacuously. Known-bad states, each proven LOUD by before/after rather than by reading code: no pytest module before: 17 misleading errors, exit 1 after: one named FATAL, exit 2, no suite started empty target dir fails on the per-directory `collected 0 tests` floor typo'd target GUARD 5 aborts naming the entry and saying "does not exist" The last two were already guarded before this PR; nothing had ever proven those paths could fire, so they are labelled REACHABILITY proofs, not regression coverage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): write the python shim via testlib.mockbin, not a hand-rolled shebang The repo-wide runtime-shebang scanner (#306, scripts/tests/ test_runtime_shebangs.py) failed this file IN THE SANDBOX ONLY. The dev-host run was green because I ran this ONE file, and the scanner lives in another — a per-file run structurally cannot see a repo-wide guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

ZacxDev merged commit a5fae58 into main Aug 3, 2026

ZacxDev mentioned this pull request Aug 3, 2026

test(gate): run every *.test.mjs suite — 529 node tests no gate had ever run #309

Merged

ZacxDev mentioned this pull request Aug 3, 2026

fix(gate): run-tests.sh could not check its own most important precondition #310

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(gate): run scripts/dl-router/tests — 989 tests no gate had ever run - #306

test(gate): run scripts/dl-router/tests — 989 tests no gate had ever run#306
ZacxDev merged 1 commit into
mainfrom
gate-dl-router-tests

ZacxDev commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ZacxDev commented Aug 3, 2026

Numbers — BOTH tiers, base ref 4eb5798 (#302)

What adding the suite surfaced — two SANDBOX-ONLY portability defects

One rule, one place

Verification — red/green matrix

Merged-tree check

Reported, NOT fixed here

Privacy

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Numbers — BOTH tiers, base ref `4eb5798` (#302)