Skip to content

fix(gate): accept a FILE target in run-tests.sh β€” unbreaks main, 913 tests were not running - #289

Merged
ZacxDev merged 1 commit into
mainfrom
fix/run-tests-file-targets
Aug 2, 2026
Merged

fix(gate): accept a FILE target in run-tests.sh β€” unbreaks main, 913 tests were not running#289
ZacxDev merged 1 commit into
mainfrom
fix/run-tests-file-targets

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

main's pytest gate is RED, and 913 tests have not been running. This unbreaks it and pins the failure mode so it cannot recur silently.

The bug

#276 (e21985a) added a file to the runner's target list:

scripts/claude-hooks/tests/test_guard_core.py

run_pytest() guarded with if [ ! -d "$d" ], which rejects a regular file:

FAIL  scripts/claude-hooks/tests/test_guard_core.py (missing directory)
RESULT: FAIL

Three consequences, all measured:

  1. the gate has been red on main ever since;
  2. the 913 tests in that file never ran (913 passed in 1.38s when invoked directly);
  3. the wording made it worse β€” "missing directory" reads as an environment fault, so the natural conclusion was "the new test(gate): count skips in the pytest gate β€” tool precondition, pinned skip set, collected-test floorsΒ #284 gate is noisy", not "a real target is being dropped". It landed after test(gate): count skips in the pytest gate β€” tool precondition, pinned skip set, collected-test floorsΒ #284 made this gate loud, which is exactly the reading that trains people to click through.

The fix

  • run_pytest() checks existence with -e and branches on -d/-f only to choose the message β€” python -m pytest accepts either. The per-target parse and attribution are untouched, so test(gate): count skips in the pytest gate β€” tool precondition, pinned skip set, collected-test floorsΒ #284's guarantee (a per-suite collapse stays attributable and cannot hide in the total) is preserved: a file target is reported under its own name exactly like a directory.
  • HERMETIC_DIRS β†’ HERMETIC_TARGETS (and DEVHOST_DIRS/DIRS likewise). The name asserted something false about its own contents, which is how the guard and the list drifted apart in the first place.
  • The header comment now states that entries may be files, and documents the new guard. (Nothing else in the repo referenced the old names β€” checked.)

New GUARD 5 β€” validate the whole list up front, naming every bad entry

The failure mode was "an entry was added, the runner silently rejected it, and the gate failed for a reason that looked environmental." A typo, a moved suite, an unexpanded glob and a legitimate file all failed the same indistinguishable way, and only once the runner reached that suite β€” minutes into the output.

GUARD 5 validates every entry before anything runs and reports each bad one with a reason. --check-targets runs just that guard (no pytest, no tool precondition), which is what makes it testable in milliseconds instead of minutes.

$ bash scripts/run-tests.sh --check-targets .
run-tests: all 15 hermetic target(s) resolve.
  dir   scripts/tests
  …
  file  scripts/claude-hooks/tests/test_guard_core.py

A claim I got wrong and corrected mid-PR: I first wrote that bash does not expand globs inside an array literal. My own reachability test disproved it β€” injecting scripts/tests/test_*.py took the list from 15 to 32 entries. So a matching glob expands harmlessly; only an unmatched one survives as a literal *. The comment and the test case now say that instead of the opposite. (claude/RULES.md: "a comment is a claim too".)

Measurements

Controlled A/B β€” same tree, only the runner differs (old runner extracted from origin/main and pointed at this worktree), so the delta is attributable to the fix alone:

collected test_guard_core.py
before (pre-fix runner) 3385 FAIL … (missing directory)
after (this branch) 4306 PASS … (collected=913 passed=913 skipped=0)

Ξ” = +921 = 913 recovered + 8 from the new test file. The count is the proof, not the exit status: accepting the target without executing it would look identical from outside the gate.

Red β†’ green matrix

New file scripts/tests/test_run_tests_targets.py (8 tests):

runner under test result
origin/main's run-tests.sh (pre-fix, base 8b0931a) 8 failed in 0.06s
this branch (ef5306a) 8 passed in 0.06s

A capability check (_require_check_targets) makes the pre-fix state fail fast and by name. Without it the pre-fix behaviour is red for the wrong reason: the old arg parser has no --check-targets case, so the flag falls through to *) ROOT="$1", the trailing ROOT overwrites it, cd succeeds and the runner executes the entire suite β€” measured, each invocation blew the 120s subprocess timeout, turning 8 tests into 8 full suite runs.

Reachability, against the REAL list

Injected a bogus entry into the actual HERMETIC_TARGETS:

run-tests: FATAL β€” 1 unusable entr(ies) in the hermetic target list:
    scripts/bogus/reachability-probe/tests  β€” does not exist (typo, or the suite moved?)

exit 2, and the regression test went red as 2 failed, 6 passed β€” the two list-acceptance tests specifically, not a blanket collapse, so the guard fails for its own reason. Reverted and re-confirmed 8 passed; the revert was verified byte-identical against the staged blob, not assumed.

The test file labels which of its tests are regression coverage and which is an invariant guard (test_hermetic_list_still_names_the_guard_core_file pins that nobody "fixes" a future failure by deleting the entry β€” the runner's own error message warns against exactly that).

πŸ”΄ A second #276 regression β€” NOT fixed here

#276 also moved SECRET_PATTERNS out of bash-guard.py into guard_core.py. session_insight's test_patterns_cover_bash_guard still ast.parses ~/.claude/hooks/bash-guard.py, gets [], and fails:

AssertionError: could not parse SECRET_PATTERNS from bash-guard.py

It fails on any host where the hook is deployed (so it breaks the pre-push tier) and skips in the nix sandbox (synthetic $HOME β†’ file absent β†’ pinned skip), so the flake gate never sees it. It passed at 16:45 today and failed at 17:03 on the same tree β€” the host's deployed hook changed under a home-manager switch mid-session.

Different file, different failure mode, and it needs a judgement call about what the test should assert now β€” so I deliberately did not bundle it. The fix is a one-line repoint of _BASH_GUARD to guard_core.py. Happy to take it as the next PR.

Process note

My own … ; echo "EXIT=$?" wrapper reported exit 0 for a run whose summary said RESULT: FAIL β€” $? read the echo, not the pipeline. Same shape as the rc=$? case documented in #287. It is why every number above is a counted total rather than an exit status.

I also discarded one contaminated baseline: I removed a worktree while a measurement was still running against it, which produced a plausible-looking collected=1133 with a cascade of "missing directory" lines that were my own doing. Caught by reading the log rather than quoting the total.

Verification

  • node: # tests 468 / # pass 468 / # fail 0 (glob form)
  • nix flake check β€” result reported in a follow-up comment.
  • shellcheck on run-tests.sh: clean apart from a pre-existing SC2155 on a line this PR does not touch.

Do not merge without the gate confirmation comment.

πŸ€– Generated with Claude Code

…tests were not running

#276 added a FILE to the runner's target list:

    scripts/claude-hooks/tests/test_guard_core.py

and `run_pytest()` guarded with `if [ ! -d "$d" ]`, which rejects a regular
file. The gate reported

    FAIL  scripts/claude-hooks/tests/test_guard_core.py (missing directory)

so the pytest check has been RED on main ever since, and the 913 tests in
that file never ran. The wording made it worse: "missing directory" reads
as an environment fault, so the natural conclusion was "the new #284 gate
is noisy" rather than "a real target is being dropped".

THE FIX
  * `run_pytest()` checks existence with `-e` and branches on `-d`/`-f`
    only to pick the message β€” `python -m pytest` accepts either. Per-target
    parse and attribution are untouched, so #284's guarantee that a
    per-suite collapse stays attributable is preserved.
  * `HERMETIC_DIRS` -> `HERMETIC_TARGETS` (`DEVHOST_DIRS`/`DIRS` likewise):
    the name asserted something false about its own contents.
  * Header comment now says entries may be files, and documents GUARD 5.

NEW GUARD 5 β€” validate the WHOLE list up front, naming every bad entry
  A typo, a moved suite, an unexpanded glob and a file all failed the same
  indistinguishable way, and only once the runner reached that suite. GUARD 5
  checks every entry before anything runs and reports each one with a reason.
  `--check-targets` runs just that guard (no pytest, no tool precondition),
  which is what makes it testable in milliseconds.

  Corrected a claim while writing it: bash DOES expand globs inside an array
  literal (measured β€” injecting `scripts/tests/test_*.py` took the list from
  15 to 32 entries), so only an UNMATCHED glob survives as a literal. The
  comment and the test case now say that instead of the opposite.

MEASUREMENTS (same tree, only the runner differs)
  before  collected=3385   FAIL … test_guard_core.py (missing directory)
  after   collected=4306   PASS … test_guard_core.py (collected=913 passed=913)
  delta   +921 = 913 recovered + 8 from the new test file

  Accepting the target without executing it would look identical from
  outside the gate, so the count is the proof β€” not the exit status.

TESTS (scripts/tests/test_run_tests_targets.py)
  red at origin/main's run-tests.sh: 8 failed in 0.06s
  green at HEAD:                     8 passed in 0.06s

  Reachability, on the REAL list: injecting a bogus entry makes GUARD 5
  exit 2 naming it, and exactly the two list-acceptance tests go red while
  the other six still pass β€” attributable, not a blanket collapse.

  The file labels which of its tests are regression coverage and which is
  an invariant guard, per claude/RULES.md.

NOT fixed here, filed separately: #276 also moved SECRET_PATTERNS out of
bash-guard.py into guard_core.py, so session_insight's
test_patterns_cover_bash_guard parses [] and FAILS on any host where the
hook is deployed. It SKIPS in the nix sandbox, so the flake gate never
sees it. Different file, different failure mode β€” not bundled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ZacxDev
ZacxDev merged commit ca088e7 into main Aug 2, 2026
@ZacxDev
ZacxDev deleted the fix/run-tests-file-targets branch August 2, 2026 22:30
ZacxDev added a commit that referenced this pull request Aug 2, 2026
…s-vs-instances instances

Both measured while unbreaking the pytest gate (#289).

TWO TIERS. 'Gate on the merged tree' extended to environments. The same
suite runs in the nix sandbox and on a dev host, and each environment
silently decides which tests execute β€” so a defect can be permanently
unobservable in the tier you happen to read. #276 shipped THREE
regressions that masked each other:

  (1) a FILE in run-tests.sh's target list rejected by `[ ! -d ]`
      -> gate red, 913 tests never ran
  (2) SECRET_PATTERNS moved to guard_core.py -> the drift test parses []
      and FAILS on a host with the hook deployed, SKIPS in the sandbox
  (3) 10 nix-instantiate tests pytest.fail() without the binary
      -> FAIL in the sandbox, pass on every dev host

(2) and (3) are exact complements; both hid behind (1)'s red.

DECLARATIONS VS INSTANCES gets its third instance in one session:
2 skipif decorators -> 123 tests; 1 list entry -> 913 tests;
1 nix_eval() helper -> 10 parametrized tests. Three is a pattern.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev added a commit that referenced this pull request Aug 2, 2026
…un in the sandbox (#290)

scripts/tests/test_opencode_config.py's `nix_eval()` shells out to
`nix-instantiate --eval` to pin the GENERATED handle values from
nix/agent-handles.nix, and calls pytest.fail() β€” NOT skip β€” when the
binary is absent, deliberately: 'a skip here is how a wrong kubeconfig
path ships'. `nix` was not in the check's nativeBuildInputs, so all 10
test_handles_resolve_to_the_exact_expected_paths[...] cases failed in the
nix sandbox and passed on every dev host.

This was INVISIBLE until #289: it hid behind the gate being red for an
unrelated reason. Its sibling defect is the exact complement β€”
session_insight's test_patterns_cover_bash_guard fails only where
~/.claude/hooks/bash-guard.py is DEPLOYED and skips in the sandbox. Both
came from #276.

MEASURED, not predicted (the pure-eval argument was a prediction; this is
the check's own output):

  scripts/tests before   collected=959 passed=949 failed=10
  scripts/tests after    collected=959 passed=959 failed=0
  TOTAL                  collected=4353 passed=4351 skipped=2 failed=0
  skips                  pinned 2, observed 2
  nodetests              files=14 tests=468 pass=468 fail=0
  nix flake check        all checks passed

Identical collected count with failures to zero β€” the 10 PASS rather than
merely stopping to fail. The 10-failure diagnosis was confirmed first by a
positive control on a dev host: stripping PATH to python alone reproduced
exactly '10 failed, 455 passed' in that file.

COST, stated as deliberately as the nodejs one above it: +30 store paths,
22 of them the nix closure (nix-{util,store,expr,main,flake,fetchers,cmd}
incl. -dev outputs, plus boost and libarchive), and a nixpkgs bump moving
`nix` now invalidates this check's cache β€” the same trade already
accepted for nodejs. Bought: 10 tests that structurally cannot run
without it, pinning that the handles every agent shell exports resolve to
exact paths. Rejected DEVHOST_TARGETS as the alternative: it keeps the
closure small but weakens the pin to 'runs only where someone remembers
to run it', which is the failure mode this whole area keeps hitting.

Also adds nix-instantiate to REQUIRED_TOOLS so a missing binary is ONE
named precondition failure instead of 10 unexplained assertion failures
deep in the run β€” that guard's entire purpose.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev added a commit that referenced this pull request Aug 2, 2026
…ations-vs-instances + output-format class, CLAUDE.md CI/byte facts, #273 tab discard, clawgate check-chart) (#287)

* docs: record what 2026-08-02 measured (RULES, CLAUDE.md, tab discard, clawgate)

RULES.md β€” three additions, all from measured ground cases:

* Verify the CONSUMER is running your artifact, not that the deploy
  reported success. `ship.sh` reported "VERIFIED β€” on branch main at
  origin/main + switched" while the browser-bridge unit was crash-looping
  on `OSError: [Errno 98] Address already in use`; an orphaned process
  from the previous day (Aug 1 16:18, in NO systemd cgroup) held
  127.0.0.1:8788 and served the OLD server.py. The converge check
  verified branch + switch and structurally could not see that the
  service never started. Recipe (`ss -lptn` -> `/proc/<pid>/cgroup` ->
  `systemctl show -p MainPID`) was executed live before being written.

* A count of DECLARATIONS is not a count of INSTANCES. A grep of
  `skipif` decorators found "2 node-related skips"; the two decorators
  gated 123 tests (initiatives: 660 passed/123 skipped sandboxed vs
  783 passed/0 skipped with node). A 60x sizing error.

* Sharpen the harness bullet from a MANIFESTATION to a CLASS: when you
  parse a tool's OUTPUT, its format is an unpinned dependency, and "no
  matches" means "possibly the wrong pattern". The file already named
  `diff`'s unified default; that rule was READ this session and the trap
  was hit anyway in three new shapes (a false CLEAN over a 1,445-byte
  difference that only `cmp` caught; node 24's reporter change emptying
  a `^# (tests|pass|fail)` grep; `rc=$?` reading `echo`'s status).

CLAUDE.md β€” correct stale facts in the browser-bridge bullet:

* Drop the hand-pinned "281 B free today". test_skill_size.py owns
  MAX_BYTES/MIN_HEADROOM_BYTES (the floor was raised to 250 in #275);
  point at it instead of re-pinning a figure that rots.
* Record that CI now gates BOTH suites β€” nodetests (#280) and the
  pytest gate's silent-coverage-collapse guards (#284) β€” and the
  headline outcome, skips 125 -> 2.
* Extension 0.7.1; `nav`/`open` accept `--wake[=MS]`.

reference/tabs-instances.md β€” record the #273 measurement: a forced
discard assigns a NEW tabId and releases ownership (484065264 ->
484065273, ownedTabId None), so the stale-documentEmulation hazard is
not reachable on this Chromium. Scope stated honestly (one build, one
profile, one mechanism; auto-discard not exercised; onReplaced inferred,
not observed) and the load-bearing caveat kept: the safety is a property
of the BROWSER, not of the bridge. Placed in reference/, not the
byte-capped core β€” SKILL.md is unchanged at 11,845 B.

clawgate/SKILL.md β€” live version 0.7.82 (embedded kubeclaw chart 0.7.1,
so the pending re-sync note is resolved); "derive from the LIVE pin"
stays primary. Adds the `make check-chart` hazard: it depends on
`sync-chart`, which rsyncs from ~/workspace/kubeclaw β€” that clone sat at
0.3.14 against a vendored 0.7.1, so running it would have clobbered the
deployed chart and reported a false failure. Fetch + `merge --ff-only`
that clone first.

Docs-only. No source file touched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(rules): two-tier suites must be green in BOTH; three declarations-vs-instances instances

Both measured while unbreaking the pytest gate (#289).

TWO TIERS. 'Gate on the merged tree' extended to environments. The same
suite runs in the nix sandbox and on a dev host, and each environment
silently decides which tests execute β€” so a defect can be permanently
unobservable in the tier you happen to read. #276 shipped THREE
regressions that masked each other:

  (1) a FILE in run-tests.sh's target list rejected by `[ ! -d ]`
      -> gate red, 913 tests never ran
  (2) SECRET_PATTERNS moved to guard_core.py -> the drift test parses []
      and FAILS on a host with the hook deployed, SKIPS in the sandbox
  (3) 10 nix-instantiate tests pytest.fail() without the binary
      -> FAIL in the sandbox, pass on every dev host

(2) and (3) are exact complements; both hid behind (1)'s red.

DECLARATIONS VS INSTANCES gets its third instance in one session:
2 skipif decorators -> 123 tests; 1 list entry -> 913 tests;
1 nix_eval() helper -> 10 parametrized tests. Three is a pattern.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant