fix(gate): accept a FILE target in run-tests.sh β unbreaks main, 913 tests were not running - #289
Merged
Merged
Conversation
β¦tests were not running #276 added a FILE to the runner's target list: scripts/claude-hooks/tests/test_guard_core.py and `run_pytest()` guarded with `if [ ! -d "$d" ]`, which rejects a regular file. The gate reported FAIL scripts/claude-hooks/tests/test_guard_core.py (missing directory) so the pytest check has been RED on main ever since, and the 913 tests in that file never ran. The wording made it worse: "missing directory" reads as an environment fault, so the natural conclusion was "the new #284 gate is noisy" rather than "a real target is being dropped". THE FIX * `run_pytest()` checks existence with `-e` and branches on `-d`/`-f` only to pick the message β `python -m pytest` accepts either. Per-target parse and attribution are untouched, so #284's guarantee that a per-suite collapse stays attributable is preserved. * `HERMETIC_DIRS` -> `HERMETIC_TARGETS` (`DEVHOST_DIRS`/`DIRS` likewise): the name asserted something false about its own contents. * Header comment now says entries may be files, and documents GUARD 5. NEW GUARD 5 β validate the WHOLE list up front, naming every bad entry A typo, a moved suite, an unexpanded glob and a file all failed the same indistinguishable way, and only once the runner reached that suite. GUARD 5 checks every entry before anything runs and reports each one with a reason. `--check-targets` runs just that guard (no pytest, no tool precondition), which is what makes it testable in milliseconds. Corrected a claim while writing it: bash DOES expand globs inside an array literal (measured β injecting `scripts/tests/test_*.py` took the list from 15 to 32 entries), so only an UNMATCHED glob survives as a literal. The comment and the test case now say that instead of the opposite. MEASUREMENTS (same tree, only the runner differs) before collected=3385 FAIL β¦ test_guard_core.py (missing directory) after collected=4306 PASS β¦ test_guard_core.py (collected=913 passed=913) delta +921 = 913 recovered + 8 from the new test file Accepting the target without executing it would look identical from outside the gate, so the count is the proof β not the exit status. TESTS (scripts/tests/test_run_tests_targets.py) red at origin/main's run-tests.sh: 8 failed in 0.06s green at HEAD: 8 passed in 0.06s Reachability, on the REAL list: injecting a bogus entry makes GUARD 5 exit 2 naming it, and exactly the two list-acceptance tests go red while the other six still pass β attributable, not a blanket collapse. The file labels which of its tests are regression coverage and which is an invariant guard, per claude/RULES.md. NOT fixed here, filed separately: #276 also moved SECRET_PATTERNS out of bash-guard.py into guard_core.py, so session_insight's test_patterns_cover_bash_guard parses [] and FAILS on any host where the hook is deployed. It SKIPS in the nix sandbox, so the flake gate never sees it. Different file, different failure mode β not bundled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev
added a commit
that referenced
this pull request
Aug 2, 2026
β¦s-vs-instances instances Both measured while unbreaking the pytest gate (#289). TWO TIERS. 'Gate on the merged tree' extended to environments. The same suite runs in the nix sandbox and on a dev host, and each environment silently decides which tests execute β so a defect can be permanently unobservable in the tier you happen to read. #276 shipped THREE regressions that masked each other: (1) a FILE in run-tests.sh's target list rejected by `[ ! -d ]` -> gate red, 913 tests never ran (2) SECRET_PATTERNS moved to guard_core.py -> the drift test parses [] and FAILS on a host with the hook deployed, SKIPS in the sandbox (3) 10 nix-instantiate tests pytest.fail() without the binary -> FAIL in the sandbox, pass on every dev host (2) and (3) are exact complements; both hid behind (1)'s red. DECLARATIONS VS INSTANCES gets its third instance in one session: 2 skipif decorators -> 123 tests; 1 list entry -> 913 tests; 1 nix_eval() helper -> 10 parametrized tests. Three is a pattern. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 2, 2026
Merged
ZacxDev
added a commit
that referenced
this pull request
Aug 2, 2026
β¦un in the sandbox (#290) scripts/tests/test_opencode_config.py's `nix_eval()` shells out to `nix-instantiate --eval` to pin the GENERATED handle values from nix/agent-handles.nix, and calls pytest.fail() β NOT skip β when the binary is absent, deliberately: 'a skip here is how a wrong kubeconfig path ships'. `nix` was not in the check's nativeBuildInputs, so all 10 test_handles_resolve_to_the_exact_expected_paths[...] cases failed in the nix sandbox and passed on every dev host. This was INVISIBLE until #289: it hid behind the gate being red for an unrelated reason. Its sibling defect is the exact complement β session_insight's test_patterns_cover_bash_guard fails only where ~/.claude/hooks/bash-guard.py is DEPLOYED and skips in the sandbox. Both came from #276. MEASURED, not predicted (the pure-eval argument was a prediction; this is the check's own output): scripts/tests before collected=959 passed=949 failed=10 scripts/tests after collected=959 passed=959 failed=0 TOTAL collected=4353 passed=4351 skipped=2 failed=0 skips pinned 2, observed 2 nodetests files=14 tests=468 pass=468 fail=0 nix flake check all checks passed Identical collected count with failures to zero β the 10 PASS rather than merely stopping to fail. The 10-failure diagnosis was confirmed first by a positive control on a dev host: stripping PATH to python alone reproduced exactly '10 failed, 455 passed' in that file. COST, stated as deliberately as the nodejs one above it: +30 store paths, 22 of them the nix closure (nix-{util,store,expr,main,flake,fetchers,cmd} incl. -dev outputs, plus boost and libarchive), and a nixpkgs bump moving `nix` now invalidates this check's cache β the same trade already accepted for nodejs. Bought: 10 tests that structurally cannot run without it, pinning that the handles every agent shell exports resolve to exact paths. Rejected DEVHOST_TARGETS as the alternative: it keeps the closure small but weakens the pin to 'runs only where someone remembers to run it', which is the failure mode this whole area keeps hitting. Also adds nix-instantiate to REQUIRED_TOOLS so a missing binary is ONE named precondition failure instead of 10 unexplained assertion failures deep in the run β that guard's entire purpose. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev
added a commit
that referenced
this pull request
Aug 2, 2026
β¦ations-vs-instances + output-format class, CLAUDE.md CI/byte facts, #273 tab discard, clawgate check-chart) (#287) * docs: record what 2026-08-02 measured (RULES, CLAUDE.md, tab discard, clawgate) RULES.md β three additions, all from measured ground cases: * Verify the CONSUMER is running your artifact, not that the deploy reported success. `ship.sh` reported "VERIFIED β on branch main at origin/main + switched" while the browser-bridge unit was crash-looping on `OSError: [Errno 98] Address already in use`; an orphaned process from the previous day (Aug 1 16:18, in NO systemd cgroup) held 127.0.0.1:8788 and served the OLD server.py. The converge check verified branch + switch and structurally could not see that the service never started. Recipe (`ss -lptn` -> `/proc/<pid>/cgroup` -> `systemctl show -p MainPID`) was executed live before being written. * A count of DECLARATIONS is not a count of INSTANCES. A grep of `skipif` decorators found "2 node-related skips"; the two decorators gated 123 tests (initiatives: 660 passed/123 skipped sandboxed vs 783 passed/0 skipped with node). A 60x sizing error. * Sharpen the harness bullet from a MANIFESTATION to a CLASS: when you parse a tool's OUTPUT, its format is an unpinned dependency, and "no matches" means "possibly the wrong pattern". The file already named `diff`'s unified default; that rule was READ this session and the trap was hit anyway in three new shapes (a false CLEAN over a 1,445-byte difference that only `cmp` caught; node 24's reporter change emptying a `^# (tests|pass|fail)` grep; `rc=$?` reading `echo`'s status). CLAUDE.md β correct stale facts in the browser-bridge bullet: * Drop the hand-pinned "281 B free today". test_skill_size.py owns MAX_BYTES/MIN_HEADROOM_BYTES (the floor was raised to 250 in #275); point at it instead of re-pinning a figure that rots. * Record that CI now gates BOTH suites β nodetests (#280) and the pytest gate's silent-coverage-collapse guards (#284) β and the headline outcome, skips 125 -> 2. * Extension 0.7.1; `nav`/`open` accept `--wake[=MS]`. reference/tabs-instances.md β record the #273 measurement: a forced discard assigns a NEW tabId and releases ownership (484065264 -> 484065273, ownedTabId None), so the stale-documentEmulation hazard is not reachable on this Chromium. Scope stated honestly (one build, one profile, one mechanism; auto-discard not exercised; onReplaced inferred, not observed) and the load-bearing caveat kept: the safety is a property of the BROWSER, not of the bridge. Placed in reference/, not the byte-capped core β SKILL.md is unchanged at 11,845 B. clawgate/SKILL.md β live version 0.7.82 (embedded kubeclaw chart 0.7.1, so the pending re-sync note is resolved); "derive from the LIVE pin" stays primary. Adds the `make check-chart` hazard: it depends on `sync-chart`, which rsyncs from ~/workspace/kubeclaw β that clone sat at 0.3.14 against a vendored 0.7.1, so running it would have clobbered the deployed chart and reported a false failure. Fetch + `merge --ff-only` that clone first. Docs-only. No source file touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(rules): two-tier suites must be green in BOTH; three declarations-vs-instances instances Both measured while unbreaking the pytest gate (#289). TWO TIERS. 'Gate on the merged tree' extended to environments. The same suite runs in the nix sandbox and on a dev host, and each environment silently decides which tests execute β so a defect can be permanently unobservable in the tier you happen to read. #276 shipped THREE regressions that masked each other: (1) a FILE in run-tests.sh's target list rejected by `[ ! -d ]` -> gate red, 913 tests never ran (2) SECRET_PATTERNS moved to guard_core.py -> the drift test parses [] and FAILS on a host with the hook deployed, SKIPS in the sandbox (3) 10 nix-instantiate tests pytest.fail() without the binary -> FAIL in the sandbox, pass on every dev host (2) and (3) are exact complements; both hid behind (1)'s red. DECLARATIONS VS INSTANCES gets its third instance in one session: 2 skipif decorators -> 123 tests; 1 list entry -> 913 tests; 1 nix_eval() helper -> 10 parametrized tests. Three is a pattern. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
main's pytest gate is RED, and 913 tests have not been running. This unbreaks it and pins the failure mode so it cannot recur silently.The bug
#276 (
e21985a) added a file to the runner's target list:run_pytest()guarded withif [ ! -d "$d" ], which rejects a regular file:Three consequences, all measured:
mainever since;913 passed in 1.38swhen invoked directly);The fix
run_pytest()checks existence with-eand branches on-d/-fonly to choose the message βpython -m pytestaccepts either. The per-target parse and attribution are untouched, so test(gate): count skips in the pytest gate β tool precondition, pinned skip set, collected-test floorsΒ #284's guarantee (a per-suite collapse stays attributable and cannot hide in the total) is preserved: a file target is reported under its own name exactly like a directory.HERMETIC_DIRSβHERMETIC_TARGETS(andDEVHOST_DIRS/DIRSlikewise). The name asserted something false about its own contents, which is how the guard and the list drifted apart in the first place.New GUARD 5 β validate the whole list up front, naming every bad entry
The failure mode was "an entry was added, the runner silently rejected it, and the gate failed for a reason that looked environmental." A typo, a moved suite, an unexpanded glob and a legitimate file all failed the same indistinguishable way, and only once the runner reached that suite β minutes into the output.
GUARD 5 validates every entry before anything runs and reports each bad one with a reason.
--check-targetsruns just that guard (no pytest, no tool precondition), which is what makes it testable in milliseconds instead of minutes.A claim I got wrong and corrected mid-PR: I first wrote that bash does not expand globs inside an array literal. My own reachability test disproved it β injecting
scripts/tests/test_*.pytook the list from 15 to 32 entries. So a matching glob expands harmlessly; only an unmatched one survives as a literal*. The comment and the test case now say that instead of the opposite. (claude/RULES.md: "a comment is a claim too".)Measurements
Controlled A/B β same tree, only the runner differs (old runner extracted from
origin/mainand pointed at this worktree), so the delta is attributable to the fix alone:test_guard_core.pyFAIL β¦ (missing directory)PASS β¦ (collected=913 passed=913 skipped=0)Ξ = +921 = 913 recovered + 8 from the new test file. The count is the proof, not the exit status: accepting the target without executing it would look identical from outside the gate.
Red β green matrix
New file
scripts/tests/test_run_tests_targets.py(8 tests):origin/main'srun-tests.sh(pre-fix, base8b0931a)ef5306a)A capability check (
_require_check_targets) makes the pre-fix state fail fast and by name. Without it the pre-fix behaviour is red for the wrong reason: the old arg parser has no--check-targetscase, so the flag falls through to*) ROOT="$1", the trailing ROOT overwrites it,cdsucceeds and the runner executes the entire suite β measured, each invocation blew the 120s subprocess timeout, turning 8 tests into 8 full suite runs.Reachability, against the REAL list
Injected a bogus entry into the actual
HERMETIC_TARGETS:exit 2, and the regression test went red as 2 failed, 6 passed β the two list-acceptance tests specifically, not a blanket collapse, so the guard fails for its own reason. Reverted and re-confirmed 8 passed; the revert was verified byte-identical against the staged blob, not assumed.
The test file labels which of its tests are regression coverage and which is an invariant guard (
test_hermetic_list_still_names_the_guard_core_filepins that nobody "fixes" a future failure by deleting the entry β the runner's own error message warns against exactly that).π΄ A second #276 regression β NOT fixed here
#276also movedSECRET_PATTERNSout ofbash-guard.pyintoguard_core.py.session_insight'stest_patterns_cover_bash_guardstillast.parses~/.claude/hooks/bash-guard.py, gets[], and fails:It fails on any host where the hook is deployed (so it breaks the pre-push tier) and skips in the nix sandbox (synthetic
$HOMEβ file absent β pinned skip), so the flake gate never sees it. It passed at 16:45 today and failed at 17:03 on the same tree β the host's deployed hook changed under ahome-manager switchmid-session.Different file, different failure mode, and it needs a judgement call about what the test should assert now β so I deliberately did not bundle it. The fix is a one-line repoint of
_BASH_GUARDtoguard_core.py. Happy to take it as the next PR.Process note
My own
β¦ ; echo "EXIT=$?"wrapper reported exit 0 for a run whose summary saidRESULT: FAILβ$?read theecho, not the pipeline. Same shape as therc=$?case documented in #287. It is why every number above is a counted total rather than an exit status.I also discarded one contaminated baseline: I removed a worktree while a measurement was still running against it, which produced a plausible-looking
collected=1133with a cascade of "missing directory" lines that were my own doing. Caught by reading the log rather than quoting the total.Verification
# tests 468 / # pass 468 / # fail 0(glob form)nix flake checkβ result reported in a follow-up comment.run-tests.sh: clean apart from a pre-existing SC2155 on a line this PR does not touch.Do not merge without the gate confirmation comment.
π€ Generated with Claude Code