Discover conformance runner tests instead of naming them (#572) by jeremy · Pull Request #594 · basecamp/basecamp-sdk

jeremy · 2026-08-03T06:22:52Z

Closes #572. First of a three-PR stack on conformance-harness false greens: this one, then #553 (whose fix edits the two files this PR makes runnable), then #573.

The defect

conformance-runner-tests selected the Python and Ruby runner suites by naming one file each — test_delay_gaps.py and delay_gaps_test.rb — and .github/workflows/test.yml named the same two a second time. Two suites in the tree matched neither name and were therefore executed by nothing:

conformance/runner/python/test_replay_runner.py
conformance/runner/ruby/replay_runner_test.rb

Go's line was go test ./..., so its replay_runner_test.go ran incidentally; Kotlin, Swift and vitest auto-discover. Only the two enumerated languages had the hole, and git grep replay_runner_test -- Makefile .github scripts returned nothing, so nothing else picked them up either.

This is the exact defect conformance-runner-tests was created to close — assertion code no target exercises — reproduced one level up in the build.

The fix

Every recipe discovers, recursively. Python runs bare pytest -q (collecting nothing is exit 5, so an emptied suite fails rather than green-passes); Ruby runs a find-driven *_test.rb loop (pruning vendor/ and .bundle/) that aborts when discovery matches nothing. Go, Kotlin and Swift already discovered and are unchanged in behavior.
Split per language. conformance-runner-tests-{go,python,ruby,kotlin,swift}, and CI calls those targets instead of respelling the commands — one definition of what "run the runner tests" means. conformance-swift-runner-tests is renamed into that family. The Python interpreter pin moves from --python to CI-side UV_PYTHON, uv's env spelling of the same flag.
A reachability guard, scripts/check-runner-test-reachability, wired into make check and the spec-gates job. Two teeth, because the gap has two ways back in:
- Enumeration ban — no runner-test recipe, in the Makefile or the workflow, may name an individual test file. It compares against the basenames found on disk rather than a "looks like a test filename" regex, so the discovery globs the recipes legitimately contain (*_test.rb) are not themselves reported.
- Discoverability — every file that contains tests must sit where its language's discovery looks. That is name AND placement, not name alone.
Both fail closed: a language whose content scan finds zero test files is a failure, since a detector that matches nothing cannot report anything unreachable.

Reachability is name AND placement, and Ruby is why

This is the review finding that reshaped the PR (thread). An earlier revision modelled reachability as the name alone: recursive find, then fnmatch the basename against the discovery glob. Correct for Go (go test ./...), pytest and vitest, all of which recurse. Ruby's discovery is not a toolchain's — it was hand-rolled as for f in *_test.rb, a glob that expands in conformance/runner/ruby and nowhere below it. The check and the recipe disagreed, and the check was the permissive one.

Demonstrated with one added file, conformance/runner/ruby/nested/probe_test.rb, whose only test is assert false. The pre-fix script is byte-identical across all three pushed heads of this stack (sha1 06011498844e), as is the pre-fix Ruby recipe (sha1 a06f51258dd8 over its indented body):

$ ./scripts/check-runner-test-reachability
  ok: Ruby — 3 test file(s), all matched by *_test.rb
==> Runner-test reachability clean (9 checks passed)
REAL_EXIT=0

$ make conformance-runner-tests-ruby
  --> delay_gaps_test.rb
  --> replay_runner_test.rb
REAL_EXIT=0

Three files counted, two opened. A guaranteed-failing file, certified reachable by the gate and executed by nothing — #572's own defect, inside #572's gate.

Fixed on both sides, because either alone leaves a way back in:

The recipe recurses. Adding a test file anywhere under the runner directory is now sufficient to run it.
The check models placement. Each glob language declares a discovery SCOPE — recursive (only the name decides) or toplevel (a correctly-named file one directory down is reported). Ruby's scope is derived from the Makefile recipe, not assumed: ruby_discovery_scope joins the recipe's backslash continuations and reads the discovery construct back out. An unrecognized shape reports unknown and fails. So reverting the recipe to a top-level glob re-arms the placement tooth rather than silently reopening the gap.

Same probe file, both halves in place:

$ ./scripts/check-runner-test-reachability
  ok: Ruby — 4 test file(s), all matched by *_test.rb (recursive discovery)
REAL_EXIT=0

$ make conformance-runner-tests-ruby
  --> ./nested/probe_test.rb
  1) Failure:
NestedProbeTest#test_guaranteed_failure [./nested/probe_test.rb:5]:
this file is executed by nothing
1 runs, 1 assertions, 1 failures, 0 errors, 0 skips
make: *** [conformance-runner-tests-ruby] Error 1
REAL_EXIT=2

Both blocks above are abbreviated: the first shows only the Ruby line of nine, the second only the failing file of the recipe's four (make stops there). The probe's own minitest output is verbatim. Note REAL_EXIT=2: through make, a recipe failure is 2, not the runner binary's own 1.

The content detector must not miss a valid test form either

Second review finding (thread). A file the content detector never recognizes is never considered, and that failure is silent in the dangerous direction: the arm's found count stays nonzero thanks to the other files, so it prints ok.

The Go marker was ^func (Test|Benchmark|Fuzz|Example)[[:upper:]] — demanding a suffix, and demanding it start with a letter. Go's rule (go help test) is that Xxx must not start with a lowercase letter, and Example takes no suffix at all. So func Example(), func Test( and func Test_helper( were invisible. Now [^a-z], that rule spelled directly.

Red proof

This PR's check against pristine origin/main (5efc52f09) — the enumeration it removes. Verbatim:

$ git archive origin/main | tar -x -C $W
$ cp scripts/check-runner-test-reachability $W/scripts/
$ REPO_ROOT=$W $W/scripts/check-runner-test-reachability
==> Checking conformance-runner test reachability
  ok: Go — 2 test file(s), all matched by *_test.go (recursive discovery)
  ok: Python — 2 test file(s), all matched by test_*.py *_test.py (recursive discovery)
  FAIL: Ruby: discovery scope is 'unknown', not recursive or toplevel
        the recipe's discovery could not be read, so placement cannot be judged
        fix the recipe or teach ruby_discovery_scope its shape — do not assume
  ok: TypeScript — 6 test file(s), all matched by *.test.ts (recursive discovery)
  ok: Kotlin — 1 test file(s), all under kotlin/conformance/src/test/kotlin
  ok: Swift — 3 test file(s), all under conformance/runner/swift/Tests/ConformanceSupportTests
  FAIL: Makefile conformance-runner-tests* recipe names individual test file(s) — discovery must be the only selector
        test_delay_gaps.py
        delay_gaps_test.rb
        a suite that does not match the named file is executed by nothing (#572)
  FAIL: .github/workflows/test.yml names individual test file(s) — discovery must be the only selector
        test_delay_gaps.py
        delay_gaps_test.rb
        a suite that does not match the named file is executed by nothing (#572)
  FAIL: .github/workflows/test.yml does not invoke every per-language runner-test target
        conformance-runner-tests-go
        conformance-runner-tests-python
        conformance-runner-tests-ruby
        conformance-runner-tests-kotlin
        conformance-runner-tests-swift

==> Runner-test reachability FAILED (4 failure(s), 5 passed)
REAL_EXIT=1

Main's Ruby recipe names an individual test file, so ruby_discovery_scope cannot read a discovery construct out of it and reports unknown — correct, and fail-closed. A revision of this PR judged that scope before scanning, so the Ruby arm returned early, never populated the basename list, and the enumeration ban below it had no delay_gaps_test.rb to report — it named test_delay_gaps.py alone. The run still failed, so nothing passed silently, but it named half the enumeration against the exact state it exists to reject. The content scan now runs first and the scope is judged after: a check that fires, fires completely.

The Python detector had the same hole, and its fix has a second edge

Third review finding (thread). The Python marker was unittest\.TestCase|^import pytest|^from unittest|^def test_. A class-based pytest suite imports nothing and indents its test method:

class TestStranded:
    def test_failure(self):
        assert False

No unittest, no import pytest, and ^def test_ is anchored at column 1 — so that file was invisible. Saved as stranded.py it is collected by nothing, and the arm still printed ok. Self-test case 5, against the old marker:

==> Self-test 5: a class-based pytest suite outside a collected filename must be reported
  FAIL: class-based pytest suite was not detected — the Python arm reports `ok`
        over a suite no collected filename covers
        ==> Checking conformance-runner test reachability
          FAIL: Go: directory conformance/runner/go does not exist
                the runner layout moved — update this check deliberately
          ok: Python — 1 test file(s), all matched by test_*.py *_test.py (recursive discovery)
          ok: Ruby — 1 test file(s), all matched by *_test.rb (recursive discovery)
          FAIL: TypeScript: directory conformance/runner/typescript does not exist
                the runner layout moved — update this check deliberately
          FAIL: Kotlin: test root kotlin/conformance/src/test/kotlin does not exist
                the runner layout moved — update this check deliberately
          FAIL: Swift: could not read the test target's path from conformance/runner/swift/Package.swift
                SwiftPM compiles the testTarget's declared path and nothing else, so the
                test root cannot be assumed — teach swift_test_target_path its shape
          ok: Makefile conformance-runner-tests* recipe enumerates no test file
          FAIL: .github/workflows/test.yml is missing
                cannot verify CI does not re-enumerate
        
        ==> Runner-test reachability FAILED (5 failure(s), 3 passed)

==> Self-test FAILED
REAL_EXIT=1

The obvious fix is wrong in the other direction. Matching the class name (^class Test) fails the real repo: conformance/runner/python/runner.py declares TestCase, TestResult, TestRunner and TestTracker as its own plumbing.

$ REPO_ROOT=$PWD $W/probe          # marker widened with `^class Test`
==> Checking conformance-runner test reachability
  ok: Go — 3 test file(s), all matched by *_test.go (recursive discovery)
  FAIL: Python: test file(s) matched by no discovery glob (test_*.py *_test.py)
        conformance/runner/python/runner.py
  ok: Ruby — 3 test file(s), all matched by *_test.rb (recursive discovery)
  ok: TypeScript — 7 test file(s), all matched by *.test.ts (recursive discovery)
  ok: Kotlin — 3 test file(s), all under kotlin/conformance/src/test/kotlin
  ok: Swift — 3 test file(s), all under conformance/runner/swift/Tests/ConformanceSupportTests
  ok: Makefile conformance-runner-tests* recipe enumerates no test file
  FAIL: .github/workflows/test.yml names individual test file(s) — discovery must be the only selector
        runner.py
        a suite that does not match the named file is executed by nothing (#572)
  ok: .github/workflows/test.yml invokes all five conformance-runner-tests-* targets

==> Runner-test reachability FAILED (2 failure(s), 7 passed)
REAL_EXIT=1

Verbatim, no elision — and note the second failure, which is the worse one. Once runner.py is treated as a test file its basename enters the enumeration ban's list, and .github/workflows/test.yml legitimately names runner.py (it is the conformance runner's entrypoint). So a name-based detector does not just report one false positive; it cascades into tooth (1) and accuses the workflow of enumerating a test file it does not have.

So the detector keys on the indented test method (^[[:space:]]+def test) — the thing pytest actually collects — not on the class name. Verified against the real tree: the widened marker matches exactly test_delay_gaps.py, test_replay_runner.py, test_request_count.py, and no other .py under conformance/runner/python.

TypeScript's marker gets the same sweep for a smaller reason: it was the literal from "vitest", so from 'vitest' — same import, same discovery — did not match. Now quote-agnostic.

Two more "recursive means the toolchain's rule, not all"

Seventh and eighth findings (pytest norecursedirs, YAML comments).

pytest's norecursedirs default — pytest --help gives it as *.egg .* _darcs build CVS dist node_modules venv {arch} — means a correctly-named test under dist/ or venv/ is matched by the glob and collected by nothing. Confirmed against the real tree with conformance/runner/python/dist/test_stranded.py present (a bare assert False): pytest -q still reported 29 passed and the check still reported ok: Python — 4 test file(s). Python now declares that ignore pattern the way Go declares its own.

And the per-language target loop grepped the workflow verbatim, so a step commented out wholesale still matched — # - run: make conformance-runner-tests-go reads as an invocation to a raw grep while CI no longer runs it. Both invocation checks now read the comment-stripped file. The enumeration ban above deliberately keeps scanning the raw file: a test filename left in a comment selects nothing, but it is a stale reference worth surfacing rather than hiding.

Self-test cases 12 and 13, against the previous revision, verbatim:

==> Self-test 12: a pytest suite under dist/ must be reported
  FAIL: a test under dist/ was not reported — the glob matched and pytest
        never collected it
        ==> Checking conformance-runner test reachability
          FAIL: Go: directory conformance/runner/go does not exist
                the runner layout moved — update this check deliberately
          ok: Python — 2 test file(s), all matched by test_*.py *_test.py (recursive discovery)
          ok: Ruby — 1 test file(s), all matched by *_test.rb (recursive discovery)
          FAIL: TypeScript: directory conformance/runner/typescript does not exist
                the runner layout moved — update this check deliberately
          FAIL: Kotlin: test root kotlin/conformance/src/test/kotlin does not exist
                the runner layout moved — update this check deliberately
          FAIL: Swift: could not read the test target's path from conformance/runner/swift/Package.swift
                SwiftPM compiles the testTarget's declared path and nothing else, so the
                test root cannot be assumed — teach swift_test_target_path its shape
          ok: Makefile conformance-runner-tests* recipe enumerates no test file
          FAIL: .github/workflows/test.yml is missing
                cannot verify CI does not re-enumerate
        
        ==> Runner-test reachability FAILED (5 failure(s), 3 passed)

==> Self-test 13: a commented-out runner-test step must not count as invoked
  FAIL: the commented-out Go step was still counted as an invocation
        ==> Checking conformance-runner test reachability
          FAIL: Go: directory conformance/runner/go does not exist
                the runner layout moved — update this check deliberately
          FAIL: Python: directory conformance/runner/python does not exist
                the runner layout moved — update this check deliberately
          ok: Ruby — 1 test file(s), all matched by *_test.rb (recursive discovery)
          FAIL: TypeScript: directory conformance/runner/typescript does not exist
                the runner layout moved — update this check deliberately
          FAIL: Kotlin: test root kotlin/conformance/src/test/kotlin does not exist
                the runner layout moved — update this check deliberately
          FAIL: Swift: could not read the test target's path from conformance/runner/swift/Package.swift
                SwiftPM compiles the testTarget's declared path and nothing else, so the
                test root cannot be assumed — teach swift_test_target_path its shape
          ok: Makefile conformance-runner-tests* recipe enumerates no test file
          ok: .github/workflows/test.yml enumerates no test file
          ok: .github/workflows/test.yml invokes all five conformance-runner-tests-* targets
          ok: .github/workflows/test.yml invokes the TypeScript runner suite
        
        ==> Runner-test reachability FAILED (5 failure(s), 5 passed)

==> Self-test FAILED
REAL_EXIT=1

ok: Python — 2 test file(s) and ok: … invokes all five conformance-runner-tests-* targets are the two false greens. (The inner FAILs are the synthetic trees' absent language directories, which every self-test case has by design.)

TypeScript could have left CI without this check noticing

Sixth finding (thread), and the first against tooth (1) rather than the detectors.

Tooth (1) verified that the workflow invokes every conformance-runner-tests-* target. TypeScript has no such target — its runner tests and its conformance suite are one vitest invocation, which CI spells npx vitest run directly. So the loop could not speak for it, and the seven TypeScript test files this check counts could vanish from CI while all nine checks reported green.

Three-way proof. npx vitest run on line 299 of the workflow replaced with npx echo STEP-REMOVED, nothing else changed:

previous revision, no TypeScript check at all
  ==> Runner-test reachability clean (9 checks passed)
  REAL_EXIT=0

this check, but grepping the workflow verbatim
  ok: .github/workflows/test.yml invokes the TypeScript runner suite
  ==> Runner-test reachability clean (10 checks passed)
  REAL_EXIT=0

this check as committed (comments stripped first)
  FAIL: .github/workflows/test.yml does not invoke the TypeScript runner suite
  ==> Runner-test reachability FAILED (1 failure(s), 9 passed)
  REAL_EXIT=1

The middle one is the reason for the sed 's/#.*//'. The workflow explains its own choice of npx vitest run over npm test in a comment three lines above the step, so a verbatim grep matched the prose and passed with the step deleted. Match what CI would run, not what it says about it. That bug was in the first version of this check, and its own red proof is what caught it.

Self-test case 11 pins it with a synthetic workflow that invokes the five make targets and nothing else.

`@Test` is not JUnit Jupiter's only test annotation

Fifth detector finding (thread). kotlin/conformance/build.gradle.kts carries testImplementation(libs.junit.jupiter) and useJUnitPlatform(), so @ParameterizedTest, @RepeatedTest, @TestFactory and @TestTemplate all declare runnable tests. The Kotlin detector matched the literal @Test and saw none of them — a @ParameterizedTest stranded under src/main/kotlin compiles into the product, runs nowhere, and DelayGapsTest keeps found nonzero so the arm prints ok.

Self-test case 10, against the literal-@Test marker, verbatim:

==> Self-test 10: a JUnit Jupiter @ParameterizedTest under src/main must be reported
  FAIL: @ParameterizedTest was not detected — the Kotlin arm reports `ok`
        over a test Gradle runs and the main source set does not host
        ==> Checking conformance-runner test reachability
          FAIL: Go: directory conformance/runner/go does not exist
                the runner layout moved — update this check deliberately
          FAIL: Python: directory conformance/runner/python does not exist
                the runner layout moved — update this check deliberately
          ok: Ruby — 1 test file(s), all matched by *_test.rb (recursive discovery)
          FAIL: TypeScript: directory conformance/runner/typescript does not exist
                the runner layout moved — update this check deliberately
          ok: Kotlin — 1 test file(s), all under kotlin/conformance/src/test/kotlin
          FAIL: Swift: could not read the test target's path from conformance/runner/swift/Package.swift
                SwiftPM compiles the testTarget's declared path and nothing else, so the
                test root cannot be assumed — teach swift_test_target_path its shape
          ok: Makefile conformance-runner-tests* recipe enumerates no test file
          FAIL: .github/workflows/test.yml is missing
                cannot verify CI does not re-enumerate
        
        ==> Runner-test reachability FAILED (5 failure(s), 3 passed)

==> Self-test FAILED
REAL_EXIT=1

ok: Kotlin — 1 test file(s) while StrandedTest.kt sat undetected in the main source set. (The inner FAILs are the synthetic tree's absent language directories, which every self-test case has by design.)

"recursive" does not mean "every subdirectory"

Fourth detector finding (thread). The Go arm was declared recursive on go test ./...'s authority — true, but not the whole rule. Per go help packages:

Directory and file names that begin with "." or "_" are ignored by the go tool, as are directories named "testdata".

So conformance/runner/go/_ignored/probe_test.go is matched by the glob, reached by find, counted as reachable — and compiled by nothing. Confirmed against the real tree with that file present:

$ (cd conformance/runner/go && go list ./...)
github.com/basecamp/basecamp-sdk/conformance/runner/go

$ (cd conformance/runner/go && go test ./...)
ok  	github.com/basecamp/basecamp-sdk/conformance/runner/go	(cached)
REAL_EXIT=0

$ ./scripts/check-runner-test-reachability
  ok: Go — 4 test file(s), all matched by *_test.go (recursive discovery)
==> Runner-test reachability clean (9 checks passed)

One package listed, four files counted. Each glob language now also declares the locations its toolchain skips, and a test-bearing file in one is reported; Go's pattern is Go's own rule spelled out, the other three declare none.

Self-test case 9, against the previous revision (ignore pattern disabled), verbatim:

==> Self-test 9: a Go test under an underscore directory must be reported
  FAIL: a _test.go under _ignored/ was not reported — the glob matched and
        `go test ./...` skipped it
        ==> Checking conformance-runner test reachability
          ok: Go — 2 test file(s), all matched by *_test.go (recursive discovery)
          FAIL: Python: directory conformance/runner/python does not exist
                the runner layout moved — update this check deliberately
          ok: Ruby — 1 test file(s), all matched by *_test.rb (recursive discovery)
          FAIL: TypeScript: directory conformance/runner/typescript does not exist
                the runner layout moved — update this check deliberately
          FAIL: Kotlin: test root kotlin/conformance/src/test/kotlin does not exist
                the runner layout moved — update this check deliberately
          FAIL: Swift: could not read the test target's path from conformance/runner/swift/Package.swift
                SwiftPM compiles the testTarget's declared path and nothing else, so the
                test root cannot be assumed — teach swift_test_target_path its shape
          ok: Makefile conformance-runner-tests* recipe enumerates no test file
          FAIL: .github/workflows/test.yml is missing
                cannot verify CI does not re-enumerate
        
        ==> Runner-test reachability FAILED (5 failure(s), 3 passed)

==> Self-test FAILED

==> Self-test FAILED
REAL_EXIT=1

ok: Go — 2 test file(s) is the false green — both counted, one never compiled. (The inner FAILs are the synthetic tree's absent language directories, which every self-test case has by design.)

Swift: "under Tests/" is not the same as "compiled"

Two more of the same class (sibling target, parameterized attribute), on the arm where discovery is a manifest rather than a convention.

conformance/runner/swift/Package.swift declares its sole test target with path: "Tests/ConformanceSupportTests". SwiftPM compiles that directory and nothing else — so Tests/ReplayTests/ReplayTests.swift, a plain sibling, is compiled by nothing while looking entirely at home. The check used Tests/ as the test root, counted the sibling as reachable, and printed ok.

The test root is now derived from the manifest (swift_test_target_path), the same way Ruby's scope is derived from the Makefile, and an unreadable manifest fails rather than defaulting. Gradle is the same story one directory up: its Kotlin test source set is src/test/kotlin, not src/test.

The scan root widened to match. Looking only under Sources/ for stranded tests could never have found a stranded test under Tests/; the whole module is scanned now, and anything test-bearing outside the compiled test directory is reported.

Second, the attribute regex demanded whitespace-or-EOL after @Test, so every parameterized form — Swift Testing's @Test("display name") and @Test(arguments:), JUnit 4's @Test(expected = ...) — was invisible. It now accepts ( too.

Both are pinned as self-test cases 7 and 8. Against the previous roots and marker, verbatim:

==> Self-test 7: a Swift test beside the declared test target must be reported
  FAIL: a test outside the declared testTarget path was not reported
        ==> Checking conformance-runner test reachability
          FAIL: Go: directory conformance/runner/go does not exist
                the runner layout moved — update this check deliberately
          FAIL: Python: directory conformance/runner/python does not exist
                the runner layout moved — update this check deliberately
          ok: Ruby — 1 test file(s), all matched by *_test.rb (recursive discovery)
          FAIL: TypeScript: directory conformance/runner/typescript does not exist
                the runner layout moved — update this check deliberately
          FAIL: Kotlin: test root kotlin/conformance/src/test/kotlin does not exist
                the runner layout moved — update this check deliberately
          ok: Swift — 2 test file(s), all under conformance/runner/swift/Tests
          ok: Makefile conformance-runner-tests* recipe enumerates no test file
          FAIL: .github/workflows/test.yml is missing
                cannot verify CI does not re-enumerate
        
        ==> Runner-test reachability FAILED (5 failure(s), 3 passed)

==> Self-test 8: a parameterized Swift Testing attribute must be detected
  FAIL: `@Test("…")` was not detected — the Swift arm reports `ok`
        over a test SwiftPM discovers and the test target does not compile
        ==> Checking conformance-runner test reachability
          FAIL: Go: directory conformance/runner/go does not exist
                the runner layout moved — update this check deliberately
          FAIL: Python: directory conformance/runner/python does not exist
                the runner layout moved — update this check deliberately
          ok: Ruby — 1 test file(s), all matched by *_test.rb (recursive discovery)
          FAIL: TypeScript: directory conformance/runner/typescript does not exist
                the runner layout moved — update this check deliberately
          FAIL: Kotlin: test root kotlin/conformance/src/test/kotlin does not exist
                the runner layout moved — update this check deliberately
          ok: Swift — 1 test file(s), all under conformance/runner/swift/Tests
          ok: Makefile conformance-runner-tests* recipe enumerates no test file
          FAIL: .github/workflows/test.yml is missing
                cannot verify CI does not re-enumerate
        
        ==> Runner-test reachability FAILED (5 failure(s), 3 passed)

==> Self-test FAILED
REAL_EXIT=1

==> Self-test FAILED
REAL_EXIT=1

Case 7's ok: Swift — 2 test file(s), all under conformance/runner/swift/Tests is the false green: two files counted, one compiled by nothing. Case 8's ok: Swift — 1 test file(s) is the other: the @Test("a display name") under Sources/ was never seen. (The inner FAILs are the synthetic trees' absent language directories, which every self-test case has by design.)

A gate that aborts is a gate that did not run

Two array expansions were unguarded under set -u. bash 3.2 — still /bin/bash on macOS — treats "${empty[@]}" as an unbound variable and kills the script, so on a machine whose env bash resolves there, the check would die partway through instead of reporting.

Reproduced against the previous revision of this file, on a synthetic tree with a toplevel Ruby recipe and no nested files — the exact shape a repo has the moment someone reverts the recipe to a top-level glob, which is the case the placement tooth exists to catch:

$ REPO_ROOT=$W /bin/bash $W/scripts/probe
  FAIL: Go: directory conformance/runner/go does not exist
  FAIL: Python: directory conformance/runner/python does not exist
.../scripts/probe: line 229: nested[@]: unbound variable
REAL_EXIT=1

Nothing elided — the script died there, so the Ruby arm and every tooth after it never ran. The same tree under this revision reports ok: Ruby — 1 test file(s), all matched by *_test.rb (toplevel discovery) and goes on to run the rest.

Both expansions are now count-guarded, matching the -gt 0 guard the others already had. An empty basename list is additionally reported rather than silently iterated: it means every tooth-(2) arm bailed out, which is worth saying out loud. Verified under bash 5.3.9 and /bin/bash 3.2.57, and shellcheck-clean.

Self-test

--self-test, wired into spec-gates, now carries thirteen cases. Three exist to keep the others honest — a detector can be wrong by missing a real test, and equally wrong by flagging something that is not one:

misnamed — replay_runner_spec.rb, matched by no glob, is reported.
misplaced — nested/probe_test.rb under a top-level Ruby recipe is reported. The committed form of the false green above.
the same nested file under a recursive recipe is not reported, because there it really does run. Without case 3, case 2 would also pass if placement were flagged unconditionally, and the scope derivation would be decorative.
undetected — examples.go holding func Example() is reported, beside a Go file that keeps the arm's found count nonzero.
the same hole in Python — a class-based pytest suite in stranded.py, importing nothing, is reported.
the converse of 5 — the runner's own Test*-named plumbing is not reported. Without case 6, case 5 would also pass under a name-based detector that fails the real repo over runner.py.
a Swift test in a sibling directory of the declared testTarget path is reported — "under Tests/" is not "compiled".
a parameterized @Test("name") under Sources/ is reported.
a Go _test.go under _ignored/ is reported — recursive means "walks subdirectories", not "walks every subdirectory".
a Jupiter @ParameterizedTest under src/main/kotlin is reported.
a workflow that no longer invokes the TypeScript suite is reported.
a pytest suite under dist/ is reported — pytest's norecursedirs default.
a runner-test step commented out wholesale is reported missing.

Case 4 against the old character class, verbatim:

==> Self-test 4: an unsuffixed Go example outside a _test.go file must be reported
  FAIL: unsuffixed `func Example()` was not detected — the Go arm reports `ok`
        over a file go test runs and no glob covers
        ==> Checking conformance-runner test reachability
          ok: Go — 1 test file(s), all matched by *_test.go (recursive discovery)
          FAIL: Python: directory conformance/runner/python does not exist
                the runner layout moved — update this check deliberately
          ok: Ruby — 1 test file(s), all matched by *_test.rb (recursive discovery)
          FAIL: TypeScript: directory conformance/runner/typescript does not exist
                the runner layout moved — update this check deliberately
          FAIL: Kotlin: test root kotlin/conformance/src/test/kotlin does not exist
                the runner layout moved — update this check deliberately
          FAIL: Swift: could not read the test target's path from conformance/runner/swift/Package.swift
                SwiftPM compiles the testTarget's declared path and nothing else, so the
                test root cannot be assumed — teach swift_test_target_path its shape
          ok: Makefile conformance-runner-tests* recipe enumerates no test file
          FAIL: .github/workflows/test.yml is missing
                cannot verify CI does not re-enumerate
        
        ==> Runner-test reachability FAILED (5 failure(s), 3 passed)

==> Self-test FAILED
REAL_EXIT=1

ok: Go — 1 test file(s) is the point — green while examples.go sat undetected beside it. ([...] elides the synthetic tree's absent-language-directory FAILs, which every self-test case has by design.)

Verification (real exit codes, measured on this commit)

./scripts/check-runner-test-reachability            10 checks passed   REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 13 cases passed    REAL_EXIT=0
make conformance-runner-tests-go      ok (cached)                      REAL_EXIT=0
make conformance-runner-tests-python  17 passed in 0.04s               REAL_EXIT=0
make conformance-runner-tests-ruby    --> ./delay_gaps_test.rb   11 runs
                                      --> ./replay_runner_test.rb 3 runs
                                      0 failures, 0 errors             REAL_EXIT=0
make conformance-runner-tests-kotlin  (--quiet, no output)             REAL_EXIT=0
make conformance-runner-tests-swift   Executed 39 tests, 0 failures    REAL_EXIT=0
make lint-actions                     No findings to report            REAL_EXIT=0
shellcheck scripts/check-runner-test-reachability                      REAL_EXIT=0
/bin/bash ./scripts/check-runner-test-reachability  (bash 3.2.57)      REAL_EXIT=0
/bin/bash ./scripts/check-runner-test-reachability --self-test         REAL_EXIT=0

That Python 17 is the target's scope — the whole conformance/runner/python directory's collection — not any one file. Broken out with pytest -q --collect-only: test_delay_gaps.py 12, test_replay_runner.py 5. The 5 in test_replay_runner.py are the tests this PR makes reachable; quoting 17 as "the newly-reachable tests" would overstate it by 12.

Both previously-unrun suites pass under discovery, so this adds coverage rather than exposing drift. Swift ran (39 tests executed, not the macOS SKIP line).

Local figures — cite the CI job's own numbers where they differ.

github-actions · 2026-08-03T06:23:02Z

Sensitive Change Detection (shadow mode)

This PR modifies control-plane files:

.github/workflows/test.yml

Shadow mode — this check is informational only. When activated, changes to these paths will require approval from a maintainer.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7946b0c40d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Pull request overview

This PR closes a build-level false-green: the conformance-runner-tests Makefile recipe and .github/workflows/test.yml each selected the Python and Ruby runner suites by naming a single file (test_delay_gaps.py / delay_gaps_test.rb), so two other suites (test_replay_runner.py, replay_runner_test.rb) that matched neither name were executed by nothing. The fix makes every recipe discover its suites, splits the target per language so CI calls the make target instead of respelling commands, and adds a fail-closed reachability guard (scripts/check-runner-test-reachability) wired into make check and the spec-gates CI job. It is the first of a three-PR stack (#553, #573 follow).

I verified: no stale references to the renamed conformance-swift-runner-tests target remain repo-wide; the guard's content markers and discovery globs match the actual runner trees; the awk recipe extractor deliberately excludes the surrounding comment blocks (so the enumeration ban won't false-positive on the #572 comments that mention the old filenames); and both previously-unrun suites (unittest-based Python, Minitest-based Ruby) are now genuinely collected by pytest -q and the *_test.rb glob loop.

Changes:

Split conformance-runner-tests into conformance-runner-tests-{go,python,ruby,kotlin,swift} (all discovery-based; renamed the old Swift target into the family) and pointed each CI language job at its make target via working-directory: ..
Added scripts/check-runner-test-reachability — a two-tooth guard (enumeration ban + discoverability, both fail-closed) with a --self-test, wired into make check and spec-gates.
Moved the Python interpreter pin from --python to the CI-side UV_PYTHON env so the make target stays version-agnostic.

Tip

If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
`scripts/check-runner-test-reachability`	New fail-closed guard asserting every runner test file is reachable from discovery and no recipe/workflow names a test file; includes a self-test.
`Makefile`	Splits the aggregate runner-test target into per-language discovery-based targets, renames the Swift target, and wires `check-runner-test-reachability` into `.PHONY`, `check`, and `help`.
`.github/workflows/test.yml`	Language jobs now invoke `make conformance-runner-tests-<lang>` (Python pin via `UV_PYTHON`); `spec-gates` runs the reachability check and its self-test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 55fadde6d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 84adba2a7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7f72bd9623

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

…couldn't see Two more, and both are the check looking past the thing it exists to find. build/ was pruned globally. `build` sat in PRUNE alongside .venv and node_modules, so find never handed conformance/runner/python/build/ to the Python arm — the arm whose norecursedirs rule exists to report exactly that directory. A guaranteed-failing test there was collected by pytest, reported by this check, and run by neither: the check exited 0. A global prune that hides a file from the arm meant to classify it is a blind spot, not a prune. Gradle's tree is unaffected (the Kotlin arm scans src/, Gradle writes to build/ beside it), SwiftPM's .build stays pruned, and every build/ under a scan root today is inside node_modules, pruned before find descends. Disabled steps counted as invocations. A step carrying `if: ${{ false }}` is skipped by Actions while its `run: make conformance-runner-tests-go` stays in the file, so the invocation check reported the Go suite as invoked. Parking a suite behind a disabled step is the natural way to reopen #572. `drop_disabled_workflow_blocks` removes statically-disabled steps and jobs before any invocation grep runs, and the bound is deliberate: only a LITERALLY false condition disables. This workflow runs the Ruby runner tests under `if: matrix.ruby == '3.3'`, so a rule that treated any `if:` as disqualifying would fail the very workflow the check protects. Self-tests 22-25. 22/23/24 fail against the state they were reported on; 25 is the bound, and it plants a condition at BOTH step and job level — an earlier draft carried only the step one, and a mutant broadening the job predicate passed the whole suite.

…iscovery * origin/main: deps(ts): bump the npm-dependencies group in /typescript with 2 updates (#609) deps(ruby): bump simplecov in /ruby in the bundler-dependencies group (#607) deps(kotlin): bump the ktor group in /kotlin with 5 updates (#608) Unbreak doc-constants-check: grant SPEC.md's two as-of pin citations (#605)

jeremy · 2026-08-03T11:21:44Z

Follow-up to my earlier note about Spec Gates: it is green now, and the
cause was upstream as diagnosed. Main fixed itself in #605
(ac11a24a7, "Unbreak doc-constants-check: grant SPEC.md's two as-of pin
citations"). CI evaluates the PR merge ref, so the job went green on this
branch as soon as main did — before this branch had the fix in its own tree.
2b378d532 merges current main in, so make check now passes locally on the
branch too:

$ make doc-constants-check
sync-doc-constants self-test: all cases passed.
REAL_EXIT=0

Nothing else changed: git diff --stat origin/main HEAD is still this PR's
three files.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b378d5320

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…t self-locating Ruby's arm had the bug the Python arm was already fixed for. `require "minitest/autorun"` counted as a test declaration, so a conventional test_helper.rb — required BY the suites rather than discovered, matching no `*_test.rb` glob — failed the build while declaring nothing runnable. It now keys on declarations: a `Minitest::Test`/`Spec` subclass, a `def test_`, or a spec-style `describe`/`it`. Both real Ruby suites match on two of those, so the counts are unchanged. The converse stays loud, and for a different reason than pytest's: minitest/autorun runs every test class LOADED into the process, so whether a declaration in a helper executes depends on some *_test.rb requiring the file — an edge this check cannot see and will not assume. A helper that declares tests is still reported. `vitest run` is not self-locating. Vitest's root is the process cwd, and the test-typescript job declares `defaults: run: working-directory: typescript`, so the identical command with the step's override dropped re-runs the SDK's own suite and collects none of the six runner files this check counts — while the text of the command is unchanged. Matching the command without its directory asserted that a string appears in a workflow, not that these tests run. vitest_invocation_dirs resolves each vitest step's effective working directory (step-level `working-directory:`, else the job default), and one of them must be conformance/runner/typescript. Self-tests 26-29, each loud case paired with its quiet one; 26 and 28 fail against the state they were reported on.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f560484790

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… its matrix Invoking a target is not running the tests. The per-language loop established only that CI calls `make conformance-runner-tests-<lang>`; the target itself is one `@true` away from running nothing, with the workflow byte-identical. The TypeScript wrapper was already followed for exactly this reason, and leaving the other five trusted by name was an asymmetry rather than a decision. Each recipe must now contain the command that runs its suite — `go test`, `pytest`, `bundle exec ruby`, `:conformance:test`, `swift test` — kept as one table so "what runs the tests" stays written down instead of inferred. It moved next to the enumeration ban, since it is a fact about the Makefile: a tree with no workflow file skipped it where it first sat, which self-test 32 caught. A matrix condition is now read against its matrix. `if: matrix.ruby == '3.3'` stops running the moment '3.3' leaves `matrix.ruby` — an ordinary consequence of moving supported runtimes on — and the step survives untouched, so nothing in that diff says a suite stopped running. The step filter now also drops a step whose pinned value the enclosing job's `strategy:` block does not offer, and one pinned in a job with no matrix at all. The bound stays narrow and is still tested: expressions are not evaluated, `!=` is not reasoned about, and a pin the matrix DOES offer still counts. Self-test 25 now carries a matrix that offers its value and asserts it stays counted. Self-tests 30, 31 and 32; all three fail against the state they were reported on, while 25 stays quiet in both.

* origin/main: Refuse a malformed GET field instead of writing it back (#576) (#597) File the event-feed api-gap entry and record the cross-team decisions (#606) # Conflicts: # .github/workflows/test.yml # Makefile

jeremy · 2026-08-03T11:54:00Z

4f93478d0 merges current main, which conflicted in Makefile and
.github/workflows/test.yml. Worth recording how, because the conflict is the
PR demonstrating itself.

Main's #576/#597 added two runner tests and wired them up by name — the
recipe grew test_error_raised.py beside test_delay_gaps.py, and the workflow
grew a second bundle exec ruby error_raised_test.rb line. That is the #572
shape the enumeration ban exists to stop, and it landed while this PR was in
review.

Resolved by keeping this branch's side in all four hunks — the enumeration is
what the PR removes — and carrying main's new errorRaised prose paragraph
across. Nothing was dropped: both files are found by discovery, with no recipe
or workflow change at all.

$ ./scripts/check-runner-test-reachability
  ok: Go — 3 test file(s), all matched by *_test.go (recursive discovery)
  ok: Python — 3 test file(s), all matched by test_*.py *_test.py (recursive discovery)
  ok: Ruby — 3 test file(s), all matched by *_test.rb (recursive discovery)
  ok: TypeScript — 7 test file(s), all matched by *.@(test|spec).?(c|m)@(j|t)s?(x) (recursive discovery)
  ok: Kotlin — 2 test file(s), all under kotlin/conformance/src/test/kotlin
  ok: Swift — 4 test file(s), all under conformance/runner/swift/Tests/ConformanceSupportTests
  ok: Makefile conformance-runner-tests* recipe enumerates no test file
[...]
==> Runner-test reachability clean (12 checks passed)
REAL_EXIT=0

Every count went up by one without anybody adding a name anywhere, which is the
whole point of the change.

Also of note: this round's work happened in a second worktree. The lane worktree
was taken over mid-session by another process — its HEAD moved to an unrelated
commit and a conflicted merge appeared in it. Nothing was lost (every fix was
already committed and pushed), and the branch was rebuilt from its own tip.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4f93478d0c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The third and last instance of one bug. A support module doing `import type { Mock } from 'vitest'`, imported BY the suites, matches no `.test.`/`.spec.` glob, so keying the TypeScript arm on the framework import failed the build over a file that declares nothing runnable — the same defect already fixed for conftest.py and test_helper.rb, in the one arm I had not carried the rule to. The marker is now `describe`/`it`/`test`/`suite`/`bench`, including the suffixed and tagged-template forms (`it.each([…])(`, `test.skip(`, ``it.each`…` ``), because a declaration written any of those ways is still runnable and missing one is the silent direction. All seven committed suites match it, between 3 and 19 declarations each; the three helper modules match none. That completes an invariant worth naming: every arm keys on a DECLARATION — `func TestX`, a `def test`, a `Minitest::Test`, a `describe`/`it`, an `@Test`, an XCTestCase — and none on the presence of a framework import. Support modules are reachable by construction in every language, and this check no longer has an opinion about them. Self-test 33, the TypeScript twin of 14 and 28; case 17 is its loud pair and still passes. Fails against the marker it was reported on.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac1f990b39

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-03T12:04:48Z

+    grep -qF -- "$target_tool" <<<"$target_recipe" \
+        || hollow_targets+=("conformance-runner-tests-$target runs no '$target_tool'")


Require an executable test-runner command

Fresh evidence beyond the prior @true self-test is that replacing the Go recipe with @echo go test leaves this checker exiting 0 with all 12 checks passing. This predicate searches the complete recipe text, so an echo, comment, or other non-executed occurrence of the tool name certifies a hollow target even though CI runs no tests; validate an executable runner command rather than a substring.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-03T12:04:48Z

+    missing_targets=()
+    for target in go python ruby kotlin swift; do
+        invoked_by_workflow "make[[:space:]]+conformance-runner-tests-$target" \
+            || missing_targets+=("conformance-runner-tests-$target")


Verify the Swift target runs on macOS

Fresh evidence beyond the earlier workflow-invocation fixes is that changing the inspected test-swift job from macos-15 to ubuntu-latest still produces Runner-test reachability clean (12 checks passed). In that environment conformance-runner-tests-swift takes its ifndef IS_MACOS branch and successfully prints SKIP, so merely finding this command does not establish that any Swift runner tests execute; associate the invocation with a macOS job or otherwise validate that the target's platform gate is satisfied.

Useful? React with 👍 / 👎.

Copilot AI review requested due to automatic review settings August 3, 2026 06:22

jeremy added the bug Something isn't working label Aug 3, 2026

github-actions Bot added the github-actions Pull requests that update GitHub Actions label Aug 3, 2026

Copilot started reviewing on behalf of jeremy August 3, 2026 06:23 View session

This was referenced Aug 3, 2026

Register the 20 missing live operations in all four replay decoders (#553) #595

Merged

requestCount is an exact count in every runner, not a lower bound (#573) #596

Merged

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread scripts/check-runner-test-reachability

Comment thread scripts/check-runner-test-reachability Outdated

Copilot AI reviewed Aug 3, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 3, 2026 09:00

jeremy force-pushed the fix/572-runner-test-discovery branch from 7946b0c to 63e434f Compare August 3, 2026 09:00

Copilot started reviewing on behalf of jeremy August 3, 2026 09:00 View session

Copilot AI reviewed Aug 3, 2026

Copilot AI review requested due to automatic review settings August 3, 2026 09:04

jeremy force-pushed the fix/572-runner-test-discovery branch from 63e434f to 55fadde Compare August 3, 2026 09:04

Copilot started reviewing on behalf of jeremy August 3, 2026 09:05 View session

Copilot AI reviewed Aug 3, 2026

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread scripts/check-runner-test-reachability Outdated

Copilot AI review requested due to automatic review settings August 3, 2026 09:13

jeremy force-pushed the fix/572-runner-test-discovery branch from 55fadde to 3e4a812 Compare August 3, 2026 09:13

Copilot started reviewing on behalf of jeremy August 3, 2026 09:13 View session

Copilot AI reviewed Aug 3, 2026

Copilot AI review requested due to automatic review settings August 3, 2026 09:18

jeremy force-pushed the fix/572-runner-test-discovery branch from 3e4a812 to 50d7402 Compare August 3, 2026 09:18

Copilot started reviewing on behalf of jeremy August 3, 2026 09:19 View session

Copilot AI reviewed Aug 3, 2026

jeremy force-pushed the fix/572-runner-test-discovery branch from 50d7402 to 84adba2 Compare August 3, 2026 09:21

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread scripts/check-runner-test-reachability Outdated

Comment thread scripts/check-runner-test-reachability Outdated

Copilot AI review requested due to automatic review settings August 3, 2026 09:41

jeremy force-pushed the fix/572-runner-test-discovery branch from 84adba2 to 5224891 Compare August 3, 2026 09:41

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread scripts/check-runner-test-reachability Outdated

Comment thread scripts/check-runner-test-reachability

Copilot AI reviewed Aug 3, 2026

jeremy added 2 commits August 3, 2026 04:20

Copilot AI review requested due to automatic review settings August 3, 2026 11:20

Copilot started reviewing on behalf of jeremy August 3, 2026 11:21 View session

Copilot AI reviewed Aug 3, 2026

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread scripts/check-runner-test-reachability Outdated

Comment thread scripts/check-runner-test-reachability Outdated

Copilot AI review requested due to automatic review settings August 3, 2026 11:41

Copilot started reviewing on behalf of jeremy August 3, 2026 11:42 View session

Copilot AI reviewed Aug 3, 2026

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread scripts/check-runner-test-reachability

Comment thread scripts/check-runner-test-reachability

jeremy added 2 commits August 3, 2026 04:51

Merge remote-tracking branch 'origin/main' into HEAD

4f93478

* origin/main: Refuse a malformed GET field instead of writing it back (#576) (#597) File the event-feed api-gap entry and record the cross-team decisions (#606) # Conflicts: # .github/workflows/test.yml # Makefile

Copilot AI review requested due to automatic review settings August 3, 2026 11:53

Copilot started reviewing on behalf of jeremy August 3, 2026 11:53 View session

Copilot AI reviewed Aug 3, 2026

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread scripts/check-runner-test-reachability Outdated

Copilot AI review requested due to automatic review settings August 3, 2026 11:59

jeremy force-pushed the fix/572-runner-test-discovery branch from 4f93478 to b4ccca1 Compare August 3, 2026 11:59

Copilot started reviewing on behalf of jeremy August 3, 2026 11:59 View session

jeremy force-pushed the fix/572-runner-test-discovery branch from b4ccca1 to 4f93478 Compare August 3, 2026 11:59

Copilot AI reviewed Aug 3, 2026

jeremy merged commit e9e55ed into main Aug 3, 2026
43 checks passed

jeremy deleted the fix/572-runner-test-discovery branch August 3, 2026 12:04

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

		grep -qF -- "$target_tool" <<<"$target_recipe" \
		\|\| hollow_targets+=("conformance-runner-tests-$target runs no '$target_tool'")

Conversation

jeremy commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The defect

The fix

Reachability is name AND placement, and Ruby is why

The content detector must not miss a valid test form either

Red proof

The Python detector had the same hole, and its fix has a second edge

Two more "recursive means the toolchain's rule, not all"

TypeScript could have left CI without this check noticing

@Test is not JUnit Jupiter's only test annotation

"recursive" does not mean "every subdirectory"

Swift: "under Tests/" is not the same as "compiled"

A gate that aborts is a gate that did not run

Self-test

Verification (real exit codes, measured on this commit)

Uh oh!

github-actions Bot commented Aug 3, 2026

Sensitive Change Detection (shadow mode)

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

jeremy commented Aug 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

jeremy commented Aug 3, 2026 •

edited

Loading

`@Test` is not JUnit Jupiter's only test annotation