Detect a conformance fixture case that every runner excludes (#602)

[jeremy]

Closes #602.

The question the census could not answer

#742 gave every runner a case census: at the end of its own run it asserts that
passed + failed + skipped equals the non-live case count under conformance/tests,
counted independently of its own load path. That catches a case executed by no runner for
a mechanical reason — an unrecognized mode, a fixture that failed to parse or was
never globbed, one nested where no runner looks, a case dropped between load and dispatch.

It cannot catch the case #602 actually names. When every runner excludes the same fixture
deliberately, all six censuses stay green — each one counted its own skip. Only a
comparison across runners sees it.

What this adds

Every runner writes the cases it did not execute, with reasons, to
conformance/manifests/<runner>.json. scripts/check-fixture-execution.rb reads all six
and fails when a case appears in every one.

On the current tree: 198 non-live cases across 6 runners; none excluded everywhere
(maximum overlap 2 of 6) — which is what #602 predicted and #596 narrowed it to.

Manifests rather than parsed output. Five runners print SKIP: <name>; TypeScript
does not, because a skip there is it.skip, reported in vitest's own format. A gate
scraping stdout would be blind to exactly one runner in the silent direction —
TypeScript would contribute an empty exclusion set and no case could ever reach all-six.

executed is recorded, not just the exclusions, and asserted against the census total
by both writer and reader. Without it, a case a runner silently dropped is simply absent
from its exclusion set — and "absent" reads identically to "ran fine", so the comparison
would score the case as covered by that runner precisely when it was not.

The absence rule is the design

A gate over six inputs is only as good as its behaviour when one is missing, and missing
is the normal state here: conformance-swift is macOS-only.

mode	requires	on a case excluded by every visible runner
FULL (default, macOS, CI)	all six manifests; fails on absence	fails
PARTIAL (--partial, Linux)	one or more	warns, exits 0

A missing manifest is never read as "that runner executed everything" — that assumption is
exactly what makes an all-six case invisible. Partial mode warns rather than fails because
five-of-six is not the all-six claim and Swift may well execute the case; a warning cannot
produce a false failure, which is the only reason it is allowed to run on partial input.
Both modes fail on zero manifests.

CI resolves it properly rather than living with the partial answer. The six language
jobs each upload their manifest; the existing conformance fan-in job — which already
needs: all six — downloads them and runs FULL mode. That step is ordered after the
results check, deliberately: the job is if: always(), so a language job that died before
its upload would otherwise be reported as a missing manifest, burying the real cause
behind a derived one.

The local target depends on conformance rather than trusting whatever manifests are on
disk, which would let it validate last week's exclusion sets.

Proving it can say no

Maximum overlap is 2 of 6, so the gate is green on arrival and a live run only proves it
can say yes. scripts/test-check-fixture-execution.rb crafts the state no committed
fixture can produce:

• a case excluded by all six → fails
• five of six → passes (the boundary; failing here would be a false alarm)
• a missing manifest in full mode → fails; the same input in partial mode → passes
• an all-visible exclusion in partial mode → warns, exits 0
• zero manifests, in both modes → fails
• executed + excluded != total → fails
• runners disagreeing on the case count → fails
• unknown runner, duplicate runner, malformed JSON, missing key → fail

It already earned its keep: it found a manifest shaped as a JSON string crashing with a
Ruby backtrace instead of naming the file.

Verified end-to-end as well — adding one shared exclusion to all six real manifests
makes the gate fail and print every runner's recorded reason.

Also

SPEC §19 now describes the gate that exists. This closes the gap #740's P1 found, where
the prose promised make check-fixture-execution after the source-text parser it referred
to had been withdrawn to conformance-skips-parser-archive.

#736 is not closed here, but this is its input: the roster stays restated-not-derived, and
these manifests are what make it checkable for set equality against what the runners
actually reported — stronger than parsing their source, which is why it waited.

Verification

• make check-fixture-execution — green in FULL mode on macOS (6 manifests), PARTIAL on Linux.
• scripts/test-check-fixture-execution.rb — all cases pass.
• make lint-actions (actionlint + zizmor) — clean on the workflow changes.
• make doc-constants-check, check-runner-test-reachability, check-gradle-serialization — green.
• Full make check running on macOS (first exercise of FULL mode inside make check) and on Linux.

---

Summary by cubic

Detects and fails a build when any conformance fixture case is excluded by all runners. Previously each runner’s census stayed green on all‑six exclusions; now each runner writes an execution manifest and a cross‑runner gate compares them.

• Manifests: every runner writes conformance/manifests/<runner>.json with runner, total_non_live, executed, and excluded entries keyed by [file, name]; files are sorted, gitignored, and written even on failing runs.
• Gate: scripts/check-fixture-execution.rb validates manifests and fails on an all‑six exclusion. FULL mode requires all six manifests; PARTIAL (--partial) accepts fewer and warns on “all‑visible” overlap. It rejects malformed JSON, missing keys, unknown/duplicate runners, duplicate case exclusions, executed + excluded != total_non_live, count disagreements, and zero manifests. A self‑test (scripts/test-check-fixture-execution.rb) proves the rejections.
• Freshness: conformance-manifests-reset is an order‑only prerequisite of all six language targets; it wipes conformance/manifests/ before any runner writes so a run never mixes stale manifests. check-fixture-execution depends on conformance and is included in check-targets.
• CI: each language job uploads its manifest (Ruby/Python uploads are gated to the matrix leg that runs conformance). The fan‑in job downloads all six, runs the gate’s self‑test, then runs FULL mode over the collected manifests. Artifact uploads set overwrite: true to allow a re‑run to replace its own manifest and avoid name conflicts.

^{Written for commit 3aed2b5. Summary will update on new commits.}

[github-actions]

Sensitive Change Detection (shadow mode)

This PR modifies control-plane files:

• .github/workflows/test.yml

Shadow mode — this check is informational only. When activated, changes to these paths will require approval from a maintainer.

[Copilot]

Pull request overview

Adds cross-runner detection for conformance cases deliberately excluded by all six SDK runners.

Changes:

• Writes execution manifests from every conformance runner.
• Adds full/partial fixture-execution validation and self-tests.
• Collects manifests in CI and documents the gate.

Tip

If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
.github/workflows/test.yml	Uploads and collects runner manifests.
.gitignore	Ignores generated manifests.
Makefile	Adds the local execution gate target.
SPEC.md	Documents census and cross-runner checks.
scripts/check-fixture-execution.rb	Implements manifest validation and overlap detection.
scripts/test-check-fixture-execution.rb	Tests gate failure modes.
conformance/runner/go/main.go	Records Go exclusions.
conformance/runner/go/manifest.go	Serializes the Go manifest.
conformance/runner/python/runner.py	Records and writes Python exclusions.
conformance/runner/ruby/runner.rb	Records and writes Ruby exclusions.
conformance/runner/typescript/case-census.ts	Adds TypeScript manifest serialization.
conformance/runner/typescript/runner.test.ts	Produces the TypeScript manifest.
conformance/runner/swift/Sources/ConformanceSupport/ExecutionManifest.swift	Serializes the Swift manifest.
conformance/runner/swift/Sources/ConformanceRunner/Runner.swift	Records Swift exclusions.
kotlin/conformance/src/main/kotlin/com/basecamp/sdk/conformance/ExecutionManifest.kt	Serializes the Kotlin manifest.
kotlin/conformance/src/main/kotlin/com/basecamp/sdk/conformance/Main.kt	Records Kotlin exclusions.

Suppressed comments (1)

.github/workflows/test.yml:695

• This upload runs in every Python matrix leg, but only the 3.13 leg runs the conformance runner and creates python.json. The other matrix legs will fail at if-no-files-found: error, so the Python job and fan-in gate cannot pass. Restrict the upload to the producing leg.

        run: |

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jeremy]

Round one at 351f62d9. Three findings, all Copilot, all correct — and two of them were defects in the gate rather than in its plumbing.

The CI failure (7 red checks, one cause): the Ruby and Python jobs are matrixed and run conformance on one version only, so my unconditional upload fired on every other leg with nothing to upload and if-no-files-found: error. Now carries the same matrix.ruby == '3.3' / matrix.python == '3.13' condition. Mirroring the condition rather than relaxing if-no-files-found, deliberately: a leg that does run conformance and produces no manifest is a real defect, and the fan-in gate's absence rule depends on that staying loud. Copilot flagged the Python half in its suppressed block too.

A soundness hole in --partial. The flag describes the input — "this run cannot produce all six" — not a softer verdict. With every runner reporting anyway (a macOS developer passing it out of habit, a CI step keeping it for safety), "excluded by all present runners" is the all-six claim, and the warning path would have let the one state this gate exists to reject exit 0. Partial handling now applies only when a manifest is genuinely absent. Self-test case added and shown to fail against the un-narrowed code first.

check-fixture-execution was not in check-targets. The edit meant to add it matched the first occurrence of its anchor — the .PHONY line — so the target existed, worked when invoked by hand, and make check never ran it. That is precisely the "present but executed by nothing" shape this PR is about, inside the PR that adds the gate. Now wired, verified by reading the check-targets line rather than grepping the file, which is what hid it.

Consequence worth stating: the make check runs I reported earlier were against a92317f, so they did not exercise this gate inside make check. Re-running both on this head.

[jeremy]

@codex review

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

Makefile:1094

• make check still does not run this target. check-targets at Makefile:1483 depends on conformance, but not on check-fixture-execution, so it only produces the manifests and never runs either the cross-runner comparison or its negative self-test. Add check-fixture-execution to that dependency list (the direct conformance dependency can then be removed because this target already includes it).

# the runners that produce them. Inside `make check` it is free, because
# conformance is phony and already a prerequisite there.

scripts/check-fixture-execution.rb:181

• --partial returns success unconditionally here, even when all six expected manifests are present. This can happen with a leftover Swift manifest on Linux, and then a genuine all-six exclusion is only warned about instead of failing. Use partial warning behavior only while at least one runner is actually missing; a complete manifest set should continue into the full failure path.

  # `--partial` is a statement about the INPUT, not a licence to soften the

.github/workflows/test.yml:941

• This CI path runs only the live, green-by-construction comparison; scripts/test-check-fixture-execution.rb is not invoked anywhere in the workflow. Because the committed manifests have no all-six overlap, the failure branch can regress to always-success while every required check stays green. Run the negative self-test here (or in spec-gates) so CI proves the new gate can reject its target state.

        run: ruby scripts/check-fixture-execution.rb

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 351f62d9fd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Makefile:1099

• Running conformance only overwrites manifests produced on this host; it never removes old ones. On Linux, a swift.json left by an earlier macOS run therefore survives, so --partial sees all six manifests and treats stale Swift exclusions as current. When the case count is unchanged, this can mask a newly all-six-excluded case (or cause a false failure). Clear the manifest directory before the conformance prerequisites run, not in this recipe after them.

check-fixture-execution: conformance

[jeremy]

Round two at e2c29d91. Four findings, all real — and two were defects in the gate's core claim rather than its plumbing.

Case identity was wrong. Codex found that fixture case names are not unique; I verified before acting, and it is worse than one collision — replace-omission-clears: sparse replace sends the request verbatim with no GET appears in three fixtures and POST operation does NOT retry (not idempotent) in two. Names are unique within a file, so identity is now [file, name], recorded by every runner and keyed on by the gate. The bad consequence was not the spurious integrity failure but the intersection: a name excluded by three runners in one file and three in another would have read as excluded by all six — a false failure on cases that all run.

Stale manifests, found by both bots, and created by my own --partial narrowing in the previous commit. On Linux conformance-swift is a no-op, so a swift.json from an earlier macOS run over the same checkout survives while the other five refresh; the gate then sees six, stops treating the run as partial, and compares five current sets against a stale sixth. Silent-wrong. The target now wipes the directory and owns the run, so "six manifests present" means "six runners reported in this run" — reset in the recipe, not as a prerequisite, since prerequisite order is not guaranteed under make -j. Verified by planting a bogus manifest and watching it get wiped.

The self-test did not run in CI. Reachable only through the Make target the workflow bypasses, so a regression making the gate accept a missing manifest or an all-six exclusion would have left CI green — check-targets is not CI coverage, in the PR that adds the gate. Now its own step.

New self-test cases: one name in two files must not fire; the same name in one file excluded everywhere still must; an exclusion missing its fixture file is rejected; one case excluded twice in a manifest is rejected.

Local: make check-fixture-execution green end to end (reset → six runners → FULL mode → self-test). make lint-actions clean. Full make check and CI re-running.

[jeremy]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: e2c29d913e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (6)

conformance/runner/go/manifest.go:76

• This consistency guard has no negative test: the normal runner always passes matching counts, and no Go test calls writeManifest, so removing or reversing the guard would leave the suite green. Add a unit test that supplies mismatched Executed/Excluded accounting and asserts that writing fails.

	if m.Executed+len(m.Excluded) != m.Total {
		return fmt.Errorf(
			"manifest for %s is internally inconsistent: %d executed + %d excluded != %d non-live "+
				"cases; the run dropped a case without recording it as either",
			m.Runner, m.Executed, len(m.Excluded), m.Total)

conformance/runner/ruby/runner.rb:185

• No Ruby runner test exercises this mismatch branch, while the live run always supplies consistent counts. Add a focused test for ExecutionManifest.write that expects ExecutionManifest::Error on inconsistent accounting; otherwise this writer-side assertion can regress without failing CI.

    if executed + excluded.length != total
      raise Error, "manifest for #{runner} is internally inconsistent: #{executed} executed + " \
                   "#{excluded.length} excluded != #{total} non-live cases; the run dropped a " \
                   "case without recording it as either"
    end

conformance/runner/python/runner.py:186

• The writer’s rejection path is not covered by the Python runner tests; committed fixtures only exercise internally consistent accounting. Add a test that calls write_execution_manifest with a mismatched total and verifies the RuntimeError, so this fail-closed invariant cannot silently disappear.

    if executed + len(excluded) != total:
        raise RuntimeError(
            f"manifest for {runner} is internally inconsistent: {executed} executed + "
            f"{len(excluded)} excluded != {total} non-live cases; the run dropped a case "
            "without recording it as either"

kotlin/conformance/src/main/kotlin/com/basecamp/sdk/conformance/ExecutionManifest.kt:55

• No Kotlin conformance unit test references ExecutionManifest, and normal fixtures always make this equation true. Add an ExecutionManifestTest mismatch case that expects ExecutionManifest.Error, so the writer-side integrity check is proven to reject bad accounting rather than only proven to accept good input.

        if (executed + excluded.size != total) {
            throw Error(
                "manifest for $runner is internally inconsistent: $executed executed + " +
                    "${excluded.size} excluded != $total non-live cases; the run dropped a case " +
                    "without recording it as either"

conformance/runner/typescript/case-census.ts:194

• writeExecutionManifest is invoked only with registered - excluded.length, so this mismatch branch is never exercised. Add a case to case-census.test.ts that passes inconsistent accounting and expects an exception; deleting this guard currently leaves all tests green.

  if (executed + excluded.length !== total) {
    throw new Error(
      `manifest for ${runner} is internally inconsistent: ${executed} executed + ` +
        `${excluded.length} excluded != ${total} non-live cases; the run dropped a case ` +
        "without recording it as either",

conformance/runner/swift/Sources/ConformanceSupport/ExecutionManifest.swift:67

• The new support writer has no corresponding Swift test, so its fail-closed accounting branch is only exercised on success by the real fixture tree. Add an ExecutionManifestTests mismatch case that asserts ManifestError.inconsistent; this target exists specifically to make support failure modes testable.

        guard executed + excluded.count == total else {
            throw ManifestError.inconsistent(
                "manifest for \(runner) is internally inconsistent: \(executed) executed + "
                    + "\(excluded.count) excluded != \(total) non-live cases; the run dropped a "
                    + "case without recording it as either")

[jeremy]

CI caught a cost I had not priced, at c62df667.

Making check-fixture-execution run $(MAKE) conformance itself (so it could wipe the manifest directory first) removed conformance from check-targets — and with it, conformance-kotlin stopped being reachable from check-targets in the graph check-gradle-serialization walks. That gate exists to keep two Gradle invocations off the same project directory, and it could no longer see one of them.

Its self-test is what failed, not the gate: conformance-kotlin edge removed: expected the gate to FAIL, it passed. A mutation case written for a different change entirely, catching this one. Worth saying plainly since this PR argues for exactly that kind of test.

Restructured so both properties hold rather than trading one for the other:

• conformance is back in check-targets; check-fixture-execution depends on it, restoring reachability.
• The reset is now a phony target that all six language targets take as an order-only prerequisite (|). That is what makes it correct under make -j — make builds a prerequisite to completion before any dependent starts, and builds a phony target once per invocation, so the reset cannot race the runners it protects. A plain prerequisite of the aggregate target would not do that; siblings may run concurrently.
• It fires for a single-language run too, deliberately: make conformance-go clears the directory, so it holds only what that invocation produced and the gate goes partial rather than silently mixing runs across machines.

Verified both directions: a planted stale swift.json is wiped and regenerated, and make conformance-go alone leaves only go.json. make check-gradle-serialization and its self-test are green.

Also worth noting the fan-in job behaved correctly through this: the results check failed first and the gate steps were skipped, rather than reporting a derived "missing manifest" for jobs that never ran. That ordering was itself a review finding earlier in this PR.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (6)

.github/workflows/test.yml:492

• On a workflow re-run, the prior attempt's artifact can still have this name. upload-artifact v7 defaults overwrite to false, so this step fails with a name-conflict error before the fan-in gate can run. Allow this runner to replace its previous-attempt manifest.

        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: execution-manifest-typescript
          path: conformance/manifests/typescript.json
          if-no-files-found: error

.github/workflows/test.yml:600

• On a workflow re-run, the prior attempt's artifact can still have this name. upload-artifact v7 defaults overwrite to false, so this step fails with a name-conflict error before the fan-in gate can run. Allow this runner to replace its previous-attempt manifest.

        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: execution-manifest-ruby
          path: conformance/manifests/ruby.json
          if-no-files-found: error

.github/workflows/test.yml:712

• On a workflow re-run, the prior attempt's artifact can still have this name. upload-artifact v7 defaults overwrite to false, so this step fails with a name-conflict error before the fan-in gate can run. Allow this runner to replace its previous-attempt manifest.

        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: execution-manifest-python
          path: conformance/manifests/python.json
          if-no-files-found: error

.github/workflows/test.yml:787

• On a workflow re-run, the prior attempt's artifact can still have this name. upload-artifact v7 defaults overwrite to false, so this step fails with a name-conflict error before the fan-in gate can run. Allow this runner to replace its previous-attempt manifest.

        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: execution-manifest-swift
          path: conformance/manifests/swift.json
          if-no-files-found: error

.github/workflows/test.yml:865

• On a workflow re-run, the prior attempt's artifact can still have this name. upload-artifact v7 defaults overwrite to false, so this step fails with a name-conflict error before the fan-in gate can run. Allow this runner to replace its previous-attempt manifest.

        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: execution-manifest-kotlin
          path: conformance/manifests/kotlin.json
          if-no-files-found: error

.github/workflows/test.yml:409

• On a workflow re-run, the prior attempt's artifact can still have this name. upload-artifact v7 defaults overwrite to false, so this step fails with a name-conflict error before the fan-in gate can run. Allow this runner to replace its previous-attempt manifest.

This issue also appears in the following locations of the same file:

• line 488
• line 596
• line 708
• line 783
• line 861

        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: execution-manifest-go
          path: conformance/manifests/go.json
          if-no-files-found: error

[jeremy]

Round three at 3aed2b54.

Copilot's six suppressed comments were all one real issue, and a good one: re-running a job creates a new attempt within the same run, and upload-artifact v4+ artifacts are immutable per run — so the second attempt fails on a name conflict with the first attempt's manifest, before the fan-in gate can run. Reachable exactly when someone is re-running to get a red PR green, which is the worst possible moment for the gate to become unreachable. overwrite: true on all six.

It is also right on the merits rather than merely avoiding the conflict: the collecting gate must read this attempt's exclusion set. An artifact left by a previous attempt is the CI-side version of the stale-manifest bug fixed two commits ago, and it takes the same answer — six manifests present must mean six runners reported in this run.

Codex's clean verdict is against e2c29d91, one commit behind the gradle-serialization restructure, so I am not counting it as a verdict on the current head; re-requested.

Status at the previous head c62df667: CI 27/27, fan-in job green with both steps running (self-test and live gate over six collected manifests), 0 unresolved threads.

[jeremy]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 3aed2b54c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.