Verify Swift against the shared conformance fixtures - #558
Merged
Merged
Conversation
github-actions
Bot
added
github-actions
Contributor
Sensitive Change Detection (shadow mode)This PR modifies control-plane files:
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
9 issues found across 9 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="conformance/runner/swift/Sources/ConformanceRunner/Runner.swift">
<violation number="1" location="conformance/runner/swift/Sources/ConformanceRunner/Runner.swift:13">
P3: The skip override is enabled by any environment-variable presence rather than by the documented value `1`; compare the value explicitly so `SWIFT_CONFORMANCE_NO_SKIPS=0` or an empty inherited variable cannot alter conformance coverage.</violation>
<violation number="2" location="conformance/runner/swift/Sources/ConformanceRunner/Runner.swift:209">
P2: Non-HTTP invalid schemes can crash the whole conformance process: the probe router only recognizes `http`, although the SDK rejects every parsed non-HTTPS scheme; route all non-HTTPS schemes through the child probe except the localhost HTTP(S) carve-out.</violation>
</file>
<file name="conformance/runner/swift/Sources/ConformanceRunner/Fixtures.swift">
<violation number="1" location="conformance/runner/swift/Sources/ConformanceRunner/Fixtures.swift:58">
P2: Non-integral JSON numbers used in semantic integer fields are silently coerced downstream into `0` or omitted, which can send requests to the wrong resource and falsely exercise a different case; validate and throw on non-integral integer parameters instead of exposing only an optional accessor.</violation>
<violation number="2" location="conformance/runner/swift/Sources/ConformanceRunner/Fixtures.swift:138">
P2: Malformed mock fixtures can become false-positive Swift conformance tests because missing `mockResponses`/`assertions` are silently treated as valid empty collections; preserving this default also undermines the documented fail-loudly policy. Require these fields for mock cases or reject their absence during decoding instead of defaulting them.</violation>
<violation number="3" location="conformance/runner/swift/Sources/ConformanceRunner/Fixtures.swift:159">
P2: A response containing `status` plus `networkError:false` bypasses the Swift runner's malformed-fixture backstop and is treated as a normal success; preserve field presence and reject any `networkError` value other than the schema's literal `true`.</violation>
</file>
<file name="conformance/runner/swift/Sources/ConformanceRunner/Assertions.swift">
<violation number="1" location="conformance/runner/swift/Sources/ConformanceRunner/Assertions.swift:43">
P2: `errorType: "ambiguous"` can never pass even though Swift maps `.ambiguous` into the conformance vocabulary; include `"ambiguous"` in `knownErrorTypes`.</violation>
<violation number="2" location="conformance/runner/swift/Sources/ConformanceRunner/Assertions.swift:82">
P2: `errorField` assertions expecting JSON null fail when the observed optional field is nil; handle `.null` as equal to a nil actual value before the string fallback.</violation>
<violation number="3" location="conformance/runner/swift/Sources/ConformanceRunner/Assertions.swift:279">
P2: Per-request `headerInjected` assertions ignore their `index`, so retries and multi-hop flows can validate the first request instead of the requested request. Resolve `assertion.requestIndex` with `resolveRequestIndex` and read `captured[idx]`, matching `headerPresent`/`headerAbsent` and the other runners.</violation>
</file>
<file name="conformance/runner/swift/Sources/ConformanceRunner/Dispatch.swift">
<violation number="1" location="conformance/runner/swift/Sources/ConformanceRunner/Dispatch.swift:265">
P2: Pagination overrides are ignored for the timeline and report operations, so fixtures that set `configOverrides.maxItems` will exercise unbounded pagination instead of the requested cap and may fail request-count assertions; pass `maxItems` through the corresponding Swift service options, as the other runners do.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
jeremy
added a commit
that referenced
this pull request
Aug 1, 2026
…DK (#563) * Pin the download hop-1 retry policy in SPEC §14 and the shared fixtures The two-hop download algorithm prescribed throw-on-error with no retry, while the conformance fixtures asserted retry the four runners had to skip. Make §14 honest: hop 1 retries on network errors plus {429, 502, 503, 504} — never 500 — with the per-SDK attempt budgets (public total-attempt caps floored at one for Python/Ruby/Kotlin, a fixed three-attempt policy for TypeScript/Swift), and hop 2 stays no-retry, no-auth. Fixture changes: the 503 case now asserts Authorization on EVERY hop-1 attempt (not just request 0) and drops its wrong all-5xx description; new cases pin network-error retry and 500 non-retry. Appendix D gains the two new rows. The eight download-retry roster lines (Python, Ruby, TypeScript, Kotlin × 2) leave §19 — the per-SDK commits remove the corresponding runner skips. * TypeScript: extract executeWithRetry and route download hop 1 through it Lift the retry loop out of createRetryingFetch into retry.ts as executeWithRetry(makeAttempt, config, emit) with four seams — attempt begin/finalize/retrying emission, caller-owned attempt preparation (request rebuild + auth refresh, terminal-marked so auth faults keep their identity past retry classification), per-attempt fetch, and the signal-aware backoff sleep. The client's middleware path delegates to it unchanged. downloadURL's hop 1 adopts the primitive with the fixed SPEC §14 policy passed directly (three attempts when enableRetry, retryOn {429, 502, 503, 504}, never 500; DownloadURL is deliberately absent from behavior-model.json). Hop 2 stays raw fetch: no retry, no auth — pinned by a native test asserting Authorization on every hop-1 attempt and never on hop 2. Native tables pin the complete retry set, 500 non-retry, the enableRetry=false single attempt, timeout-abort terminality, balanced start/end/onRetry hooks across retries, and client-level Retry-After. The runner drops its two download skips and runs downloads.json retry-enabled. * Python: route download hop 1 through the retry loop with the declared set get_no_retry becomes get_download: the authenticated hop-1 GET now runs through _request_with_retry with an explicit retry_on set — SPEC §14's {429, 502, 503, 504}, never 500 — passed directly because DownloadURL is deliberately absent from behavior-model.json. The declared set is authoritative in both directions, carving downloads out of the ungoverned GET taxonomy (which retries all 5xx). Network errors keep §7's classification; the public max_retries total-attempt cap applies, floored at one. Hop 2 stays a fresh unauthenticated client: no retry, no auth. Sync and async transports change identically. Native tables (sync + async) pin the complete declared set, 500 non-retry, network-error retry, cap exhaustion, the max_retries: 0 single-attempt floor, Authorization on every hop-1 attempt and never on hop 2, balanced start/end/on_retry hooks, Retry-After on 429, and async cancellation as terminal. The runner drops its two download skips (zero skips remain) and aligns delayBetweenRequests to the first-gap semantic the Go, TypeScript, and Kotlin runners already use — the download flow's final gap is the redirect hop to the signed URL, which is deliberately un-delayed. * Ruby: route download hop 1 through the retry loop with the declared set get_no_retry becomes get_download: the authenticated hop-1 GET now runs through request_with_retry with an explicit retry_on set — SPEC §14's {429, 502, 503, 504}, never 500 — passed directly because DownloadURL has no behavior-model entry. The declared set is authoritative in both directions, carving downloads out of the ungoverned GET taxonomy (which retries all retryable 5xx). Status-less network errors keep the taxonomy's judgment. The download attempt budget is the public max_retries total-attempt cap floored at one — FOR DOWNLOADS ONLY, by sharing the governed caller_cap shape. The ungoverned general path keeps its unfloored cap; its max_retries: 0 zero-attempt behavior is tracked separately (#532). Hop 2 stays a bare Net::HTTP GET: no retry, no auth. Native tests pin the complete declared set status by status, 500 non-retry, network-error retry, cap exhaustion, the max_retries: 0 single-attempt floor, Authorization on every hop-1 attempt and never on hop 2, balanced start/end/on_retry hooks, and Retry-After on 429. The runner drops its two unwaivered download skips (11 waivered 2B.3 GET-only skips remain) and aligns delayBetweenRequests to the first-gap semantic the Go, TypeScript, and Kotlin runners already use — the download flow's final gap is the redirect hop to the signed URL, which is deliberately un-delayed. * Kotlin: route download hop 1 through the retry loop with the declared set downloadURL built a one-shot Ktor client with followRedirects off and fired a single unguarded request through it, so the authenticated hop surfaced every 429/502/503/504 straight to the caller and the two download-retry conformance fixtures had to be skipped. Hop 1 now runs in downloadHop1, a directive-shaped loop mirroring BasecampHttpClient.requestWithRetry (#517): the catch clauses only classify the attempt's outcome, and the retry side effects — onRetry, the backoff sleep, the next attempt — run outside any catch, so a CancellationException from the sleep propagates raw and no phantom request events fire for an attempt that already ended. The declared set is {429, 502, 503, 504} plus network errors; 500 stays out, matching the main GET loop's declared-set discipline rather than the error taxonomy's broader all-5xx flag. The budget is the public maxRetries total-attempt cap coerced to at least one, so an accepted maxRetries = 0 still sends exactly one attempt, and enableRetry = false collapses it to one. Every attempt re-runs the auth strategy so a rotated token is picked up. A throwing strategy is a configuration fault, not a transport fault: it surfaces raw through the DownloadAuthFailure tag, spends no retry budget, and matches BasecampHttpClient's auth-phase classification. The signed second hop is untouched — no retry, no auth. KOTLIN_SKIPS is now empty: the two download-retry fixtures run, and the remaining architectural skip is the tag-based link-header branch. * Swift: route download hop 1 through a retry loop with the declared set performDownloadRequest fired exactly one authenticated request, so downloadURL surfaced every 429/502/503/504 from the API hop straight to the caller — the one leg of the two-hop flow where a transient gateway blip is worth another attempt. It now runs the #517 directive loop: the do/catch classifies the attempt's outcome into done/retry/fail and the loop tail acts on it, so onRetry, the backoff sleep, and re-authentication never run inside a catch clause and a CancellationError from the sleep propagates raw. The policy is fixed and passed directly rather than looked up by operation — DownloadURL is deliberately absent from behavior-model.json, so there is no per-operation RetryConfig to resolve and no public numeric knob. Three attempts when enableRetry is true, exactly one when false; the declared set is {429, 502, 503, 504} plus network errors, never 500; exponential backoff from the 1-second base, honoring Retry-After on 429. Every attempt re-authenticates so a rotated token is picked up. The signed second hop is untouched: fetchSignedDownload still fires once, bare. Native tests pin the complete retry set status by status, the 500 non-retry, the fixed three-attempt exhaustion, the enableRetry = false single attempt, Authorization on every hop-1 attempt and never on hop 2, balanced start/end/onRetry hooks across retries, Retry-After, and cancellation during backoff. The Swift conformance runner lands separately (#558). * Go: pin the download hop-1 retry set that the implementation already honors Go's fetchAPIDownload has always retried the declared hop-1 set, but the tests only covered three of its five edges: 503, 429-with-Retry-After, and network errors. 502 and 504 rode along untested, the 500 carve-out was only implied by a 404 case, and nothing asserted that a retried attempt still carries Authorization — the property that makes hop-1 retry safe to add elsewhere. Two pins, no behavior change. A table walks the complete set status by status, including the 500 row that must stay at one attempt and surface a non-retryable error. A second test records the Authorization header on every authenticated-hop attempt and on the signed hop, asserting the credential boundary holds across a retry. Both fail against mutated implementations: narrowing the switch to {429, 503} reddens the 502 and 504 rows, widening it to all 5xx reddens the 500 row, and authenticating only on attempt 1 reddens the header test with an empty Authorization on attempt 2. * Point the retry-metadata token smoke at the relocated consumption sites Routing download hop 1 through the shared retry primitives moved two of the guard's watch points. TypeScript lifted the loop out of client.ts into retry.ts, so client.ts no longer spells retryOn.includes or the backoff math; Ruby's status gate became retry_eligible?, where the declared set arrives either from the operation's metadata or from an explicit set like the download flow's, so fetch("retryOn") and the membership test are no longer one expression. The guard now watches where the behavior actually lives: a client.ts row for resolving the per-operation tuple and handing it to executeWithRetry, a new retry.ts row for the loop that consumes it, and a Ruby row that requires the metadata read and the membership test as separate tokens. Strength is unchanged — stubbing config.retryOn.includes in retry.ts or declared.include? in http.rb still reddens the check. * Scope the download hop-1 headers to what SPEC §14 allows in Python and Ruby §14 has always said hop 1 sets Authorization and User-Agent only — "no Accept or Content-Type — this is a binary download, not a JSON API call". Python and Ruby never honored it: their download hop borrows the generic request path, which stamps Accept: application/json on every request. Go, TypeScript, Kotlin and Swift build the hop-1 headers themselves and were already correct, so this was a two-SDK divergence against an explicit clause, predating the retry work. Both transports gain an accept seam threaded from the retry entry point down to the header builder, defaulting to the JSON Accept so every other caller is untouched; the download hop passes none. Python's 401-refresh replay and Ruby's carry it through, so a refreshed retry does not silently re-acquire the header. What remains on the wire is the HTTP library's own default (httpx sends Accept: */*, Faraday sends nothing) — not something the SDK sets. The tests pin that honestly: Python asserts hop 1's Accept is identical to the bare hop-2 client's and never application/json; Ruby asserts no attempt carries the JSON Accept. Both fail against the un-fixed transports with the header present on every hop-1 attempt. * Share Kotlin's auth-phase tag and bound the Swift cancellation poll The download hop-1 loop had declared its own DownloadAuthFailure, an exact copy of BasecampHttpClient's AuthPhaseFailure — same shape, same job: mark an auth-strategy throw so it surfaces raw and spends no retry budget. Two copies means the classification can drift per hop, so AuthPhaseFailure is now internal and the download loop uses it. The Swift cancellation test spun on `while transport.requests.isEmpty` with no ceiling, so a regression that never issued the first attempt would hang the suite instead of failing it. It now gives up after five seconds with a named failure. * Use assert_not_equal for the Ruby Accept-header pin rubocop's Rails/RefuteMethods prefers the assert_not_* spelling. * Apply ruff format to the Python accept seam ruff's line-length allows the single-line call. * Tighten the round-two review findings across the spec, fixtures and tests Five corrections from the second review pass. SPEC §14 claimed the download set "stays aligned with the main GET loop's @retryable classification". That is true only of Go's hand-written client. Every operation in behavior-model.json declares retry_on {429, 503}, so the download set is broader than the per-operation sets and narrower than the taxonomy's all-5xx flag — a third, independently declared set. §14 now says so instead of implying an inheritance that does not exist. The 500 no-retry fixture asserted its request count and status but not that its one hop-1 attempt carried Authorization, leaving the PR's own credential invariant unchecked on the one case that never retries. It asserts it now. Ruby's Accept pin only forbade the literal application/json. Faraday supplies a */* default once the SDK stops setting one, so the header cannot be absent outright — but the exact value is pinnable, and that catches any Accept the SDK might start attaching, not just the JSON one. Red against the un-seamed transport. The async Python transport had no delay assertion at all; its sync twin pinned Retry-After and it did not. It does now. Kotlin's download loop re-declared safeOnRequestStart, safeOnRequestEnd and safeOnRetry verbatim from BasecampHttpClient. Same consolidation as the auth-phase tag: the three are internal in the http package and the download loop imports them, so hook-exception behavior cannot drift between the two request paths. * Count the 401 refresh replay, make Swift cancellation terminal, restore runner strictness Six findings from the third review pass, three of them real defects. Swift treated a mid-flight CancellationError as a retryable network error: a cancelled download made all three attempts and fired onRetry twice before surfacing the cancellation wrapped as a network failure. It is now classified terminal inside the generic catch — after the request-end event, so start and end stay paired and an observer never leaks a span — and propagates raw. Fixed in performRequest as well: same code, and Kotlin, Python and TypeScript all treat cancellation as terminal on both paths, so Swift was the outlier. Swift also crashed the process on a large Retry-After. UInt64(_:) on an out-of-range Double is a runtime trap, and Retry-After: 99999999999 is 9.9e19 ns against a 1.8e19 ceiling — SIGTRAP, from a response header. Both loops now convert through a clamped sleepNanoseconds. The 401 token-refresh replay lived inside the single-request primitive with its own counter, so max_retries: 0 still put two requests on the wire — an uncounted attempt, contradicting the total-attempt semantics settled in #461. Python and Ruby now replay from the retry loop, where it draws from the same budget. Direct single-request callers keep the in-primitive replay, so mutations do not lose 401 refresh. The conformance runners regressed the other way: Ruby and Python were the two runners validating EVERY inter-request gap, and this branch loosened them to the first gap to accommodate the download flow's un-delayed redirect hop. delayBetweenRequests now honors the schema's existing index as a gap selector, the strict every-gap default is back, and the download fixtures name their real retry gaps — so the 503 case validates both of its backoff gaps, which the old rule could not. Rounding out: SPEC §14 documents the per-attempt-timeout carve-out Kotlin inherits from its main GET loop, and the two Kotlin cap tests assert BasecampException.Api rather than the root type. * Await the async refresh, not its boolean bool(await tp.refresh()), not await bool(...) — mypy caught the inversion. * Restore SPEC §4's refresh replay and teach every runner the retry-gap index Two corrections from the fourth review pass. The previous commit made the 401 token-refresh replay draw from the transient-retry budget. That regressed SPEC §4, which is explicit: refresh is attempted at most once PER REQUEST, tracked "with a boolean (e.g. refresh_attempted) rather than a counter", and it is not subordinate to the retry cap. Under the change, max_retries: 1 refreshed the token and then rethrew the stale 401 without ever sending the refreshed request — for every GET in Python and Ruby, not just downloads. Reverted. The tension that prompted it is real but is a spec question, not a bug: §4 replays once per request, §14 promises exactly one hop-1 attempt when retry is disabled, and a 401 with retry off satisfies only one of them. #565 is where that gets settled. Both SDKs now pin the behavior the spec actually prescribes, so changing it later has to be a decision rather than a drift. The download fixtures also asserted a second retry gap that three runners could not see: Go, TypeScript and Kotlin always measured requestTimes[1] - requestTimes[0] and ignored the index, so an implementation that dropped its second backoff still passed there. All five runners now resolve the gap index, and a named gap that does not exist fails rather than passing silently — otherwise a dropped retry makes the assertion vanish instead of firing. Proven by raising the gap-1 minimum: Go reports "Expected delay >= 16m39.999s at gap 1, got 2.028111541s", the second exponential backoff it previously never looked at.
jeremy
added a commit
that referenced
this pull request
Aug 1, 2026
…annot reach
Second review round on this PR. Five findings from cubic, two from the B4
agent, and one class the round exposed that neither had named: the runners'
assertion logic had no tests of its own, so every bounds branch here was
written blind and verified only by fixtures that pass.
An index of Int.MAX_VALUE crashed the Kotlin runner instead of failing its
assertion: `gap + 1 >= size` does the addition first, so the value wraps
negative and sails through the guard into an out-of-bounds read. Go had the
identical shape. Both now compare against the gap count and never add. Ruby
and Python were already safe — they compare against the gap count directly.
TypeScript did not reject a negative gap index at all: `toBeGreaterThan(index
+ 1)` is trivially satisfied by a negative, so `times[-1]` produced NaN and a
message about a delay rather than an out-of-range index — contradicting what
the other four runners did. All five now distinguish the two failures: a
negative index is refused categorically, which has nothing to do with how many
requests were made.
A `min` of zero silently disabled the whole assertion in Python and Ruby. Both
gated the check on the value's truthiness, and `0` is falsy in Python while an
absent `min` is nil in both — so `{"type": "delayBetweenRequests", "min": 0}`
asserted nothing at all. That is the exact false-green class this PR exists to
kill, reintroduced one layer up. The default now lands INSIDE the checked
function rather than at the call site, so no caller can gate it away, and a
zero minimum still requires the gap to EXIST.
Making the Kotlin index nullable to tell "omitted" from "0" also regressed
diagnostics: requestPath, requestMethod, requestBody and the header assertions
interpolated `assertion.index` into their failure messages, which printed
`index null` for an assertion that had actually used index 0. Each branch now
resolves the index once and reports the resolved value.
The Swift cancellation test caught every error and asserted nothing about it,
so it would have passed even if the cancellation were wrapped as
BasecampError.network — the exact regression it exists to catch. It now
requires URLError(.cancelled) to arrive raw. Inserting isCancellation above
sleepNanoseconds had also orphaned a doc comment, leaving generated
documentation claiming isCancellation clamps Retry-After; the comments are
reordered onto their own functions.
Underneath all of that: the delayBetweenRequests contract is now one function
per runner (checkDelayGaps / DelayGaps.check / check_delay_gaps) with a
committed unit test beside it, wired into `make check` and CI through a new
conformance-runner-tests target. Ten cases per runner, each named for a
behavior that regressed here or on #563 — omitted index catching a LATER
failing gap, omitted index with zero gaps failing, a named gap the run never
produced failing, a negative index rejected, an enormous index failing without
overflow, and a zero minimum still requiring the gap to exist. The runners are
test harnesses, but their assertion logic is code, and its bounds branches
never execute against a fixture that passes — which is precisely how #563
shipped an assertion that vacuously passed.
One residual false-green closed with them: an omitted index over zero gaps
passed in all five runners while checking nothing. Existing fixtures happen to
pair the delay assertion with requestCount, which limited the exposure, but the
contract itself was unsound and #558 is about to build conformance claims on
it.
conformance/schema.json and SPEC §19 now state the whole contract: never
vacuous, bounds-checked unconditionally, negatives rejected.
jeremy
added a commit
that referenced
this pull request
Aug 1, 2026
…sion's cancellation (#568) * Make the gap index bite in every runner and catch URLSession's cancellation Three follow-ups from the fourth review pass, all on code this branch added. The schema documents an omitted delayBetweenRequests index as "require the minimum on every gap", but Go, TypeScript and Kotlin still measured only gap 0 — so the documentation promised coverage three runners did not provide. All five now iterate every gap when the index is omitted. A NAMED gap that does not exist now fails in all five. The bounds check had been sitting inside each runner's "did we record two requests" guard, so a fully dropped retry left one request and the assertion evaporated instead of firing — the worst outcome for a timing pin. Ruby and Python additionally reject a negative index rather than wrapping to the end the way the per-request assertions do; there is no sensible "last gap" semantic when the point is to name a specific backoff. Swift only recognised Swift-concurrency cancellation. URLSession reports a cancelled task as URLError(.cancelled), not CancellationError, so a genuinely cancelled download was still classified a retryable network blip and spent the whole budget. Both loops now test for either shape through a shared isCancellation, pinned by a test that fails four ways against the previous commit. * Make the gap-index bounds bite, and unit-test the branches fixtures cannot reach Second review round on this PR. Five findings from cubic, two from the B4 agent, and one class the round exposed that neither had named: the runners' assertion logic had no tests of its own, so every bounds branch here was written blind and verified only by fixtures that pass. An index of Int.MAX_VALUE crashed the Kotlin runner instead of failing its assertion: `gap + 1 >= size` does the addition first, so the value wraps negative and sails through the guard into an out-of-bounds read. Go had the identical shape. Both now compare against the gap count and never add. Ruby and Python were already safe — they compare against the gap count directly. TypeScript did not reject a negative gap index at all: `toBeGreaterThan(index + 1)` is trivially satisfied by a negative, so `times[-1]` produced NaN and a message about a delay rather than an out-of-range index — contradicting what the other four runners did. All five now distinguish the two failures: a negative index is refused categorically, which has nothing to do with how many requests were made. A `min` of zero silently disabled the whole assertion in Python and Ruby. Both gated the check on the value's truthiness, and `0` is falsy in Python while an absent `min` is nil in both — so `{"type": "delayBetweenRequests", "min": 0}` asserted nothing at all. That is the exact false-green class this PR exists to kill, reintroduced one layer up. The default now lands INSIDE the checked function rather than at the call site, so no caller can gate it away, and a zero minimum still requires the gap to EXIST. Making the Kotlin index nullable to tell "omitted" from "0" also regressed diagnostics: requestPath, requestMethod, requestBody and the header assertions interpolated `assertion.index` into their failure messages, which printed `index null` for an assertion that had actually used index 0. Each branch now resolves the index once and reports the resolved value. The Swift cancellation test caught every error and asserted nothing about it, so it would have passed even if the cancellation were wrapped as BasecampError.network — the exact regression it exists to catch. It now requires URLError(.cancelled) to arrive raw. Inserting isCancellation above sleepNanoseconds had also orphaned a doc comment, leaving generated documentation claiming isCancellation clamps Retry-After; the comments are reordered onto their own functions. Underneath all of that: the delayBetweenRequests contract is now one function per runner (checkDelayGaps / DelayGaps.check / check_delay_gaps) with a committed unit test beside it, wired into `make check` and CI through a new conformance-runner-tests target. Ten cases per runner, each named for a behavior that regressed here or on #563 — omitted index catching a LATER failing gap, omitted index with zero gaps failing, a named gap the run never produced failing, a negative index rejected, an enormous index failing without overflow, and a zero minimum still requiring the gap to exist. The runners are test harnesses, but their assertion logic is code, and its bounds branches never execute against a fixture that passes — which is precisely how #563 shipped an assertion that vacuously passed. One residual false-green closed with them: an omitted index over zero gaps passed in all five runners while checking nothing. Existing fixtures happen to pair the delay assertion with requestCount, which limited the exposure, but the contract itself was unsound and #558 is about to build conformance claims on it. conformance/schema.json and SPEC §19 now state the whole contract: never vacuous, bounds-checked unconditionally, negatives rejected. * Report the real request count, and install the Ruby bundle before its helper test Third review round, two findings. The out-of-range diagnostic in Python and Ruby inferred the request count as `len(delays) + 1`. That inference assumes at least one request was made, so a run that failed during construction or dispatch — before the tracker recorded anything — reported "only 1 request(s) were made" when the true count was zero, pointing at the wrong failure. Both now take the count from the tracker and report it; Go, TypeScript and Kotlin already measured request times directly and were accurate. `make conformance-runner-tests` also ran `bundle exec` in the Ruby runner before anything had installed its gems. `make conformance` reaches the helper tests ahead of the `conformance-ruby` recipe that does the install, and invoking the target on its own had no installer at all, so a clean checkout hit a Bundler error instead of the test. It now mirrors the runner recipe with `bundle install --quiet` first.
Swift was the only SDK not fixture-verified — the #508/#509 retry-lifecycle bugs lived exactly where no runner watched. This runner executes every mock fixture in conformance/tests through the SDK's public Transport seam (no @testable): a scripted per-test response queue with network-error injection, monotonic request timing, and full request capture, feeding the shared assertion vocabulary (headerAbsent, index-aware headerPresent, errorMessage substrings, responseMeta, responseBody with 64-bit integer fidelity). Ported from the Kotlin runner at current main, including its policies: a mock body that flunks required-field decoding is a FAIL (a fixture bug to fix), not a silent skip; link-header requestCount fixtures are skipped because the SDK auto-paginates by design. Swift-specific mechanics: - The package depends on the ROOT distribution manifest: SwiftPM derives package identity from the directory name, so a path dependency on ".../swift" collides with conformance/runner/swift's own identity and is treated as a self-reference. Both manifests build the same swift/Sources/Basecamp sources. - HTTPS enforcement is a preconditionFailure (a trap, not a thrown error), so the runner re-executes itself as a --https-probe subprocess and expects the child to die; a surviving child is a FAIL, never a silent pass. Result today: 121 passed, 0 failed, 3 skipped of 124 — identical to Kotlin. The two download hop-1 retry skips are TEMPORARY until B4 threads retry through performDownloadRequest (proven to fail today via SWIFT_CONFORMANCE_NO_SKIPS=1: 1 request where 4 and 3 are expected).
Run conformance-swift inside the test-swift job (via the IS_MACOS-gated make target, same single-source-of-truth pattern as the drift gate) and add test-swift to the required "Conformance Tests" fan-in's needs — without the fan-in edge, Swift conformance could fail while the required check stayed green. The conformance-swift target lives in the Makefile's Swift section because its IS_MACOS ifdef must parse after the variable is defined; the conformance aggregate picks it up by name.
A verification sweep found the new runner reporting green on things it was not checking. A conformance runner that does that is worse than no runner: it converts an untested SDK into a tested-looking one. The delayBetweenRequests evaluator measured gap 0 and ignored the assertion's index, and skipped the check entirely below two requests — the identical defect #568 had just fixed in the other five. It mattered here and not hypothetically: the two download-retry fixtures this branch un-skips carry index 0 AND index 1 assertions, so the second backoff was never measured. checkDelayGaps moves into an SDK-free ConformanceSupport target with #568's semantics and its committed test roster ported; 12 of the 13 cases fail against the evaluator as shipped. An executable target carrying @main cannot host XCTest, which is why the split exists. The two download skips are gone. #563 landed the authenticated hop-1 retry, and both cases now pass live rather than being deleted on faith. Nothing capability-shaped is skipped any more: the roster is empty and the only standing exclusion is the architectural link-header branch Kotlin and TypeScript share. The rest are review findings on this branch's own code, each a way a malformed fixture could have passed: - A case with no assertions ran an operation and verified nothing. An empty assertions array is schema-legal, so the fixture gate does not catch it. - An absent mockResponses key decoded as an empty queue, which is a deliberate declaration for the HTTPS case and a malformed fixture everywhere else. The two no longer collapse. - networkError: false alongside a status slipped past the exactly-one-of backstop and was served as a plain success. The schema pins the literal true; so does the runner now. - Path parameters coerced to 0 when missing or non-integral, so the request went to a different resource and the scripted queue answered it anyway. They throw. The two timesheet arms used a `== 0` sentinel to pick between two spellings of one key, which could not tell an absent key from an id of zero; they ask for the first key present instead. - configOverrides.maxItems reaches the SDK through one dispatch arm. Any other operation would have paginated unbounded while the fixture believed it had capped the walk, so that now fails loudly instead. - headerInjected ignored its index and validated the first request, which the schema documents as index-aware and the other runners implement. - errorType: "ambiguous" could never pass despite Swift mapping it. - An explicit expected: null compared nil against the literal "null" and always failed. - The HTTPS probe routed only http:// to the crash child, but the SDK traps on every non-HTTPS scheme outside the localhost carve-out, so an ftp:// or ws:// fixture would have taken the whole run down mid-suite. - SWIFT_CONFORMANCE_NO_SKIPS was enabled by the variable's presence, so an inherited empty value silently changed coverage. It compares to "1". One of these was not a runner bug at all. The single-key array unwrap that lets list fixtures decode also rewrote {"payload_url": ["is not a valid URL"]} into a bare array, so the SDK found no field errors and reported "bad request" — a false FAIL, and the same heuristic could as easily manufacture a pass. Kotlin took the success-only status guard in #549; this port predated it and now carries it too. SPEC retires the "where a runner does not exist yet, e.g. Swift" carve-out, adds swift to the §21 gate roster, and rosters the one architectural skip.
jeremy
force-pushed
the
feat/swift-conformance-runner
branch
from
August 1, 2026 09:28
35432a0 to
97e700e
Compare
jeremy
changed the title
Member
Author
|
Landing status at `97e700ee3`. Review. cubic's first pass raised nine findings, all on this branch's own code. All nine are fixed and each thread carries a reply naming the fix — including the two where I disagreed with part of the reasoning and said so: the Census — cursor-paginated, count from Review bodies audited too, not just the thread ledger: cubic's summary reports "9 issues found across 9 files" and carries no CI. All checks settled, all green. The four reported as Six-runner proof, local, verified exit code (
CI's own numbers are the ones that gate; these are the pre-push check. Swift's single skip is the architectural |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 97e700ee34
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
requestCount only checked a lower bound on fixtures carrying a Link
rel="next" header, so an SDK that fetched too many pages passed. That is
backwards for the fixtures it covered: "Pagination stops at maxPages safety
cap" and "maxItems caps results across pages" both queue THREE pages and
expect TWO requests, and the whole point is that the walk stops early. A
lower bound cannot fail in the direction those cases exist to test.
The one fixture where a count genuinely does not apply to an auto-paginating
SDK — first page only, Link header present — is excluded by its own
link-header tag before it reaches the evaluator, so the relaxation was
protecting nothing that was still running.
Proven both ways against a simulated regression that drops the maxPages cap:
with the exact comparison the count assertion itself fails ("Expected 2
requests, got 3"); with the lower bound it stays silent and only the
unrelated meta.truncated assertion fires. The multi-page fixture has no such
companion assertion at all — requestCount and noError are the only two, so
an over-fetch there had nothing to catch it.
The transport keeps its auto-pagination tolerance. Answering an over-walk
with a terminal empty page rather than a 500 is what lets the tightened
assertion report a clean count mismatch instead of a decode error.
The suite is unchanged at 136 passed, 0 failed, 1 skipped: Swift already
made exactly the expected number of requests everywhere, so the lower bound
was pure false-green surface with no coverage behind it.
Kotlin's evaluator has the identical shape and the same fixtures; filed
separately rather than widened here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c1387898e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The runner checked the first request's METHOD against the fixture but never its PATH, and the scripted transport answers any URL. So an operation aimed at the wrong endpoint consumed the queued responses and passed anyway: its retry, status, error-mapping, auth-header and pagination assertions all hold just as well against a resource the fixture never named. Both invariants exist for the same reason and only one of them was there. The first request must now match the fixture's declared path with pathParams substituted, unless the fixture carries an explicit requestPath assertion — the same carve-out the method invariant already used. The comparison is EXACT, against both the account-scoped form the SDK builds and the absolute form the download fixtures state. Deliberately not a suffix test: /999/my/ projects.json ends with /projects.json and both endpoints exist, so a suffix match would wave through exactly the confusion this is meant to catch. A placeholder the params do not supply is reported by name rather than left to fail as a mismatch, which is the difference between a fixable fixture and a puzzling diff. Proven by pointing GetProject at a hardcoded id: 20 fixtures that passed before now fail, each naming the endpoint it expected and the one it got. Every one of those had been verifying error mapping or retry behavior against the wrong project. The renderer and the match rule live in ConformanceSupport with nine tests of their own, including the suffix collision and the wrong-account case. Unchanged at 136 passed, 0 failed, 1 skipped: all 135 fixtures that reach the check already went where they said they would.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c24d42b3b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Every invariant in the evaluator is guarded on having captured a request, so
an operation that short-circuits before the transport skips all of them at
once. The "HTTP allowed for localhost" fixture asserts only noError, so a
ListProjects that returned locally without dialing would have reported PASS
while exercising nothing — the runner green on a call it never watched.
A fixture that queues mock responses is testing a wire operation, so one has
to have happened. An EMPTY queue stays the deliberate no-request case: the
HTTPS-enforcement fixture makes no call at all and says so with requestCount
0. It is the only fixture in the suite with an empty queue, and no fixture
queues responses while expecting none to be used.
Proven by returning early from the ListProjects dispatch arm:
with the guard: FAIL: HTTP allowed for localhost
fixture queues 1 mock response(s) but the operation made
no request — it never reached the transport
without it: PASS, suite 136/0/1
Six runners green: Go 135/0/2, Kotlin 136/0/1, TypeScript 168 passed,
Ruby 126/0/11, Python 137/0/0, Swift 136/0/1.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ca900e4f6b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The path invariant covers every request; the method invariant still stopped
at the first. So a download could POST its signed final hop and satisfy
everything else — the transport answers any verb, and the path,
authorization, count and noError assertions were all still true.
Both now cover every request not named by an indexed assertion. Measuring
first: no hop in the suite currently diverges. Retries repeat the verb, a
redirect followed after a GET stays a GET, and the read-modify-write
composites — the one place a later hop legitimately differs — already pin
their hops with indexed requestMethod assertions.
Proven by making the redirect-following hop a POST in HTTPClient:
every hop: FAIL x5, e.g.
Expected request 1 to use method GET, got POST
Expected request 3 to use method GET, got POST
first only: PASS x5, suite 136/0/1
Six runners green: Go 135/0/2, Kotlin 136/0/1, TypeScript 168 passed,
Ruby 126/0/11, Python 137/0/0, Swift 136/0/1.
Member
Author
|
Landing at Review. Three reviewers over seven rounds. cubic raised nine findings in its first pass; Codex raised seven more across six rounds after undrafting, each one a genuine false green and each progressively narrower — from "the path is never checked at all" down to "this one fixture's guard." All sixteen are fixed, every thread carries a reply naming the fix, and eight of them carry a red proof showing the check fails against the un-fixed code. Two places I disagreed with part of a reviewer's reasoning and said so rather than quietly complying: the I also recorded one proof of my own that was wrong: an early attempt to red-prove the request-zero exemption moved both hops of the composite instead of one, so it went red under either behavior — it looked like a proof and demonstrated nothing. The real proof needed a patch to the SDK composite. Copilot errored on all eight attempts ( Census — cursor-paginated, count from Review bodies audited, not just the ledger: no CI. All settled, all green. The four reporting Six runners, local, verified exit code (
Swift skip roster: one line, architectural. "List operation returns first page with Link header" — the Follow-up filed: #573 — Kotlin's evaluator carries the same |
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header", where a first-page-only count genuinely does not
apply to an auto-paginating SDK. Kotlin and Swift already excluded that fixture
through a `link-header` tag branch; Go, Python, Ruby and TypeScript now do too,
which leaves nothing reaching the evaluator that needs a lower bound, so `>=`
becomes `!=` everywhere. Swift took this in #558 and its shape is what the five
now match. The MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance
stays: answering an over-walk with a terminal empty page rather than an error is
what lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Red proof 1 — the runners on the real fixture. Mutating "Auto-pagination
follows Link headers across multiple pages" to expect 2 while the SDK makes 3
is exactly the over-fetch the lower bound waves through. origin/main (5efc52f)
green, this branch red, in all five languages:
REAL_EXIT[main-go]=0 PASS: Auto-pagination follows Link headers ...
REAL_EXIT[branch-go]=1 FAIL: ... Expected 2 requests, got 3
REAL_EXIT[main-py]=0 PASS / REAL_EXIT[branch-py]=1 FAIL
REAL_EXIT[main-rb]=0 PASS / REAL_EXIT[branch-rb]=1 FAIL
REAL_EXIT[main-kt]=0 PASS: Passed: 142, Failed: 0
REAL_EXIT[branch-kt]=1 FAIL: ... Expected 2 requests, got 3
REAL_EXIT[branch-ts]=1 Error: [Auto-pagination ...] Expected 2 requests, got 3
(The fixture edit was reverted; conformance/tests/pagination.json is unchanged
in this commit.)
Red proof 2 — the new unit tests against the old predicate. Restoring the lower
bound in each support module fails the same three cases in every language:
Go 3 FAIL: RejectsAnOverFetch, MessageNamesBothCounts,
ZeroExpectedRequiresZeroActual
Python 3 failed, 3 passed
Ruby 6 runs, 3 failures
TS 3 failed | 3 passed
Kotlin 6 tests, 3 failures
Verification, exact-count code, real exit codes:
make conformance-go Passed: 140, Failed: 0, Skipped: 3 REAL_EXIT=0
make conformance-python 142 passed, 0 failed, 1 skipped REAL_EXIT=0
make conformance-ruby 131 passed, 0 failed, 12 skipped REAL_EXIT=0
make conformance-typescript 179 passed | 3 skipped REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go REAL_EXIT=0
make conformance-runner-tests-python 26 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 6 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin DelayGaps 10, ReplayDecoders 7,
RequestCount 6, 0 failures REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability REAL_EXIT=0
./scripts/check-replay-decoder-parity REAL_EXIT=0
make lint-actions REAL_EXIT=0
Every skip count rises by exactly one, the newly tag-excluded fixture, and no
test that was passing stops running.
SPEC §19's roster is restructured: the `link-header` exclusion is now one
shared architectural line covering all six runners rather than a per-runner
repeat, since Go, Python, Ruby and TypeScript joined Kotlin and Swift.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 4 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 4 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 4 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 4 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 4 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 6 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 6 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 6 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 6 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 6 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 6 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount`
assertion as a LOWER bound whenever any mock response carried
`Link: rel="next"`:
if autoPaginates {
if requestCount < expected { fail }
} else if requestCount != expected { fail }
That is backwards for the fixtures it covered. conformance/tests/pagination.json
queues MORE pages than it expects requests in two cases, because stopping early
is the behavior under test:
Pagination stops at maxPages safety cap 3 pages queued, 2 expected
maxItems caps results across pages 3 pages queued, 2 expected
Auto-pagination follows Link headers ... 3 pages queued, 3 expected
An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is
the exposed case: its only assertions are requestCount and noError, so an
over-fetch had nothing else to catch it — the other two happen to carry a
`responseMeta` truncated assertion that fires instead, coverage by luck.
The relaxation was protecting exactly one fixture, "List operation returns
first page with Link header" (tagged `link-header`), whose `requestCount: 1`
counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK.
With that one fixture's count taken out of scope, nothing reaching the
evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took
this in #558 and its shape is what the five now match. The
MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays:
answering an over-walk with a terminal empty page rather than an error is what
lets the tightened assertion report a clean count mismatch.
Each runner's predicate moves into that language's SDK-free support module
alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py,
`RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit
test per language. Those five test files are discovered automatically thanks to
#572; before it, three of them would have been executed by nothing.
Take the ASSERTION out of scope, not the CASE
-----------------------------------------------------------------------
An earlier revision of this commit did it the blunt way: it skipped the whole
`link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and
Swift already did. That is wrong, and wrong in this stack's own signature
shape.
The fixture carries THREE assertions, not one:
requestCount: 1 inapplicable — the SDK follows the Link header
statusCode: 200 perfectly applicable
noError perfectly applicable
Kotlin and Swift had always skipped the case, so the moment the other four
joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError`
assertions had been running in four runners; they then ran in zero. And nothing
reports that: the fixture still sits in `conformance/tests/pagination.json`,
still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the
build stays green over a fixture no runner executes. That is precisely #572's
defect — present in the tree, run by nothing — one layer down, committed inside
the stack that exists to close it.
So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and
its per-language spellings) returns false for `link-header`, the evaluator
skips that one assertion, and the case runs. Verbatim before/after from the
TypeScript runner, which is the arm where a skip is directly observable —
pre-narrowing files taken from this stack's pushed head a378d89, everything
else identical:
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design)
[...]
Tests 182 skipped (182)
REAL_EXIT=0
$ npx vitest run --reporter=verbose -t "List operation returns first page with Link header"
✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms
[...]
Tests 1 passed | 184 skipped (185)
REAL_EXIT=0
([...] elides vitest's per-file listing of the other 180-odd cases the `-t`
filter skipped, and its Test Files / Duration lines. The two quoted lines and
the Tests summary are verbatim. Zero passed becomes one passed: the case was
executed by nothing and now runs.)
Kotlin and Swift keep the whole-case skip, deliberately
-----------------------------------------------------------------------
They are not narrowed, and this is a stated exception rather than an oversight.
Both derive the response status from the last mock response the SDK consumed,
and an auto-paginating SDK walks past the end of a one-response queue, so
`statusCode` has nothing to report. Narrowing them was tried; both fail.
Verbatim, one line each from the pagination.json section plus the run tail:
$ make conformance-kotlin
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
$ make conformance-swift
FAIL: List operation returns first page with Link header
Expected status code 200, but got no response
[...]
Passed: 142, Failed: 1, Skipped: 0, Total: 143
REAL_EXIT=2
([...] elides the other 142 result lines of each run. Note the exit code:
through `make` a recipe failure is 2, not the runner binary's own 1.)
Widening those two runners' status model is separate work, not a skip to delete
here. Both call sites now carry that reasoning in a comment, so the asymmetry
does not read as drift to be "aligned" away.
Net effect on coverage: the fixture goes from run by four runners (before this
stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES
NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification
blocks below.
The general gap remains: nothing in the build detects a fixture that every
runner skips. That is filed as #602 and referenced from SPEC §19; this commit
fixes the instance, not the class.
Red proof 1 — the runners on the real fixture
-----------------------------------------------------------------------
Mutating "Auto-pagination follows Link headers across multiple pages" to expect
2 while the SDK makes 3 is exactly the over-fetch the lower bound waves
through. Base is 07482b9ba, this stack's #553 tip, byte-identical to
origin/main for every runner file involved. Each line is the fixture's own
result line plus that command's real exit status:
$ make conformance-<lang> # base 07482b9ba (lower bound)
PASS: Auto-pagination follows Link headers across multiple pages
REAL_EXIT[base-conformance-go]=0
REAL_EXIT[base-conformance-python]=0
REAL_EXIT[base-conformance-ruby]=0
REAL_EXIT[base-conformance-typescript]=0
REAL_EXIT[base-conformance-kotlin]=0
$ make conformance-<lang> # this commit (exact count)
FAIL: Auto-pagination follows Link headers across multiple pages
Expected 2 requests, got 3
REAL_EXIT[branch-conformance-go]=2
REAL_EXIT[branch-conformance-python]=2
REAL_EXIT[branch-conformance-ruby]=2
REAL_EXIT[branch-conformance-typescript]=2
REAL_EXIT[branch-conformance-kotlin]=2
(The PASS/FAIL lines above are one per language and identical in each; TS
prints its as `FAIL runner.test.ts > conformance/pagination.json >
Auto-pagination follows Link headers across multiple pages` with `Error:
[Auto-pagination follows Link headers across multiple pages] Expected 2
requests, got 3`. Every exit code is real and from `make`, which reports a
recipe failure as 2. The fixture edit was reverted; pagination.json is
unchanged in this commit.)
Red proof 2 — the new unit tests against the old predicate
-----------------------------------------------------------------------
Restoring the lower bound in each support module fails the same three cases in
every language. These are the bare test binaries, so the exit code is 1, not
make's 2:
Go --- FAIL: TestRequestCountRejectsAnOverFetch
--- FAIL: TestRequestCountMessageNamesBothCounts
--- FAIL: TestRequestCountZeroExpectedRequiresZeroActual
FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1
Python 3 failed, 6 passed in 0.10s REAL_EXIT=1
Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1
TS Tests 3 failed | 6 passed (9) REAL_EXIT=1
Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests,
3 failures — "an over-fetch fails", "the failure message names
both counts", "zero expected requires zero actual") REAL_EXIT=1
The three new `requestCountApplies` tests per language are not part of that
red: they pin the SCOPE of the exclusion, asserting that a `link-header`
fixture keeps its statusCode and noError assertions live. If the suppression
ever widens back to the whole case, those fail.
Verification, exact-count code, real exit codes, measured on this commit
-----------------------------------------------------------------------
make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0
make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0
make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0
make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0
make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0
make conformance-runner-tests-go ok (cached) REAL_EXIT=0
make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0
make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0
make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0
make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0
./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0
./scripts/check-runner-test-reachability --self-test 6 cases REAL_EXIT=0
./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0
cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0
make lint-actions No findings to report REAL_EXIT=0
And the same six suites at base 07482b9ba, for the skip-count comparison — the
figures the "no skip count changes" claim above rests on:
base conformance-go Passed: 141, Failed: 0, Skipped: 2
base conformance-python 143 passed, 0 failed, 0 skipped
base conformance-ruby 132 passed, 0 failed, 11 skipped
base conformance-typescript 174 passed | 2 skipped (176)
base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1
Identical skip counts throughout. TypeScript's total rises 176 → 185 because
this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on
both sides. (Those base figures come from the red-proof-1 base runs, which
carry the mutated expectation — a changed expectation moves no skip.)
The Go runner's `hasTag` helper is deleted. It existed only for the whole-case
branch this commit no longer has, and it had been inserted between
`goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment
as documenting `hasTag`.
SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner
repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" —
untrue, TypeScript had no `link-header` handling at all. It is now one block
that states what each runner excludes, why Kotlin and Swift differ, and what
the all-six-skipped shape would cost.
Local figures. Cite the CI job's own numbers when they differ.
Closes #573
jeremy
added a commit
that referenced
this pull request
Aug 3, 2026
…) (#596) Go, Python, Ruby, TypeScript and Kotlin all evaluated the `requestCount` assertion as a LOWER bound whenever any mock response carried `Link: rel="next"`: if autoPaginates { if requestCount < expected { fail } } else if requestCount != expected { fail } That is backwards for the fixtures it covered. conformance/tests/pagination.json queues MORE pages than it expects requests in two cases, because stopping early is the behavior under test: Pagination stops at maxPages safety cap 3 pages queued, 2 expected maxItems caps results across pages 3 pages queued, 2 expected Auto-pagination follows Link headers ... 3 pages queued, 3 expected An SDK that ignored the cap and walked all three passed `3 >= 2`. The third is the exposed case: its only assertions are requestCount and noError, so an over-fetch had nothing else to catch it — the other two happen to carry a `responseMeta` truncated assertion that fires instead, coverage by luck. The relaxation was protecting exactly one fixture, "List operation returns first page with Link header" (tagged `link-header`), whose `requestCount: 1` counts FIRST-PAGE requests only and so cannot apply to an auto-paginating SDK. With that one fixture's count taken out of scope, nothing reaching the evaluator needs a lower bound, and `>=` becomes `!=` everywhere. Swift took this in #558 and its shape is what the five now match. The MockEngine/httptest/MSW/respx/WebMock auto-pagination tolerance stays: answering an over-walk with a terminal empty page rather than an error is what lets the tightened assertion report a clean count mismatch. Each runner's predicate moves into that language's SDK-free support module alongside `delay_gaps` — request_count.go, `check_request_count` in runner.py, `RequestCount` in runner.rb, request-count.ts, RequestCount.kt — with a unit test per language. Those five test files are discovered automatically thanks to #572; before it, three of them would have been executed by nothing. Take the ASSERTION out of scope, not the CASE ----------------------------------------------------------------------- An earlier revision of this commit did it the blunt way: it skipped the whole `link-header` CASE in Go, Python, Ruby and TypeScript, copying what Kotlin and Swift already did. That is wrong, and wrong in this stack's own signature shape. The fixture carries THREE assertions, not one: requestCount: 1 inapplicable — the SDK follows the Link header statusCode: 200 perfectly applicable noError perfectly applicable Kotlin and Swift had always skipped the case, so the moment the other four joined them the fixture was skipped by ALL SIX. Its `statusCode` and `noError` assertions had been running in four runners; they then ran in zero. And nothing reports that: the fixture still sits in `conformance/tests/pagination.json`, still passes `conformance-fixtures-check` and `check-fixture-coverage`, so the build stays green over a fixture no runner executes. That is precisely #572's defect — present in the tree, run by nothing — one layer down, committed inside the stack that exists to close it. So the exclusion is now one assertion wide. `requestCountApplies(tags)` (and its per-language spellings) returns false for `link-header`, the evaluator skips that one assertion, and the case runs. Verbatim before/after from the TypeScript runner, which is the arm where a skip is directly observable — pre-narrowing files taken from this stack's pushed head a378d89, everything else identical: $ npx vitest run --reporter=verbose -t "List operation returns first page with Link header" ↓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header (TS SDK auto-paginates; follows Link headers by design) [...] Tests 182 skipped (182) REAL_EXIT=0 $ npx vitest run --reporter=verbose -t "List operation returns first page with Link header" ✓ runner.test.ts > conformance/pagination.json > List operation returns first page with Link header 19ms [...] Tests 1 passed | 184 skipped (185) REAL_EXIT=0 ([...] elides vitest's per-file listing of the other 180-odd cases the `-t` filter skipped, and its Test Files / Duration lines. The two quoted lines and the Tests summary are verbatim. Zero passed becomes one passed: the case was executed by nothing and now runs.) Kotlin and Swift keep the whole-case skip, deliberately ----------------------------------------------------------------------- They are not narrowed, and this is a stated exception rather than an oversight. Both derive the response status from the last mock response the SDK consumed, and an auto-paginating SDK walks past the end of a one-response queue, so `statusCode` has nothing to report. Narrowing them was tried; both fail. Verbatim, one line each from the pagination.json section plus the run tail: $ make conformance-kotlin FAIL: List operation returns first page with Link header Expected status code 200, but got no response [...] Passed: 142, Failed: 1, Skipped: 0, Total: 143 REAL_EXIT=2 $ make conformance-swift FAIL: List operation returns first page with Link header Expected status code 200, but got no response [...] Passed: 142, Failed: 1, Skipped: 0, Total: 143 REAL_EXIT=2 ([...] elides the other 142 result lines of each run. Note the exit code: through `make` a recipe failure is 2, not the runner binary's own 1.) Widening those two runners' status model is separate work, not a skip to delete here. Both call sites now carry that reasoning in a comment, so the asymmetry does not read as drift to be "aligned" away. Net effect on coverage: the fixture goes from run by four runners (before this stack) to run by four runners (after), instead of by zero. THIS COMMIT CHANGES NO SKIP COUNT IN ANY RUNNER — see the identical figures in the two verification blocks below. The general gap remains: nothing in the build detects a fixture that every runner skips. That is filed as #602 and referenced from SPEC §19; this commit fixes the instance, not the class. Red proof 1 — the runners on the real fixture ----------------------------------------------------------------------- Mutating "Auto-pagination follows Link headers across multiple pages" to expect 2 while the SDK makes 3 is exactly the over-fetch the lower bound waves through. Base is 07482b9ba, this stack's #553 tip, byte-identical to origin/main for every runner file involved. Each line is the fixture's own result line plus that command's real exit status: $ make conformance-<lang> # base 07482b9ba (lower bound) PASS: Auto-pagination follows Link headers across multiple pages REAL_EXIT[base-conformance-go]=0 REAL_EXIT[base-conformance-python]=0 REAL_EXIT[base-conformance-ruby]=0 REAL_EXIT[base-conformance-typescript]=0 REAL_EXIT[base-conformance-kotlin]=0 $ make conformance-<lang> # this commit (exact count) FAIL: Auto-pagination follows Link headers across multiple pages Expected 2 requests, got 3 REAL_EXIT[branch-conformance-go]=2 REAL_EXIT[branch-conformance-python]=2 REAL_EXIT[branch-conformance-ruby]=2 REAL_EXIT[branch-conformance-typescript]=2 REAL_EXIT[branch-conformance-kotlin]=2 (The PASS/FAIL lines above are one per language and identical in each; TS prints its as `FAIL runner.test.ts > conformance/pagination.json > Auto-pagination follows Link headers across multiple pages` with `Error: [Auto-pagination follows Link headers across multiple pages] Expected 2 requests, got 3`. Every exit code is real and from `make`, which reports a recipe failure as 2. The fixture edit was reverted; pagination.json is unchanged in this commit.) Red proof 2 — the new unit tests against the old predicate ----------------------------------------------------------------------- Restoring the lower bound in each support module fails the same three cases in every language. These are the bare test binaries, so the exit code is 1, not make's 2: Go --- FAIL: TestRequestCountRejectsAnOverFetch --- FAIL: TestRequestCountMessageNamesBothCounts --- FAIL: TestRequestCountZeroExpectedRequiresZeroActual FAIL github.com/basecamp/basecamp-sdk/conformance/runner/go REAL_EXIT=1 Python 3 failed, 6 passed in 0.10s REAL_EXIT=1 Ruby 9 runs, 13 assertions, 3 failures, 0 errors, 0 skips REAL_EXIT=1 TS Tests 3 failed | 6 passed (9) REAL_EXIT=1 Kotlin 23 tests completed, 3 failed (RequestCountTest: 6 tests, 3 failures — "an over-fetch fails", "the failure message names both counts", "zero expected requires zero actual") REAL_EXIT=1 The three new `requestCountApplies` tests per language are not part of that red: they pin the SCOPE of the exclusion, asserting that a `link-header` fixture keeps its statusCode and noError assertions live. If the suppression ever widens back to the whole case, those fail. Verification, exact-count code, real exit codes, measured on this commit ----------------------------------------------------------------------- make conformance-go Passed: 141, Failed: 0, Skipped: 2 REAL_EXIT=0 make conformance-python 143 passed, 0 failed, 0 skipped REAL_EXIT=0 make conformance-ruby 132 passed, 0 failed, 11 skipped REAL_EXIT=0 make conformance-typescript 183 passed | 2 skipped (185) REAL_EXIT=0 make conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0 make conformance-swift Passed: 142, Failed: 0, Skipped: 1 REAL_EXIT=0 make conformance-runner-tests-go ok (cached) REAL_EXIT=0 make conformance-runner-tests-python 29 passed, 31 subtests REAL_EXIT=0 make conformance-runner-tests-ruby 11 + 6 + 9 runs, 0 fail REAL_EXIT=0 make conformance-runner-tests-kotlin (--quiet, no output) REAL_EXIT=0 make conformance-runner-tests-swift 39 tests, 0 failures REAL_EXIT=0 ./scripts/check-runner-test-reachability 9 checks passed REAL_EXIT=0 ./scripts/check-runner-test-reachability --self-test 6 cases REAL_EXIT=0 ./scripts/check-replay-decoder-parity 5 checks, 31 operations REAL_EXIT=0 cd conformance/runner/go && go build ./... && go vet ./... REAL_EXIT=0 make lint-actions No findings to report REAL_EXIT=0 And the same six suites at base 07482b9ba, for the skip-count comparison — the figures the "no skip count changes" claim above rests on: base conformance-go Passed: 141, Failed: 0, Skipped: 2 base conformance-python 143 passed, 0 failed, 0 skipped base conformance-ruby 132 passed, 0 failed, 11 skipped base conformance-typescript 174 passed | 2 skipped (176) base conformance-kotlin Passed: 142, Failed: 0, Skipped: 1 Identical skip counts throughout. TypeScript's total rises 176 → 185 because this commit adds request-count.test.ts's 9 tests; its SKIPPED count is 2 on both sides. (Those base figures come from the red-proof-1 base runs, which carry the mutated expectation — a changed expectation moves no skip.) The Go runner's `hasTag` helper is deleted. It existed only for the whole-case branch this commit no longer has, and it had been inserted between `goSDKSkips`' doc comment and `goSDKSkips` itself, so godoc read that comment as documenting `hasTag`. SPEC §19: the `link-header` entry is rewritten. Previously it was a per-runner repeat that claimed Swift's skip was "identical to Kotlin and TypeScript" — untrue, TypeScript had no `link-header` handling at all. It is now one block that states what each runner excludes, why Kotlin and Swift differ, and what the all-six-skipped shape would cost. Local figures. Cite the CI job's own numbers when they differ. Closes #573
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Swift was the only SDK not fixture-verified; the #508/#509 retry-lifecycle bugs lived exactly where no runner watched. This adds the sixth conformance runner, executing every mock fixture in
conformance/tests/through the SDK's publicTransportseam — no@testableanywhere.Rebased over all six prerequisites (#563, #568, #571, #561, #560, #549). The gate is open and the temporary skips are gone.
Result — 136 passed / 0 failed / 1 skipped of 137
The one skip is architectural and shared with Kotlin and TypeScript:
link-header-tagged cases assert a first-page-onlyrequestCount, which does not apply to an SDK that auto-paginates by design. No capability skips.temporarySkipsis empty.Cross-runner, same fixture set (
make conformance,REAL_EXIT_CONFORMANCE=0): Go 135/0/2, Kotlin 136/0/1, TypeScript 168 passed, Ruby 126/0/11, Python 137/0/0, Swift 136/0/1.Acceptance evidence —
SWIFT_CONFORMANCE_NO_SKIPS=1, liveThe four named gates, by name, from
SWIFT_CONFORMANCE_NO_SKIPS=1 swift run ConformanceRunner:The two download-retry cases are the ones this branch previously skipped. They pass because #563 landed the hop-1 retry — deleted on evidence, not on faith.
REAL_EXITis grep-backed from the process, not read off a mid-run banner.The runner's own false greens, fixed before it gates anything
A verification sweep found this branch reporting green on things it was not checking. A runner that does that is worse than no runner: it converts an untested SDK into a tested-looking one.
The delay contract now bites in six runners, not five. The
delayBetweenRequestsevaluator measured gap 0, ignored the assertion'sindex, and skipped the check entirely below two requests — the identical defect #568 had just fixed in the other five. Not hypothetical here: the two download fixtures this branch un-skips carryindex: 0andindex: 1, so the second backoff was never measured.checkDelayGapsmoves into an SDK-freeConformanceSupporttarget with #568's semantics and its committed test roster ported — omitted index checks every gap, zero gaps fails, out-of-range fails, negative is rejected,Int.maxdoes not overflow.Red proof. Swapping the shipped evaluator back in, 12 of the 13 delay-gap cases fail (14 assertion failures). The one that passes is a green case that should. The split into a library target exists because an executable carrying
@maincannot host XCTest.resolveRequestIndexmoved alongside it, mainly to pin that it wraps on negatives whilecheckDelayGapsrejects them — they share theindexfixture key and are easy to conflate. Those six cases pass against the pre-fix resolver; they are characterization, not regression, and the file says so.Seventeen review findings, each a way a malformed fixture could have passed:
assertions: []is schema-legal)mockResponsesdecoded as an empty queuenetworkError: false+statusslipped past the exactly-one-of backstoptrue; so does the runner0/"", sending requests to the wrong resource== 0sentinels becamelongParam(anyOf:)configOverrides.maxItemssilently ignored outside one dispatch armheaderInjectedignored itsindexerrorType: "ambiguous"could never passexpected: nullalways failedAssertiondecodes presence;compareValuehandles.nullhttp://to the crash childftp://fixture would have killed the run mid-suiteSWIFT_CONFORMANCE_NO_SKIPStriggered on presence"1"requestCountwas a lower bound on auto-paginating fixturesrel="next"pins the following URLOne was not a runner bug at all. The single-key array unwrap that lets list fixtures decode also rewrote
{"payload_url": ["is not a valid URL"]}into a bare array, so the SDK found no field errors and reported"bad request". That surfaced as a false FAIL, but the same heuristic could as easily manufacture a pass. Kotlin took the success-only status guard in #549; this port predated it and now carries it too.And one more, found by Codex after undrafting.
requestCountchecked only a lower bound on fixtures carrying aLink rel="next"header. That is backwards for the fixtures it covered: three of them queue MORE pages than they expect requests, because stopping early is the behavior under test.An SDK that ignored
maxPagesand walked all three satisfied3 >= 2, so the cap assertions were unfailable in the only direction they exist to test. The third is the exposed case —requestCountandnoErrorare its only assertions, so an over-fetch had nothing else to catch it.Proven both ways against a simulated regression that drops the cap: with the exact comparison the count assertion itself fails (
Expected 2 requests, got 3); with the lower bound it stays silent and only the unrelatedmeta.truncatedassertion fires. The suite is unchanged at 136/0/1 — Swift already made exactly the expected number of requests everywhere, so the relaxation was pure false-green surface. The one fixture where a count genuinely does not apply to an auto-paginating SDK is excluded by itslink-headertag before reaching the evaluator, so it was protecting nothing still running.The transport keeps its auto-pagination tolerance on purpose: answering an over-walk with a terminal empty page rather than a 500 is what lets the tightened assertion report a clean count mismatch instead of a decode error. Kotlin's evaluator has the identical shape — filed as #573 rather than widening this PR.
And the one that mattered most, also Codex. The runner checked the first request's METHOD against the fixture but never its PATH, and the transport answers any URL — so an operation aimed at the wrong endpoint consumed the queued responses and passed anyway. Its retry, status, error-mapping, auth-header and pagination assertions all hold just as well against a resource the fixture never named. Both invariants exist for the same reason and only one was there.
The first request must now match the fixture's declared path with
pathParamssubstituted, unless the fixture carries an explicitrequestPathassertion. Exact, against both the account-scoped form the SDK builds and the absolute form the download fixtures state — deliberately not a suffix test, since/999/my/projects.jsonends with/projects.jsonand both endpoints exist.Red proof: pointing
GetProjectat a hardcoded id makes 20 previously-passing fixtures fail, each naming the endpoint it expected and the one it got. Every one had been verifying error mapping or retry behavior against the wrong project.Unchanged at 136/0/1 on the real code: all 135 fixtures that reach the check already went where they said they would.
Codex then found the follow-on in the same round: both implicit invariants stood down whenever the fixture carried any assertion of that type, regardless of which request it named. One fixture has exactly that shape —
edit-clearpinsrequestPathat index 1, the composite's PUT, and nothing at index 0 — so a read-modify-write composite's leading GET was unchecked. The exemption now requires an assertion that resolves to request 0.Isolated by diverging the composite's GET alone (the SDK's
editreads todo 777777, writes todo 456):Recorded because it nearly fooled me: my first attempt at that proof changed the
todoIdthe runner passes, which moves both hops — the run went red under either carve-out, looking like a proof while demonstrating nothing.Then the mirror image, same class from the other end: the invariant checked
captured.firstand stopped, so a composite's second hop was unpinned.update-preserves-due-onGETs a card and PUTs it back; a PUT that regressed while the GET stayed correct kept method, body, count andnoErrorall green.Rather than pin that one fixture, I measured whether the rule generalizes. Probing every hop across the suite, exactly five diverge, and all five are the download redirect and delegation flows (a 302 to a signed URL;
UploadsDownloadfollowing an upload'sdownload_url). Retries, pagination and every composite already stay on the fixture's path.So the rule is now every hop matches the fixture path unless an explicit
requestPathassertion names that index — and those five fixtures carry their own indexed assertions rather than the runner encoding a carve-out, since a carve-out is precisely what produced the previous round's bug. Those fixture assertions are the better half: they are shared, so all six runners now pin the redirect target instead of only counting hops.Red proof, diverging the card composite's PUT alone:
The last two, and the deepest. The path match accepted either the account-scoped form or the bare fixture path, so an SDK that dropped the account prefix and asked for
/projects.jsonsatisfied a fixture meaning/999/projects.json. Which form is required is now decided by the fixture, not by whichever the SDK sent.And
request.url?.pathdiscards the query — so refetching page 1 and following the link to page 2 were the same string to the runner, and the transport served page 2's body anyway. Three requests, three pages, green, while production would have looped. Requests now carry their query, and a response advertisingrel="next"pins the request that follows it to exactly that URL.Tightening the first exposed why the second is the right shape: with scoping enforced, five pagination fixtures failed, because their
Linkheaders are root-relative and the SDK correctly follows them unscoped. The hop was being judged by two rules that disagreed. A link-following hop is now governed by the link rule alone — the most specific available, and the stronger, since it pins the query too.Red proof, dropping the query inside the SDK's own
parseNextLink:And the last. Every invariant above is guarded on having captured a request, so an operation that short-circuits before the transport skips all of them at once.
HTTP allowed for localhostasserts onlynoError, so nothing was left holding it. A fixture that queues responses is testing a wire operation, so one must have happened — while an empty queue stays the deliberate no-request case (the HTTPS fixture, the only one, which declaresrequestCount: 0).Finally the symmetry: the path invariant covered every hop while the method invariant still stopped at the first, so a download could POST its signed final hop and satisfy path, authorization, count and
noError. Both cover every request now — measured first, and no hop in the suite diverges.Design
conformance/runner/swift/, port model = KotlinMain.kt: Codable fixture models, per-test scripted response queue, full assertion evaluator, 57-operation dispatch table.swift/Package.swift— SwiftPM derives package identity from the directory name, so a path dependency on.../swiftcollides withconformance/runner/swift's own identity and is silently treated as a self-reference. Documented in the manifest.preconditionFailure— a trap, not a thrown error — so the runner re-executes itself as a--https-probechild and requires it to die. A surviving child is a FAIL, never a silent pass.CI fan-in
Verified against
.github/workflows/test.ymlas it stands post-rebase, not assumed:test-swiftruns onmacos-15, so the Makefile'sIS_MACOSgate is satisfied rather than silently printingSKIP.make conformance-swift-runner-testsandmake conformance-swift.test-swiftis in the required Conformance Tests job'sneeds:[test-go, test-typescript, test-ruby, test-kotlin, test-python, test-swift].if: always()and fails when any need's result is notsuccess, so a Swift conformance failure fails the required check instead of leaving it green.make conformance-runner-testsalso reaches the Swift unit tests through the platform-gated sub-make, so the cross-language aggregate stays complete.SPEC
swiftadded to theconformancerunner roster.link-headerline, and an explicit note that Swift carries no capability skips.network-retry.json's "passes in all five runners" is now six.