Skip to content

ci: bulletproof test gates + binding test-quality contract for the port - #4

Merged
LukasParke merged 6 commits into
mainfrom
LukasParke/e2e-CI
Jul 30, 2026
Merged

ci: bulletproof test gates + binding test-quality contract for the port#4
LukasParke merged 6 commits into
mainfrom
LukasParke/e2e-CI

Conversation

@LukasParke

Copy link
Copy Markdown
Contributor

Why

This repo is a living port — every sync is a machine-generated diff to load-bearing concurrent code (tool loop, streaming fan-out, approval/HITL ordering, hooks). The only thing between a subtly-wrong port and main is what CI and the two gates actually check. They weren't checking enough.

I probed the tree rather than reading it. The first probe found a real bug:

Probe Before
go test ./... passes, 65.9% coverage
go test -race ./... FAILED — data race in TestHooksManagerAsyncDrain
Deterministic tests driving a successful stream zero (1 streaming fake, error-only; 26 non-streaming)
retryCurrentRequest — runs on the last turn of every tool conversation 0%; StrictFinalResponse had zero test references
FinishReasonIs — contract-Required API 0% — exported, so the presence check passed
stream_guards.go, tool_event_broadcaster.go, +2 zero callers anywhere

The race is the headline. It's in test code, but that's luck, not process: CI had never run -race on a package whose headline feature is concurrent stream fan-out and whose hooks manager is documented as concurrency-safe. Upstream ships a dedicated turn-end-race-condition.test.ts; this port had no counterpart and no gate.

FinishReasonIs is the second: listed in the contract's parity floor, exported, and never called by any test. A presence check structurally cannot see that. It's the exact failure mode PORTING.md says the gates exist to prevent — "a port which compiles cleanly while sitting a version behind on real behavior."

What changed

Gates-race (blocking), -shuffle, -count=1 (the cache was reporting (cached)), pinned staticcheck, a dependency-free goroutine-leak TestMain, and two coverage gates shared between CI and the port verifier:

  • Ratchet — coverage can't fall below .upstreamer/coverage-floor.txt; a real gain must be locked in by raising it. Measured deterministic across three runs, so it won't flake. Floor at 72.0 vs 72.2 actual, leaving headroom.
  • Per-symbol — every contract-Required symbol must be exercised, not just exported. This is the hole that hid FinishReasonIs.

Tests — 65.9% → 72.2%. consumeCreateResponse 42% → 88%, ReasoningStream 0% → 100%. All Required-API symbols now covered. The unlock was a fake EventStream that emits real SSE frames decoded by the SDK's own decoder; three SDK details each produce a fake that looks fine and silently carries no events, which is why TestSanityFakeStreamDecodesThroughSDK guards the fake itself.

Contract — a binding Test Quality section in upstreamer.md, upstream-test-parity criteria in eval.md (an untested changed behavior is now a FAIL), and a new porting-tests skill. Per PORTING.md's own rule, fixing the contract rather than only the code is what stops this re-rotting next sync.

Judgment calls worth reviewing

  • Every gate was proven to fail. Reverting the mutex reproduces the race; disabling the FinishReasonIs tests makes Gate B name that exact symbol; a 40-goroutine probe trips the leak check; floors of 99 and 50 both fail Gate A. Mutating Drain and the retry branch makes those tests fail. A gate I hadn't watched fail would be decoration.
  • I reverted one of my own changes. I had wrapped the completed-event check in IsResponseCompletedEvent(data) && …. That narrows behavior rather than deduplicating — any event with the pointer set but Type unset would turn a working run into "stream ended without response.completed". Not worth the risk.
  • I did not wire in ExecuteToolLoop. It looks like the tool loop but passes a zero TurnContext and nil emitter, silently dropping hooks, approval gating, and generator streaming. Wiring it in would be a regression; testing it would legitimize a footgun.
  • verify-port stays advisory. Its required-API check fails by design until the port catches up to upstream. Making that red on every PR teaches people to ignore CI.

User-visible change

The wrapped Stop-hook error is now stop hook: … (was Stop hook: …, flagged ST1005). Noted in the changelog — code matching on that text rather than errors.Is/errors.As needs updating. Also fixed a real nil-deref in an existing test that staticcheck caught.

Follow-ups (deliberately not here)

  • tool_orchestrator.go and next_turn_params.go are still uncalled. Deleting them touches the port's API surface, so it needs a contract amendment rather than being smuggled into a CI PR.
  • stream_transformers.go's three Extract*Deltas remain unused; I didn't manufacture call sites for them.
  • Upstream has whole test files with no Go counterpart — the six *-adversarial suites, schema-sanitization, shared-context, tool-context, turn-end-race-condition. They map onto this repo's lowest-coverage files, so they're the best guide for what to cover next.

Verification

gofmt -l .                                  # clean
go test -race -shuffle=on -count=1 ./...    # ok
staticcheck ./...                           # clean
.upstreamer/scripts/verify.sh               # PASS: 0 failures
env -u OPENROUTER_API_KEY go test -run TestE2E -v .   # 5 SKIP, not fail

All six commits build and pass independently (git bisect-safe): the race fix lands before the gate that would catch it, and the streaming tests land before the loop refactor they protect.

🤖 Generated with Claude Code

LukasParke and others added 6 commits July 30, 2026 15:26
handlerCompleted was written by a detached goroutine inside the hook handler
and read by the test goroutine with no synchronization, so `go test -race`
failed on this test. Guard it with a mutex.

Production code is unaffected -- the race is entirely in the test's own shared
state. But CI never ran the race detector, so nothing would have caught the
same mistake in production code either. This lands first so the blocking race
gate added in the next commit is green from day one rather than shipping a
knowingly-red required check.

The assertions still mean what they did: emit returns before the async work
settles, and Drain waits for it. Verified by neutering Drain, which still fails
the test.

Co-Authored-By: Claude <noreply@anthropic.com>
The suite had no deterministic test that drove a *successful* stream: one
streaming fake existed and it emitted a broken frame to assert error
propagation, while the other 26 fake responses were non-streaming. So
consumeCreateResponse's success loop sat at 42%, and because that loop never
ran, ReasoningStream was structurally unreachable and TextStream was only ever
exercised through its single-chunk non-streaming fallback -- for a package whose
headline feature is streaming fan-out.

Adds stream_fake_test.go: events built with the SDK's Create* constructors and
serialized to real SSE frames, decoded back by the SDK's own decoder. Three SDK
details each produce a fake that looks fine and silently carries no events (a
hand-built event can leave `type` empty; marshalling via map[string]any drops
the union discriminator; the SDK re-wraps the `data:` payload before the decoder
sees it), all with no error returned -- hence
TestSanityFakeStreamDecodesThroughSDK, which guards the fake itself.

Also covers, in rough order of risk:

- retryCurrentRequest (0%) and StrictFinalResponse (zero test references). Live
  code on the terminal turn of every tool-using conversation, and a documented
  divergence from upstream.
- validateValue's number/integer/boolean/array branches and nested recursion
  (50%). This is the guard that stops malformed model output reaching user tool
  code.
- serverToolImpl and NewServerTool (0%), asserting the wrapped SDK union reaches
  the request verbatim rather than just exercising getters.
- The paused-run read path: ItemsStream, ToolCalls, PendingToolCalls,
  RequiresApproval, Cancel. Existing tests only reached a paused run via State().
- FinishReasonIs and ToChatMessage, both in the contract's required API with no
  test at all -- exported, so the verifier's presence check passed.

Coverage 65.9% -> 72.2%; consumeCreateResponse 42% -> 88%, ReasoningStream
0% -> 100%. Each test was checked by breaking the production code it covers and
confirming it fails.

Co-Authored-By: Claude <noreply@anthropic.com>
stream_guards.go and tool_event_broadcaster.go had zero callers in production or
test code -- they were parallel reimplementations of logic model_result.go did
inline. ToolEventBroadcaster wraps exactly the ReusableStream[ToolStreamEvent]
that ModelResult built directly, and the guards answered "is this a text or
reasoning delta" by checking the union's `type` while production nil-checked the
member pointer: two mechanisms for one question, with only the unused copy
exported.

Rather than test dead code (which raises coverage while adding maintenance
surface) or delete required-API symbols, wire production to use them. The
duplication -- the thing that drifts and misleads -- goes away, and they are
covered for free.

Behavior-preserving. The SDK sets `type` and the member pointer together in both
its Create* constructors and UnmarshalJSON, so the type-based guards agree with
the previous pointer checks for any event the SDK produced. Error(nil) forwards
to Complete(nil), matching the sibling streams' completion on both paths. The
full test suite produces byte-identical results before and after.

Lands after the streaming tests in the previous commit so there is a real
regression net under a change to the load-bearing loop.

Not wired in: tool_orchestrator.go's ExecuteToolLoop, which looks like the tool
loop but passes a zero TurnContext and nil emitter, so it silently drops hooks,
approval gating, and generator event emission. Using it would be a regression
and testing it would legitimize a footgun; the real loop is
executeToolCallsForTurn. It and next_turn_params.go are still uncalled -- both
are deletion candidates needing a contract amendment.

Co-Authored-By: Claude <noreply@anthropic.com>
Both pre-existing, surfaced by adding staticcheck to CI:

- SA5011 in TestAllowFinalResponseSendsNoToolsRequestOnStop: the test
  dereferenced `directive` to read .Content before its own nil check, so a nil
  directive would panic instead of failing with the intended message. Check
  first, then dereference.
- ST1005: the wrapped Stop-hook error was capitalized ("Stop hook: ..."), against
  Go convention. Now lower-cased.

The error-string change is user-visible, so it is noted in the changelog: code
matching on that text rather than using errors.Is/errors.As needs updating. No
test depended on the casing.

Co-Authored-By: Claude <noreply@anthropic.com>
CI ran gofmt/build/vet/test and nothing else. For a package whose whole job is
concurrent stream fan-out plus a hooks manager documented as concurrency-safe,
the notable omission was the race detector -- it had never run, and it fails on
the current tree (fixed in the first commit of this branch). Upstream considers
this enough of a risk to ship a dedicated turn-end-race-condition test.

CI (.github/workflows/ci.yaml):

- `go test -race`, blocking. A data race here is a real defect that a plain
  `go test` reports as a pass.
- `-count=1` defeats Go's test cache, which was reporting `(cached)` on re-runs
  -- a cached ok says some earlier tree passed, not this one.
- `-shuffle=on` catches order-dependent tests as the suite grows.
- staticcheck, pinned to 2025.1.1. Unpinned, an upstream release turns into a
  surprise red CI on an unrelated PR.
- The coverage gates below, so they apply to hand-written PRs and not just to
  port syncs.

Verifier (.upstreamer/scripts/verify.sh) gains the same race run plus two
coverage gates:

- Gate A, ratchet: coverage may not fall below .upstreamer/coverage-floor.txt,
  and a gain above 1.5 points must be locked in by raising the floor. Coverage
  measured deterministically (three consecutive runs, identical), so the ratchet
  will not flake. Floor committed at 72.0 against 72.2 actual, leaving headroom
  so noise cannot redden an unrelated PR.
- Gate B, per-symbol: every symbol in the contract's required public API must be
  *exercised*, not merely exported. This is the hole the existing presence check
  cannot see -- FinishReasonIs was listed, exported, and 0% covered. Matching
  takes the max across same-named functions, since Execute/Push/CallModel and
  friends repeat across types.

Both gates read only this Go tree, deliberately: verify.sh has no upstream
dependency, which is what lets ci.yaml run it in a job that does a plain
actions/checkout with no upstream clone. Upstream-vs-port test comparison needs
the upstream tree and so belongs in eval.md.

verify.sh also now asserts the CI workflow and coverage floor still exist, so a
port run cannot delete its own gates and still pass verification.

main_test.go adds goroutine-leak detection over the whole run: this package
spawns a goroutine per run and per subscriber, so a missed Complete() leaks
invisibly -- tests pass, and a long-lived process grows. Dependency-free rather
than uber-go/goleak, since this is a load-bearing SDK whose contract pins its
dependency graph and a test-only dep still lands in go.mod for every consumer.
It only polices leaks on an otherwise-green run, so a failing test is not
reported as a leak.

Every gate was verified to actually fail: raising the floor to 99 and dropping it
to 50 both fail Gate A; disabling the FinishReasonIs tests makes Gate B name that
symbol; a 40-goroutine probe trips the leak check; reverting the mutex fix
reproduces the race.

verify-port stays advisory -- its required-API check fails by design until the
port catches up to upstream, and making that red on every PR just teaches people
to ignore CI.

Co-Authored-By: Claude <noreply@anthropic.com>
PORTING.md's own rule is that a code-only fix gets re-broken on the next sync,
so the gates in the previous commit are only half the job -- the standard has to
live in the contract, which is the durable artifact.

.upstreamer/upstreamer.md gains a binding Test Quality section: port upstream's
own test cases rather than inventing them, assert upstream-observable behavior
(ordering, error surfaces, stream boundaries, state shape) never the port's
internal shape, cover the edge case the upstream fix exists for, ship a test with
every new required-API symbol, keep concurrency tests meaningful under -race, use
real streaming events for streaming behavior, and never lower the coverage floor.
It also records .github/ and the coverage floor as repo-owned, since a run that
deleted its own gates would otherwise verify clean.

.upstreamer/eval.md gains a test-parity criterion. This belongs in the eval
rather than the verifier because the eval has the upstream checkout and the
verifier deliberately does not. It asks the evaluator to diff upstream's
*.test.ts across the delta and report behaviors that changed with no
corresponding test -- and explicitly not to grade on test counts, since Go
table-driven subtests bundle cases and the ~619-it-blocks-to-70-Test-funcs ratio
is meaningless. An untested changed behavior is now a FAIL.

New .upstreamer/skills/porting-tests/SKILL.md carries the mechanics the
converter skill's four-line Tests step did not: how to find the upstream case for
a changed behavior, a table of the existing fakes to reuse instead of writing new
ones, the three SDK streaming pitfalls that silently produce a fake carrying no
events, the anti-patterns (asserting internal shape, happy-path-only,
unsynchronized test state, coverage theater on dead code), and how to prove a new
test can actually fail. The TestHooksManagerAsyncDrain race and
ExecuteToolLoop-as-footgun are used as the worked examples, since both are real
and in-tree.

Every file, helper, and upstream path referenced was verified to exist.

Co-Authored-By: Claude <noreply@anthropic.com>
@LukasParke
LukasParke merged commit 712749e into main Jul 30, 2026
4 checks passed
@LukasParke
LukasParke deleted the LukasParke/e2e-CI branch July 30, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant