ci: bulletproof test gates + binding test-quality contract for the port - #4
Merged
Conversation
handlerCompleted was written by a detached goroutine inside the hook handler and read by the test goroutine with no synchronization, so `go test -race` failed on this test. Guard it with a mutex. Production code is unaffected -- the race is entirely in the test's own shared state. But CI never ran the race detector, so nothing would have caught the same mistake in production code either. This lands first so the blocking race gate added in the next commit is green from day one rather than shipping a knowingly-red required check. The assertions still mean what they did: emit returns before the async work settles, and Drain waits for it. Verified by neutering Drain, which still fails the test. Co-Authored-By: Claude <noreply@anthropic.com>
The suite had no deterministic test that drove a *successful* stream: one streaming fake existed and it emitted a broken frame to assert error propagation, while the other 26 fake responses were non-streaming. So consumeCreateResponse's success loop sat at 42%, and because that loop never ran, ReasoningStream was structurally unreachable and TextStream was only ever exercised through its single-chunk non-streaming fallback -- for a package whose headline feature is streaming fan-out. Adds stream_fake_test.go: events built with the SDK's Create* constructors and serialized to real SSE frames, decoded back by the SDK's own decoder. Three SDK details each produce a fake that looks fine and silently carries no events (a hand-built event can leave `type` empty; marshalling via map[string]any drops the union discriminator; the SDK re-wraps the `data:` payload before the decoder sees it), all with no error returned -- hence TestSanityFakeStreamDecodesThroughSDK, which guards the fake itself. Also covers, in rough order of risk: - retryCurrentRequest (0%) and StrictFinalResponse (zero test references). Live code on the terminal turn of every tool-using conversation, and a documented divergence from upstream. - validateValue's number/integer/boolean/array branches and nested recursion (50%). This is the guard that stops malformed model output reaching user tool code. - serverToolImpl and NewServerTool (0%), asserting the wrapped SDK union reaches the request verbatim rather than just exercising getters. - The paused-run read path: ItemsStream, ToolCalls, PendingToolCalls, RequiresApproval, Cancel. Existing tests only reached a paused run via State(). - FinishReasonIs and ToChatMessage, both in the contract's required API with no test at all -- exported, so the verifier's presence check passed. Coverage 65.9% -> 72.2%; consumeCreateResponse 42% -> 88%, ReasoningStream 0% -> 100%. Each test was checked by breaking the production code it covers and confirming it fails. Co-Authored-By: Claude <noreply@anthropic.com>
stream_guards.go and tool_event_broadcaster.go had zero callers in production or test code -- they were parallel reimplementations of logic model_result.go did inline. ToolEventBroadcaster wraps exactly the ReusableStream[ToolStreamEvent] that ModelResult built directly, and the guards answered "is this a text or reasoning delta" by checking the union's `type` while production nil-checked the member pointer: two mechanisms for one question, with only the unused copy exported. Rather than test dead code (which raises coverage while adding maintenance surface) or delete required-API symbols, wire production to use them. The duplication -- the thing that drifts and misleads -- goes away, and they are covered for free. Behavior-preserving. The SDK sets `type` and the member pointer together in both its Create* constructors and UnmarshalJSON, so the type-based guards agree with the previous pointer checks for any event the SDK produced. Error(nil) forwards to Complete(nil), matching the sibling streams' completion on both paths. The full test suite produces byte-identical results before and after. Lands after the streaming tests in the previous commit so there is a real regression net under a change to the load-bearing loop. Not wired in: tool_orchestrator.go's ExecuteToolLoop, which looks like the tool loop but passes a zero TurnContext and nil emitter, so it silently drops hooks, approval gating, and generator event emission. Using it would be a regression and testing it would legitimize a footgun; the real loop is executeToolCallsForTurn. It and next_turn_params.go are still uncalled -- both are deletion candidates needing a contract amendment. Co-Authored-By: Claude <noreply@anthropic.com>
Both pre-existing, surfaced by adding staticcheck to CI:
- SA5011 in TestAllowFinalResponseSendsNoToolsRequestOnStop: the test
dereferenced `directive` to read .Content before its own nil check, so a nil
directive would panic instead of failing with the intended message. Check
first, then dereference.
- ST1005: the wrapped Stop-hook error was capitalized ("Stop hook: ..."), against
Go convention. Now lower-cased.
The error-string change is user-visible, so it is noted in the changelog: code
matching on that text rather than using errors.Is/errors.As needs updating. No
test depended on the casing.
Co-Authored-By: Claude <noreply@anthropic.com>
CI ran gofmt/build/vet/test and nothing else. For a package whose whole job is concurrent stream fan-out plus a hooks manager documented as concurrency-safe, the notable omission was the race detector -- it had never run, and it fails on the current tree (fixed in the first commit of this branch). Upstream considers this enough of a risk to ship a dedicated turn-end-race-condition test. CI (.github/workflows/ci.yaml): - `go test -race`, blocking. A data race here is a real defect that a plain `go test` reports as a pass. - `-count=1` defeats Go's test cache, which was reporting `(cached)` on re-runs -- a cached ok says some earlier tree passed, not this one. - `-shuffle=on` catches order-dependent tests as the suite grows. - staticcheck, pinned to 2025.1.1. Unpinned, an upstream release turns into a surprise red CI on an unrelated PR. - The coverage gates below, so they apply to hand-written PRs and not just to port syncs. Verifier (.upstreamer/scripts/verify.sh) gains the same race run plus two coverage gates: - Gate A, ratchet: coverage may not fall below .upstreamer/coverage-floor.txt, and a gain above 1.5 points must be locked in by raising the floor. Coverage measured deterministically (three consecutive runs, identical), so the ratchet will not flake. Floor committed at 72.0 against 72.2 actual, leaving headroom so noise cannot redden an unrelated PR. - Gate B, per-symbol: every symbol in the contract's required public API must be *exercised*, not merely exported. This is the hole the existing presence check cannot see -- FinishReasonIs was listed, exported, and 0% covered. Matching takes the max across same-named functions, since Execute/Push/CallModel and friends repeat across types. Both gates read only this Go tree, deliberately: verify.sh has no upstream dependency, which is what lets ci.yaml run it in a job that does a plain actions/checkout with no upstream clone. Upstream-vs-port test comparison needs the upstream tree and so belongs in eval.md. verify.sh also now asserts the CI workflow and coverage floor still exist, so a port run cannot delete its own gates and still pass verification. main_test.go adds goroutine-leak detection over the whole run: this package spawns a goroutine per run and per subscriber, so a missed Complete() leaks invisibly -- tests pass, and a long-lived process grows. Dependency-free rather than uber-go/goleak, since this is a load-bearing SDK whose contract pins its dependency graph and a test-only dep still lands in go.mod for every consumer. It only polices leaks on an otherwise-green run, so a failing test is not reported as a leak. Every gate was verified to actually fail: raising the floor to 99 and dropping it to 50 both fail Gate A; disabling the FinishReasonIs tests makes Gate B name that symbol; a 40-goroutine probe trips the leak check; reverting the mutex fix reproduces the race. verify-port stays advisory -- its required-API check fails by design until the port catches up to upstream, and making that red on every PR just teaches people to ignore CI. Co-Authored-By: Claude <noreply@anthropic.com>
PORTING.md's own rule is that a code-only fix gets re-broken on the next sync, so the gates in the previous commit are only half the job -- the standard has to live in the contract, which is the durable artifact. .upstreamer/upstreamer.md gains a binding Test Quality section: port upstream's own test cases rather than inventing them, assert upstream-observable behavior (ordering, error surfaces, stream boundaries, state shape) never the port's internal shape, cover the edge case the upstream fix exists for, ship a test with every new required-API symbol, keep concurrency tests meaningful under -race, use real streaming events for streaming behavior, and never lower the coverage floor. It also records .github/ and the coverage floor as repo-owned, since a run that deleted its own gates would otherwise verify clean. .upstreamer/eval.md gains a test-parity criterion. This belongs in the eval rather than the verifier because the eval has the upstream checkout and the verifier deliberately does not. It asks the evaluator to diff upstream's *.test.ts across the delta and report behaviors that changed with no corresponding test -- and explicitly not to grade on test counts, since Go table-driven subtests bundle cases and the ~619-it-blocks-to-70-Test-funcs ratio is meaningless. An untested changed behavior is now a FAIL. New .upstreamer/skills/porting-tests/SKILL.md carries the mechanics the converter skill's four-line Tests step did not: how to find the upstream case for a changed behavior, a table of the existing fakes to reuse instead of writing new ones, the three SDK streaming pitfalls that silently produce a fake carrying no events, the anti-patterns (asserting internal shape, happy-path-only, unsynchronized test state, coverage theater on dead code), and how to prove a new test can actually fail. The TestHooksManagerAsyncDrain race and ExecuteToolLoop-as-footgun are used as the worked examples, since both are real and in-tree. Every file, helper, and upstream path referenced was verified to exist. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This repo is a living port — every sync is a machine-generated diff to load-bearing concurrent code (tool loop, streaming fan-out, approval/HITL ordering, hooks). The only thing between a subtly-wrong port and
mainis what CI and the two gates actually check. They weren't checking enough.I probed the tree rather than reading it. The first probe found a real bug:
go test ./...go test -race ./...TestHooksManagerAsyncDrainretryCurrentRequest— runs on the last turn of every tool conversationStrictFinalResponsehad zero test referencesFinishReasonIs— contract-Required APIstream_guards.go,tool_event_broadcaster.go, +2The race is the headline. It's in test code, but that's luck, not process: CI had never run
-raceon a package whose headline feature is concurrent stream fan-out and whose hooks manager is documented as concurrency-safe. Upstream ships a dedicatedturn-end-race-condition.test.ts; this port had no counterpart and no gate.FinishReasonIsis the second: listed in the contract's parity floor, exported, and never called by any test. A presence check structurally cannot see that. It's the exact failure mode PORTING.md says the gates exist to prevent — "a port which compiles cleanly while sitting a version behind on real behavior."What changed
Gates —
-race(blocking),-shuffle,-count=1(the cache was reporting(cached)), pinned staticcheck, a dependency-free goroutine-leakTestMain, and two coverage gates shared between CI and the port verifier:.upstreamer/coverage-floor.txt; a real gain must be locked in by raising it. Measured deterministic across three runs, so it won't flake. Floor at 72.0 vs 72.2 actual, leaving headroom.FinishReasonIs.Tests — 65.9% → 72.2%.
consumeCreateResponse42% → 88%,ReasoningStream0% → 100%. All Required-API symbols now covered. The unlock was a fakeEventStreamthat emits real SSE frames decoded by the SDK's own decoder; three SDK details each produce a fake that looks fine and silently carries no events, which is whyTestSanityFakeStreamDecodesThroughSDKguards the fake itself.Contract — a binding Test Quality section in
upstreamer.md, upstream-test-parity criteria ineval.md(an untested changed behavior is now aFAIL), and a newporting-testsskill. Per PORTING.md's own rule, fixing the contract rather than only the code is what stops this re-rotting next sync.Judgment calls worth reviewing
FinishReasonIstests makes Gate B name that exact symbol; a 40-goroutine probe trips the leak check; floors of 99 and 50 both fail Gate A. MutatingDrainand the retry branch makes those tests fail. A gate I hadn't watched fail would be decoration.IsResponseCompletedEvent(data) && …. That narrows behavior rather than deduplicating — any event with the pointer set butTypeunset would turn a working run into"stream ended without response.completed". Not worth the risk.ExecuteToolLoop. It looks like the tool loop but passes a zeroTurnContextand nil emitter, silently dropping hooks, approval gating, and generator streaming. Wiring it in would be a regression; testing it would legitimize a footgun.verify-portstays advisory. Its required-API check fails by design until the port catches up to upstream. Making that red on every PR teaches people to ignore CI.User-visible change
The wrapped Stop-hook error is now
stop hook: …(wasStop hook: …, flagged ST1005). Noted in the changelog — code matching on that text rather thanerrors.Is/errors.Asneeds updating. Also fixed a real nil-deref in an existing test that staticcheck caught.Follow-ups (deliberately not here)
tool_orchestrator.goandnext_turn_params.goare still uncalled. Deleting them touches the port's API surface, so it needs a contract amendment rather than being smuggled into a CI PR.stream_transformers.go's threeExtract*Deltasremain unused; I didn't manufacture call sites for them.*-adversarialsuites,schema-sanitization,shared-context,tool-context,turn-end-race-condition. They map onto this repo's lowest-coverage files, so they're the best guide for what to cover next.Verification
All six commits build and pass independently (
git bisect-safe): the race fix lands before the gate that would catch it, and the streaming tests land before the loop refactor they protect.🤖 Generated with Claude Code