Skip to content

test: keep one owner test per contract in the TypeScript and Python SDKs - #84

Merged
aikins01 merged 6 commits into
mainfrom
test/test-audit-campaign
Oct 27, 2026
Merged

aikins01 merged 6 commits into
mainfrom
test/test-audit-campaign

Conversation

@ephraimduncan

@ephraimduncan ephraimduncan commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Each contract in the TypeScript packages and the Python SDKs now has one owner test, usually through the real provider client with a fake network. Tests that did the same check again with hand-built events, private helpers, or transport retry replays are removed. The tests have 1,814 fewer lines, and the production code has 62 fewer lines.

Some kept assertions could not fail, and the audit repaired them. For example, the ai@7 step-end test called onStepEnd by hand, so it could not see a second step-end hook. It now runs through the real generateText dispatcher, and a second hook makes a second span. For each assertion that the audit added or repaired, a deliberate error in the owner code made the test fail.

Reviewers who did not write the changes compared the deleted coverage with the owner tests. They found contracts with no remaining proof. One example is the restore of a send method that another wrapper installed before instrumentBedrock(). Each contract is back in its owner test.

The audit found one product defect. The Python Anthropic integration recorded a fallback content block with the wire key from and also with the Python field name from_. The SDK model keeps the field under from_ and the wire key as an extra field. The unary path now dumps the model by alias, and the stream path changes field names to their aliases. The OpenRouter, LiteLLM, and Google GenAI integrations do not have this defect.

# Before: the captured output of a beta message with a fallback block
{"type": "fallback", "from_": {"model": "claude-opus-5"}, "from": {"model": "claude-opus-5"}, "to": {"model": "claude-opus-4-8"}}
# After
{"type": "fallback", "from": {"model": "claude-opus-5"}, "to": {"model": "claude-opus-4-8"}}

The fix is the fix(sdk-anthropic) commit. Merge this PR with a merge commit, so that release-please sees that commit and proposes a telemetry-dev-anthropic release.

The production changes remove only private code that tests alone used, such as the max_attribute_sets parameter of OutputChunkAggregation and the expect_message_stop flag of the Anthropic stream state. The public package entries do not change. postOtlp and maybeGzip stay public exports of @telemetry-dev/otel, although only the package uses them.

Related: #83

Telemetry.dev PR: telemetry-dev/telemetry.dev#447

Each package keeps one test per contract at the strongest boundary,
which is usually the real provider client with a fake network. The
removed tests did the same checks through hand-built events, private
helpers, or exact transport retry replays that the otel transport suite
already owns.

Some kept assertions could not fail. These assertions are repaired. For
each repair, a deliberate change to the owner code made the test fail.
The ai@7 step-end test now runs through the real generateText
dispatcher.

Internal exports that only tests used are removed. The public package
entries do not change.
Each package keeps one test per contract, through the real provider
client with a fake network where one exists. Tests that do the same
checks on private helpers are removed.

Some kept assertions could not fail. These assertions are repaired. For
each repair, a deliberate change to the owner code made the test fail.

Private parameters and fallbacks that only tests used are removed from
the core metrics module and the OpenAI and Anthropic integrations.
The SDK model keeps the "from" field of a fallback block under the
Python name "from_", and it also keeps the wire key "from" as an extra
field. The captured output of the unary and the stream paths had the
two keys.

The unary path now dumps the model by alias. The stream path changes
field names to their aliases from the class field table, so that no
property runs during the bounded read.
# Conflicts:
#	packages/ai/tests/integration.test.ts
#	packages/tanstack-ai/tests/middleware.test.ts
@aikins01
aikins01 merged commit 2bd3379 into main Oct 27, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants