fix(anthropic): lay server-tool blocks out per hop in non-streaming turns - #152
Conversation
…urns
A non-streaming turn that ran a server tool came back with every tool
block hoisted ahead of all the prose, and each hop's reasoning and text
merged into one block apiece:
[server_tool_use] [tool_result] [thinking1+thinking2] [text1+text2]
Anthropic lays an assistant turn out per hop instead — thinking and text,
then the tools they led to, repeated — so a client replaying the message
expects [thinking] [text] [tool_use] [tool_result] [thinking] [text].
The old shape put each search ahead of the reasoning that asked for it
and merged hops that were never contiguous.
The merged strings are not fixable at the renderer: `withIntermediateTurns`
joins every hop into one `content` and one `reasoning_content`, because
that is all the OpenAI-shaped payload can carry, and the join happens
before any block exists. So the loop now also emits `turns`, keeping each
hop's reasoning, text, and calls grouped and in order, and the Anthropic
map builds blocks from that when present.
Paths that never group the hops — no server tool ran — keep the previous
behaviour of thinking and text followed by their tool blocks.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c105e11bb6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // content blocks needs to know where one hop's reasoning ends and the next | ||
| // begins, which a joined string has already lost. The closing hop is the | ||
| // model's final answer, so it carries no further calls. | ||
| turns: buildIntermediateTurns({ |
There was a problem hiding this comment.
When a non-streaming /v1/chat/completions request executes a local web tool, this property is serialized by Response.json, and app/v1/chat/completions/route.ts returns proxyChatCompletions unchanged. The OpenAI-compatible endpoint therefore exposes a non-protocol turns field containing internal inputs and results; strict response validators can reject it, and every client receives duplicated tool data. Carry the grouping out of band, as already done for serverToolExecutions, or add it only inside the Anthropic adapter.
Useful? React with 👍 / 👎.
| message: withIntermediateTurns({ | ||
| payload, | ||
| executions: intermediateExecutions, |
There was a problem hiding this comment.
When a later hop mixes a local server tool with a client-owned tool, withIntermediateTurns(...) creates turns only on its returned payload, but this branch extracts just .message and then gives buildMixedTurnPayload the original payload. The response consequently has no hop metadata and never groups the current executions, so the Anthropic mapper flattens all reasoning and text before all server-tool blocks—for example, after a search-only hop followed by search plus client_tool. Preserve the prior turns and append the current execution group when constructing the mixed response.
Useful? React with 👍 / 👎.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #152 +/- ##
==========================================
- Coverage 95.43% 95.41% -0.02%
==========================================
Files 36 36
Lines 6329 6368 +39
Branches 1809 1817 +8
==========================================
+ Hits 6040 6076 +36
- Misses 289 292 +3
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
The changed-branch coverage check gates at 90% and the previous commit landed at 88.24%. Two of the gaps were fallbacks I had added that cannot be reached: `reasonings[index] ?? ''` and `texts[index] ?? ''` in `buildIntermediateTurns`. The loop appends to all three hop arrays on every hop and the closing prose is appended to `texts` and `reasonings` together, so those two are always the same length and the fallback never runs. Indexing them directly makes that invariant load-bearing instead of papered over. `executions` genuinely does run one short — the closing hop answers without calling anything — so its fallback stays. The other gap was real: `execution.result.url ?? execution.input.url`. A fetch backend reports the URL it actually read, which follows redirects and so can differ from the one the model asked for. The existing test mocked a backend that omits `url`, covering only the fallback, so the mock now returns one and the test asserts the result carries it.
#152 laid non-streaming server-tool blocks out per hop, but put the grouping it needed on the chat-completion payload itself. That payload is what an OpenAI-protocol client receives, so three things went wrong. ## 1. Hop metadata leaked into chat-completions responses `withIntermediateTurns` attached `turns` to the payload, and `Response.json` serialized it. `app/v1/chat/completions/route.ts` returns `proxyChatCompletions` unchanged, so `/v1/chat/completions` exposed a non-protocol `turns` field holding internal tool inputs and results — strict validators can reject it, and every client received the tool data twice. Carry the grouping out of band through a WeakMap, the way `serverToolExecutions` already is: `attachServerToolTurns` / `getServerToolTurns`. The Anthropic adapter reads it off the response instead of the payload. ## 2. Mixed server/client-tool turns lost the grouping When a later hop mixed a local server tool with a client-owned tool, `withIntermediateTurns` built the turns but the mixed branch took only `.message` off its result and never grouped the current hop. With no hop metadata the mapper flattened all reasoning and text ahead of all server-tool blocks — the exact ordering bug #152 set out to fix. Preserve the prior turns and add the current hop's calls to its entry. ## 3. A first hop that mixed tools dropped its own prose The mixed branch built the current hop with empty text and reasoning. When that hop was also the first, `withIntermediateTurns` had no earlier hops to fold and so returned no turns, and a block renderer renders purely from `turns` once it is non-empty — so everything the model said in that hop disappeared from the Anthropic response. `withIntermediateTurns` already builds this hop as its closing entry, so the calls are added to that entry rather than appended as a new one, which would repeat the prose. Only when there are no earlier hops is a fresh entry built, seeded with this iteration's text and reasoning. ## Verification - Three regression tests, one per issue. Each was confirmed to fail with its fix reverted and pass with it applied, so they guard the behaviour rather than the implementation. - `withIntermediateTurns` and `buildMixedTurnPayload` now return `{ payload, turns }`, and `ServerToolLoopResult.turns` is required rather than optional, so every return path states its grouping explicitly instead of relying on a `?? []` fallback. - 733 tests pass; typecheck, eslint, prettier, and build are clean. Coverage 94.67% statements. Rebased onto #156, which exports `withIntermediateTurns` to the image-generation loop; those call sites now take `.payload`.
1 participant
Problem
A non-streaming turn that ran a server tool came back with every tool block hoisted ahead of all the prose, and each hop's reasoning and text merged into one block apiece:
Anthropic lays an assistant turn out per hop instead — each hop contributes its own thinking and text, followed by the tool blocks that hop triggered, repeated:
The old shape put each search ahead of the reasoning that asked for it, and merged hops that were never contiguous.
Root cause
The merging is not fixable at the renderer:
withIntermediateTurnsjoins every hop into a singlecontentand a singlereasoning_content, because that is all the OpenAI-shaped payload can carry — and the join happens before any content block exists.Fix
turns— one entry per hop, keeping that hop's reasoning, text, and calls grouped and in order. The joined strings stay as the OpenAI-shaped view, so chat-completions clients are unaffected. Adds aServerToolTurntype.buildAnthropicTurnBlockswhenturnsis present.turnskeep the previous behaviour — thinking and text followed by their tool blocks, which is the correct order when no server tool ran and there is only one hop.Tests
Scope
Non-streaming only. The streaming path already emits the correct layout and is untouched.
Separately: this investigation also confirmed that the
thinking1||thinking2 / content1||content2rendering seen in the lobehub UI is not caused by this project. It originates in lobehub's own stream accumulation (two+=scalar accumulators discard ordering before anything is stored or rendered), not in the API responses this proxy emits.