feat(responses): emit image_generation_call output items on the chat path - #156
Merged
Merged
Conversation
…path The chat upstream has no notion of image generation, so a locally executed call previously disappeared: the image was folded back into the transcript for the model, but the client only received the assistant's text and had no way to see the image it asked for. The loop now records one execution per call and the Responses payload carries a standard `image_generation_call` output item, with the generated image in `result` as base64 and the prompt in `revised_prompt`. A failed generation still emits the item, with `result: null` and `status: 'failed'`, so the client can tell an image was attempted rather than seeing silence. This mirrors how `web_search_call` items are already synthesized from locally executed server tools. Streaming had a worse problem: the call was forwarded as an ordinary function_call for the client to resolve, and nothing ever generated the image. Streaming requests that declare the tool are now buffered so the call can be inspected before any delta reaches the client, then replayed as the same event sequence a live stream produces. Requests where the model makes no image call fall through to the live stream unchanged. `partial_images` is still passthrough-only: the generations endpoint returns a finished image, with no intermediate frames to stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Raises changed-branch coverage to 92.31%. Covers a generation that returns only a hosted URL — marked completed, but with no base64 to hand back — and a failed upstream on the streaming path, which is surfaced to the client rather than swallowed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 20ef195e44
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #156 +/- ##
==========================================
+ Coverage 95.41% 95.47% +0.06%
==========================================
Files 37 37
Lines 6585 6633 +48
Branches 1899 1910 +11
==========================================
+ Hits 6283 6333 +50
+ Misses 302 300 -2
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Four fixes, three of them real bugs the review caught.
**Streamed turns were billed twice.** The loop returned null when the model
made no image call, and the caller re-issued the request. For a client that
declares the optional tool, that is the common case, so most streaming turns
paid for two upstream calls and could return an answer different from the
one already inspected. The loop now always returns its final response, and
neither caller re-issues.
**The stream advertised the wrong response id.** `response.created` carried
the id the payload mapper created and stored, while later events used a
freshly generated one — so a client passing the completed id as
`previous_response_id` got "Unknown or expired". The stream now reuses the
mapper's id.
**A non-string prompt crashed the request.** Model arguments are untrusted
JSON; `{"prompt":123}` threw on `.trim()` and turned a handled generation
failure into a 500. Now coerced, so the intended failed item is emitted.
**Prose written before the tool call was dropped.** A model that explains
itself and then calls the tool lost the explanation, because only the final
hop's message survived the replay. The web-search loop already solved this,
so `withIntermediateTurns` is now exported and shared rather than
reimplemented — along with `ChatCompletionMessage`,
`ChatCompletionPayload` and `ChatCompletionToolCall`, removing three
duplicate type declarations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Raises changed-branch coverage to 90.63%. Covers a failed upstream on the non-streaming path, which is surfaced to the client rather than swallowed. Also removes two unreachable fallbacks: the buffered stream helper re-derived a response id the payload mapper always returns, and the loop's terminal branch had a dead guard because its body always runs at least once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The streaming image path sat inside the branch taken only when neither web search nor web fetch was enabled. A turn declaring both an image tool and an enabled server search tool therefore skipped generation entirely: the model's call was forwarded as an ordinary function_call for the client to resolve, and no image was ever generated. Image generation does not depend on the search backends, so it is now handled before that branch. A turn declaring both runs generation first, and one declaring only server tools is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
orangeboyChen
enabled auto-merge (squash)
September 17, 2026 10:08
orangeboyChen
added a commit
that referenced
this pull request
Sep 17, 2026
#152 laid non-streaming server-tool blocks out per hop, but put the grouping it needed on the chat-completion payload itself. That payload is what an OpenAI-protocol client receives, so three things went wrong. ## 1. Hop metadata leaked into chat-completions responses `withIntermediateTurns` attached `turns` to the payload, and `Response.json` serialized it. `app/v1/chat/completions/route.ts` returns `proxyChatCompletions` unchanged, so `/v1/chat/completions` exposed a non-protocol `turns` field holding internal tool inputs and results — strict validators can reject it, and every client received the tool data twice. Carry the grouping out of band through a WeakMap, the way `serverToolExecutions` already is: `attachServerToolTurns` / `getServerToolTurns`. The Anthropic adapter reads it off the response instead of the payload. ## 2. Mixed server/client-tool turns lost the grouping When a later hop mixed a local server tool with a client-owned tool, `withIntermediateTurns` built the turns but the mixed branch took only `.message` off its result and never grouped the current hop. With no hop metadata the mapper flattened all reasoning and text ahead of all server-tool blocks — the exact ordering bug #152 set out to fix. Preserve the prior turns and add the current hop's calls to its entry. ## 3. A first hop that mixed tools dropped its own prose The mixed branch built the current hop with empty text and reasoning. When that hop was also the first, `withIntermediateTurns` had no earlier hops to fold and so returned no turns, and a block renderer renders purely from `turns` once it is non-empty — so everything the model said in that hop disappeared from the Anthropic response. `withIntermediateTurns` already builds this hop as its closing entry, so the calls are added to that entry rather than appended as a new one, which would repeat the prose. Only when there are no earlier hops is a fresh entry built, seeded with this iteration's text and reasoning. ## Verification - Three regression tests, one per issue. Each was confirmed to fail with its fix reverted and pass with it applied, so they guard the behaviour rather than the implementation. - `withIntermediateTurns` and `buildMixedTurnPayload` now return `{ payload, turns }`, and `ServerToolLoopResult.turns` is required rather than optional, so every return path states its grouping explicitly instead of relying on a `?? []` fallback. - 733 tests pass; typecheck, eslint, prettier, and build are clean. Coverage 94.67% statements. Rebased onto #156, which exports `withIntermediateTurns` to the image-generation loop; those call sites now take `.payload`.
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #151, which added
image_generationsupport but left the chat path returning a non-standard shape.What was wrong
Non-streaming. The chat upstream has no notion of image generation, so the call was executed locally and the image folded back into the transcript — the model saw it and replied, but the client received only the assistant's text. The image never left the proxy.
Streaming was worse. The model's call was forwarded as an ordinary
function_callfor the client to resolve, and nothing ever generated the image. A client that declaredimage_generationand streamed got a tool call it had no way to satisfy.What changed
executeImageGenerationLoopnow records one execution per call and returns them alongside the response, andmapChatResponseToResponsesPayloadsynthesizes a standardimage_generation_calloutput item:{"id":"ig_…","result":"<base64>","revised_prompt":"a cat","status":"completed","type":"image_generation_call"}This follows the existing
web_search_callprecedent, which is already synthesized from locally executed server tools. A failed generation still emits the item withresult: nullandstatus: 'failed'— dropping it would leave the client unable to tell an image was attempted.Streaming requests that declare the tool are buffered (
stream: false) so the call can be inspected before any delta reaches the client, then replayed throughmapChatResponseToResponsesStreamas the same event sequence a live stream produces:response.created→output_item.added/done→response.completed. Requests where the model makes no image call fall through to the live stream unchanged, so ordinary turns keep streaming as before.Limits
partial_imagesremains passthrough-only./v2/images/generationsreturns a finished image with no intermediate frames, so there is nothing to stream mid-generation; only theresponsespassthrough, which forwards the upstream SSE byte-for-byte, can surface them./v1/messagesis unaffected — Anthropic has no image-generation tool type, so it still supports image input only.Verification
bun run lint,format:check,typecheck,test:coverage(94.6%) andbuildpass; 731 tests pass, 7 new. Changed-branch coverage is 92.31%.Co-Authored-By: Claude Fable 5 noreply@anthropic.com