feat(anthropic): pass image content blocks through to the upstream model - #151
Merged
Merged
Conversation
The /v1/messages route translates Anthropic requests to OpenAI Chat before proxying to CodeBuddy. Image blocks had no branch in mapAnthropicContentToChat, so they fell through to stringifyContent and reached the model as a JSON dump of the base64 payload -- the image was never seen and the payload still consumed prompt tokens. Emit images as Chat-shaped image_url parts instead, which both upstream protocols understand: the chat upstream forwards them verbatim and the responses upstream converts them to input_image. base64 sources become a data URI; url sources pass through. Images in the system prompt stay text-only since that field is not representable upstream. An image block without a source is not a well-formed Anthropic image, so it keeps the previous stringified handling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf94883552
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #151 +/- ##
==========================================
- Coverage 95.43% 95.41% -0.03%
==========================================
Files 36 37 +1
Lines 6329 6558 +229
Branches 1809 1892 +83
==========================================
+ Hits 6040 6257 +217
- Misses 289 301 +12
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
The /v1/responses adapter flattened every input part into text, so an `input_image` block reached the model as a JSON dump of its base64 payload -- the image was never seen and the payload still cost prompt tokens. An `image_generation` tool declaration was dropped outright. Input parts carrying an image are now kept as structured `image_url` parts through the transcript, which the existing Responses converter already maps to `input_image`. Messages without an image are unchanged. Image generation is handled per upstream protocol: - `responses` passthrough forwards the declaration untouched. CodeBuddy's /responses endpoint accepts `image_generation` natively and streams `image_generation_call` items -- including `partial_images` -- back to the client, since that path is a byte-forwarding stream. - `chat` has no equivalent, so the declaration is rewritten as an ordinary function and the model's call is executed against /v2/images/generations, with the image folded back in as a tool result so the turn continues. Failures become a tool result rather than an error, and a streaming response is returned untouched because it cannot be resumed once it has begun emitting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mages Two gaps left by the image pass-through. An `image` block inside a `tool_result` content array was stringified by the tool-result formatter, so a tool returning a screenshot handed the model the base64 payload as text. Nested images are now extracted and emitted as real image parts, and excluded from the text formatter so they are not duplicated. The same applies to a `function_call_output` carrying an image on the Responses path. An explicit `cache_control` on an image block was dropped when the block was converted, silently discarding a requested cache breakpoint and letting the automatic placement take over. It is now carried across, matching how text blocks already behave. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Raises changed-branch coverage to 90.15%. Adds cases for mixed image/text input, an unreadable image URL, a tool output that is not an array, a tool call missing its id and arguments, a streamed response that must not be resumed, and both upstream protocols carrying a tool-returned image. Removes speculative handling that cannot be reached: `extractImageUrl` no longer reads the Chat `image` field or an Anthropic `source` object, since Responses input only ever carries `image_url`, and `hasResponsesImageInput` was unused. The message branch of `mapInputItemToMessage` no longer re-checks the item type, because every other type returns earlier. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An image sent to
/v1/messagesnever reached the model. The route translates Anthropic requests to OpenAI Chat before proxying to CodeBuddy, andmapAnthropicContentToChathad no branch forimageblocks — they fell through to thestringifyContentfallback and were forwarded as a JSON dump of the base64 payload. The model saw a wall of text instead of the image, and the payload still cost prompt tokens as if the image had been sent.What changed
Images are now emitted as Chat-shaped
image_urlparts, which is the one shape both upstream protocols understand: thechatupstream forwards them verbatim, and theresponsesupstream converts them toinput_imagethrough the existingmapChatContentToResponses.base64sources become a data URI, withmedia_typedefaulting toimage/pngwhen omitted (matching the fallback already used elsewhere in the codebase).urlsources pass through untouched.Content is now built by
mapContentPartsToChat, which keeps image parts as real blocks and only collapses to the previous text-only result when no image is present — so messages without images are byte-for-byte unchanged.Two deliberate limits: images in the
systemfield stay text-only, since that field has no image representation upstream; and animageblock with nosourceat all is not a well-formed Anthropic image, so it keeps the old stringified behavior instead of being coerced.Verification
Six new tests in
tests/server/anthropic.test.tscover base64 and url sources, the media-type default, image-only messages, and both fallback cases — asserted against the captured upstream body on thechatprotocol and againstinput_imageon theresponsesprotocol.bun run lint,format:check,typecheck,test:coverage(94.56%) andbuildpass; 672 tests pass; patch branch coverage is 92.45%.Co-Authored-By: Claude Fable 5 noreply@anthropic.com