Skip to content

Carry Gemini thought signatures on the transcript - #219

Merged
mattt merged 2 commits into
huggingface:mainfrom
NoWaY233851:fix/gemini-thought-signature-transcript
Sep 5, 2026
Merged

Carry Gemini thought signatures on the transcript#219
mattt merged 2 commits into
huggingface:mainfrom
NoWaY233851:fix/gemini-thought-signature-transcript

Conversation

@NoWaY233851

@NoWaY233851 NoWaY233851 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #203.

Gemini attaches a thought_signature to each functionCall part and rejects any later request whose function calls have lost it, so tool calling could never complete on thinking models.

The signature was dropped twice. GeminiPart decoded it away, and Transcript.ToolCall had nowhere to keep it, so every request rebuilt from the transcript lost it again — the follow-up inside the tool loop, and the replay of that call on later turns.

Transcript.ToolCall gains providerMetadata: [String: String]? for opaque state a provider needs back verbatim, and the Gemini adapter keeps the signature there. It's documented as an AnyLanguageModel extension, following the - Note: This property is exclusive to AnyLanguageModel convention already used for the tool execution delegate. The property is optional, so transcripts encoded before this change still decode. Its key stays private for now, per review.

On the wire the signature sits on the enclosing part rather than inside functionCall, so GeminiPart reads and writes it and GeminiFunctionCall.thoughtSignature stays out of CodingKeys. Parallel calls carry a signature only on the first functionCall part; since it's held per ToolCall and rebuilt per part, that shape round-trips as issued.

The second commit is the transcriptEntries fix. respond reported ArraySlice(transcript) at all three exits — the whole conversation rather than the entries it added — and LanguageModelSession.respond appends that back, so history doubled every turn: request contents went 1 → 3 → 6 → 14 across three turns. That capped the signature fix at the second turn, because by the third the same signed functionCall part went out twice in one request. It now collects a local entries array like the other adapters, and the same three turns are 1 → 3 → 5 → 7.

The tests are offline: StubURLProtocol records request bodies and replays canned responses, so the Gemini tool path is covered without GEMINI_API_KEY and the wire types stay private.

  • echoesSignatureOnFollowUpRequest — the signature comes back with the function results.
  • keepsSignatureOnLaterTurn — it survives into the next turn of the same session.
  • doesNotReplayHistoryOnLaterTurns — request sizes stay linear over three turns and the tool call is replayed exactly once, still signed. Both of its assertions fail without the second commit.
  • Two TranscriptTests for the new property: omitted from the encoding when nil, round-tripped when set.

swift test --skip OllamaLanguageModel passes (325 tests); swift format lint --strict --recursive . is clean.

Signatures on text parts are still dropped; I'll open a separate issue for those as suggested.

Gemini attaches a `thought_signature` to each `functionCall` part and rejects a
request whose function calls have lost it:

    400 INVALID_ARGUMENT "Function call is missing a thought_signature in
    functionCall parts."

`GeminiPart` decoded the signature away, and `Transcript.ToolCall` had nowhere
to keep it, so every request rebuilt from the transcript dropped it — the
follow-up inside the tool loop, and the replay of that tool call on later turns.

Give `Transcript.ToolCall` a `providerMetadata` dictionary for opaque state that
a provider requires back verbatim, and have the Gemini adapter put the signature
there. The field is optional, so transcripts encoded before this change still
decode, and it is marked as an AnyLanguageModel extension rather than something
carried over from Foundation Models.

Other providers need the same shape: Anthropic's `encrypted_content` for server
tool results, and encrypted reasoning items on the OpenAI Responses API.
@mattt

mattt commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Hi @NoWaY233851. Thank you. This is exactly the shape I was hoping for, and the offline tests are great.

On your three questions:

  • Yes, please fold the transcriptEntries fix into this PR. Returning the whole transcript is a bug in its own right, and the signature fix needs it to hold past the second turn.
  • Keep the key private for now. Once a second provider needs the slot, we'll know whether a shared constant or per-provider keys is the right shape.
  • Agreed that signatures on text parts are a separate change. Feel free to open an issue for it.

I'll merge as soon as that's in.

`GeminiLanguageModel.respond` returned `transcriptEntries: ArraySlice(transcript)`
— the whole conversation rather than the entries it appended — where the other
adapters accumulate a local array. `LanguageModelSession.respond` appends that
back into the session transcript, so history doubled on every turn: request
`contents` went 1 -> 3 -> 6 -> 14 over three turns.

Until thought signatures were echoed, that was a token cost. It now also replays a
signed `functionCall` part at a position its signature was never issued for, so
the signature fix does not hold past the second turn without this.

Collect the tool call and tool output entries in a local `entries` array, as
`AnthropicLanguageModel` and the other adapters do, and report that. `transcript`
still carries the whole conversation, because each iteration of the tool loop
rebuilds the request from it.
@NoWaY233851

Copy link
Copy Markdown
Contributor Author

Second commit: respond now reports only the entries it added, so those same three turns go 1 → 3 → 5 → 7 instead of 1 → 3 → 6 → 14, and the tool call is replayed exactly once, still signed, however far out the conversation runs. doesNotReplayHistoryOnLaterTurns pins both; both of its assertions fail without that commit.

Key left private, and I'll open the text-part issue separately.

@mattt

mattt commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Merging now. Thank you, @NoWaY233851, for the fix and for pinning it with a test that fails without it.

@mattt
mattt merged commit 56a512f into huggingface:main Sep 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemini tool calling fails with HTTP 400: thought_signature is discarded when decoding functionCall parts

2 participants