Skip to content

Cache the repeated prompt prefix on Claude chat requests - #183

Closed
duaneatat wants to merge 1 commit into
mainfrom
db/chat-prompt-caching
Closed

Cache the repeated prompt prefix on Claude chat requests#183
duaneatat wants to merge 1 commit into
mainfrom
db/chat-prompt-caching

Conversation

@duaneatat

@duaneatat duaneatat commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Each question is several model requests that re-send the same tools, instructions and earlier tool results; this lets Claude read that prefix from cache.

  • Sets pydantic-ai's openrouter_cache_tool_definitions, openrouter_cache_instructions and openrouter_cache_messages
  • The turn's usage (and so saved eval runs) now reports cache_read_tokens and cache_write_tokens
  • Tests play a turn against a fake OpenRouter: Claude requests carry the markers, GPT requests are unchanged
  • Not run against the live provider; the saving needs an eval run to measure (cache writes cost more than plain input, reads much less)

@duaneatat
Cache the repeated prompt prefix on Claude chat requests
cdc3943 
One question is several model requests, each re-sending the tool
definitions, the instructions, and every earlier tool result. Mark all
three as cacheable through OpenRouter so later requests read them at
the cached rate. Other providers ignore the markers. The turn's usage
now reports cache read and write tokens.
@duaneatat
duaneatat marked this pull request as draft November 12, 2026 22:39
@duaneatat duaneatat mentioned this pull request Nov 12, 2026
Open
@duaneatat

duaneatat commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #182.

@duaneatat duaneatat closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@duaneatat