Skip to content

feat(llm): add dynamic cache breakpoint on last message for Anthropic - #828

Merged
lizhengfeng101 merged 1 commit into
alibaba:mainfrom
yingjiexu2002:feat/anthropic-cache-breakpoint
Aug 10, 2026
Merged

feat(llm): add dynamic cache breakpoint on last message for Anthropic#828
lizhengfeng101 merged 1 commit into
alibaba:mainfrom
yingjiexu2002:feat/anthropic-cache-breakpoint

Conversation

@yingjiexu2002

Copy link
Copy Markdown
Contributor

Description

Mark the last content block of the final message with ephemeral cache_control so multi-turn conversations cache the growing history, not just the static system+tools prefix. Each turn then reads the previous full prefix and only writes the new delta.

Previously only two static breakpoints were set (last system block, last tool definition), so everything after the tools — the growing multi-turn conversation — was re-processed as fresh input on every turn of the per-file review loop.

This change adds a dynamic breakpoint on the last content block of the final message (user text, tool_result, or tool_use — all handled generically via the SDK's GetCacheControl() helper). Since request params are rebuilt from the append-only history on every turn, the marker never leaks back into stored history, and the breakpoint simply advances with each turn. Total breakpoints stay at 3, within Anthropic's limit of 4.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring (no functional changes)
  • Documentation update
  • CI / Build / Tooling

How Has This Been Tested?

  • make test passes locally
  • Manual testing (describe below)

Added TestBuildAnthropicParams_DynamicCacheBreakpoint covering:

  • conversations ending in a tool_result block (the typical agent-loop turn shape) — the last block gets cache_control: {"type":"ephemeral"} and earlier messages stay unmarked
  • conversations ending in user text

Also verified against the Anthropic API in a real multi-turn review session: turns after the first report cache_read tokens for the full prior prefix and only write the delta.

Checklist

  • My code follows the project's coding style (go fmt, go vet)
  • I have performed a self-review of my code
  • I have added tests that prove my fix is effective or my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated the documentation accordingly (if applicable)
  • I have signed the CLA

Related Issues

@github-actions

Copy link
Copy Markdown
Contributor

OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Mark the last content block of the final message with ephemeral
cache_control so multi-turn conversations cache the growing history,
not just the static system+tools prefix. Each turn then reads the
previous full prefix and only writes the new delta.
@yingjiexu2002
yingjiexu2002 force-pushed the feat/anthropic-cache-breakpoint branch from 7b52673 to 29a9821 Compare August 10, 2026 13:39

@lizhengfeng101 lizhengfeng101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lizhengfeng101
lizhengfeng101 merged commit db117ad into alibaba:main Aug 10, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants