Hi — app.py's llm_anthropic sends system=bot_config["identity"] plus the whole reconstructed chat history concatenated into a single user prompt, with no cache_control — so every turn re-buys the bot identity and the entire conversation so far at full input rate. Because the history rides inside one user string, the fix is a small restructure: keep the identity in system as a block list marked {"cache_control": {"type": "ephemeral"}}, and pass history as proper messages (marking the latest content block) — later turns then read the shared prefix at ~10% of base input price, which for long conversations is most of the bill. One honest caveat: caching engages once the prefix passes the model's minimum (~1k tokens), so short chats won't see it and long ones will. Happy to open a PR if useful.
(Context: I build a cost-analysis tool for agent workloads — https://lens.r-lattice.com — and this repo came up when scanning public Claude agents for uncached call sites. No affiliation needed to take the fix.)
Hi — app.py's
llm_anthropicsendssystem=bot_config["identity"]plus the whole reconstructed chat history concatenated into a single user prompt, with nocache_control— so every turn re-buys the bot identity and the entire conversation so far at full input rate. Because the history rides inside one user string, the fix is a small restructure: keep the identity insystemas a block list marked{"cache_control": {"type": "ephemeral"}}, and pass history as propermessages(marking the latest content block) — later turns then read the shared prefix at ~10% of base input price, which for long conversations is most of the bill. One honest caveat: caching engages once the prefix passes the model's minimum (~1k tokens), so short chats won't see it and long ones will. Happy to open a PR if useful.(Context: I build a cost-analysis tool for agent workloads — https://lens.r-lattice.com — and this repo came up when scanning public Claude agents for uncached call sites. No affiliation needed to take the fix.)