Repository navigation
Trim AI prompts and cache Anthropic system prompts - #177
Conversation
Research prompts list each source once: a page read in full is no longer repeated as a search snippet, a URL several queries found appears once, and at most 12 snippets go in. Fetched pages, including links pasted into Ask, keep 2,000 characters instead of 8,000. Non-streamed Ask keeps the last 6 earlier messages, and only user and assistant turns. Anthropic models get a cache breakpoint on the system prompt. The limits live in supabase/functions/_shared/prompt-budget.ts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reachedEnable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Next included review available in 52 minutes. View limit details
Note 🎁 Summarized by CodeRabbit Free
Comment |
|
Codex review (gpt-6.1-sol, high), round 1 at 76f077e: no findings. It covered the research consumers (Ask citation filtering, factoids verification, Generate ranking, the timeout partial copy), every reader of the system message including the JSON repair retry and the fake provider, non-Anthropic passthrough with |
# Conflicts: # supabase/functions/ai/handler-research.test.ts
Cost fix 5 of 7. This makes AI prompts smaller and turns on Anthropic prompt caching. The other six cost fixes are separate PRs.
What changes
Research goes in once. Before, a research prompt listed every search's results under that search, then repeated the pages it had read in full. A page read in full therefore appeared twice, and a URL that several queries found appeared once per query. Now each source appears once. Pages read in full stay, and the 12 highest-ranked search snippets for other URLs go in beside them. The model still sees which queries ran.
Pages are cut to 2,000 characters. Research page reads and links pasted into Ask used to keep 8,000 characters (research prompts then used 4,000 of them). Both now keep 2,000, and we ask Exa for only that much.
Non-streamed Ask keeps the last 6 earlier messages. The server used to forward whatever history the browser sent (the browser caps it at 8). It now keeps the last 6, cuts each to 6,000 characters, and accepts only user and assistant turns. Before this, a request could put a
systemmessage into the history and the server passed it to the model. The new test caught that on the old code. Streamed Ask (ai-stream) used 8 and now reads the same setting, so it also keeps 6.Anthropic system prompts are cached. For
anthropic/*models, the system message goes to OpenRouter as a text part withcache_control: { type: "ephemeral" }. When the same prefix comes back within five minutes, Anthropic charges a tenth of the input price for it. The first call pays 1.25x on those tokens. Other models get exactly the request they got before.The limits are config. They all live in
supabase/functions/_shared/prompt-budget.ts:MAX_FETCHED_PAGE_CHARS(2,000),MAX_RESEARCH_SNIPPETS(12),MAX_ASK_HISTORY_MESSAGES(6),MAX_ASK_HISTORY_MESSAGE_CHARS(6,000),MAX_ASK_CONTEXT_CHARS(32,000, unchanged) andANTHROPIC_PROMPT_CACHING(on).How much it saves
On a fixture shaped like a Generate or Factoids run (6 overlapping searches of 10 results each and 5 pages read), the research block drops from about 100,000 characters to about 26,700, roughly 25,000 tokens down to 6,700. Real runs vary with how much the queries overlap.
I checked caching against real OpenRouter through the actual
callOpenRoutercode. I sent each prompt twice, and the second call read the system prompt from cache every time:So every feature's system prompt is long enough to cache, even without Library sources or debate context.
OpenRouter reported
cost: 0on all of these calls. Cost fix 6 covers that; this PR doesn't change cost logging.What the fake provider changed
e2e/full-stack/fake-openrouter.mjsnow reads a system prompt sent as text parts, because the verify stack's default model is Anthropic. It also recordsmessageCountandcacheBreakpointfor each call so a verify run can show what was sent.Tests
research.test.ts: a page read in full isn't repeated as a snippet, a URL two searches found appears once, and page text stays within the cap.handler-research.test.ts: non-streamed Ask sends only the last 6 earlier messages and drops asystemrole. This fails onmain.ai-stream/handler_test.ts: the streamed Ask request to an Anthropic model carries the cache breakpoint.Verification
Local runs at
76f077e. GitHub Actions isn't running for this repo, so there is no CI result.npm run verifysteps: docs check, lint, typecheck, collaboration tests (30), Edge tests (357), build and the service worker test all pass. Unit tests: 298 of 305 passed in the full run. The 7 others hit the 5-second timeout while the VPS load average was over 40. All 7 are frontend tests this PR doesn't touch, and they pass when run one file at a time (53 of 53). Smoke e2e: 28 passed, plus 2 flaky that passed on retry (announcements and the home page).verify-debatelab: not run. The parallel cost-fix threads kept the shared local Supabase stack lock busy the whole time. This change is backend-only with no UI, so instead I called the realaiEdge Function handler under Deno with real Exa research and real OpenRouter (Haiku 5.5). Only the Supabase auth, quota and logging calls were stubbed. I sent an Ask with web search, citations and 10 earlier messages, twice:Trade-offs
MAX_FETCHED_PAGE_CHARSor give pasted links their own setting.🤖 Generated with Claude Code