Skip to content

Commit 6364102

Browse files
Programcodeclaude
andauthored
Trim AI prompts and cache Anthropic system prompts (#177)
Research prompts list each source once: a page read in full is no longer repeated as a search snippet, a URL several queries found appears once, and at most 12 snippets go in. Fetched pages, including links pasted into Ask, keep 2,000 characters instead of 8,000. Non-streamed Ask keeps the last 6 earlier messages, and only user and assistant turns. Anthropic models get a cache breakpoint on the system prompt. The limits live in supabase/functions/_shared/prompt-budget.ts. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent e5850ee commit 6364102

10 files changed

Lines changed: 157 additions & 31 deletions

File tree

‎docs/architecture.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,8 @@ Research makes two small model calls of its own: Ask decides whether a question
9494

9595
The import endpoint validates public HTTP URLs, resolves redirect targets, blocks private networks, and enforces response timeout, content type, and size limits. Research paths bound source count and content before prompt construction.
9696

97+
[`supabase/functions/_shared/prompt-budget.ts`](../supabase/functions/_shared/prompt-budget.ts) holds the limits that set how many input tokens a request pays for: characters kept per fetched page, search snippets per research prompt, earlier Ask messages, and the Ask context cap. A research prompt lists each source once: a page read in full is not repeated as a search snippet, and a URL that several queries found appears a single time. For `anthropic/*` models the system prompt carries a `cache_control` breakpoint, so a repeat of the same prefix within five minutes is billed at the cache-read price; a prompt below the model's minimum cacheable length is sent uncached at the normal price.
98+
9799
Every AI request in a Current Debate carries a `debateContext` of IDs and preferences, never text: the debate, the side the request prepares, the Library source IDs, and the saved results to read. [`src/lib/debate/context.ts`](../src/lib/debate/context.ts) chooses the default with the rules in [`supabase/functions/_shared/debate-context.ts`](../supabase/functions/_shared/debate-context.ts): the debate's active sources, pinned results first, then up to three of the newest results of the types that feature uses, for the requested side. A compact line above each tool's primary button shows that selection, and Review changes it for one request, or for a whole Crossfire or Sparring round, where Review appears only before the round starts; pins persist in `debate_context_pins`. [`supabase/functions/_shared/debate-context-loader.ts`](../supabase/functions/_shared/debate-context-loader.ts) loads the motion, settings, instructions, sources, and result digests with every query filtered by the caller and the one debate, before quota or provider work. The prompt renders that material in delimited blocks followed by rules that it cannot override. Live sparring stays transcript-only: the scorecard sees only the motion and settings. Crossfire omits free-form instructions. Whole results are dropped when their budget is spent, and a Crossfire round that would pass its 100,000-character prompt budget drops its saved results. In both cases `meta.debateContext` reports what reached the model and what was set aside. A debate deleted on another device returns 404, and the browser reloads its debate list. One-off requests send no debate context. The retired per-user custom system prompt is no longer sent or honored; operator prompts in `global_prompts` remain. Jev scores relevance, importance, and evidence strength after the existing deterministic quality floor, while failures preserve the prior order. Generate also records a bounded source-use snapshot in the saved case output. Semantic counts and the latest case-use flag live in the existing source `metadata` JSON, so the Library can default to Smart ordering without a schema migration.
98100

99101
Ordinary Ask remains stateless while a conversation is in progress. The browser sends only the newest eight messages within a UTF-8 byte budget. Inside a debate, each finished exchange is recorded into one chat for that debate; in one-off mode, pressing Save persists the chat and its messages; Workspace threads use their separate durable server-side context path. Legacy local-only sources are not sent to the authenticated AI endpoint because they have no persisted UUID to authorize and load.

‎e2e/full-stack/fake-openrouter.mjs‎

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -176,16 +176,28 @@ const routes = [
176176
["ask", "You are a debate coach chat bot", ask],
177177
];
178178

179+
// A cacheable system prompt arrives as text parts instead of a plain string.
180+
function messageText(message) {
181+
const content = message?.content;
182+
if (Array.isArray(content)) return content.map((part) => part?.text ?? "").join("");
183+
return String(content ?? "");
184+
}
185+
179186
function classify(body) {
180187
const messages = Array.isArray(body.messages) ? body.messages : [];
181-
const system = String(messages.find((message) => message.role === "system")?.content ?? "");
188+
const systemMessage = messages.find((message) => message.role === "system");
189+
const system = messageText(systemMessage);
182190
const user = String(messages.at(-1)?.content ?? "");
191+
const shape = {
192+
messageCount: messages.length,
193+
cacheBreakpoint: Array.isArray(systemMessage?.content) && systemMessage.content.some((part) => part?.cache_control),
194+
};
183195
if (body.stream === true && system.includes("Answer in clear Markdown that can stream smoothly")) {
184-
return { feature: "ask-stream", system, user };
196+
return { feature: "ask-stream", system, user, ...shape };
185197
}
186198
const route = routes.find(([, marker]) => system.includes(marker));
187-
if (!route) return { feature: null, system, user };
188-
return { feature: route[0], system, user, output: route[2](user, system) };
199+
if (!route) return { feature: null, system, user, ...shape };
200+
return { feature: route[0], system, user, ...shape, output: route[2](user, system) };
189201
}
190202

191203
function jevAnswers(questions) {
@@ -358,8 +370,9 @@ createServer(async (req, res) => {
358370
}
359371

360372
const body = JSON.parse(await readBody(req));
361-
const { feature, system, user, output } = classify(body);
362-
requests.push({ feature, model: body.model, stream: body.stream === true, system, user });
373+
const { output, ...record } = classify(body);
374+
const { feature } = record;
375+
requests.push({ ...record, model: body.model, stream: body.stream === true });
363376
if (!feature) return json(res, 500, { error: { message: "Fake OpenRouter does not recognize this prompt" } });
364377

365378
const failure = failures.find((item) => item.count > 0 && (!item.feature || item.feature === feature));
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
// Prompt size and caching limits for the AI Edge Functions. Every number here
2+
// sets how many input tokens a request pays for, so raise them deliberately.
3+
4+
/** Characters kept from each fetched web page: research reads and links pasted into Ask. */
5+
export const MAX_FETCHED_PAGE_CHARS = 2_000;
6+
7+
/** Search snippets rendered into a research prompt, besides the pages read in full. */
8+
export const MAX_RESEARCH_SNIPPETS = 12;
9+
10+
/** Earlier chat messages an Ask request sends to the model. */
11+
export const MAX_ASK_HISTORY_MESSAGES = 6;
12+
13+
/** Characters kept from each earlier Ask message. */
14+
export const MAX_ASK_HISTORY_MESSAGE_CHARS = 6_000;
15+
16+
/** Characters of pasted-page and research context a non-streamed Ask request may carry. */
17+
export const MAX_ASK_CONTEXT_CHARS = 32_000;
18+
19+
/**
20+
* Mark the system prompt cacheable for Anthropic models. OpenRouter passes the
21+
* breakpoint to Anthropic, which bills a repeat of the same prefix within five
22+
* minutes at a tenth of the input price. Prompts below the model's minimum
23+
* cacheable length are sent uncached at the normal price.
24+
*/
25+
export const ANTHROPIC_PROMPT_CACHING = true;
26+
27+
type ChatMessage = { role: string; content: string };
28+
type CacheableMessage = ChatMessage | {
29+
role: string;
30+
content: Array<{ type: "text"; text: string; cache_control: { type: "ephemeral" } }>;
31+
};
32+
33+
/** Adds an Anthropic cache breakpoint to the system message; other models get the messages unchanged. */
34+
export function withPromptCaching(model: string, messages: ChatMessage[]): CacheableMessage[] {
35+
if (!ANTHROPIC_PROMPT_CACHING || !model.startsWith("anthropic/")) return messages;
36+
return messages.map((message) => message.role === "system"
37+
? { role: "system", content: [{ type: "text", text: message.content, cache_control: { type: "ephemeral" } }] }
38+
: message);
39+
}
40+
41+
/** The last earlier messages of an Ask conversation, each cut to the per-message cap. */
42+
export function boundAskHistory(history: unknown): ChatMessage[] {
43+
if (!Array.isArray(history)) return [];
44+
return history.slice(-MAX_ASK_HISTORY_MESSAGES).flatMap((item) => {
45+
if (!item || typeof item !== "object") return [];
46+
const { role, content } = item as Record<string, unknown>;
47+
if ((role !== "user" && role !== "assistant") || typeof content !== "string" || !content) return [];
48+
return [{ role, content: content.slice(0, MAX_ASK_HISTORY_MESSAGE_CHARS) }];
49+
});
50+
}

‎supabase/functions/_shared/research.test.ts‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,18 +12,40 @@ import {
1212
MAX_PROVIDER_RESPONSE_BYTES,
1313
type ResearchContext,
1414
} from "./research.ts";
15+
import { MAX_FETCHED_PAGE_CHARS } from "./prompt-budget.ts";
1516

1617
function ctxWith(overrides: Partial<ResearchContext>): ResearchContext {
1718
return { ...emptyResearchContext(), ...overrides };
1819
}
1920

2021
Deno.test("empty research does not render a research context block", () => {
2122
const rendered = buildResearchContext(ctxWith({
22-
searches: [{ query: "empty search", results: "" }],
23+
searches: [{ query: "empty search", results: [] }],
2324
}));
2425
if (rendered !== "") throw new Error(`empty research rendered unexpected context: ${rendered}`);
2526
});
2627

28+
Deno.test("research context lists each source once and caps page text", () => {
29+
const ranked = (url: string, query: string, qualityScore: number) => ({
30+
url, query, title: `Title ${qualityScore}`, content: "snippet", sourceType: "news" as const, qualityScore, qualityLabel: "strong" as const, qualityReasons: [],
31+
});
32+
const read = ranked("https://news.example/read", "first", 90);
33+
const shared = ranked("https://news.example/shared", "first", 80);
34+
const rendered = buildResearchContext(ctxWith({
35+
searches: [
36+
{ query: "first", results: [read, shared] },
37+
{ query: "second", results: [{ ...shared, query: "second" }, ranked("https://news.example/other", "second", 60)] },
38+
],
39+
rankedResults: [read, shared],
40+
readContent: [{ url: read.url, title: "Read", content: "p".repeat(MAX_FETCHED_PAGE_CHARS + 500), result: read }],
41+
}));
42+
const count = (needle: string) => rendered.split(needle).length - 1;
43+
if (count(read.url) !== 1) throw new Error("a page read in full was repeated as a search snippet");
44+
if (count(shared.url) !== 1) throw new Error("a URL found by two searches was listed twice");
45+
if (count("https://news.example/other") !== 1) throw new Error("a distinct search result was dropped");
46+
if (count("p".repeat(MAX_FETCHED_PAGE_CHARS + 1)) !== 0) throw new Error("page text exceeded the configured cap");
47+
});
48+
2749
Deno.test("factoid citations backed by strong or moderate fetched sources stay verified", () => {
2850
const research = ctxWith({
2951
readContent: [
@@ -242,7 +264,7 @@ Deno.test("Exa contents sends the documented maxCharacters field", async () => {
242264
try {
243265
await exaGetContents("https://example.com/report");
244266
const text = requestBody?.text as { maxCharacters?: number; max_characters?: number } | undefined;
245-
if (text?.maxCharacters !== 8000) throw new Error("contents maxCharacters was not sent");
267+
if (text?.maxCharacters !== MAX_FETCHED_PAGE_CHARS) throw new Error("contents maxCharacters was not sent");
246268
if (text?.max_characters !== undefined) throw new Error("legacy contents casing leaked into request");
247269
} finally {
248270
globalThis.fetch = originalFetch;

‎supabase/functions/_shared/research.ts‎

Lines changed: 27 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ import {
1818
export type { ExaResult, RankedSearchResult, ResearchSource };
1919

2020
import type { ReasoningEffort } from "./model-catalog.ts";
21+
import { MAX_FETCHED_PAGE_CHARS, MAX_RESEARCH_SNIPPETS } from "./prompt-budget.ts";
2122

2223
export type Provider = "openrouter";
2324

@@ -35,7 +36,7 @@ export type Source = {
3536
};
3637

3738
export interface ResearchContext {
38-
searches: Array<{ query: string; results: string }>;
39+
searches: Array<{ query: string; results: RankedSearchResult[] }>;
3940
rankedResults: RankedSearchResult[];
4041
readContent: Array<{ url: string; title: string; content: string; result?: RankedSearchResult }>;
4142
verifiedUrls: Set<string>;
@@ -298,8 +299,8 @@ export async function exaSearch(search: string | ExaSearchRequest, signal?: Abor
298299
}
299300
}
300301

301-
// exaGetContents: extract full text from a specific URL using Exa's /contents endpoint
302-
export async function exaGetContents(url: string, signal?: AbortSignal): Promise<string> {
302+
// exaGetContents: extract a page's text, up to maxChars, using Exa's /contents endpoint
303+
export async function exaGetContents(url: string, signal?: AbortSignal, maxChars = MAX_FETCHED_PAGE_CHARS): Promise<string> {
303304
const apiKey = Deno.env.get("EXA_API_KEY");
304305
if (!apiKey) {
305306
console.warn("[EXA] EXA_API_KEY not configured");
@@ -310,15 +311,15 @@ export async function exaGetContents(url: string, signal?: AbortSignal): Promise
310311
const { response, data } = await fetchJsonWithTimeout<{ results?: unknown }>("https://api.exa.ai/contents", {
311312
method: "POST",
312313
headers: { "Content-Type": "application/json", "x-api-key": apiKey },
313-
body: JSON.stringify({ urls: [url], text: { maxCharacters: 8000 } }),
314+
body: JSON.stringify({ urls: [url], text: { maxCharacters: maxChars } }),
314315
}, 30000, signal);
315316
throwIfAborted(signal);
316317
if (!response.ok) {
317318
console.warn(`[EXA] Contents fetch failed (${response.status}) urlLength=${url.length}`);
318319
return "";
319320
}
320321
if (!data?.results || !Array.isArray(data.results) || data.results.length === 0) return "";
321-
return (data.results[0]?.text || "").slice(0, 8000);
322+
return (data.results[0]?.text || "").slice(0, maxChars);
322323
} catch (err) {
323324
if (signal?.aborted) throw err;
324325
console.warn(`[EXA] Contents fetch failed urlLength=${url.length}`);
@@ -464,7 +465,7 @@ async function deepResearchInner(
464465
throwIfAborted(signal);
465466
const ranked = rankSearchResults(results.map((result) => ({ ...result, query })), claim, preferredSourceType);
466467
const visibleRanked = preferredSourceType === "any" ? ranked : ranked.filter(matchesPreferredSourceType);
467-
context.searches.push({ query, results: formatRankedSearchResults(visibleRanked) });
468+
context.searches.push({ query, results: visibleRanked });
468469
console.log(`[RESEARCH] Seed queryLength=${query.length}: ${results.length} results`);
469470
return results.map((result) => ({ ...result, query }));
470471
}));
@@ -484,7 +485,7 @@ async function deepResearchInner(
484485
throwIfAborted(signal);
485486
const ranked = rankSearchResults(results.map((result) => ({ ...result, query })), claim, preferredSourceType);
486487
const visibleRanked = preferredSourceType === "any" ? ranked : ranked.filter(matchesPreferredSourceType);
487-
context.searches.push({ query, results: formatRankedSearchResults(visibleRanked) });
488+
context.searches.push({ query, results: visibleRanked });
488489
allResults.push(...results.map((result) => ({ ...result, query })));
489490
return results.length;
490491
});
@@ -574,9 +575,24 @@ export function buildResearchContext(research: ResearchContext): string {
574575
let ctx = "=== WEB RESEARCH RESULTS ===\n\n";
575576
ctx += "IMPORTANT: Only cite sources marked 'strong' or 'moderate'. Sources marked 'weak' are fallback context only, do NOT cite them as evidence. If only weak sources are available, state that evidence is limited.\n\n";
576577

577-
// Add search results
578-
for (const search of research.searches) {
579-
ctx += `--- Search: "${search.query}" ---\n${search.results}\n\n`;
578+
// Each source appears once. A page read in full is not repeated as a search
579+
// snippet, and a URL several queries found is listed a single time.
580+
const listed = new Set(research.readContent.map((page) => page.url));
581+
const snippets = research.searches
582+
.flatMap((search) => search.results)
583+
.sort((a, b) => b.qualityScore - a.qualityScore)
584+
.filter((result) => {
585+
if (!result.url || listed.has(result.url)) return false;
586+
listed.add(result.url);
587+
return true;
588+
})
589+
.slice(0, MAX_RESEARCH_SNIPPETS);
590+
591+
if (research.searches.length > 0) {
592+
ctx += `Searches run: ${research.searches.map((search) => `"${search.query}"`).join("; ")}\n\n`;
593+
}
594+
if (snippets.length > 0) {
595+
ctx += `--- Search results ---\n${formatRankedSearchResults(snippets)}\n\n`;
580596
}
581597

582598
// Add full page content
@@ -585,7 +601,7 @@ export function buildResearchContext(research: ResearchContext): string {
585601
for (const page of research.readContent) {
586602
const quality = page.result ? `${page.result.sourceType}, ${page.result.qualityLabel}, ${page.result.qualityScore}/100` : "verified";
587603
const warning = page.result?.qualityLabel === "weak" ? " [WEAK, do not cite as evidence]" : "";
588-
ctx += `--- From: ${page.title} (${page.url}) ---\nSource quality: ${quality}${warning}\n${page.content.slice(0, 4000)}\n\n`;
604+
ctx += `--- From: ${page.title} (${page.url}) ---\nSource quality: ${quality}${warning}\n${page.content.slice(0, MAX_FETCHED_PAGE_CHARS)}\n\n`;
589605
}
590606
}
591607

‎supabase/functions/ai-stream/handler_test.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,10 @@ Deno.test("streaming Ask forwards the shared prompt and returns provider SSE", a
155155
assertIncludes(body, '"requestedModel":"auto-fast"');
156156
assertIncludes(body, '"resolvedModel":"anthropic/claude-haiku-5.5"');
157157

158-
const messages = providerRequest?.messages as Array<{ role: string; content: string }> | undefined;
158+
const messages = providerRequest?.messages as Array<{
159+
role: string;
160+
content: string | Array<{ text: string; cache_control?: unknown }>;
161+
}> | undefined;
159162
if (providerRequest?.model !== "anthropic/claude-haiku-5.5") {
160163
throw new Error(`unexpected OpenRouter model: ${String(providerRequest?.model)}`);
161164
}
@@ -170,7 +173,11 @@ Deno.test("streaming Ask forwards the shared prompt and returns provider SSE", a
170173
if ("sort" in preferences || "order" in preferences) {
171174
throw new Error(`sort/order disables OpenRouter balanced routing: ${JSON.stringify(preferences)}`);
172175
}
173-
const system = messages?.find((message) => message.role === "system")?.content ?? "";
176+
const systemContent = messages?.find((message) => message.role === "system")?.content;
177+
if (!Array.isArray(systemContent) || !systemContent.some((part) => part.cache_control)) {
178+
throw new Error("Anthropic system prompt was sent without a cache breakpoint");
179+
}
180+
const system = systemContent.map((part) => part.text).join("");
174181
assertIncludes(system, "Answer in clear Markdown that can stream smoothly token by token.");
175182
assertIncludes(system, "Follow any explicit output-language instruction.");
176183
assertIncludes(system, "Otherwise match the language used by the user.");

‎supabase/functions/ai-stream/index.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import { HttpError, jsonError, requireUser } from "../_shared/auth.ts";
2222
import { getCorsHeaders } from "../_shared/cors.ts";
2323
import { checkRateLimit, logUsage } from "../_shared/entitlements.ts";
2424
import { OPENROUTER_API_URL } from "../_shared/research.ts";
25+
import { MAX_ASK_HISTORY_MESSAGE_CHARS, MAX_ASK_HISTORY_MESSAGES, withPromptCaching } from "../_shared/prompt-budget.ts";
2526
import {
2627
AUTO_PRO,
2728
DEFAULT_MODEL,
@@ -51,7 +52,6 @@ const supabaseServiceKey = Deno.env.get("SUPABASE_SERVICE_ROLE_KEY")!;
5152
const supabase = createClient(supabaseUrl, supabaseServiceKey);
5253

5354
const MAX_INPUT_LENGTH = 50000;
54-
const MAX_HISTORY_ITEMS = 8;
5555
const OPENROUTER_PROVIDER = "openrouter" as const;
5656

5757
function jsonResponse(body: Record<string, unknown>, status: number, corsHeaders: Record<string, string>) {
@@ -73,10 +73,10 @@ function sanitizeText(value: unknown, limit: number) {
7373

7474
function normalizeHistory(value: unknown): ChatMessage[] {
7575
if (!Array.isArray(value)) return [];
76-
return value.slice(-MAX_HISTORY_ITEMS).flatMap((item) => {
76+
return value.slice(-MAX_ASK_HISTORY_MESSAGES).flatMap((item) => {
7777
if (!item || typeof item !== "object") return [];
7878
const role = (item as Record<string, unknown>).role;
79-
const content = sanitizeText((item as Record<string, unknown>).content, 6000);
79+
const content = sanitizeText((item as Record<string, unknown>).content, MAX_ASK_HISTORY_MESSAGE_CHARS);
8080
if ((role === "user" || role === "assistant") && content) return [{ role, content }];
8181
return [];
8282
});
@@ -264,7 +264,7 @@ export async function handleRequest(req: Request): Promise<Response> {
264264

265265
const payload: Record<string, unknown> = {
266266
model,
267-
messages,
267+
messages: withPromptCaching(model, messages),
268268
max_tokens: Math.min(modelEntry.maxTokens, MAX_REQUEST_OUTPUT_TOKENS),
269269
reasoning: { effort: reasoningEffort, exclude: true },
270270
// Omitting sort and order keeps OpenRouter's balanced provider load balancing.

0 commit comments

Comments
 (0)