inference budget: replace 400-row / 3.5M-char fold with model-window token budget #551

Parent

#548 (agent session architecture)

Problem

Inference is capped like a CRUD payload:

• maxMessages default 400
• maxChars default 3_500_000
• then out.slice(out.length - maxChars)

See formatPromptWithHistory (lib/sessionStore.ts). Catalog models expose
up to ~1M token windows. We store 8 MiB and send a character tail
that neither respects tokenizer reality nor the model’s window.

#327 / PR #547 paint provider usage on the context slot. That is a
meter, not a budget. We still fold with 400 / 3.5M.

Goal

The fold budget is model capability:

• Input sized to the selected model’s context window (catalog / provider
  metadata), minus a reserve for the completion (Pi-style
  contextWindow - reserveTokens, not a magic 3.5M chars).
• Prefer tokens (provider tokenizer or a documented estimator). Character
  slice is a last-resort backstop only, and must not be the product rule.
• Drop 400 as an intelligence cap. A row count may remain only as a
  safety rail against pathological message count, not as the way we
  “save payload size.”

Success: a long session on a 1M-context model actually uses that window
for prior work (after A1/A2 content exists to put in it). A small-window
model compact-first (A4) instead of silently tail-slicing prose.

Caps (for the future plan — do not change in this issue)

Any raise/lower of 400 / 3.5M is a human decision on the phase plan
(plan-review cap rule).

Non-goals

• Implementing compaction (A4) — this issue is the policy + accounting
• Viewport / localStorage (A5)
• Fake % of window without a real catalog max (forbidden; feat(#539): context/usage status slot — bounded provider-usage summary (chat+agent, JSON + stream done) folded into the Wasm context slot #547 honesty)

Suggested next

create-plan against #548 A3. Need a per-model window source (catalog
field vs provider vs conservative default) before deleting 3.5M.

Peer harness notes

Pi: compact when contextTokens > contextWindow - reserveTokens (default
reserve 16 384). Occupancy for that comparison is last/max step, not
the summed turn total painted today (1.3M in can be 15×90k). Meter work
is #556 (not this board); this issue consumes occupancy once it
exists.

Also shrink what we send before arguing the window:

• agent memory: structured truncated tool_result on the wire (not Tool: crumbs) #549 structured + truncated tool results (not 400-row prose)
• skills: inject a catalog, not full bodies, every turn #557 skill catalog instead of 256 KiB bodies (the 1.2M cached prefix)
• inference: cache-stable two-block system prompt (don’t bust the KV prefix) #558 two-block system so the prefix stays cacheable

Orrery (2026-08-15)

Budget still comes from last/max-step occupancy (#556, not this board) vs catalog window. Extra law: a family / model switch (operator Next, or a future router) is a cache miss — price the re-read of the warm prefix at fresh rates. Prefer switching only at the same boundaries as compaction (#552). Stay-put is the default decision.