agent session architecture — model-context memory, compaction, and viewport #548

Title intent

This is the project-tracking / roadmap issue for agent session intelligence
and viewport. It sequences work that #408 (session object: id, cwd, sandbox,
Redis, Blob) deliberately did not cover: what the model actually sees,
how the canvas pages history, and how first paint stays cheap.

#408 is the durable addressable session. This issue is the durable
agent mind + scroll window on top of that object.

Estimates and phase boundaries are directional. Each phase must be planned as
its own parent create-plan issue before implementation.

Status (2026-09-07): OPEN (umbrella). Diagnosis is locked. Shipped:
A1 #549 (plan #936 / PR #937,
Buffer follow-up PR #939),
A1-adj #693 (plan #941 / PR #943),
A2 #550 (plan #938 / PR #940),
A3 #551 (plan #944 / PR #945),
A4 #552 (plan #947 / PRs #953–#957; living docs PR #957 merge b31ca51),
A6 #557 (plan #931 / PR #932).
Next on this board: A5 #553 (must close #924; follow-up requires e2e int; likely multi-phase). A7 #558 / A8 #560 still OPEN.
#694 stays Later. Still do not open a fifth identity issue.

Goal

Stop treating the session as an app payload (400 rows, 3.5M char slice,
8 MiB Blob, 8 MiB localStorage, paint everything). Treat it as:

1. Agent intelligence — the model should remember what it did and what it
   learned across turns, up to the model’s real context window (hundreds
   of thousands to ~1M tokens on current catalog models). Asking “what did you
   already find?” should work.
2. A viewport, not a dump — the canvas scrolls a window. Almost never
   the whole 8 MiB. First paint must not JSON.stringify megabytes onto
   localStorage (Stop freeze).
3. Viewport A5 (spike: viewport hydrate — latest-page F5 (#924), slim local, scrollbar (A5 rebase) #553) — latest-page F5 that closes bug: F5 / reload fast-forward replays entire durable stream (need latest-page hydrate, no thinking replay) #924, slim local
   tail, scrollbar/load-earlier that does not require the host to hold the
   whole transcript. Mid-turn prev chunks + terminal flatten already exist;
   HTTP Range over one messages[] object is obsolete. Do not shard into
   thousands of part files.

The operator should not need to scroll at all if the agent still knows the work.

The lobotomy (2026-08 diagnosis)

Production durable turns no longer match this table for inference: A1
shipped a structured model-messages projection (meta.modelMessagesPointer);
A1-adj shipped a volatile per-turn freshness reminder (meta.freshnessReminderPointer);
A2 shipped session-owned meta.workingNotes; A3 shipped a window-derived
token fold budget; A6 shipped catalog-only skill inject; A4 shipped
model-window compaction (meta.compactionPointer checkpoint + retained tail).
Remaining first-turn / warehouse pain is A5 (viewport / slim local / #924).
The rows below are the 2026-08 baseline.

Symptom: the assistant refers to “the previous assistant,” forgets research,
and the tab freezes on Stop — orthogonal problems, same root: we store a
warehouse and ship crumbs to the Gateway.

See Identity dissociation (2026-08-18) below for the four fault lines
and the streamText response.messages lock on A1.

Target (directional — lock in phase plans)

Session (already shipped, #408)     = id, tenant, cwd, sandbox, pointer
AgentMemory (this umbrella)         = working notes + folded tool evidence
                                       + compacted history sized to model window
Viewport (this umbrella)            = slim local tail + latest-page F5 (#924)
                                       + scrollbar that feels infinite
                                       (not byte-Range on one JSON array)

Key invariant: inference budget ≠ storage budget ≠ paint budget.

• Storage may keep the full Blob (8 MiB object ceiling, not a Function body).
• Paint may keep a ring window (2048 is a canvas cap, not an intelligence cap).
• Inference uses model context tokens (catalog window − reserve; A3 inference budget: replace 400-row / 3.5M-char fold with model-window token budget #551 shipped). Occupancy metering is status bar: show current context occupancy in addition to summed turn usage #556. Compaction when over budget is compaction: model-capability context compaction (not API payload slices) #552 shipped. 400 / 3.5M are not intelligence caps.

Source issues

Baseline (do not re-litigate): #408 core session object, envelope + Blob
carrier (#515), tool_run display-only (#345), thinking collapse, status bar
#328 / #538. File mutate freshness (#277 RunFileFreshness) is shipped and
stays per-POST — do not put fingerprints on the session (#693).

Sequence / phases

Intelligence first (forgetting is the product bug). Viewport can spike in
parallel once slim-local is specified; do not block memory work on pagination.

A3 shipped separately from A4 (budget is the policy; compaction is the
mechanism). A4 shipped. Next: A5 #553 (must close #924; follow-up requires e2e int; likely multi-phase).

A1 lock (2026-08-15): #549 is native / structured tool_result on the
Gateway payload, not more prose in formatPromptWithHistory.

A1-adj lock (2026-08-19): #693 is not a fifth identity issue. A1 has
shipped, so the model does see last turn’s read_file and can think it has
an edit grant; #277’s ledger is empty every Function POST. Reminder plan is
#941 (source #693).
Shipped 2026-09-05: PR #943
(merge b0a81d6). Advisory volatile reminder; #277 stays the fail-closed gate.

A3 lock (2026-09-05): #551 is a window-derived token budget, not a tokenizer
and not occupancy. Durable seed trim at the /api/turns route boundary
(trimModelMessagesToBudget); host legacy fold (formatPromptWithHistory) is
token-budget first. Reserve is Pi-style max(16384, 15% × window); tokens are
ceil(chars/4). Plan #944 /
PR #945 (merge 9a049c6).
Occupancy meter stays #556; compaction shipped (#552 / plan #947 / PRs #953–#957).

A4 lock (2026-09-07): #552 is a model-window compaction engine, not a
payload slice. Trigger is the pre-trim seed vs catalog window −
COMPACTION_RESERVE_TOKENS; one server-only tools-off summarizer step; checkpoint
is a session-bound Blob meta.compactionPointer {summary, filesTouched, retainedTail} (persist-time livePostCompactTail then boundCheckpointForPersist
drop-oldest; over-cap fail-closes compaction_write_failed). Seed preference:
honesty-prefixed live model-messages → valid checkpoint → ordinary model-messages
→ legacy promptHistory. Failed/empty/unusable summary fails open (no fabricated
checkpoint); 'cancelled' / 'wall_clock' skip the first tools-on round.
Logical compact ≠ delete Blob. Wasm never summarizes. Plan
#947 / PRs
#953–#957
(phase 5 living docs PR #957
merge b31ca51). Occupancy meter stays #556. Viewport stays #553 / #924.

Peer harness map (OpenCode / Pi / Oh My Pi / Codex / DeepSeek)

Steal payload ideas. Skip their disk/process model (JSONL, SQLite,
part/*.json, Hashline, LSP-on-write, subagent worktrees, mailboxes,
homedir ~/.codex/memories).

Identity dissociation — operator report + research (2026-08-18)

Operator: agents do not recognize the previous session as themselves.
Each new send feels like the model’s first turn. This is not a foundation-model
defect.

It is the same “previous assistant / new hire each send” symptom in the
lobotomy table. A 2026-08-18 review of Pi, Oh My Pi, OpenCode, Codex CLI,
DeepSeek, and Vercel AI SDK streamText maps it onto four harness fault
lines we already have children for. Do not open a fifth issue.
A1 (#549) has since shipped (PRs #937 / #939); A1-adj (#693 / plan #941 /
PR #943) shipped 2026-09-05.
A3 (#551 / plan #944 / PR #945)
shipped 2026-09-05. A4 (#552 / plan #947 / PRs
#953–#957)
shipped 2026-09-07. Remaining first-turn / warehouse pain is A5 (viewport / #924).

#550 A2 (working notes) is complementary, not the first-turn fix. A notes
block on top of a pruned transcript is still a new assistant reading someone
else’s diary. A1 shipped first (evidence to remember); A2 notes now carry
identity; A4 compaction now writes the checkpoint those notes sit beside.

A1 mechanism — streamText response.messages (shipped)

lib/agent/runAgent.ts uses streamText + stepCountIs. The SDK’s
onFinish / result.response.messages is the complete ModelMessage[]
for the turn: assistant tool-calls, role: 'tool' results, reasoning parts,
final text — not the terminal assistant string the 2026-08 fold flattened.

A1 persists that array (bounded / truncated) as the model-facing history
and sends it as a messages array on turn N+1. Do not add more Tool:
lines to formatPromptWithHistory. Paint stays tool_run cards
(feature-divide). Abort/cancel: persist committed tool pairs the same way we
persist change_dir.

Peer split-result (Pi): LLM payload ≠ UI payload. We already have that split
in spirit (tool_run paint vs what A1 must send). Do not stuff paint JSON
into the Gateway.

Reasoning lock (do not silently flip)

• Paint / SessionStore: thinking rows stay ephemeral. Do not dump CoT
  into Redis/Blob as the memory mechanism.
• Model-facing replay: A1 locked, per provider, whether reasoning
  parts + providerMetadata ride the structured messages array (required
  for DeepSeek-R1 / Anthropic thinking continuation) or stay dropped. That
  remains an A1 lock, not a new issue.

Compaction (A4) — tool-pair atomicity (shipped)

When we compact, an assistant tool-call and its tool-result are kept or
summarized together. Never slice(-N) across that boundary. Checkpoint
is labeled summary, never live assistant prose (#552 shipped).

What this research does not change

• session architecture — addressable per-user sessions with session-owned state (cwd, sandbox binding) #408 session object / envelope / Blob
• Dual chat / Wasm talking to Redis or Blob
• Pi JSONL DAG, OMP loopback workers, or Codex /responses/compact as our
  runtime — steal the invariant, not the process
• Implementing any phase in this issue

Orrery takeaways (2026-08-15)

Steal when and measure. Do not steal their runtime.

• Compaction is a cache flush — phase boundary or hard ceiling only; write into a durable-spec slot, not a churning history head.
• Stay-put is a decision. Operator Next mid-session is priced as a prefix re-read.
• feat: session TODO lists — one item per turn, host auto-continues with full list in the prompt #536 phase tags are how compaction: model-capability context compaction (not API payload slices) #552 knows the boundary. Keep feat: session TODO lists — one item per turn, host auto-continues with full list in the prompt #536 off this board until the loop is phase-driven.
• A fold record (telemetry: per-turn fold records (know if cache/memory work landed) #560) is how we know A1/A6/A7 worked. Policy/eval/hashline/spawn stay out.

Skip: Go binary, SQLite SoT, hashline, first-party multi-provider router, localhost-no-auth, learned policy, replay eval.

Source-level peer notes (2026-08-19)

Four investigations of non-cloud harnesses (Codex, DeepSeek Harness, OpenCode,
Oh My Pi). Comment on this issue has the ranked steal/skip list. Locks that
belong on children, not a new identity bug:

1. Two systems. Live thread (agent memory: structured truncated tool_result on the wire (not Tool: crumbs) #549–compaction: model-capability context compaction (not API payload slices) #552, agent memory: durable working notes / findings across turns (identity, not one-shot) #550) ≠ deferred handbook (later: cross-session handbook (idle extract, default off) — not the first-turn fix #694).
   Mixing them is how compaction wipes a cat while the handbook never had it.
2. Persist ≠ fresh. After A1, tool_result is a snapshot. plan: agent read-before-edit + external file freshness (parent) #277 already
   empties the observation ledger every Function POST. Prompt-side reminder is
   agent memory: tool_result snapshots are not live file views (re-read or they’re stale) #693. Do not rewrite results; do not put fingerprints on meta.
3. Reload a durable array. OpenCode re-queries SQLite every llm.stream.
   We already reload (stateless Functions). A1 persists response.messages
   and stopped flattening tool_run paint (agent memory: structured truncated tool_result on the wire (not Tool: crumbs) #549).
4. Persist the compact replacement. Codex Compacted.replacement_history /
   OpenCode typed compaction message so resume == live post-compact view
   (compaction: model-capability context compaction (not API payload slices) #552). We have no FTS over shadowed history — files-touched + notes must
   carry the investigation.
5. Diff the world. Codex world-state / OpenCode Context Epoch: freeze block 1;
   chronological system diffs; rewrite the prefix only at compact (inference: cache-stable two-block system prompt (don’t bust the KV prefix) #558).
   OMP sticky RULES.md: re-attach standing orders near the leaf so A4 cannot
   bury the persona snapshot.
6. Don’t write the handbook on the hot path. Idle extract, no-op is valid,
   default off (OMP, not Codex-on-by-default). Cloud-shaped, not ~/.codex.
   later: cross-session handbook (idle extract, default off) — not the first-turn fix #694 Later.

Skip (unchanged): homedir JSONL, client SQLite, snapcompact PNGs, hashline as
the edit dialect, mnemopi embeddings as core, file-watcher inbox as v1.

Non-goals (this umbrella)

• Replacing Redis / envelope / server-minted ids (session architecture — addressable per-user sessions with session-owned state (cwd, sandbox binding) #408 stays)
• CRDT / real-time co-edit
• OpenCode’s thousands of part/*.json files or client SQLite as SoT
• Putting raw thinking blobs into SessionStore by default
• Sending 8 MiB through a Vercel Function (4.5 MB ceiling; 2 MiB function-body cap stays)
• Dual DOM chat / Wasm talking to Blob
• Implementing any phase in this issue
• Hashline, in-process bash, mid-stream token abort, subagent worktrees
• Provider anthropic.txt / beast.txt packs (optional later, not this board)
• Default-on cross-session handbook / live MEMORY.md writes during runAgent (later: cross-session handbook (idle extract, default off) — not the first-turn fix #694 is later, off)
• Persisting plan: agent read-before-edit + external file freshness (parent) #277 file fingerprints on the session blob (agent memory: tool_result snapshots are not live file views (re-read or they’re stale) #693)

Board

Only agent session architecture (user project 3).

The agent-session label is the auto-add trigger for this board. Do not also put #548–#558 / #560 / #372 / #693 / #694 on the #408 session-architecture board (project 2). That board is the shipped session object track. This umbrella is a different product problem (model memory + viewport).

Phase issues

• agent memory: structured truncated tool_result on the wire (not Tool: crumbs) #549 A1 — structured truncated tool results on the wire (PR #937)
• agent memory: tool_result snapshots are not live file views (re-read or they’re stale) #693 A1-adj — snapshots ≠ live files (PR #943; plan plan: volatile per-turn freshness reminder — snapshots are not live file views (A1-adj, source #693) #941)
• agent memory: durable working notes / findings across turns (identity, not one-shot) #550 A2 — working notes / identity (PR #940)
• inference budget: replace 400-row / 3.5M-char fold with model-window token budget #551 A3 — model-window budget (PR #945; plan plan: context-window fold budget (A3, source #551) #944)
• compaction: model-capability context compaction (not API payload slices) #552 A4 — compaction engine (plan plan: A4 compaction engine — checkpoint + retained tail, model-window budget (parent, source #552) #947; PRs #953–#957; living docs PR #957)
• spike: viewport hydrate — latest-page F5 (#924), slim local, scrollbar (A5 rebase) #553 A5 — viewport hydrate spike (must close bug: F5 / reload fast-forward replays entire durable stream (need latest-page hydrate, no thinking replay) #924; follow-up requires e2e int; likely multi-phase)
• skills: inject a catalog, not full bodies, every turn #557 A6 — skill catalog (not full bodies) (PR #932)
• inference: cache-stable two-block system prompt (don’t bust the KV prefix) #558 A7 — cache-stable system blocks
• telemetry: per-turn fold records (know if cache/memory work landed) #560 A8 — fold records
• feat: agent templates / forking (save good context, reuse later) #372 templates / fork — A4 shipped; still templates track
• later: cross-session handbook (idle extract, default off) — not the first-turn fix #694 Later — cross-session handbook (default off; A4 shipped, still Later)