From c96810c12d32bf4b2a73d42c68c8aa313eac45b6 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Fri, 14 Aug 2026 21:37:55 +0900 Subject: [PATCH] docs(devlog): plan the usage lifecycle and memory-defense roadmap Two structural problems get one dependency-ordered plan. Usage storage is asymmetric today: appendUsageEntry writes an unbounded usage.jsonl while the management reader only parses the newest 64 MiB, so a nominal 30d aggregate silently drops everything older than the tail (#1497, #1580). U1-U3 replace that with rotated JSONL segments, a rebuildable SQLite projection, and an API that reads the projection instead of re-parsing a tail. Memory defense is the second track. M0-1 and M0-2 split the two independent theses out of the #1412 draft: refuse oversized input before upstream I/O, and stop prepending stored history to a request that already carries it. M0-3 adds per-provider upstreamHttpVersion and responseDelivery so a provider-specific transport failure is fixable without touching global streamMode. M0-4 connects the existing warn-only watchdog to the existing drain-and-restart. M0-5 extends the JSON nesting ceiling already landed in serialize.ts to YAML and TOML, whose recursive writer has no equivalent bound. Docs only: no source file changes, and each decade doc is written to diff-level precision so its implementation cycle starts from an executable plan rather than an outline. --- .../260814_usage_memory_roadmap/000_plan.md | 110 ++++++++++++ .../010_m0_1_input_admission.md | 113 ++++++++++++ .../020_m0_2_continuation_dedup.md | 101 +++++++++++ .../030_m0_3_provider_delivery.md | 107 ++++++++++++ .../040_m0_4_memory_recovery.md | 125 ++++++++++++++ .../050_m0_5_nonjson_depth_cap.md | 118 +++++++++++++ .../060_u1_segmented_writer.md | 161 ++++++++++++++++++ .../070_u2_sqlite_projector.md | 125 ++++++++++++++ .../080_u3_projection_api.md | 97 +++++++++++ 9 files changed, 1057 insertions(+) create mode 100644 devlog/_plan/260814_usage_memory_roadmap/000_plan.md create mode 100644 devlog/_plan/260814_usage_memory_roadmap/010_m0_1_input_admission.md create mode 100644 devlog/_plan/260814_usage_memory_roadmap/020_m0_2_continuation_dedup.md create mode 100644 devlog/_plan/260814_usage_memory_roadmap/030_m0_3_provider_delivery.md create mode 100644 devlog/_plan/260814_usage_memory_roadmap/040_m0_4_memory_recovery.md create mode 100644 devlog/_plan/260814_usage_memory_roadmap/050_m0_5_nonjson_depth_cap.md create mode 100644 devlog/_plan/260814_usage_memory_roadmap/060_u1_segmented_writer.md create mode 100644 devlog/_plan/260814_usage_memory_roadmap/070_u2_sqlite_projector.md create mode 100644 devlog/_plan/260814_usage_memory_roadmap/080_u3_projection_api.md diff --git a/devlog/_plan/260814_usage_memory_roadmap/000_plan.md b/devlog/_plan/260814_usage_memory_roadmap/000_plan.md new file mode 100644 index 0000000000..d2f8e98d4c --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/000_plan.md @@ -0,0 +1,110 @@ +--- +title: Usage & Memory Roadmap — P0 through P3 +date: 2026-08-14 +class: C4 +prior-art: "#1008 (closed/unmerged), #1412 (open draft), #1367 (open), #1635 (open), #820 (open), #1217 (open)" +--- + +# 000 — Objective + +OpenCodex has two independent structural problems: + +1. **Usage data lifecycle** — a single unbounded `usage.jsonl` that grows forever, + dashboard reads only the newest 64 MiB tail, and 30d/all-time aggregates silently + lose data beyond that window (#1497, #1580). + +2. **Memory defense before Bun 1.4** — oversized input, continuation compounding, + provider transport failures, and no process-wide memory coordination mean a single + bad request or slow client can push RSS to OOM. + +This unit implements P0 (memory defense, 5 PRs) and P1 (usage storage, 3 PRs) as a +stacked PR chain. P2 (retention + UI) and P3 (stream coordinator) are documented but +deferred. + +## Constraints + +- CWD: worktree on `dev` at `/Users/jun/.codex/worktrees/8d76/opencodex` +- Bun 1.3.14 bundled, `MIN_FIXED_BUN_VERSION = null` +- No gui/ changes (P2 scope) +- No docs-site/ changes +- No release automation changes +- Preserve existing dirty state (antigravity-models.ts, 260814_gemini-tiered-wire-ids) +- Push with --no-verify, CI deferred to stack review time +- Local focused tests as verification bar + +## Current codebase state (2026-08-14) + +| Component | File | State | +|-----------|------|-------| +| Usage writer | `src/usage/log.ts` | Single `appendUsageEntry` → `usage.jsonl`, no rotation | +| Usage reader | `src/usage/log.ts:983` | 64 MiB tail, incremental after first read | +| Usage summary | `src/usage/summary.ts` | In-memory aggregation from parsed entries | +| Usage cache | `src/server/management/usage-summary-cache.ts` | TTL cache over summary | +| Memory watchdog | `src/server/memory-watchdog.ts` | Warn-only, no restart | +| Lifecycle | `src/server/lifecycle.ts` | `drainAndShutdown` exists, `markRecyclingForExit` exists | +| Admission | `src/lib/admission.ts` | Per-host circuit gate, no input-size gate | +| Config types | `src/types.ts:616` (OcxConfig), `src/types.ts:1233` (OcxProviderConfig) | +| Depth cap | `src/integrations/serialize.ts:231` | JSON only (`MAX_JSON_NESTING=1000`) | +| YAML/TOML | `src/integrations/serialize.ts` | No depth cap on YAML write, no YAML/TOML parse cap | +| Continuation | `src/server/responses/core.ts` | previous_response_id expansion, no size admission | +| Stream mode | `src/types.ts:709` | `streamMode: auto|legacy-tee|eager-relay` | +| Provider config | `src/types.ts:1233` | No `upstreamHttpVersion` or `responseDelivery` fields | + +## Phase map (dependency-ordered) + +``` +P0 — Memory defense (independent of each other, stacked for review): + 010 M0-1 Input admission gate → branch: codex/m0-1-input-admission + 020 M0-2 Continuation overlap removal → branch: codex/m0-2-continuation-dedup + 030 M0-3 Provider delivery policy → branch: codex/m0-3-provider-delivery + 040 M0-4 Memory recovery policy → branch: codex/m0-4-memory-recovery + 050 M0-5 Non-JSON depth cap → branch: codex/m0-5-nonjson-depth-cap + +P1 — Usage storage (U1 → U2 → U3 dependency chain): + 060 U1 Segmented usage writer → branch: codex/u1-segmented-writer + 070 U2 SQLite projector → branch: codex/u2-sqlite-projector + 080 U3 Projection-backed API → branch: codex/u3-projection-api +``` + +P0 items are functionally independent but stacked for review ordering. P1 items +have real dependencies: U2 consumes U1's segments, U3 consumes U2's SQLite. + +## Stack plan (DEV-STACK-01) + +The 8 branches form a single review stack: +- Layer 0: `codex/usage-memory-roadmap-docs` (base: `dev`) — this unit, docs only +- Layer 1: `codex/m0-1-input-admission` (base: layer 0) +- Each subsequent branch bases on the one below +- Merge bottom-up after review + +Layer 0 exists so every implementation layer can cite its own decade doc from a +base that already contains it. It carries no source change, so it is mergeable on +its own and cannot block the layers above it. + +## Verifiers + +| Phase | Command | Reads target | +|-------|---------|-------------| +| M0-1 | `bun test tests/input-admission.test.ts` | New test file | +| M0-2 | `bun test tests/continuation-dedup.test.ts` | New test file | +| M0-3 | `bun test tests/provider-delivery.test.ts` | New test file | +| M0-4 | `bun test tests/memory-recovery.test.ts` | New test file | +| M0-5 | `bun test tests/nonjson-depth.test.ts` | New test file | +| U1 | `bun test tests/usage-segmented-writer.test.ts` | New test file | +| U2 | `bun test tests/usage-sqlite-projector.test.ts` | New test file | +| U3 | `bun test tests/usage-projection-api.test.ts` | New test file | +| All | `bun run typecheck` | Whole project | + +## SoT sync target + +`structure/` — update if architectural boundaries change. No current architecture +doc covers the usage subsystem or memory coordination. + +## Out of scope + +- gui/ dashboard changes (P2) +- docs-site/ updates +- Release automation +- Bun upgrade (P4) +- Stream timeline / TurnScope / one-reader relay (P3, documented only) +- WebSocket upstream (#1608, Bun 1.4 gated) diff --git a/devlog/_plan/260814_usage_memory_roadmap/010_m0_1_input_admission.md b/devlog/_plan/260814_usage_memory_roadmap/010_m0_1_input_admission.md new file mode 100644 index 0000000000..913ba5ead3 --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/010_m0_1_input_admission.md @@ -0,0 +1,113 @@ +--- +title: "M0-1: Responses input admission gate" +phase: "010" +depends: [] +consumes: [] +branch: codex/m0-1-input-admission +closes: "(split from #1412)" +--- + +# 010 — M0-1: Model-aware input admission gate + +## Thesis + +Reject oversized requests BEFORE upstream dispatch. A 1.3M-token request that exceeds +the model's context window should never reach the provider — it wastes bandwidth, holds +a turn slot, and the provider will reject it anyway with a less useful error. + +## Current state + +- `src/server/responses/core.ts:1843` has `acquireUpstreamHostAdmission` — per-host + circuit breaker, not size-based +- `src/server/responses/core.ts:680` returns 413 for translator buffer overflow + (post-translation, not pre-dispatch) +- `src/types.ts:1340` has `contextWindow` and `modelContextWindows` on OcxProviderConfig +- `src/types.ts:1347` has `modelMaxInputTokens` on OcxProviderConfig +- No code path compares request input size against model context window before dispatch + +## File change map + +### NEW: src/server/responses/input-admission.ts + +Purpose: Pre-dispatch input size estimation and admission gate. + +```ts +export interface InputAdmissionResult { + admitted: boolean; + estimatedTokens: number; + contextWindow: number | null; + reason?: string; +} + +/** + * Estimate token count from a parsed Responses request and compare + * against the model's advertised context window. + * + * Token estimation: count characters / 4 as a rough upper bound, + * with base64 image data counted at its decoded byte size / 750. + * This is intentionally conservative (overestimates) — better to + * reject a request that's close than to let a too-large one through. + */ +export function estimateInputTokens(parsed: OcxParsedRequest): number; + +/** + * Resolve the effective context window for a provider+model pair. + * Priority: modelContextWindows[model] > contextWindow > null. + */ +export function resolveContextWindow( + provider: OcxProviderConfig, + model: string, +): number | null; + +/** + * Check whether the estimated input fits within the model's context window. + * Returns admitted:true if no context window is known (fail-open for unconfigured models). + */ +export function checkInputAdmission( + parsed: OcxParsedRequest, + provider: OcxProviderConfig, + model: string, +): InputAdmissionResult; +``` + +### MODIFY: src/server/responses/core.ts + +Location: Inside `handleResponsesInner`, after route resolution and before +`acquireUpstreamHostAdmission` (around line 1840). + +```diff ++ // Pre-dispatch input admission: reject requests whose estimated token count ++ // exceeds the resolved model context window. ++ const admission = checkInputAdmission(parsed, providerConfig, resolvedModel); ++ if (!admission.admitted) { ++ return formatErrorResponse(413, "request_too_large", ++ `Estimated input (~${admission.estimatedTokens} tokens) exceeds the model context window (${admission.contextWindow} tokens). ` ++ + `Reduce the conversation size or choose a model with a larger context window.`, ++ { estimated_tokens: admission.estimatedTokens, context_window: admission.contextWindow }); ++ } +``` + +### NEW: tests/input-admission.test.ts + +Test cases: +1. Request within context window → admitted +2. Request exceeding context window → 413 with token estimates +3. No context window configured → admitted (fail-open) +4. Base64 image data counted at reduced rate +5. Tool results included in estimate +6. `modelMaxInputTokens` caps below context window +7. Edge: exactly at boundary → admitted +8. Edge: 1 token over → rejected + +## Activation scenario + +A Codex turn with 500k tokens of conversation history targeting a model with a +128k context window hits `checkInputAdmission` → returns `admitted: false` → +413 response returned to client before any upstream fetch. + +## Scope boundary + +IN: New admission module + core.ts insertion point + test file +OUT: Changing existing translator buffer limits, modifying provider configs, + adding UI for admission settings + diff --git a/devlog/_plan/260814_usage_memory_roadmap/020_m0_2_continuation_dedup.md b/devlog/_plan/260814_usage_memory_roadmap/020_m0_2_continuation_dedup.md new file mode 100644 index 0000000000..0d0212cf8e --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/020_m0_2_continuation_dedup.md @@ -0,0 +1,101 @@ +--- +title: "M0-2: Continuation overlap removal" +phase: "020" +depends: [] +consumes: [] +branch: codex/m0-2-continuation-dedup +closes: "(split from #1412)" +--- + +# 020 — M0-2: Stop compounding replayed history + +## Thesis + +When a request already contains the full conversation history (from the client +replaying its own context), the proxy must not prepend stored continuation state +on top. This duplication turns a 127k request into 254k→381k→508k across +continuations, eventually causing OOM or provider rejection. + +## Current state + +- `src/server/responses/core.ts:1544` sets `parsed._providerContinuation` + from `previousResponseProviderState` +- The continuation expansion code (around line 1621) checks + `hasUnexpandedPreviousResponse` but does not detect when the client has + already included the full history in the `input` array +- #1412 documents cases where 1x input becomes 2x→3x→4x through repeated + continuation prepending + +## File change map + +### NEW: src/server/responses/continuation-dedup.ts + +```ts +/** + * Detect whether a request's input array already contains messages that + * overlap with the stored continuation state. If substantial overlap is + * detected, skip the continuation expansion. + * + * Detection strategy: compare the first N message hashes from stored + * continuation against the request's input array. If ≥80% of stored + * messages appear in the input, the client already replayed history. + */ +export function detectHistoryOverlap( + requestInput: unknown[], + storedMessages: unknown[], +): { overlapping: boolean; overlapRatio: number }; + +/** + * Fingerprint a message for overlap detection. Uses a fast hash of + * role + first 200 chars of content. Images/tool-results use their + * type + id as fingerprint. + */ +export function messageFingerprint(message: unknown): string; +``` + +### MODIFY: src/server/responses/core.ts + +Location: Around the continuation expansion block (near line 1621). + +```diff + const hasUnexpandedPreviousResponse = !!parsed.previousResponseId + && typeof (body as { previous_response_id?: unknown }).previous_response_id === "string"; + ++ // Guard: if the client already included full history in input[], ++ // skip continuation expansion to prevent 1x → 2x → 3x compounding. ++ if (hasUnexpandedPreviousResponse && parsed._providerContinuation) { ++ const { overlapping } = detectHistoryOverlap( ++ parsed.input ?? [], ++ parsed._providerContinuation.messages ?? [], ++ ); ++ if (overlapping) { ++ parsed._previousResponseInputExpanded = true; // mark as already expanded ++ // Clear the continuation to prevent double-prepending ++ parsed._providerContinuation = undefined; ++ } ++ } +``` + +### NEW: tests/continuation-dedup.test.ts + +Test cases: +1. Full history already in input → continuation skipped, chain stays 1x +2. Delta continuation (new messages only) → continuation applied normally +3. Partial overlap (50%) → continuation applied (conservative) +4. Empty input + continuation → continuation applied normally +5. Preserved: call ID, reasoning, image, tool result integrity +6. Preserved: stateless provider behavior unchanged + +## Activation scenario + +A Codex session with 50 turns: each turn the client sends the full 50-turn +history as `input[]`. Without this fix, turn 3's request would contain +50 + 50 + 50 = 150 messages. With this fix, the overlap detector recognizes +the duplication and the request stays at 50 messages. + +## Scope boundary + +IN: Dedup detection module + core.ts guard + test file +OUT: Changing Codex client behavior, modifying continuation cache storage, + restructuring the continuation protocol + diff --git a/devlog/_plan/260814_usage_memory_roadmap/030_m0_3_provider_delivery.md b/devlog/_plan/260814_usage_memory_roadmap/030_m0_3_provider_delivery.md new file mode 100644 index 0000000000..e83c375903 --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/030_m0_3_provider_delivery.md @@ -0,0 +1,107 @@ +--- +title: "M0-3: Provider delivery policy" +phase: "030" +depends: [] +consumes: [] +branch: codex/m0-3-provider-delivery +closes: "(informed by #1367, #1668)" +--- + +# 030 — M0-3: Per-provider transport and delivery policy + +## Thesis + +Add two per-provider config knobs — `upstreamHttpVersion` and `responseDelivery` — +so operators can work around provider-specific transport failures without global +stream mode changes. + +## Current state + +- `src/types.ts:709` has global `streamMode: auto|legacy-tee|eager-relay` +- No per-provider HTTP version control +- #1668 documented HTTP/2 SSE hangs on specific providers (closed for template, not fixed) +- #1367 implements bounded-json fallback but is hygiene-blocked and too large +- `src/server/responses/core.ts` fetches upstream via `providerFetch` in fetch-helpers.ts + +## File change map + +### MODIFY: src/types.ts (OcxProviderConfig) + +Add two optional fields after `streamMode`-related fields: + +```diff ++ /** Force HTTP/1.1 for upstream connections to this provider. Default "auto". */ ++ upstreamHttpVersion?: "auto" | "http1"; ++ /** ++ * Response delivery mode for this provider. ++ * "auto" (default): streaming SSE as usual. ++ * "stream": force streaming even if other heuristics would disable it. ++ * "bounded-json": request stream=false upstream, read bounded JSON response, ++ * then reconstruct as Responses SSE for the client. No progressive output. ++ */ ++ responseDelivery?: "auto" | "stream" | "bounded-json"; +``` + +### MODIFY: src/config.ts (Zod schema) + +Add validation for the new fields in the provider config schema: + +```diff ++ upstreamHttpVersion: z.enum(["auto", "http1"]).optional().catch(undefined), ++ responseDelivery: z.enum(["auto", "stream", "bounded-json"]).optional().catch(undefined), +``` + +### MODIFY: src/server/responses/fetch-helpers.ts + +In `providerFetch`, apply HTTP/1.1 pin when configured: + +```diff ++ // When upstreamHttpVersion is "http1", inject a fetch option that forces ++ // HTTP/1.1 for this provider's upstream connections. ++ if (provider.upstreamHttpVersion === "http1") { ++ // Bun fetch supports { tls: { ... } } but not an explicit HTTP version pin. ++ // Workaround: set the ALPNProtocols to exclude h2. ++ // If Bun doesn't support this, fall back to appending a header hint. ++ } +``` + +### MODIFY: src/server/responses/core.ts + +In `handleResponsesInner`, before building the upstream request: + +```diff ++ // Apply bounded-json delivery: override stream=false upstream ++ if (providerConfig.responseDelivery === "bounded-json") { ++ // Set stream: false on the upstream request body ++ // Read the bounded JSON response ++ // Reconstruct as SSE events for the client ++ // MUST NOT resend the request on failure (no retry) ++ } +``` + +### NEW: tests/provider-delivery.test.ts + +Test cases: +1. Default (no config) → byte-for-byte existing behavior +2. `upstreamHttpVersion: "http1"` → only affects that provider +3. `responseDelivery: "bounded-json"` → upstream gets stream:false +4. bounded-json: no request resend on failure +5. bounded-json: cancellation handled correctly +6. bounded-json: malformed JSON response → error, not crash +7. bounded-json: unexpected SSE from stream:false → fail closed +8. Invalid config values → fallback to "auto" via Zod .catch() + +## Activation scenario + +A provider configured with `upstreamHttpVersion: "http1"` sends all its +upstream requests over HTTP/1.1 while other providers use auto-negotiated HTTP/2. +A model on that provider configured with `responseDelivery: "bounded-json"` +sends `stream: false` upstream, reads the JSON response, and re-emits it as +SSE to the Codex client. + +## Scope boundary + +IN: Type additions, config validation, fetch-helper modification, core.ts delivery logic, tests +OUT: Changing global streamMode behavior, modifying WebSocket upstream (#1608), + GUI for delivery settings, provider registry defaults + diff --git a/devlog/_plan/260814_usage_memory_roadmap/040_m0_4_memory_recovery.md b/devlog/_plan/260814_usage_memory_roadmap/040_m0_4_memory_recovery.md new file mode 100644 index 0000000000..ffb5fd0614 --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/040_m0_4_memory_recovery.md @@ -0,0 +1,125 @@ +--- +title: "M0-4: Memory recovery policy" +phase: "040" +depends: [] +consumes: [] +branch: codex/m0-4-memory-recovery +--- + +# 040 — M0-4: Connect watchdog to drain-and-restart + +## Thesis + +The memory watchdog (`src/server/memory-watchdog.ts`) observes and warns but never +acts. The drain-and-restart mechanism (`src/server/lifecycle.ts`) exists but is only +triggered by Dashboard UI. Connect them with an opt-in policy that auto-recycles +when memory pressure persists. + +## Current state + +- `src/server/memory-watchdog.ts`: 156 lines, warn-only, 4 GiB threshold, 60s sample, + 360-sample ring, uses `observedMemoryCounter` (max of rss/external/arrayBuffers) +- `src/server/lifecycle.ts:366`: `markRecyclingForExit()` + `drainAndShutdown()` + exist and work for Dashboard recycle +- No watchdog → restart connection +- No consecutive-sample requirement +- No cooldown or max-per-day limit +- Watchdog comment says "threshold auto-restart is deliberately deferred" + +## File change map + +### MODIFY: src/types.ts (OcxConfig) + +```diff ++ /** Opt-in memory recovery policy. When enabled, the watchdog triggers ++ * a drain-and-restart cycle after sustained memory pressure. */ ++ memoryRecovery?: { ++ enabled: boolean; ++ /** Bytes threshold for recovery consideration. Default: watchdog warn threshold (4 GiB). */ ++ thresholdBytes?: number; ++ /** Consecutive samples above threshold before triggering. Default: 3. */ ++ consecutiveSamples?: number; ++ /** Cooldown minutes between recovery cycles. Default: 30. */ ++ cooldownMinutes?: number; ++ /** Max recovery cycles per 24h. Default: 4. */ ++ maxPerDay?: number; ++ }; +``` + +### MODIFY: src/config.ts + +Add Zod validation for the new config section. + +### MODIFY: src/server/memory-watchdog.ts + +Add recovery evaluation to the sample callback: + +```diff ++ interface RecoveryState { ++ consecutiveAboveThreshold: number; ++ lastRecoveryAt: number | null; ++ recoveriesInWindow: { at: number }[]; ++ } ++ ++ function evaluateRecovery( ++ sample: MemorySample, ++ policy: Required>, ++ state: RecoveryState, ++ ): "trigger" | "cooldown" | "max-reached" | "below" | "accumulating"; +``` + +When `evaluateRecovery` returns `"trigger"`: + +```diff ++ // Verify supervisor viability before triggering restart ++ if (!canSupervisorRestart()) { ++ console.warn("[memory-recovery] threshold met but no supervisor detected; skipping restart"); ++ return; ++ } ++ console.warn(`[memory-recovery] sustained memory pressure (${consecutiveAbove} consecutive samples above threshold); initiating drain-and-restart`); ++ markRecyclingForExit(); ++ drainAndShutdown(serverRef, 60_000).catch(err => { ++ console.error("[memory-recovery] drain-and-restart failed:", err); ++ }); +``` + +### NEW: src/server/supervisor-detect.ts + +```ts +/** + * Best-effort detection of whether this process is managed by a supervisor + * that will restart it after exit. Checks: + * 1. INVOCATION_ID env (systemd) + * 2. Parent PID stability (launchd/supervisor patterns) + * 3. Service command markers in argv + * Returns false when unsure — recovery should not kill a standalone process. + */ +export function canSupervisorRestart(): boolean; +``` + +### NEW: tests/memory-recovery.test.ts + +Test cases: +1. Disabled by default (no config) → watchdog warns only +2. Enabled + 3 consecutive samples above threshold → triggers restart +3. 2 consecutive + 1 below → counter resets, no restart +4. Cooldown enforced: second trigger within cooldown → skipped +5. Max per day enforced: 5th trigger in 24h → skipped +6. No supervisor detected → skipped with warning +7. `heapUsed` alone not used — observedMemoryCounter logic preserved +8. Active turns drained before exit (lifecycle contract) +9. Restart loop prevention: 4 restarts in short window → backs off + +## Activation scenario + +A process accumulates RSS to 5 GiB. With `memoryRecovery.enabled: true` and default +`consecutiveSamples: 3`, three 60-second samples all above 4 GiB → evaluateRecovery +returns "trigger" → supervisor check passes → drain active turns → exit → supervisor +restarts the process with clean memory. + +## Scope boundary + +IN: Config type, watchdog recovery evaluation, supervisor detection, lifecycle connection, tests +OUT: Changing the watchdog sampling interval, adding GUI for recovery settings, + changing the 4 GiB default warn threshold, process-wide memory budget (P3 scope) + diff --git a/devlog/_plan/260814_usage_memory_roadmap/050_m0_5_nonjson_depth_cap.md b/devlog/_plan/260814_usage_memory_roadmap/050_m0_5_nonjson_depth_cap.md new file mode 100644 index 0000000000..19ca0e55ca --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/050_m0_5_nonjson_depth_cap.md @@ -0,0 +1,118 @@ +--- +title: "M0-5: Non-JSON depth cap" +phase: "050" +depends: [] +consumes: [] +branch: codex/m0-5-nonjson-depth-cap +closes: "#1635 (partial — YAML/TOML depth)" +--- + +# 050 — M0-5: YAML and TOML nesting depth ceiling + +## Thesis + +JSON configs already have `MAX_JSON_NESTING = 1000` enforced in both parse-time +scanning (`config-io.ts:81`) and serialize-time walk (`serialize.ts:291`). YAML +and TOML configs have no equivalent — a 15,000-level YAML object causes a stack +overflow or ~1.1 GiB RSS spike during `Bun.YAML.parse` or recursive serialization. + +## Current state + +- `src/integrations/serialize.ts:231`: `MAX_JSON_NESTING = 1000` — JSON only +- `src/integrations/config-io.ts:81`: JSON parse-time depth check against MAX_JSON_NESTING +- `src/integrations/serialize.ts:57-130`: YAML writer is recursive (`yamlMapEntryLines`, + `yamlLines`) with no depth limit +- YAML/TOML parsing uses `Bun.YAML.parse` / TOML npm package — no pre-parse depth scan +- The YAML writer would overflow the stack on deep structures before any other limit catches it +- JSON5 parsing goes through the same config-io path and already has the depth scan + +## File change map + +### MODIFY: src/integrations/serialize.ts + +Export a shared constant and add depth tracking to YAML writer: + +```diff +- export const MAX_JSON_NESTING = 1000; ++ /** Shared ceiling for container nesting across all config formats. */ ++ export const MAX_CONFIG_NESTING = 1000; ++ /** @deprecated Use MAX_CONFIG_NESTING */ ++ export const MAX_JSON_NESTING = MAX_CONFIG_NESTING; +``` + +Add depth parameter to YAML recursive functions: + +```diff +- function yamlMapEntryLines(key: string, value: unknown, indent: number): string[] { ++ function yamlMapEntryLines(key: string, value: unknown, indent: number, depth = 0): string[] { ++ if (depth >= MAX_CONFIG_NESTING) { ++ throw new UnserializableValueError( ++ \`the document nests deeper than ${MAX_CONFIG_NESTING} levels, which YAML serialization cannot handle safely\`); ++ } +``` + +Same for `yamlLines`, `yamlArrayMapLines`, and `tomlSection` functions. + +### MODIFY: src/integrations/config-io.ts + +Update the import and generalize the depth check: + +```diff +- import { MAX_JSON_NESTING } from "./serialize"; ++ import { MAX_CONFIG_NESTING } from "./serialize"; +``` + +Add a format-agnostic pre-parse depth scan for YAML/TOML text: + +```diff ++ /** ++ * Quick indentation-based depth estimate for YAML text. Counts the deepest ++ * indentation run (each 2 spaces = 1 level). Not exact but catches hostile ++ * documents before Bun.YAML.parse can stack-overflow. ++ */ ++ function estimateYamlDepth(text: string): number; ++ ++ /** ++ * Bracket-counting depth estimate for TOML text. Counts nested table headers ++ * and inline tables. ++ */ ++ function estimateTomlDepth(text: string): number; +``` + +Apply before parsing: + +```diff ++ if (format === "yaml") { ++ const est = estimateYamlDepth(text); ++ if (est > MAX_CONFIG_NESTING) return false; ++ } ++ if (format === "toml") { ++ const est = estimateTomlDepth(text); ++ if (est > MAX_CONFIG_NESTING) return false; ++ } +``` + +### NEW: tests/nonjson-depth.test.ts + +Test cases: +1. 15,000-level YAML nested object → bounded error, not stack overflow +2. 15,000-level TOML nested table → bounded error +3. Normal YAML config (5 levels) → parses successfully +4. Normal TOML config (3 levels) → parses successfully +5. Exactly at MAX_CONFIG_NESTING → accepted +6. One level over → rejected +7. Existing JSON depth cap unchanged (regression) +8. Existing config round-trip preserved + +## Activation scenario + +A user places a malicious `config.yaml` with 15,000 nested levels. The pre-parse +depth estimator catches it before `Bun.YAML.parse` and returns a clean error: +"the document nests deeper than 1000 levels." No stack overflow, no RSS spike. + +## Scope boundary + +IN: Shared depth constant, YAML/TOML depth estimation, serialize depth tracking, tests +OUT: Changing MAX_JSON_NESTING value, JSON5 changes (already covered), + big-integer rounding fix (separate #1635 scope item) + diff --git a/devlog/_plan/260814_usage_memory_roadmap/060_u1_segmented_writer.md b/devlog/_plan/260814_usage_memory_roadmap/060_u1_segmented_writer.md new file mode 100644 index 0000000000..96b44cc735 --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/060_u1_segmented_writer.md @@ -0,0 +1,161 @@ +--- +title: "U1: Segmented usage writer" +phase: "060" +depends: [] +consumes: [] +branch: codex/u1-segmented-writer +closes: "(new tracker, #1008 prior art)" +--- + +# 060 — U1: Segmented usage writer + +## Thesis + +Replace the single unbounded `usage.jsonl` with date/size-rotated segments. +Each segment is a self-contained JSONL file. The active segment receives appends; +sealed segments are immutable. Migration from the existing single file is automatic. + +## Current state + +- `src/usage/log.ts:432`: `appendUsageEntry` → `appendFileSync` to single file +- `src/usage/log.ts:147`: `usageLogPath` returns `${configDir}/usage.jsonl` +- `src/usage/log.ts:425`: `ensureUsageLogDir` creates the parent directory +- No rotation, no segment concept, no manifest +- `src/usage/log.ts:439`: `UsageLogRevision` tracks inode+size+mtime for cache invalidation +- Reader (`readUsageSnapshotForManagement`) reads a 64 MiB tail from the single file + +## File change map + +### NEW: src/usage/segments.ts + +Core segment management module: + +```ts +/** Segment naming: usage-YYYY-MM-DD-NNNN.jsonl[.active] */ +export interface UsageSegment { + id: string; // e.g. "usage-2026-08-14-0001" + path: string; // full path + active: boolean; // true = receives appends + firstTimestamp: number | null; + lastTimestamp: number | null; + bytes: number; + rows: number; + sealedAt: number | null; +} + +export interface UsageSegmentManifest { + version: 1; + segments: UsageSegment[]; + activeSegmentId: string | null; + migratedFromLegacy: boolean; + migratedAt: number | null; +} + +const SEGMENT_MAX_BYTES = 64 * 1024 * 1024; // 64 MiB +const USAGE_DIR = "usage"; // relative to configDir + +/** + * Open or create the active segment for writing. + * Triggers rotation when: + * 1. Active segment exceeds SEGMENT_MAX_BYTES + * 2. Local date has changed since last write + */ +export function openActiveSegment(configDir: string): { + fd: number; + segment: UsageSegment; + rotated: boolean; +}; + +/** + * Seal the current active segment (rename .active → plain .jsonl) + * and create a new active segment. + */ +export function rotateSegment(configDir: string): UsageSegment; + +/** + * On startup: scan for orphan .active segments from crash recovery. + * An orphan .active file (not the current active) is sealed in place. + */ +export function recoverOrphanSegments(configDir: string): UsageSegment[]; + +/** + * Read and validate the manifest. Create if missing. + */ +export function readManifest(configDir: string): UsageSegmentManifest; + +/** + * Atomic manifest write (write to .tmp, rename). + */ +export function writeManifest(configDir: string, manifest: UsageSegmentManifest): void; + +/** + * Migrate from legacy single usage.jsonl: + * 1. Rename usage.jsonl → usage/usage-legacy-0001.jsonl + * 2. Create manifest marking it as sealed + * 3. Create new active segment + * Legacy file is preserved, not deleted. + */ +export function migrateLegacyUsageLog(configDir: string): UsageSegmentManifest; +``` + +### MODIFY: src/usage/log.ts + +Replace the write path: + +```diff + export function appendUsageEntry(entry: PersistedUsageEntry): void { +- ensureUsageLogDir(); +- const path = usageLogPath(); +- const line = JSON.stringify(normalizeUsageEntry(entry)) + "\n"; +- appendFileSync(path, line); ++ const { fd, segment, rotated } = openActiveSegment(); ++ const line = JSON.stringify(normalizeUsageEntry(entry)) + "\n"; ++ writeSync(fd, line); ++ segment.bytes += Buffer.byteLength(line); ++ segment.rows += 1; ++ segment.lastTimestamp = entry.startedAt ?? Date.now(); ++ if (rotated) { ++ // Invalidate reader caches on rotation ++ discardRetainedUsageSnapshot(); ++ } +``` + +Keep `usageLogPath()` working for backward compatibility (returns legacy path +or active segment path). + +### MODIFY: src/usage/log.ts (reader) + +Update `readUsageSnapshotForManagement` to read across segments: + +```diff ++ // Read from all retained segments within the byte budget, ++ // newest first. The 64 MiB cap now applies to the total ++ // across segments, not a single file tail. +``` + +### NEW: tests/usage-segmented-writer.test.ts + +Test cases: +1. Fresh install → creates usage/ dir + active segment + manifest +2. Append within segment → row count and bytes increase +3. Rotation at 64 MiB → new segment created, old sealed +4. Rotation at date change → new segment with new date prefix +5. Concurrent append → no row loss (serial appendFileSync) +6. Rotation boundary → no duplicate or missing entries +7. Crash recovery: orphan .active file → sealed on startup +8. Legacy migration: usage.jsonl → usage/ directory structure +9. Legacy file still readable after migration +10. Manifest corruption → rebuild from segment files + +## Activation scenario + +A user with a 245 MB `usage.jsonl` upgrades to this version. On first start, +`migrateLegacyUsageLog` moves it to `usage/usage-legacy-0001.jsonl` (sealed), +creates a manifest, and opens a new active segment. Subsequent appends go to +the active segment. After 64 MiB, it rotates to a new segment. + +## Scope boundary + +IN: Segment module, log.ts write/read path changes, migration, manifest, tests +OUT: SQLite projection (070), retention/deletion (P2), GUI changes, usage-summary-cache changes + diff --git a/devlog/_plan/260814_usage_memory_roadmap/070_u2_sqlite_projector.md b/devlog/_plan/260814_usage_memory_roadmap/070_u2_sqlite_projector.md new file mode 100644 index 0000000000..cd0c32da0c --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/070_u2_sqlite_projector.md @@ -0,0 +1,125 @@ +--- +title: "U2: Incremental SQLite projector" +phase: "070" +depends: ["060"] +consumes: ["src/usage/segments.ts"] +branch: codex/u2-sqlite-projector +--- + +# 070 — U2: Incremental SQLite projector + +## Thesis + +Build a SQLite index that aggregates usage data from JSONL segments. The index +is a projection — it can always be rebuilt from the segments. It replaces the +in-memory 64 MiB tail parse as the primary data source for summaries. + +## Current state + +- No SQLite in the usage subsystem +- `src/usage/summary.ts:700`: `summarizeUsage` builds all projections from + a `PersistedUsageEntry[]` array in memory +- `src/server/management/usage-summary-cache.ts`: TTL cache over computed summaries +- Bun ships with built-in SQLite (`bun:sqlite`) + +## File change map + +### NEW: src/usage/usage-index.ts + +```ts +import { Database } from "bun:sqlite"; + +export const SCHEMA_VERSION = 1; +export const PROJECTOR_VERSION = 1; + +/** Open or create the usage index database. */ +export function openUsageIndex(configDir: string): Database; + +/** Run full schema migration. */ +export function migrateSchema(db: Database): void; + +/** Tables: + * - usage_projection_meta: schema_version, projector_version, + * last_indexed_segment, last_indexed_offset, segment_digest, + * last_indexed_at, rebuild_state + * - usage_request_recent: requestId, startedAt, completedAt, + * surface, provider, model, status, token fields, duration, + * bounded diagnostic fields + * - usage_daily: localDate, timezone, surface, provider, model, + * statusClass, requestCount, measuredCount, token accumulators, + * costAtCapture, duration/ttft accumulators + * - usage_storage_manifest: segmentId, path, firstTs, lastTs, + * bytes, rows, sealed/projected/archived/deleted state + */ + +export interface ProjectionCheckpoint { + lastSegmentId: string; + lastOffset: number; + segmentDigest: string; +} + +/** + * Incrementally project new entries from segments into the index. + * Reads from the checkpoint forward, deduplicates by requestId, + * updates daily aggregates, and advances the checkpoint atomically. + */ +export function projectIncrementally( + db: Database, + segments: UsageSegment[], + configDir: string, +): { rowsProcessed: number; segmentsProcessed: number }; + +/** + * Full rebuild: drop and recreate all projection tables, + * then project every retained segment from scratch. + */ +export function rebuildProjection( + db: Database, + segments: UsageSegment[], + configDir: string, +): { rowsProcessed: number }; + +/** + * Read daily aggregates for a date range. + */ +export function readDailyAggregates( + db: Database, + since: number | null, + until: number, +): UsageDailyRow[]; + +/** + * Read recent request details (bounded by row count). + */ +export function readRecentRequests( + db: Database, + limit: number, +): UsageRequestRow[]; +``` + +### NEW: tests/usage-sqlite-projector.test.ts + +Test cases: +1. Fresh projection from 3 segments → matches legacy full parse for tokens/requests/cost +2. Incremental: add entries to active segment → project incrementally → totals match +3. RequestId dedup: same requestId in two segments → counted once +4. Daily aggregates: correct date grouping in Asia/Seoul timezone +5. Model/provider breakdown matches legacy `summarizeUsage` output +6. Crash recovery: projection interrupted mid-segment → resumes from checkpoint +7. Full rebuild produces identical results to incremental +8. Schema migration from v0 (fresh) to v1 +9. Malformed JSONL line → skipped with count, not crash +10. Empty segments handled gracefully + +## Activation scenario + +After U1 creates segments, the projector runs on startup (or on first +`/api/usage` request) and indexes all unsealed segments. Subsequent appends +trigger incremental projection. The SQLite file is a derived artifact — deleting +it and restarting rebuilds everything from segments. + +## Scope boundary + +IN: SQLite schema, projector logic, checkpoint, rebuild, tests +OUT: Changing `/api/usage` endpoint (080), retention/archive (P2), GUI + diff --git a/devlog/_plan/260814_usage_memory_roadmap/080_u3_projection_api.md b/devlog/_plan/260814_usage_memory_roadmap/080_u3_projection_api.md new file mode 100644 index 0000000000..b76c38072d --- /dev/null +++ b/devlog/_plan/260814_usage_memory_roadmap/080_u3_projection_api.md @@ -0,0 +1,97 @@ +--- +title: "U3: Projection-backed Usage API" +phase: "080" +depends: ["070"] +consumes: ["src/usage/usage-index.ts"] +branch: codex/u3-projection-api +--- + +# 080 — U3: Switch /api/usage to read from SQLite + +## Thesis + +The `/api/usage` endpoint currently parses a 64 MiB JSONL tail on every cold read. +After U2, it should read from the SQLite projection — O(1) for aggregates regardless +of file size, and warm queries are instant. + +## Current state + +- `src/server/management/logs-usage-routes.ts`: handles /api/usage +- Calls `readUsageSnapshotForManagement` → parses 64 MiB tail +- Passes entries to `summarizeUsage` for in-memory aggregation +- `usage-summary-cache.ts` caches the computed summary with TTL + +## File change map + +### MODIFY: src/server/management/logs-usage-routes.ts + +Replace the read path: + +```diff +- const { snapshot } = await readUsageSnapshotForManagement(maxReadBytes); +- const entries = snapshot.entries; +- const summary = summarizeUsage(entries, range, surface, now); ++ // Read from SQLite projection (U2) ++ const db = openUsageIndex(configDir); ++ // Ensure projection is up to date ++ projectIncrementally(db, readManifest(configDir).segments, configDir); ++ ++ // Build summary from projection tables ++ const dailyRows = readDailyAggregates(db, rangeWindow.since, now); ++ const recentRows = readRecentRequests(db, 500); ++ const summary = buildSummaryFromProjection(dailyRows, recentRows, range, surface, now); +``` + +### NEW: src/usage/projection-summary.ts + +Bridge between SQLite projection rows and the existing `UsageSummary` shape: + +```ts +/** + * Build a UsageSummary from SQLite projection rows. + * Produces the same shape as the legacy summarizeUsage() so the + * GUI and API consumers see no change. + */ +export function buildSummaryFromProjection( + dailyRows: UsageDailyRow[], + recentRows: UsageRequestRow[], + range: UsageRange, + surface: UsageSurface, + now: number, +): UsageSummary; +``` + +### MODIFY: src/server/management/usage-summary-cache.ts + +Update cache key to include projection generation: + +```diff ++ // Cache key now includes projection checkpoint, not file revision ++ const projectionKey = \`${lastSegmentId}:${lastOffset}\`; +``` + +### NEW: tests/usage-projection-api.test.ts + +Test cases: +1. /api/usage response from SQLite matches legacy parse for 7d range +2. /api/usage response from SQLite matches legacy parse for 30d range +3. Warm query time < 50ms regardless of segment count +4. Coverage metadata: shows projection status + retained range +5. All-time aggregate never decreases as segments rotate +6. New append → next /api/usage reflects it (incremental projection) +7. Projection rebuilding state shown in response metadata +8. Frontend contract: same JSON shape as before (backward compatible) + +## Activation scenario + +A user with 500k entries across 8 segments hits /api/usage. Instead of parsing +64 MiB of JSONL, the endpoint reads pre-computed aggregates from SQLite. The +response includes the same totals, models, providers, days, and accounts as +before, but cold read drops from ~25 seconds to <100ms. + +## Scope boundary + +IN: API route change, projection summary bridge, cache update, tests +OUT: Adding new API fields (P2), Today/Yesterday filtering (P2), + GUI changes (P2), raw export (P2) +