Spend Dashboard Codex publish delayed for hours — stale scanSinceKey reseeds lookback queue on every scan

[kesslerio]

TL;DR

The Usage & Spend dashboard took ~5 hours to publish Codex spend for one of my two Codex accounts because the lookback queue reseeded on every scan pass. Root cause: a stale scanSinceKey in the persisted lookback state triggered activeLookbackStateWasReset on every scan, which reseeded the entire queue (3,367 files) even though all files were already parsed and priced. The queue eventually drained fully between two scan boundaries and the dashboard published, but the recovery should take one scan pass, not five hours.

Bug Type

behavior bug

Release Blocker

• No

Steps to Reproduce

1. Have a large Codex session corpus (3,879 files, ~3,500 in ambient ~/.codex/sessions, rest in CodexBar-managed homes). Two Codex accounts configured.
2. Let CodexBar build its cost cache. scan_metadata reaches completedFiles: 3879 / totalFiles: 3879.
3. Observe lookback_state.pendingFilePaths drain to zero during background passes (queue shrinks ~250-500 files per ~15 min pass).
4. Trigger a new scan (open Settings → Usage & Spend, or wait for the next background cycle). The scan re-enumerates and re-enqueues the entire lookback set. pendingFilePaths jumps back to ~3,367.
5. catchUpPending stays true for hours. The dashboard shows "Spend unavailable" for the affected account row until a drain pass happens to finish the whole queue between scan boundaries.

Expected Behavior

Once all files are complete and the lookback queue is empty, catchUpPending should flip to false and the dashboard should publish Codex spend on the next refresh. The queue should not re-enqueue files that were already processed and marked complete. A scan-window change should not reseed already-completed files.

Actual Behavior

The queue reseeded on every scan pass for ~5 hours before one drain pass completed between scan boundaries.

Timeline from a single session (all times UTC):

Time	pending	completed	trigger
04:42:48	0	3879/3879	background pass, queue drained
04:46:03	3,367	3878/3879	pane opened, scan re-enqueued entire lookback set
04:46:36	3,262	3878/3879	draining again
04:48:14	307	3878/3879	continued draining

Root cause: scanSinceKey mismatch triggers lookback state reset on every scan

CostUsageScanner.swift:5616 checks activeLookbackStateWasReset by comparing the cached lookback state's scanSinceKey and rootPaths against the freshly computed values:

let activeLookbackStateWasReset = cache.codexActiveLookbackState.map {
    $0.scanSinceKey != activeLookbackState.scanSinceKey
        || $0.rootPaths != activeLookbackState.rootPaths
} ?? true

When activeLookbackStateWasReset is true, shouldSeedBoundedQueue becomes true (line 5780), which calls seedOrExtendCodexActiveLookbackQueue and reseeds the entire pending queue from all known files.

The dashboard uses a 365-day scan window (SpendDashboardController.swift:168-170: activityDays = 365, scanDays = activityDays). So the fresh scanSinceKey is 2025-09-03 (365 days back from today). But the persisted lookback_state.scanSinceDay was stuck at 2026-06-05:

scan_metadata.scanSinceDay:     2025-09-03   (365-day window)
lookback_state.scanSinceDay:   2026-06-05   (stale, ~90-day window)

Every scan saw the scanSinceKey mismatch, activeLookbackStateWasReset = true, shouldSeedBoundedQueue = true, entire queue reseeded. The stale scanSinceDay couldn't update because the scanner only persists the new lookback state at the end of a completed scan, and the scan couldn't complete because the reseeded queue was too large to drain in one bounded pass.

The lookback_state.scanSinceDay of 2026-06-05 corresponds to a ~90-day window, likely the original scan window before the dashboard switched to 365 days. When the window widened, the cached lookback state's scanSinceKey became stale. The codexActiveLookbackState function (line 2775) returns the cached state when scanSinceKey matches, but when it doesn't match, it creates a fresh state with retainedPendingFilePaths (line 2787) and the new scanSinceKey. The fresh state is used in-memory but never persisted until a scan completes.

Resolution observed

After ~5 hours, one drain pass finished the entire queue before the next scan boundary. The scan completed, the fresh lookback state persisted (the lookback_state row is now gone entirely), and the publish gate opened:

lookback_state row: gone (0 rows)
catchUpPending: false
completedFiles: 3839 / 3839

The dashboard now renders the previously blocked account row. The bug is not a permanent deadlock on every corpus, but a race: the reseed loop continues until a drain pass happens to complete the full queue between scan boundaries. On a large corpus (3,879 files) with ~15-minute background cadence, that took roughly 5 hours.

Suggested fix direction

A scanSinceKey-only mismatch should not reseed already-completed files. When the window widens, the scanner should enqueue only files whose content days fall in the newly added range (between the old scanSinceKey and the new one), not re-enqueue all 3,879 files. Alternatively, the lookback state's scanSinceKey could be updated in-place when a window change is detected, without requiring a full scan completion. Either approach collapses the 5-hour recovery into a single scan pass.

Cost data was computed the whole time

The dashboard's JSON export (Usage & Spend > export) shows the affected account's data was fully computed while the UI showed "Spend unavailable":

{
  "displayName": "Codex · #2",
  "id": "codex:subs@kessler.io",
  "sourceKind": "native",
  "totalCost": 4256.232196680002,
  "totalTokens": 14660778502
}

The cost engine correctly attributed and priced all sessions. The "Spend unavailable" state was purely the publish gate. The model breakdown was also computed (gpt-5.6-sol, gpt-5.6-luna, gpt-5.6-terra, gpt-5.5, gpt-5.4).

Version

0.56.4 (build 135)

Environment

macOS 26.6.2 (25G83), Apple Silicon. Two Codex accounts (one team account managed home, one personal account managed home). 3,879 indexed session files.

Configuration Details

Two Codex accounts configured via CodexBar managed homes. Sessions routed through a wrapper that sets CODEX_HOME per account (not auth_profile). Ambient ~/.codex/sessions has ~3,500 files; managed homes have a small number of recently routed files. Both managed homes were re-authed with fresh codex login (previous tokens had expired).

Logs

Redacted log excerpts (current rotation, ~/Library/Logs/CodexBar/CodexBar.log):

[2026-09-03T22:18:38Z] [WARNING] Spend Dashboard Codex cost catch-up stopped because all pending account caches stalled
[2026-09-03T23:08:32Z] [WARNING] Spend Dashboard Codex cost catch-up stopped because all pending account caches stalled
[2026-09-04T02:45:50Z] [WARNING] Spend Dashboard Codex cost catch-up stopped because all pending account caches stalled
[2026-09-04T03:06:01Z] [WARNING] Spend Dashboard Codex cost catch-up stopped because all pending account caches stalled
[2026-09-04T04:42:57Z] [INFO]    cost usage success provider=codex duration=8.79s today=$0.00 historyDays=90 windowCost=$10,615.66

Screenshots

Dashboard overview (30d) after the publish gate opened, showing Codex · #2 at $4,256.23 · 15B tokens in "By subscription" and the model breakdown:

Session breakdown, showing per-session rows for Codex · #2 all rendering with individual costs:

Note: the banner still reads "Local estimated history · Refreshing" with "stale data · Updated 14 minutes ago", 71 KB / 71 KB. This may clear on the next successful refresh; I'll report back if it persists.

Impact and Severity

• Affected: users with large Codex session corpora and multiple accounts
• Severity: blocks workflow. The Usage & Spend dashboard is the primary spend surface, and one account row shows "Spend unavailable" for hours
• Frequency: every time the scan window changes on a large corpus (window widen, or possibly the daily rolling-window shift)
• Consequence: cannot see per-account Codex spend in the dashboard for hours; must rely on the menu bar aggregate which doesn't break down by account

Additional Information

Related: #3071 (closed, fix in #3075), #3316 (open, restart loop variant).

This is distinct from #3316 in two ways:

1. Re-enumeration across scan boundaries: my queue drains to zero but a new scan repopulates it. Spend Dashboard no-progress pause auto-restarts, loops at 100% CPU #3316 reports 25 already-complete paths that never drain. My corpus shows the queue can reach nil (the Drain processed zero-row files from Codex cost catch-up #3075 drain fix partially works), but the next scan re-enqueues everything.
2. Corpus size: 3,879 files vs 537. The re-enumeration scales with corpus size, which may explain why the drain fix works on smaller corpora but stalls here.

What I tried that didn't help (before understanding the root cause):

• Deleting the lookback_state row: triggered a full cold rebuild that re-enqueued the entire lookback set (3,367 files). Made it worse.
• Waiting for the drain: the queue drains between scans but repopulates on the next pass.
• Keeping the Usage & Spend pane open (accelerated 10s cadence per fix(spend): clear final-file stall for local estimated history #3114): triggers more frequent scans, which trigger more re-enumerations.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed September 4, 2026, 1:49 AM ET / 05:49 UTC.

Summary
Keep open for focused diagnosis. The screenshots show a real stale Refreshing panel while the dashboard displays current Codex data, but current source and the reported v0.56.4 source both persist a refreshed lookback state on each changed save, so the claimed repeated stale-key reseed is not yet established.

Reproducibility: no. The screenshots establish the visible stale panel, but current source contradicts the proposed repeated stale-key mechanism unless a save fails or does not run; neither condition was reproduced here.

Ways to help us reproduce this

• Add expected vs actual behavior.
• Share version, platform, channel/provider, and relevant config details.

Next step
The report is actionable, but its stated root cause conflicts with current and released persistence code; collect a two-pass stored-state trace or save failure before queueing a repair PR.

Review details

Best possible solution:

Preserve the existing per-pass persistence behavior, add diagnostics or a regression spanning two bounded refreshes, and ensure the catch-up panel is removed once fresh statuses are complete.

Do we have a high-confidence way to reproduce the issue?

No. The screenshots establish the visible stale panel, but current source contradicts the proposed repeated stale-key mechanism unless a save fails or does not run; neither condition was reproduced here.

Is this the best way to solve the issue?

No for the proposed queue-only repair. The current implementation already carries the fresh key into the finalized state and persists it, so the narrow next step is to establish whether persistence fails or dashboard activity becomes stale before altering queue selection.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• A save-baseline or transaction failure could leave the old persisted state in place, but the report does not include evidence that the scanner save returned its catch-up-required failure path.
• The stale Refreshing panel may be a separate in-memory catch-up activity issue rather than a queue-reseed defect; its transition was not reproduced in the read-only environment.

Codex review notes: model internal, reasoning high; reviewed against 392310c66548.

Label changes

Label justifications:

• P2: The primary per-account spend view can remain misleadingly unavailable or refreshing for hours on large local corpora, without evidence of data loss or process unavailability.
• impact:ux-friction: Users can eventually see computed spend data, but the dashboard’s stale availability state creates substantial confusion and reliance on less detailed views.

Evidence reviewed

What I checked:

• Visible stale dashboard state: The supplied screenshots show the Refreshing catch-up panel at 71 KB / 71 KB while Codex · Homebrew #2 and its session/model spend data are rendered, supporting the follow-up report that the visible panel can be stale after data publication.
• Window mismatch reseeds one pass: A changed scan-window key creates a new lookback state and permits bounded-queue seeding in that pass. This source alone does not show that the same stale key persists into the next pass. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:5611, 392310c66548)
• Changed lookback state is persisted: The scanner assigns the finalized lookback state before saving; the storage layer compares the full cache on its identical-content path and otherwise writes the lookback state in the save transaction. A changed scanSinceKey therefore requires a changed save unless the save fails or the scan never reaches it. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageStore+CodexCache.swift:95, 392310c66548)
• Reported release has the same persistence path: The v0.56.4 release contains the same reset comparison and full-cache persistence logic, so the stated explanation is not a known pre-release-only implementation gap. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:5611, fb9d295304af)
• Related queue fixes and recent ownership: History shows the same scanner received queue scheduling, processed-file draining, and bounded-target updates; exact line-introduction provenance could not be completed because the partial clone lacks parent blobs. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:5611, 01da8fc6ea2d)

Likely related people:

• LeoLin990405: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Yuxin Qiao: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[kesslerio]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.