fix(daemon): Heal sessions failed by a locked store on boot

[4ndreello]

Problem

Before #126, the event loop marked a session failed on database is locked and stopped reading its log, while the harness kept running. Everything past the persisted logOffset stayed on disk and never reached the store. One such session was a codex resume turn: its tool calls, final message and second-turn usage were all lost.

#126 stops new cases. Rows already failed that way stay failed forever, because recover() only looks at active sessions.

Change

Heal on boot (6c62d5a)

reviveStoreBusyFailures() runs at the start of recover().

• Which rows: failed rows with failure.code = STORE_BUSY, or the old UNKNOWN rows whose detail says database is locked. SessionStore.listStoreBusyFailures() finds them.
• Conditions: the row needs a log file and a pid, and is never origin = open.
• Mechanism: the row goes back to working, then through the same reattach as a daemon restart.
  • Live harness (same pid identity): it keeps being tailed from the offset.
  • Dead harness, or a recycled pid: the log is drained from the offset and the outcome is classified. A healing row with a reused pid counts as dead instead of failing with pid_reused, since its log is still its own.
• Timestamps: parsers stamp events with read time, so the drained backlog and completedAt are pinned to the log's mtime (finishHeal). Usage-by-hour buckets by created_at, so only the timeline and completedAt change.
• Idempotence: the new outcome no longer carries a locked-store detail, so the next boot does not revive the row again.

Complete a reattached turn that ended on turn.completed (750123d)

A reattach learns of a death by polling the pid and never sees an exit code. Codex writes no terminal frame and completes only through exit 0. So every codex turn that ended while the daemon was away closed as failed with "exited without reporting a terminal event", healed rows included.

synthesizeTerminalEvent now returns session.completed with reason: "turn completed; exit not observed" when all of these hold:

• the exit code is unknown and there is no signal;
• the last non-usage event is turn.completed (SessionRuntime passes endedOnTurnCompleted);
• stderr carries no crash signature.

It stays failed when the log stops before turn.completed, on a fatal signal, on an observed non-zero exit, or on a crash signature in stderr.

Limitation: the runtime only sees events read since the reattach. A turn whose turn.completed was persisted before the daemon went away, and whose exit was then missed, still closes failed.

Spec: .specs/features/store-busy-healing/spec.md.

Validation

Tests were written first and seen red:

tests/daemon-heal-store-busy.test.ts (6 tests) drives recover() with the real codex driver against a real ndjson log:

• dead harness: backlog drained, completed, events and completedAt at the log mtime;
• live harness: back to working, new lines persisted;
• STORE_BUSY rows are healed;
• non-lock failures and rows without a log are left alone;
• a log cut before turn.completed closes failed.

tests/terminal-synth.test.ts (5 tests) covers the completion rule and each case that keeps it failed.

Full suite in a Node 22 container: 1648 pass, 1 fail. The failure (open-claude-probes "probes again when the binary size changes") also fails on main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R6o1wJTTJWcHm9mZMGvDTn

---

Generated by Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud