[Bug]: a task that never reaches a terminal status stays at running forever -- _is_expired has no branch for RUNNING #6130

Issue Origin

Observed or reproduced in a real environment

Bug Description

TaskTracker._finalize_task (openviking/service/task_tracker.py:737) writes the terminal
state only when the task owns no work any more:

async def _finalize_task(self, task_id, account_id=None, user_id=None):
    """Apply the task's terminal outcome after all owned work is settled."""
    if self._work_index.has_work(task_id):
        return

If that call is missed — the process dies between persisting result and finalizing, or the
last work item is released through a path that does not re-finalize — the record stays at
running.

Nothing ever closes it. _is_expired (task_tracker.py:308) gives a TTL only to terminal
states:

TTL_COMPLETED = 86_400   # 24 hours
TTL_FAILED = 604_800     # 7 days

def _is_expired(self, task: TaskRecord, now: float) -> bool:
    age = now - task.updated_at
    if task.status in (TaskStatus.COMPLETED, TaskStatus.CANCELLED):
        return age > self.TTL_COMPLETED
    return task.status == TaskStatus.FAILED and age > self.TTL_FAILED

RUNNING matches neither branch, so the record is never expired, never pruned by the
cleanup loop (_evict_expired, every CLEANUP_INTERVAL = 300 s), and stays on disk
indefinitely. There is also no reconciliation at startup: prepare_task_tracking
(storage/queuefs/queue_manager.py:170-175) rebuilds owners from the queues and calls
restore_work_tasks (task_tracker.py:228), which only loads the referenced records — it
does not close a running record whose work is gone.

The result is a record that reports work in progress forever, while the work finished long
ago.

Steps to Reproduce

1. Create a task and move it to running (_record_outcome persists result without a
   terminal_status).
2. Kill the process before _finalize_task runs, so the work index entry disappears with the
   process and no further finalize is triggered.
3. Restart. The queue snapshot no longer references the task, so has_work is false, but
   nothing calls _finalize_task for it.
4. Inspect the record: status: running, stage: null, a populated result, error: null.

Observed over 19-21 hours on ten session_commit tasks.

Expected Behavior

A task record eventually reaches a terminal state, or is at least reported as abandoned.
A record that claims to be running while owning no work should not survive indefinitely.

Actual Behavior

Ten session_commit tasks stayed at running for 19-21 hours.

They were not lost work. Every one had a fully populated result and no error:

Each also had only two execution events (created, status_changed: running) where a
completed task has three — the third being the transition that never happened.

Minimal Reproducible Example

import time

# the shape found on disk: work finished, terminal status never written
record = {
    "task_id": "5b3cbe7a-...", "task_type": "session_commit", "status": "running",
    "stage": None, "error": None,
    "result": {"memories_extracted": {"memory_write": 7}},
    "created_at": time.time() - 21 * 3600,
    "updated_at": time.time() - 20 * 3600,
    "execution_events": {"items": [
        {"seq": 1, "kind": "created", "status": "pending"},
        {"seq": 2, "kind": "status_changed", "status": "running"},
    ], "dropped_count": 0, "started_mid_task": False},
}

TTL_COMPLETED, TTL_FAILED = 86_400, 604_800

def is_expired(status, updated_at, now):
    age = now - updated_at
    if status in ('completed', 'cancelled'):
        return age > TTL_COMPLETED
    return status == 'failed' and age > TTL_FAILED

print(is_expired(record['status'], record['updated_at'], time.time()))
# False -- 21 hours old, and the cleanup loop will never touch it

Error Logs

# none -- that is part of the problem.
#
# No log line, no error and no metric reports the stuck record.
# It is visible only by reading the task store or /api/v1/tasks.

OpenViking Version

0.5.0

Python Version

3.13.15

Operating System

Windows

Model Backend

Other

Additional Context

Model Backend
Other — local llama-server (llama.cpp) behind an OpenAI-compatible endpoint

Impact. A record reporting work in progress is misleading in three ways:

1. It never disappears, so /api/v1/tasks and any dashboard built on it overstate running
   work; task counts stop being trustworthy.

2. has_running(task_type, resource_id) (task_tracker.py:1059) counts _ACTIVE_STATUSES,
   which includes RUNNING, and session_service.py:564 consults it before scheduling an
   auto-commit — return False skips the commit.

   In our incident it did not actually block anything, and I want to be precise about why,
   because the code reads as if it should. Measured on the preserved records:

   orphan 192c3e7a (session dsh-session-1159e9c6..., 18:41) -> 8 later tasks in that session
       19:00:32 running / 19:13:44 completed / 19:23:55 failed / ... / 20:57:19 completed
   orphan 5b3cbe7a (session dsh-session-eb197836..., 17:10) -> 2 later tasks in that session
   

   Later commits in the same session did run, so has_running returned false despite a
   running record for that resource_id. I did not isolate which guard let them through —
   _matches_owner (task_tracker.py:352) and the per-worker _auto_commit_inflight set are
   both plausible. I am reporting the observed behaviour rather than a mechanism I did not
   prove. The risk is nonetheless real: the guard is written to return false on a stale
   running record, and nothing prevents a future interaction from actually blocking commits.

3. Any operator diagnosing a queue or extraction problem has to first rule out these ghosts.

Suggested fix. Two independent parts; the first is the important one.

1. Give active statuses a recovery window. A record that owns no work and has not been updated
   for longer than that window cannot be in progress, so close it. The outcome is already
   stored, so completed/failed can be derived rather than guessed:

TTL_ABANDONED = 3_600  # an active record with no work and no update is not alive

def _is_expired(self, task: TaskRecord, now: float) -> bool:
    age = now - task.updated_at
    if task.status in (TaskStatus.COMPLETED, TaskStatus.CANCELLED):
        return age > self.TTL_COMPLETED
    if task.status == TaskStatus.FAILED:
        return age > self.TTL_FAILED
    return age > self.TTL_ABANDONED and not self._work_index.has_work(task.task_id)

The has_work guard keeps this safe: a task that genuinely owns queue work is never
collected, however old it is.

2. Reconcile at startup. restore_work_tasks knows exactly which task ids still own work;
   any persisted record in an active state that is not in that set, and is older than the
   window above, should be finalized then. That closes the gap at the moment it is created
   instead of waiting for a cleanup pass.

Operator-side workaround (what we run): a script that reads the task store and, for each
running record with a non-empty result, no error, no message in any queue and an age
over an hour, writes the terminal state the tracker would have written — status: completed,
stage: completed, plus a status_changed event with the correct seq. Ten records
repaired; re-running is a no-op.

Note on verifying such a repair: the running server serves task records from its
process-local cache (_publish_task / _merge_loaded_tasks), so a store edit is not visible
through the API until the server restarts. Worth knowing before concluding a change did
nothing.

Suggested area / owners (from CONTRIBUTING.md): Platform → server, API, auth, identity,
admin, tasks (openviking/server, openviking/service) → @qin-ctx. Memory → session,
memory extraction (session_commit is the affected task type) → @chenjw, @heaoxiang-ai,
@fujiajie666. My guess on routing, not a fact.