Keep a timed-out handler visible after runWithTimeout stops waiting for it #266

runWithTimeout in packages/outpost/queue/src/worker.ts is a Promise.race between the handler and a timer. The race settling does not cancel the handler, so on a timeout the job is marked failed, processJob's finally frees the slot and deletes the job's entry from activeJobStarts — and the handler keeps running.

That matters more since #262, because overdueJobCount is measured from activeJobStarts. An orphaned handler is removed from the map at the moment it becomes orphaned, so the count is zero by construction for exactly the work that escaped its timeout. The docstring on that field says a non-zero count means the timeout machinery failed; this is a failure of the timeout machinery that it cannot see.

What it looks like in production

The LLM provider stalls, with no AbortSignal on the pipeline's fetch. Every AI_RESPONSE job times out at 120s, handleFailure schedules a retry, and the poll immediately claims more. Each bad job leaves an orphan behind and is retried up to maxAttempts, so orphans accumulate — each holding a socket and a connection-pool checkout.

Steady state after ten minutes is a worker running noticeably more work than it believes it is, while /health answers 200 {"status":"ok"} and activeJobCount reports only whatever is currently raced. The eventual symptom is pool exhaustion surfacing as an unrelated Prisma error somewhere else entirely, which is a long way from the cause.

Shape of a fix

Keep the orphan visible rather than deleting it:

const started = this.activeJobStarts.get(job.id);
void promise.finally(() => this.orphanedHandlers.delete(job.id));
this.orphanedHandlers.set(job.id, started);

Publish orphanedHandlerCount on WorkerHealthStatus and answer 503 above a threshold — >= maxConcurrency is a defensible line, since past it the worker is running more work than it thinks it is. At minimum, console.error on every timeout naming the job id and type: today the only record is the console.warn inside handleFailure, which describes a retry rather than an abandoned handler.

The deeper fix is an AbortSignal threaded through the handler contract so a timeout actually stops the work. That is a larger change to a shared package and worth doing on its own.

Pre-existing on main; #262 neither introduced nor worsened it. Filed because #262's docs now reference the limit, and a documented limit with no tracking behind it tends to become a documented permanence.