Repository navigation
ob status and execution list do not show an in-flight scheduled job, making an exclusive lock look orphaned #188
Description
What happens
An operation was refused with:
✗ ob: application scheduling rendezvous remained busy — wait for the current
scheduled job or application operation to finish
while, at the same moment, onebox reported that nothing was running:
$ ob execution list
no durable executions recorded
$ ob status
schedule <weekly-job> last run failed: timeout (exit 143), and nothing has run since ⚠
schedule <5min-job> skipped 20 firings in a row: an application operation is taking its lock ⚠
Host inspection showed the truth: a scheduled job declaring deploy_lock: exclusive had been running for nearly eight hours and legitimately held the rendezvous.
PID 365044 started 10:00:11 elapsed 07:57:46 PPID 1
/bin/sh /etc/systemd/system/ob-<app>-<job>.run
PID 982137 started 15:47:12 docker compose run --rm --no-deps ... <job>
both holding fd 8 on <base>/<app>/schedule.lock. The application lock file itself did not exist, so nothing was stuck — the lock was doing exactly its job.
The bug
An in-flight scheduled job is invisible to ob status and ob execution list.
ob execution listreturnsno durable executions recordedwhile a job has been running for hours.ob statusreports the previous run's outcome and adds "and nothing has run since", which is actively wrong and points away from the truth.
ob schedule list does show the current run's start time as LAST TRIGGER, so the information exists and is simply not joined up.
Why it matters
The refusal message is accurate but gives nothing to act on. Every diagnostic onebox offers then says the system is idle, so the natural conclusion is an orphaned lock from the earlier timeout — that is what I concluded, and it is wrong. The real answer, "a job with an exclusive lock has been running since 10:00 and will hold it until it finishes", is not reachable through any ob command. It took /proc/locks and an fd scan on the host to find it.
The confusion is compounded by skipped being the same word used for a healthy skip, so a schedule that has done nothing for hours looks unremarkable.
Suggested
- Show in-flight scheduled jobs in
ob statusandob execution list. A running exclusive job is the single most useful thing to know when an operation is refused. - Name the holder in the
rendezvous remained busyerror — job name and start time. The data is discoverable from/proc/locksplus an fd scan; surfacing it turns a dead end into a wait. - Do not report "nothing has run since" when something is running.
- Consider whether repeated
skippedoutcomes deserve a distinct signal from ordinary ones.
Environment
- Runner:
ob v2026.9.3 (ffe61080c66b) - Host: Ubuntu, docker 29.1.3, compose 2.40.3
- Jobs: one weekly with
deploy_lock: exclusiveand an 8h timeout, two withdeploy_lock: pinned
Activity
vishr commentedon Sep 14, 2026
Additional finding while investigating: the recovery path I reached for is itself gated behind the same rendezvous.
ob schedule applyregenerates the timer units, and is the documented way to reconcile them after a runner or configuration change. It acquires the rendezvous first:That is defensible on its own — reconciling units while a job runs from them is not obviously safe. But combined with the reporting gap above it leaves an operator with no way forward and no way to find out why:
ob execrefused,ob schedule applyrefused, and every status command reporting an idle system.Worth noting an asymmetry in the locking while this is being looked at.
AcquireLockhas a take-over path for the application lock —age > lockTTL→ "expired Ns ago — taking over", default 10 minutes. The rendezvous has no equivalent. That is correct while a real holder exists, which was the case here, but it means a genuinely leaked flock would have no recovery path at all.vishr commentedon Sep 14, 2026
Correcting my own analysis: the lock was not orphaned.
My original report concluded that timer units generated before 3e7fe57 never release the lock. That was wrong, and I have rewritten the issue body to describe the actual defect.
What was happening: a scheduled job with
deploy_lock: exclusivewas running normally and holding the rendezvous for its whole run, exactly as documented.<base>/<app>/lockdid not exist, so no application lock was stuck, and every skipped firing behind it was correct behaviour.The units on that host are older than 3e7fe57 — the last deploy predates the fix — but that is not what caused this, and the older code path would have skipped correctly here too. Apologies for the noise.
What led me to the wrong conclusion is the part still worth fixing: with that job running for eight hours,
ob execution listreportedno durable executions recordedandob statusreported the previous run's timeout with "and nothing has run since". I trusted both, and they described an idle system while an exclusive job held the lock.The remaining separate observation — that a runner upgrade does not regenerate timer units, so a fix to the generated scripts stays inert until someone runs
ob schedule apply— still seems worth its own issue, but it did not cause this one.[-]Scheduled jobs skip forever after a runner upgrade: timer units keep pre-fix lock logic[/-][+]ob status and execution list do not show an in-flight scheduled job, making an exclusive lock look orphaned[/+]vishr commentedon Sep 14, 2026
One more command gated behind the same rendezvous:
ob schedule pause.So while a scheduled job is running, it cannot be paused through
ob. Combined with the earlier findings, an operator facing a misbehaving long-running job has:ob exec— refusedob schedule apply— refusedob schedule pause— refusedob status/ob execution list— report an idle systemand no supported way to stop the job or find out what it is. The only route left is
systemctl stopon the host, outside onebox entirely, which also means it happens without the journallingobwould otherwise provide.pausein particular seems worth exempting. It does not mutate the release or the units — it stops a timer — and its value is highest exactly when a job is misbehaving, which is when it is currently unavailable. The error message even makes the case: pausing is how you stop the next occurrence of the run that is holding the lock.vishr commentedon Sep 14, 2026
Reproduced again, this time with an operator-initiated sealed-plan job rather than a timer-driven one, which rules out the timer path being the cause.
While the job is running — container up, actively logging progress, exclusive lock held:
ob statusis describing the previous run. There is no indication anywhere inob's output that a job is running right now, despiteob job plan/ob job runhaving just sealed, approved and started it seconds earlier, and despite the audit recording it:So the state is known to the audit log and absent from the two commands an operator would actually consult.
Two adjacent observations from the same session, offered as context rather than separate reports:
ob job runis synchronous and attached. For a job whose declaredtimeoutis measured in hours, that commits a terminal for the duration. The container is detached and survives a dropped connection, butobstops tracking and the run is recordedINCOMPLETE— the same marker an abandoned approval leaves, so a completed-but-disconnected run and a cancelled one are indistinguishable afterwards.INCOMPLETEis also what a cancelled approval leaves behind. In this session a run cancelled at the typed-confirmation prompt and a run that started and detached both producedINCOMPLETErows, which makes the audit ambiguous about whether anything actually executed.None of this is data-unsafe. It is that the tool's own reporting cannot answer "is something running, and did it do anything", which is the question an operator has when an exclusive lock is refusing their work.
vishr commentedon Oct 5, 2026
Closing the scheduled-job visibility defect as completed by #191 and verified on
v2026.10.1-alpha.ob statusnow reports a running scheduled job's live phase, elapsed time, trigger, release, start time, lock policy, attempt, timeout, and retry budget. It suppresses the previous-run failure verdict while a run is active, and repeated skipped firings have a distinct warning. Operator runs of scheduled non-migration jobs use the host supervisor from #190/#191;ob job historymerges timer and operator evidence.#212 intentionally removed
ob executionand the Python workflow layer. It is no longer a diagnostic command for ordinary jobs.This does not claim that every attached sealed job is durable: jobs without a schedule and migration-effect operator runs still use the attached path. Their disconnect, ownership, reconciliation, and inspection gaps remain tracked in #179. The focused history/status/scheduled-job regression run passed all 47 tests.