[16.0][ADD] base_cron_reconnect: restart threaded-mode cron workers after a lost database connection - #3717
Open
baguenth wants to merge 1 commit into
Open
Conversation
This was referenced Aug 27, 2026
baguenth
force-pushed
the
16.0-add-base_cron_reconnect
branch
from
August 27, 2026 15:14
48943f8 to
e7be24b
Compare
… lost database connection ThreadedServer.cron_thread has no exception handling around its polling loop, so a dropped PostgreSQL connection kills the thread permanently with no respawn - unlike PreforkServer, which already reaps and restarts a crashed WorkerCron. This wraps cron_thread in the same catch-log-sleep-retry pattern bus.ImDispatch.run() already uses, and adds INFO-level lifecycle logging cron previously lacked.
baguenth
force-pushed
the
16.0-add-base_cron_reconnect
branch
from
August 27, 2026 15:18
e7be24b to
a795b5d
Compare
Member
|
At first instance, this seems to be a bug in Odoo core that should be fixed there. |
Member
Author
I agree. Several issues have been raised. Statement from Odoo anyway was, that they do not plan to fix it. See odoo/odoo#88984 (comment) or odoo/odoo#128571 Maybe there is a chance to convince them, but i am not optimistic, tbh :D |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi folks! 👋 This is a new module, so there's no existing maintainer to loop in automatically — tagging a few people active in this repo in case anyone has a spare moment: @pedrobaeza @hbrunn @StefanRijnhart . No worries if you're busy, just trying to get some eyes on it!
Quick context: this fixes a real gap where
ThreadedServer.cron_thread()has no recovery from a dropped DB connection — the thread just dies silently. I know Odoo core has closed similar issues as "Won't fix" before, but I want to be clear this isn't trying to relitigate that decision — it's a small, opt-in module for people stuck on threaded mode for whatever reason.Happy to adjust anything — tests, scope, naming — based on feedback. Thanks for taking a look!
Problem
In threaded mode (
workers=0),ThreadedServer.cron_thread()has no exception handling around its outer polling loop. If the database connection is lost — a PostgreSQL restart, failover, or a brief network blip —pg_conn.poll()raises an uncaughtpsycopg2.OperationalErrorthat escapes the thread entirely. The thread dies, and sincecron_spawn()only runs once at server startup, nothing ever respawns it. Every scheduled action on that process then silently stops running, permanently, until the whole server is restarted.HTTP requests are unaffected — each borrows a fresh connection from the pool — which is what makes this so easy to miss in production: the application looks completely healthy while cron has been dead for days.
By contrast,
ir_cron._process_jobs(), called from inside the same loop, is wrapped in try/except — job-level failures are correctly isolated and logged. Only the connection-polling loop itself is missing that protection.History: not overlooked, deliberately out of scope for core
This class of failure has come up repeatedly over the years. The pattern isn't "nobody noticed" - it's "core has taken a position on it":
8.0branch). That predates theLISTEN/NOTIFY-basedpg_conn.poll()callcron_threaduses today, and per the current 16.0 source (verified directly), that specific call is still unguarded - the 2017 fix didn't carry forward to cover it.ThreadedServer... any failure to establish a connection or failure in the connection link is considered a fatal error that needs to be addressed by the system administrator." This is a deliberate design position, not an oversight.Bus.loop's retry pattern as the fix; unaddressed.Given that stance, this isn't proposed as a core PR. It's an opt-in
server-toolsmodule: it doesn't argue with core's position that DB availability is an operational concern, it just gives the subset of deployments stuck in threaded mode - Windows (no prefork support), small or trial instances, anyone who hasn't moved toworkers>0- a way to opt into the same resiliencePreforkServeralready gets for free.What this module does
base_cron_reconnectis aserver_wide_modules-only addon (no models, no database install needed) that patchesThreadedServer.cron_threadfrom apost_loadhook, at server startup, beforecron_spawn()runs. The patch wraps the existing method — it doesn't reimplement Odoo's cron polling — in the same catch/log/sleep/retry supervisorbus.ImDispatch.run()already uses for its own long-livedLISTEN/poll()connection.The retry itself is logged as
WARNING, notERROR— the underlying disconnect is already logged elsewhere (e.g.odoo.sql_db); this just confirms the thread is recovering on its own. The lifecycle is also logged atINFO(start / restarting / stopped gracefully), matching whatqueue_job's jobrunner already does — todaycron_thread's own start/poll only logs atDEBUG, and a graceful stop isn't logged at all.It's a no-op under prefork mode (
workers>0), sincePreforkServeralready respawns a crashedWorkerCronprocess on its own andThreadedServeris never instantiated there.Test plan
cron_threadinvocation raising is retried (mocked wait) instead of propagating and killing the thread; no retry is logged if a clean shutdown wins the race against the wait; the patch is idempotent; start/stop lifecycle is logged.