Skip to content

[16.0][ADD] base_cron_reconnect: restart threaded-mode cron workers after a lost database connection - #3717

Open
baguenth wants to merge 1 commit into
OCA:16.0from
AmetrasIntelligence:16.0-add-base_cron_reconnect
Open

[16.0][ADD] base_cron_reconnect: restart threaded-mode cron workers after a lost database connection#3717
baguenth wants to merge 1 commit into
OCA:16.0from
AmetrasIntelligence:16.0-add-base_cron_reconnect

Conversation

@baguenth

@baguenth baguenth commented Aug 27, 2026

Copy link
Copy Markdown
Member

Hi folks! 👋 This is a new module, so there's no existing maintainer to loop in automatically — tagging a few people active in this repo in case anyone has a spare moment: @pedrobaeza @hbrunn @StefanRijnhart . No worries if you're busy, just trying to get some eyes on it!

Quick context: this fixes a real gap where ThreadedServer.cron_thread() has no recovery from a dropped DB connection — the thread just dies silently. I know Odoo core has closed similar issues as "Won't fix" before, but I want to be clear this isn't trying to relitigate that decision — it's a small, opt-in module for people stuck on threaded mode for whatever reason.

Happy to adjust anything — tests, scope, naming — based on feedback. Thanks for taking a look!

Problem

In threaded mode (workers=0), ThreadedServer.cron_thread() has no exception handling around its outer polling loop. If the database connection is lost — a PostgreSQL restart, failover, or a brief network blip — pg_conn.poll() raises an uncaught psycopg2.OperationalError that escapes the thread entirely. The thread dies, and since cron_spawn() only runs once at server startup, nothing ever respawns it. Every scheduled action on that process then silently stops running, permanently, until the whole server is restarted.

HTTP requests are unaffected — each borrows a fresh connection from the pool — which is what makes this so easy to miss in production: the application looks completely healthy while cron has been dead for days.

By contrast, ir_cron._process_jobs(), called from inside the same loop, is wrapped in try/except — job-level failures are correctly isolated and logged. Only the connection-polling loop itself is missing that protection.

History: not overlooked, deliberately out of scope for core

This class of failure has come up repeatedly over the years. The pattern isn't "nobody noticed" - it's "core has taken a position on it":

Given that stance, this isn't proposed as a core PR. It's an opt-in server-tools module: it doesn't argue with core's position that DB availability is an operational concern, it just gives the subset of deployments stuck in threaded mode - Windows (no prefork support), small or trial instances, anyone who hasn't moved to workers>0 - a way to opt into the same resilience PreforkServer already gets for free.

What this module does

base_cron_reconnect is a server_wide_modules-only addon (no models, no database install needed) that patches ThreadedServer.cron_thread from a post_load hook, at server startup, before cron_spawn() runs. The patch wraps the existing method — it doesn't reimplement Odoo's cron polling — in the same catch/log/sleep/retry supervisor bus.ImDispatch.run() already uses for its own long-lived LISTEN/poll() connection.

The retry itself is logged as WARNING, not ERROR — the underlying disconnect is already logged elsewhere (e.g. odoo.sql_db); this just confirms the thread is recovering on its own. The lifecycle is also logged at INFO (start / restarting / stopped gracefully), matching what queue_job's jobrunner already does — today cron_thread's own start/poll only logs at DEBUG, and a graceful stop isn't logged at all.

It's a no-op under prefork mode (workers>0), since PreforkServer already respawns a crashed WorkerCron process on its own and ThreadedServer is never instantiated there.

Test plan

  • Unit tests: a cron_thread invocation raising is retried (mocked wait) instead of propagating and killing the thread; no retry is logged if a clean shutdown wins the race against the wait; the patch is idempotent; start/stop lifecycle is logged.
  • Manual: killed the PostgreSQL connection under a running threaded dev server; the thread logged a warning and resumed polling after the retry interval instead of dying.

… lost database connection

ThreadedServer.cron_thread has no exception handling around its
polling loop, so a dropped PostgreSQL connection kills the thread
permanently with no respawn - unlike PreforkServer, which already
reaps and restarts a crashed WorkerCron. This wraps cron_thread in
the same catch-log-sleep-retry pattern bus.ImDispatch.run() already
uses, and adds INFO-level lifecycle logging cron previously lacked.
@baguenth
baguenth force-pushed the 16.0-add-base_cron_reconnect branch from e7be24b to a795b5d Compare August 27, 2026 15:18
@baguenth baguenth changed the title [ADD] base_cron_reconnect: restart threaded-mode cron workers after a lost database connection [16.0][ADD] base_cron_reconnect: restart threaded-mode cron workers after a lost database connection Aug 27, 2026
@pedrobaeza

Copy link
Copy Markdown
Member

At first instance, this seems to be a bug in Odoo core that should be fixed there.

@baguenth

baguenth commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

At first instance, this seems to be a bug in Odoo core that should be fixed there.

I agree. Several issues have been raised. Statement from Odoo anyway was, that they do not plan to fix it. See odoo/odoo#88984 (comment) or odoo/odoo#128571

Maybe there is a chance to convince them, but i am not optimistic, tbh :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mod:base_cron_reconnect Module base_cron_reconnect series:16.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants