Skip to content

Manager startup stalls in MQTT connection monitoring when Camel consumers are not ready #3587

Open

Description

@ebariaux

User Story

As an operator, I want Manager to finish startup reliably when MQTT devices reconnect after a host reboot, so a normal restart does not cause prolonged downtime.

Description

A production custom project using core MQTT connection monitoring and USER_CONNECTED metadata experienced a prolonged Manager startup after a host reboot. Manager v1.32.2 logged runtime startup completion after 12m 56s, including 11m 28.433s inside MQTTBrokerService.start(). Detailed logs show connection-status initialization running on main, with 22 pauses of 30.001–30.004 seconds, totaling 660.029 seconds.

This points to a core startup ordering problem: attribute events are submitted to a Camel direct: endpoint before its consumer is running. Under shared-executor saturation, CallerRunsPolicy executes ostensibly asynchronous initialization and sends on the startup thread. That thread cannot advance to the service which starts Camel until the repeated consumer-wait timeouts expire.

The relevant core code is unchanged between v1.32.2 and v1.34.1. A subsequent v1.34.1 deployment started much faster, but still exhibited a 30-second wait on the background initialization worker. The comparison also involved image and memory-setting changes, so it does not establish that a version upgrade or memory limit fixes the issue.

Relationship to earlier issues

Production evidence

Incident: 2026-11-17. Host: ARM64 Linux, 2 vCPUs, approximately 8 GiB RAM and 2 GiB host swap; Docker 25.0.14. Manager used Camel 4.22.0. A Docker package update restarted the daemon, followed by a host reboot after a kernel update; those two shutdown sequences were intentional.

All times below are UTC. The detailed application log used UTC+1 and has been converted for comparison with the container logs. Excerpts are not contiguous; user identifiers are replaced with <user>.

Observation Reboot: Manager v1.32.2 Later deployment: Manager v1.34.1
MQTT service start → next service start 20:30:29.202 → 20:41:57.635 20:48:09.385 → 20:48:10.195
Time inside MQTT service startup 688.433 s 0.810 s
Thread performing connection-status initialization main ContainerExecutor-6
Connection-status log entries on that thread 291 192
Approximately 30-second gaps between consecutive status updates on that thread 22 1, on the background worker
MQTT connections before the first initialization status update 149 distinct clients 0
Camel message broker starts 20:42:36.541 20:48:52.986
Runtime startup complete 20:42:43.810 20:48:57.651

Representative slow-start excerpts, including the first and last of the 22 gaps:

2026-11-17T20:30:29.202Z  INFO    [main                          ] org.openremote.container.Container       : Starting service: org.openremote.manager.mqtt.MQTTBrokerService
2026-11-17T20:30:57.420Z  FINE    [main                          ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=false
2026-11-17T20:31:27.424Z  FINE    [main                          ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=false
2026-11-17T20:41:27.625Z  FINE    [main                          ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=true
2026-11-17T20:41:57.626Z  FINE    [main                          ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=true
2026-11-17T20:41:57.635Z  INFO    [main                          ] org.openremote.container.Container       : Starting service: org.openremote.manager.provisioning.ProvisioningService
2026-11-17T20:42:36.541Z  INFO    [main                          ] e.container.message.MessageBrokerService : Starting Camel message broker
2026-11-17T20:42:43.810Z  INFO    [main                          ] org.openremote.container.Container       : >>> Runtime container startup complete

The later deployment still shows a consumer-wait-shaped pause, but on a worker rather than main:

2026-11-17T20:48:09.385Z  INFO    [main                          ] org.openremote.container.Container       : Starting service: org.openremote.manager.mqtt.MQTTBrokerService
2026-11-17T20:48:10.195Z  INFO    [main                          ] org.openremote.container.Container       : Starting service: org.openremote.manager.provisioning.ProvisioningService
2026-11-17T20:48:14.129Z  FINE    [ContainerExecutor-6           ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=false
2026-11-17T20:48:44.129Z  FINE    [ContainerExecutor-6           ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=false
2026-11-17T20:48:52.986Z  INFO    [main                          ] e.container.message.MessageBrokerService : Starting Camel message broker
2026-11-17T20:48:57.651Z  INFO    [main                          ] org.openremote.container.Container       : >>> Runtime container startup complete

PostgreSQL was accepting connections by 20:29:39.872; Keycloak finished bootstrapping at 20:30:11.028, and Manager logged a successful connection at 20:30:11.904. The long MQTT startup interval therefore continued well after initial dependency startup. The supplied kernel journal contains no OOM kill in this incident window.

After the reboot, the existing containers, including the proxy, started almost together. In the later deployment, the proxy started only after Manager became healthy. Early device reconnections are therefore a plausible trigger for shared-executor saturation. The logs confirm the thread, timing, and traffic differences; no historical thread dump was captured to identify every worker's task during this incident.

Acceptance Criteria

  • Add regression coverage for connection-status initialization while Camel routes are not yet running, with the shared executor saturated and MQTT clients reconnecting.
  • Manager startup does not accumulate 30-second consumer waits on main through this path.
  • Initial USER_CONNECTED values and connection changes during startup are correctly reconciled once processing is ready; fixing the wait must not silently discard required state updates.
  • Validate both concurrent container restart and dependency-ordered deployment in a test environment.

Implementation Notes

The relevant code in v1.34.1 (also unchanged in v1.32.2):

  1. Container creates a shared executor with a SynchronousQueue and CallerRunsPolicy. When saturated, submissions can execute on the submitting thread.
  2. MessageBrokerService makes Camel's default-profile executors use that same pool.
  3. ConnectionMonitorHandler.start() submits initialization with the intention of not blocking startup. CallerRunsPolicy can instead run this task on main.
  4. Connection-status changes send attribute events. sendAttributeEvent() uses asyncSend() to the direct attribute-processing endpoint without checking route readiness.
  5. Camel direct: defaults to waiting up to 30 seconds for a consumer. If its asynchronous send is also executed by the caller, the initialization loop blocks inside the send.
  6. MessageBrokerService.start() starts Camel later in the startup sequence. Thus the startup thread waits for a consumer whose startup depends on that same thread making progress. Timeouts eventually allow startup to finish.

Consider deferring or safely buffering attribute-producing initialization until the required routes are active, and reviewing other startup-time producers. The existing syslog guard is useful precedent, but simply returning and dropping initial connection-state events may not preserve correct behavior. Increasing the pool size or shortening timeouts alone would not remove the ordering dependency.

Local reproduction of the blocking mechanism

An isolated probe using Java 21 and Camel 4.22.0 reproduced the mechanism without touching production. It uses a one-worker shared executor held busy by a latch, the same default-profile override, and a route defined before the Camel context starts. The timeout is shortened to 200 ms to make the probe quick.

The initialization task ran on main. Each asyncSend() itself took approximately 200 ms to return, with an already-completed future containing DirectConsumerNotAvailableException; three sends blocked the startup thread for approximately 687 ms. After starting Camel, the same send succeeded. Timing was measured before calling Future.get(), so the measured wait was inside submission, not introduced by awaiting the future.

This validates the shared-executor / unavailable-consumer mechanism; it is not a full reproduction of the production traffic pattern.

Standalone probe source (Camel 4.22.0 dependencies, including camel-core and camel-direct)
import java.util.concurrent.*;
import org.apache.camel.*;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import org.apache.camel.support.DefaultThreadPoolFactory;
import org.apache.camel.spi.ThreadPoolProfile;

public class StartupWaitProbe {
    public static void main(String[] args) throws Exception {
        var executor = new ThreadPoolExecutor(1, 1, 60, TimeUnit.SECONDS,
            new SynchronousQueue<Runnable>(), new ThreadPoolExecutor.CallerRunsPolicy());
        var busy = new CountDownLatch(1);
        var release = new CountDownLatch(1);
        var context = new DefaultCamelContext();
        context.getExecutorServiceManager().setThreadPoolFactory(new DefaultThreadPoolFactory() {
            @Override public ExecutorService newThreadPool(ThreadPoolProfile profile, ThreadFactory factory) {
                return profile.isDefaultProfile() ? executor : super.newThreadPool(profile, factory);
            }
        });
        context.addRoutes(new RouteBuilder() {
            @Override public void configure() {
                from("direct:AttributeEventProcessor").process(exchange -> {});
            }
        });
        var producer = context.createFluentProducerTemplate();
        try {
            producer.start();
            executor.submit(() -> {
                busy.countDown();
                try { release.await(); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
            });
            if (!busy.await(5, TimeUnit.SECONDS)) throw new IllegalStateException("Worker did not start");
            long totalStart = System.nanoTime();
            var monitor = executor.submit(() -> {
                System.out.println("Monitor task thread: " + Thread.currentThread().getName());
                for (int i = 0; i < 3; i++) {
                    long start = System.nanoTime();
                    var future = producer.withBody(false).to("direct:AttributeEventProcessor?timeout=200").asyncSend();
                    double submitMillis = (System.nanoTime() - start) / 1e6;
                    if (!future.isDone()) throw new AssertionError("Expected caller-runs completion before asyncSend returns");
                    try {
                        var result = future.get(5, TimeUnit.SECONDS);
                        System.out.printf("Event %d submit waited %.1f ms; failure=%s%n", i,
                            submitMillis,
                            result.getException() == null ? "none" : result.getException().getClass().getSimpleName());
                    } catch (Exception e) { throw new RuntimeException(e); }
                }
            });
            monitor.get(5, TimeUnit.SECONDS);
            System.out.printf("Startup thread blocked %.1f ms for 3 sends (configured timeout 200 ms each)%n",
                (System.nanoTime() - totalStart) / 1e6);
            release.countDown();
            context.start();
            var ready = producer.withBody(false).to("direct:AttributeEventProcessor?timeout=200").asyncSend().get(5, TimeUnit.SECONDS);
            System.out.println("After Camel starts: " + (ready.getException() == null ? "event succeeds" : ready.getException()));
        } finally {
            release.countDown();
            producer.stop();
            context.stop();
            executor.shutdownNow();
        }
    }
}

Activity

  1. self-assigned this
    on Nov 18, 2026
  2. richturner commented on Nov 18, 2026

    @richturner
    Member

    This could potentially be the same issue observed in another instance which showed startup stuck in the SolarForecastService for ~5mins.

    A structural approach that either:

    1. Tries to ensure correct startup ordering (given custom projects can also add container services this could be unreliable)
    2. Guards in core services to relating to event processing to mitigate this risk

    Are there any decisions that need to be made about startup behaviour? Event processing needs to be fully started before services start pushing events through otherwise they would be silently dropped.

  3. ebariaux commented on Nov 18, 2026

    @ebariaux
    ContributorAuthor

    The startup order needs to cover Java callers as well as public listeners. For example, ApplyPredictedDataPointsService.start() can submit a due prediction through AssetProcessingService.sendAttributeEvent() before Camel starts. The two MQTT integration cases also reproduced the startup-thread waits and stale USER_CONNECTED values with the shared executor saturated.

    1. Make startup a three-phase lifecycle. The container completes each phase for all services before moving to the next.

      • init() prepares services. Resolve dependencies, configure components, and register resources, routes, and handlers. The public listeners stay closed.
      • start() brings up internal processing. Camel starts its event consumers. Artemis starts with its in-process acceptor so MQTT handlers can create their internal sessions. Startup work can submit attribute events to the queue, and processing begins once the attribute route is ready.
      • startIngress() opens public listeners. Undertow opens the shared REST and WebSocket listener, and Artemis opens the public MQTT acceptor. Any startup work a listener depends on must finish before it opens. For MQTT, that includes initializing the monitored USER_CONNECTED values. The container marks itself running after all ingress hooks complete.
    2. Use a bounded Camel seda: queue for asynchronous Java attribute submissions. Initialize it before startup producers can submit, then enable its consumer once the attribute processor is ready. sendAttributeEvent() can accept events during startup and report an immediate error if the queue is full or unavailable. The queue feeds the existing attribute processor. REST opens after readiness and can return the result of processing the write.

    A standalone Camel 4.22.0 probe verified the queue behavior with the shared executor saturated. With the queue explicitly initialized, three events were accepted before Camel started, the fourth failed immediately because the queue was full, and all three accepted events were delivered when the consumer started. Camel SEDA documentation

    Alternative for Java submissions. Keep direct: and make sendAttributeEvent() fail immediately and visibly when the route is unavailable. That’s a smaller change with no additional queue to size or manage. The trade-off is that each producer of a required startup write must defer or retry it, including producers in custom projects. The SEDA approach centralizes that handling, at the cost of a bounded in-memory backlog and a distinction between an event being queued and its write being complete. I’d use SEDA here because required startup writes can come from several services.

  4. ebariaux commented on Nov 19, 2026

    @ebariaux
    ContributorAuthor

    Following up on the earlier proposal, the implementation uses four startup phases and keeps Camel’s direct: route for Java attribute submissions.

    The container completes each phase for every registered service before moving to the next.

    Phase Responsibility
    init() Wire dependencies and register routes, handlers, and subscriptions.
    start() Start internal services and event consumers, including Camel and Artemis’s in-process acceptor.
    startProducers() Enable rules and predicted datapoint processing, and complete startup writes required before ingress.
    startIngress() Open Undertow’s shared REST and WebSocket listener and Artemis’s public MQTT acceptor.

    The producer phase gives internal services a clear point to submit events after consumers are ready. It also keeps public listeners closed while producers initialize, regardless of custom service ordering. In particular, the connection monitor waits for the initial USER_CONNECTED writes before MQTT clients can reconnect.

    AssetProcessingService.sendAttributeEvent() now throws an explicit IllegalStateException if the broker or the attribute route is unavailable. The Java send uses block=false so it also avoids Camel’s consumer wait if the route stops between the readiness check and the send. Built-in startup producers defer their work to startProducers(). Custom callers that submit early get a clear error and can retry when processing is ready.

    The seda: probe showed that buffering is feasible. It would also introduce a queue capacity, an overflow policy, and a distinction between accepting a write and completing it. The lifecycle ordering and fail-fast guard address the startup dependency with the existing route.

    All 24 targeted integration tests passed. They cover custom ordering with Camel starting before Rules, early Java rejection and retry, a stopped attribute route, MQTT connection-state reconciliation, due predictions, rules restarts, and the public transports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions