Repository navigation
Description
User Story
As an operator, I want Manager to finish startup reliably when MQTT devices reconnect after a host reboot, so a normal restart does not cause prolonged downtime.
Description
A production custom project using core MQTT connection monitoring and USER_CONNECTED metadata experienced a prolonged Manager startup after a host reboot. Manager v1.32.2 logged runtime startup completion after 12m 56s, including 11m 28.433s inside MQTTBrokerService.start(). Detailed logs show connection-status initialization running on main, with 22 pauses of 30.001–30.004 seconds, totaling 660.029 seconds.
This points to a core startup ordering problem: attribute events are submitted to a Camel direct: endpoint before its consumer is running. Under shared-executor saturation, CallerRunsPolicy executes ostensibly asynchronous initialization and sends on the startup thread. That thread cannot advance to the service which starts Camel until the repeated consumer-wait timeouts expire.
The relevant core code is unchanged between v1.32.2 and v1.34.1. A subsequent v1.34.1 deployment started much faster, but still exhibited a 30-second wait on the background initialization worker. The comparison also involved image and memory-setting changes, so it does not establish that a version upgrade or memory limit fixes the issue.
Relationship to earlier issues
- This investigation in #2005 describes the same shared-executor / unavailable-Camel-consumer failure pattern through rule compilation, syslog, and
ClientEventService.publishEvent(). - The readiness guard discussed there is already present in both v1.32.2 and v1.34.1:
ClientEventService.publishEvent()checks both its own state andmessageBrokerService.isStarted(). - This report concerns another path:
ConnectionMonitorHandler→AssetProcessingService.sendAttributeEvent()→direct://AttributeEventProcessor. That send does not have an equivalent readiness guard. - Also related: MQTT Broker: delay broker startup to ensure system is fully online before devices try to connect #947, which requested delaying MQTT availability until the system is fully online.
Production evidence
Incident: 2026-11-17. Host: ARM64 Linux, 2 vCPUs, approximately 8 GiB RAM and 2 GiB host swap; Docker 25.0.14. Manager used Camel 4.22.0. A Docker package update restarted the daemon, followed by a host reboot after a kernel update; those two shutdown sequences were intentional.
All times below are UTC. The detailed application log used UTC+1 and has been converted for comparison with the container logs. Excerpts are not contiguous; user identifiers are replaced with <user>.
| Observation | Reboot: Manager v1.32.2 | Later deployment: Manager v1.34.1 |
|---|---|---|
| MQTT service start → next service start | 20:30:29.202 → 20:41:57.635 | 20:48:09.385 → 20:48:10.195 |
| Time inside MQTT service startup | 688.433 s | 0.810 s |
| Thread performing connection-status initialization | main |
ContainerExecutor-6 |
| Connection-status log entries on that thread | 291 | 192 |
| Approximately 30-second gaps between consecutive status updates on that thread | 22 | 1, on the background worker |
| MQTT connections before the first initialization status update | 149 distinct clients | 0 |
| Camel message broker starts | 20:42:36.541 | 20:48:52.986 |
| Runtime startup complete | 20:42:43.810 | 20:48:57.651 |
Representative slow-start excerpts, including the first and last of the 22 gaps:
2026-11-17T20:30:29.202Z INFO [main ] org.openremote.container.Container : Starting service: org.openremote.manager.mqtt.MQTTBrokerService
2026-11-17T20:30:57.420Z FINE [main ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=false
2026-11-17T20:31:27.424Z FINE [main ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=false
2026-11-17T20:41:27.625Z FINE [main ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=true
2026-11-17T20:41:57.626Z FINE [main ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=true
2026-11-17T20:41:57.635Z INFO [main ] org.openremote.container.Container : Starting service: org.openremote.manager.provisioning.ProvisioningService
2026-11-17T20:42:36.541Z INFO [main ] e.container.message.MessageBrokerService : Starting Camel message broker
2026-11-17T20:42:43.810Z INFO [main ] org.openremote.container.Container : >>> Runtime container startup complete
The later deployment still shows a consumer-wait-shaped pause, but on a worker rather than main:
2026-11-17T20:48:09.385Z INFO [main ] org.openremote.container.Container : Starting service: org.openremote.manager.mqtt.MQTTBrokerService
2026-11-17T20:48:10.195Z INFO [main ] org.openremote.container.Container : Starting service: org.openremote.manager.provisioning.ProvisioningService
2026-11-17T20:48:14.129Z FINE [ContainerExecutor-6 ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=false
2026-11-17T20:48:44.129Z FINE [ContainerExecutor-6 ] anager.mqtt.ConnectionMonitorHandler.API : Updating connected status for '<user>' on 1 attribute(s) connected=false
2026-11-17T20:48:52.986Z INFO [main ] e.container.message.MessageBrokerService : Starting Camel message broker
2026-11-17T20:48:57.651Z INFO [main ] org.openremote.container.Container : >>> Runtime container startup complete
PostgreSQL was accepting connections by 20:29:39.872; Keycloak finished bootstrapping at 20:30:11.028, and Manager logged a successful connection at 20:30:11.904. The long MQTT startup interval therefore continued well after initial dependency startup. The supplied kernel journal contains no OOM kill in this incident window.
After the reboot, the existing containers, including the proxy, started almost together. In the later deployment, the proxy started only after Manager became healthy. Early device reconnections are therefore a plausible trigger for shared-executor saturation. The logs confirm the thread, timing, and traffic differences; no historical thread dump was captured to identify every worker's task during this incident.
Acceptance Criteria
- Add regression coverage for connection-status initialization while Camel routes are not yet running, with the shared executor saturated and MQTT clients reconnecting.
- Manager startup does not accumulate 30-second consumer waits on
mainthrough this path. - Initial
USER_CONNECTEDvalues and connection changes during startup are correctly reconciled once processing is ready; fixing the wait must not silently discard required state updates. - Validate both concurrent container restart and dependency-ordered deployment in a test environment.
Implementation Notes
The relevant code in v1.34.1 (also unchanged in v1.32.2):
Containercreates a shared executor with aSynchronousQueueandCallerRunsPolicy. When saturated, submissions can execute on the submitting thread.MessageBrokerServicemakes Camel's default-profile executors use that same pool.ConnectionMonitorHandler.start()submits initialization with the intention of not blocking startup.CallerRunsPolicycan instead run this task onmain.- Connection-status changes send attribute events.
sendAttributeEvent()usesasyncSend()to the direct attribute-processing endpoint without checking route readiness. - Camel
direct:defaults to waiting up to 30 seconds for a consumer. If its asynchronous send is also executed by the caller, the initialization loop blocks inside the send. MessageBrokerService.start()starts Camel later in the startup sequence. Thus the startup thread waits for a consumer whose startup depends on that same thread making progress. Timeouts eventually allow startup to finish.
Consider deferring or safely buffering attribute-producing initialization until the required routes are active, and reviewing other startup-time producers. The existing syslog guard is useful precedent, but simply returning and dropping initial connection-state events may not preserve correct behavior. Increasing the pool size or shortening timeouts alone would not remove the ordering dependency.
Local reproduction of the blocking mechanism
An isolated probe using Java 21 and Camel 4.22.0 reproduced the mechanism without touching production. It uses a one-worker shared executor held busy by a latch, the same default-profile override, and a route defined before the Camel context starts. The timeout is shortened to 200 ms to make the probe quick.
The initialization task ran on main. Each asyncSend() itself took approximately 200 ms to return, with an already-completed future containing DirectConsumerNotAvailableException; three sends blocked the startup thread for approximately 687 ms. After starting Camel, the same send succeeded. Timing was measured before calling Future.get(), so the measured wait was inside submission, not introduced by awaiting the future.
This validates the shared-executor / unavailable-consumer mechanism; it is not a full reproduction of the production traffic pattern.
Standalone probe source (Camel 4.22.0 dependencies, including camel-core and camel-direct)
import java.util.concurrent.*;
import org.apache.camel.*;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import org.apache.camel.support.DefaultThreadPoolFactory;
import org.apache.camel.spi.ThreadPoolProfile;
public class StartupWaitProbe {
public static void main(String[] args) throws Exception {
var executor = new ThreadPoolExecutor(1, 1, 60, TimeUnit.SECONDS,
new SynchronousQueue<Runnable>(), new ThreadPoolExecutor.CallerRunsPolicy());
var busy = new CountDownLatch(1);
var release = new CountDownLatch(1);
var context = new DefaultCamelContext();
context.getExecutorServiceManager().setThreadPoolFactory(new DefaultThreadPoolFactory() {
@Override public ExecutorService newThreadPool(ThreadPoolProfile profile, ThreadFactory factory) {
return profile.isDefaultProfile() ? executor : super.newThreadPool(profile, factory);
}
});
context.addRoutes(new RouteBuilder() {
@Override public void configure() {
from("direct:AttributeEventProcessor").process(exchange -> {});
}
});
var producer = context.createFluentProducerTemplate();
try {
producer.start();
executor.submit(() -> {
busy.countDown();
try { release.await(); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
});
if (!busy.await(5, TimeUnit.SECONDS)) throw new IllegalStateException("Worker did not start");
long totalStart = System.nanoTime();
var monitor = executor.submit(() -> {
System.out.println("Monitor task thread: " + Thread.currentThread().getName());
for (int i = 0; i < 3; i++) {
long start = System.nanoTime();
var future = producer.withBody(false).to("direct:AttributeEventProcessor?timeout=200").asyncSend();
double submitMillis = (System.nanoTime() - start) / 1e6;
if (!future.isDone()) throw new AssertionError("Expected caller-runs completion before asyncSend returns");
try {
var result = future.get(5, TimeUnit.SECONDS);
System.out.printf("Event %d submit waited %.1f ms; failure=%s%n", i,
submitMillis,
result.getException() == null ? "none" : result.getException().getClass().getSimpleName());
} catch (Exception e) { throw new RuntimeException(e); }
}
});
monitor.get(5, TimeUnit.SECONDS);
System.out.printf("Startup thread blocked %.1f ms for 3 sends (configured timeout 200 ms each)%n",
(System.nanoTime() - totalStart) / 1e6);
release.countDown();
context.start();
var ready = producer.withBody(false).to("direct:AttributeEventProcessor?timeout=200").asyncSend().get(5, TimeUnit.SECONDS);
System.out.println("After Camel starts: " + (ready.getException() == null ? "event succeeds" : ready.getException()));
} finally {
release.countDown();
producer.stop();
context.stop();
executor.shutdownNow();
}
}
}
Activity
richturner commentedon Nov 18, 2026
This could potentially be the same issue observed in another instance which showed startup stuck in the SolarForecastService for ~5mins.
A structural approach that either:
Are there any decisions that need to be made about startup behaviour? Event processing needs to be fully started before services start pushing events through otherwise they would be silently dropped.
ebariaux commentedon Nov 18, 2026
The startup order needs to cover Java callers as well as public listeners. For example,
ApplyPredictedDataPointsService.start()can submit a due prediction throughAssetProcessingService.sendAttributeEvent()before Camel starts. The two MQTT integration cases also reproduced the startup-thread waits and staleUSER_CONNECTEDvalues with the shared executor saturated.Make startup a three-phase lifecycle. The container completes each phase for all services before moving to the next.
init()prepares services. Resolve dependencies, configure components, and register resources, routes, and handlers. The public listeners stay closed.start()brings up internal processing. Camel starts its event consumers. Artemis starts with its in-process acceptor so MQTT handlers can create their internal sessions. Startup work can submit attribute events to the queue, and processing begins once the attribute route is ready.startIngress()opens public listeners. Undertow opens the shared REST and WebSocket listener, and Artemis opens the public MQTT acceptor. Any startup work a listener depends on must finish before it opens. For MQTT, that includes initializing the monitoredUSER_CONNECTEDvalues. The container marks itself running after all ingress hooks complete.Use a bounded Camel
seda:queue for asynchronous Java attribute submissions. Initialize it before startup producers can submit, then enable its consumer once the attribute processor is ready.sendAttributeEvent()can accept events during startup and report an immediate error if the queue is full or unavailable. The queue feeds the existing attribute processor. REST opens after readiness and can return the result of processing the write.A standalone Camel 4.22.0 probe verified the queue behavior with the shared executor saturated. With the queue explicitly initialized, three events were accepted before Camel started, the fourth failed immediately because the queue was full, and all three accepted events were delivered when the consumer started. Camel SEDA documentation
Alternative for Java submissions. Keep
direct:and makesendAttributeEvent()fail immediately and visibly when the route is unavailable. That’s a smaller change with no additional queue to size or manage. The trade-off is that each producer of a required startup write must defer or retry it, including producers in custom projects. The SEDA approach centralizes that handling, at the cost of a bounded in-memory backlog and a distinction between an event being queued and its write being complete. I’d use SEDA here because required startup writes can come from several services.ebariaux commentedon Nov 19, 2026
Following up on the earlier proposal, the implementation uses four startup phases and keeps Camel’s
direct:route for Java attribute submissions.The container completes each phase for every registered service before moving to the next.
init()start()startProducers()startIngress()The producer phase gives internal services a clear point to submit events after consumers are ready. It also keeps public listeners closed while producers initialize, regardless of custom service ordering. In particular, the connection monitor waits for the initial
USER_CONNECTEDwrites before MQTT clients can reconnect.AssetProcessingService.sendAttributeEvent()now throws an explicitIllegalStateExceptionif the broker or the attribute route is unavailable. The Java send usesblock=falseso it also avoids Camel’s consumer wait if the route stops between the readiness check and the send. Built-in startup producers defer their work tostartProducers(). Custom callers that submit early get a clear error and can retry when processing is ready.The
seda:probe showed that buffering is feasible. It would also introduce a queue capacity, an overflow policy, and a distinction between accepting a write and completing it. The lifecycle ordering and fail-fast guard address the startup dependency with the existing route.All 24 targeted integration tests passed. They cover custom ordering with Camel starting before Rules, early Java rejection and retry, a stopped attribute route, MQTT connection-state reconciliation, due predictions, rules restarts, and the public transports.