From 77df4aaf4942d663a644704fc1eaf13098d9d24f Mon Sep 17 00:00:00 2001 From: "Scott R. Shinn" Date: Wed, 5 Aug 2026 17:29:16 -0400 Subject: [PATCH] Document analysisd/remoted tuning for UDP Recv-Q backlog. Answer the common 1514 Recv-Q symptom: remoted backpressure from a slow analysisd, with FAQ guidance, remoted notes, and missing pipeline knobs (#2066). --- docs/faq/ossec.rst | 48 +++++++++++++++++++++ docs/manual/agent/remoted-architecture.rst | 11 +++++ docs/syntax/internal_options.analysisd.trst | 46 ++++++++++++++++++++ 3 files changed, 105 insertions(+) diff --git a/docs/faq/ossec.rst b/docs/faq/ossec.rst index f3f6550c..646ce8af 100644 --- a/docs/faq/ossec.rst +++ b/docs/faq/ossec.rst @@ -104,6 +104,54 @@ Do the rules get pushed to the agents automatically? Agents do not send alerts to the manager, they only send the raw logs. +UDP port 1514 shows a large Recv-Q — is analysisd too slow? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + Often yes, indirectly. Agent traffic lands on ``ossec-remoted`` (UDP/TCP + ``secure`` listener, commonly port ``1514``). Remoted decrypts and immediately + forwards each event to ``ossec-analysisd`` over the local message queue. When + analysisd cannot keep up, remoted blocks or slows on that hand-off, the kernel + UDP receive buffer fills, and ``ss`` / ``netstat`` reports a growing + **Recv-Q**. + + On **OSSEC 3.6** (the version in the original report), ``ossec-analysisd`` was + effectively single-threaded. On **current Linux managers** (4.2+), analysisd + always runs a multi-threaded pipeline. Prefer upgrading before heavy tuning. + + Diagnose on the manager: + + 1. Confirm Recv-Q on the agent port (``ss -ulnp | grep 1514`` or equivalent). + 2. Watch ``/var/ossec/var/run/ossec-analysisd.state`` (see + ``analysisd.state_interval``) for queue depth, drops, and + ``events_processed_per_second``. + 3. Check ``ossec.log`` for remoted "Unable to send message to queue" and + analysisd pipeline pressure / drop warnings. + + Tuning (``/var/ossec/etc/local_internal_options.conf``, then restart): + + * Raise worker counts if CPUs are idle: + ``analysisd.event_threads``, ``analysisd.rule_matching_threads``, + ``analysisd.input_demux_threads`` (``0`` = auto for most ``*_threads``). + * Under bursty load, enlarge ring buffers + (``analysisd.raw_input_queue_size``, ``analysisd.decode_*_queue_size``, + ``analysisd.alerts_queue_size``, …) and/or mid-pipe soak + (``analysisd.*_push_wait_ms``). + * Reduce work: disable ```` if enabled, quiet noisy rules, avoid + expensive regex where possible, and lower agent EPS at the source. + + Kernel socket buffer (remoted side) can also help absorb short bursts:: + + # example — persist via sysctl.d as appropriate + sysctl -w net.core.rmem_max=16777216 + sysctl -w net.core.rmem_default=16777216 + + That only buys time; sustained Recv-Q still means analysisd or the remoted → + analysisd path cannot keep up. + + See :ref:`intopt_analysisd`, :ref:`manual-remoted`, and the 4.2 migration notes + in :ref:`upgrade-migration` for the threaded pipeline. + + How can I get ossec.log to rotate daily? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/manual/agent/remoted-architecture.rst b/docs/manual/agent/remoted-architecture.rst index 0a1e520f..57347ecd 100644 --- a/docs/manual/agent/remoted-architecture.rst +++ b/docs/manual/agent/remoted-architecture.rst @@ -58,6 +58,17 @@ Syslog UDP The listener thread receives datagrams and forwards them to ``ossec-analysisd`` with no worker pool. +Agent port Recv-Q (backpressure) +-------------------------------- + +A large **Recv-Q** on the secure agent listener (often UDP ``1514``) usually means +``ossec-remoted`` is not draining datagrams as fast as agents send them. The common +cause is backpressure from ``ossec-analysisd``: remoted forwards every event into the +local analysisd queue, so a slow or overloaded analyzer fills the kernel UDP buffer. + +See the FAQ entry *UDP port 1514 shows a large Recv-Q* in :ref:`faq_ossec` for +diagnostics and ``internal_options.conf`` knobs on the analysisd pipeline. + Shutdown -------- diff --git a/docs/syntax/internal_options.analysisd.trst b/docs/syntax/internal_options.analysisd.trst index 48841f0e..76220c4c 100644 --- a/docs/syntax/internal_options.analysisd.trst +++ b/docs/syntax/internal_options.analysisd.trst @@ -114,6 +114,31 @@ **Allowed:** 0 to 32 + - analysisd.input_demux_threads + + Workers that demux raw messages from the Unix MQ receive path onto the + per-type decode queues after the single receiver thread. + + **Default:** 2 + + **Allowed:** 1 to 8 + + - analysisd.sid_prev_matched_max + + Maximum retained events per ``if_matched_sid`` correlation list. + + **Default:** 128 + + **Allowed:** Any positive integer (implementation-defined upper bound) + + - analysisd.raw_input_queue_size + + Front-door ring buffer between the Unix MQ receiver and demux workers. + + **Default:** 16384 + + **Allowed:** 128 to 2000000 + - analysisd.decode_event_queue_size Ring buffer size (minus one slot) for raw messages awaiting general decode. @@ -162,6 +187,27 @@ **Allowed:** 128 to 2000000 + - analysisd.statistical_queue_size + + Ring buffer size for the statistical / hourly alert writer backlog. + + **Default:** 16384 + + **Allowed:** 128 to 2000000 + + - analysisd.raw_input_push_wait_ms + - analysisd.demux_push_wait_ms + - analysisd.shard_push_wait_ms + - analysisd.alerts_push_wait_ms + + Milliseconds to wait when a mid-pipe push would block before dropping. + ``0`` means immediate push/drop (no soak). Raise under bursty load together + with queue sizes; lower EPS if drops persist. + + **Defaults:** 25 / 50 / 75 / 100 + + **Allowed:** 0 to 5000 + - analysisd.archives_queue_size Ring buffer size for archive writer thread backlog. When full, archive