Missing matching EOE markers cause unbounded NMEA collection, high CPU/memory usage and delayed ROS output #118

Summary

When nmea_epoch is set to GNSS but the sensor stream contains GSA/GSV messages and only FP_A-EOE_FUSION markers, the ROS 2 driver's NMEA collector is never reset. Its accumulated data grows, and processing eventually falls behind the live sensor stream.

We understand that the epoch configuration must match the sensor output. This report concerns robustness when that requirement is not met: the mismatch can progressively delay the driver's other ROS outputs by several minutes, rather than producing a clear configuration warning and keeping collection bounded.

Environment

• Vision-RTK 2 connected over TCP.
• fixposition_driver tag 8.1.0, built in Release mode (-O3).
• ROS 2 Humble in Docker on an NVIDIA Jetson Orin Nano, Ubuntu 22.04, aarch64.
• The TCP read-buffer-size fix using read(sensor_fd_, buf, size) was already applied.
• Sensor firmware version was not recorded during this investigation.
• The behavior has not been retested against current main.

Reproduction conditions

1. Configure the sensor's TCP output to include NMEA GSA/GSV messages and FP_A-EOE_FUSION, without FP_A-EOE_GNSS.
2. Enable these messages in the driver and leave nmea_epoch set to GNSS.
3. Run the driver for an extended period while monitoring RSS, CPU, TCP receive backlog and the age of ROS message timestamps.

Relevant configuration excerpt, with other settings omitted:

/**:
  ros__parameters:
    nmea_epoch: "GNSS"
    messages:
      - "FP_A-EOE"
      - "FP_A-ODOMETRY"
      - "GSA"
      - "GSV"

A 12-second capture of our live sensor output contained 120 FUSION EOE markers and no GNSS EOE markers, alongside continuous GSA/GSV messages. We have not yet built a standalone replay reproducer.

Observed behavior

After approximately 18 hours of operation:

• The driver process consumed approximately 92% of one CPU core and 790,096 KiB RSS (~772 MiB).
• The TCP receive queue held approximately 5.4 MB.
• ROS odometry/status messages were approximately 562–570 seconds old when received.
• A separate connection to the sensor still delivered current native output continuously at 10 Hz.

This affected position and status publications beyond the composite NMEA output. The ROS driver also logged significant-delay warnings. Native sensor status confirmed that both GNSS receivers were RTK-fixed and corrections were being used.

A 10-second CPU profile attributed 99.29% of sampled cycles to:

fpsdk::common::parser::nmea::NmeaCollectGsaGsv::AddGsv
  <- FixpositionDriverNode::StartNode() NMEA callback
  <- FixpositionDriver::NotifyNmeaObservers
  <- FixpositionDriver::Worker

The profile contained 630 samples and reported no lost samples.

Apparent cause

The driver's EOE callback calls nmea_epoch_data_.CompleteAndReset() only when the received EOE type matches params_.nmea_epoch_. With the configuration above, that condition is never satisfied, while the GSA/GSV callbacks continue collecting observations.

In the SDK collector, AddGsv() searches the accumulated GSA observations and appends signal observations. Without regular resets, storage grows and those searches become increasingly expensive. This is consistent with the CPU profile, memory usage and delayed TCP processing.

Relevant driver code: EOE handling and NMEA callbacks in version 8.1.0.

Workaround and validation

Changing nmea_epoch to FUSION and restarting the driver resolved the issue for our current stream:

nmea_epoch: "FUSION"

A subsequent 120.32-second observation of the deployed driver measured:

These are short-term post-restart measurements, not an 18-hour validation of the workaround. The regular composite NMEA publications confirm that epoch completion/reset is now occurring.

Expected behavior / suggested improvement

Could the driver detect that the configured epoch's EOE markers are missing and warn explicitly? It would also be useful to bound or expire incomplete NMEA collections so that a mismatched configuration or missing EOE messages cannot cause unbounded accumulation and delay unrelated ROS outputs.

The driver need not silently select a different epoch; an explicit diagnostic and a bounded recovery policy would already make this failure much easier to detect and contain.