Research: concurrency capacity, bottlenecks, and scaling/optimization options

[NotYuSheng]

Objective

Determine how many concurrent users/uploads TracePcap can support, identify the bottlenecks, and document the levers for scaling (software + hardware) and the optimizations found. This is a research/findings write-up to inform capacity planning and future work; it is not a bug.

TL;DR

• Concurrent users (analysts browsing results) is not the constraint — read endpoints are sub-second and trivially handle 10+ users even on modest hardware. The real constraint is concurrent analyses (ingestion throughput).
• The analysis pipeline is CPU-bound and dominated by Suricata (~94% of per-file time). Throughput is capped by the in-process async pool (max-pool-size, default 10).
• The system degrades gracefully (queues, latency grows) until the queue overflows — at which point jobs are silently dropped (tracked separately in Async analysis queue silently drops jobs on overflow; files stuck in PROCESSING forever #451).

Methodology

• Load generated with k6 (containerised, grafana/k6) driving the real user journey: POST /files (multipart upload) → poll GET /files/{id} until completed → fetch /analysis/{id}/summary, /analysis/{id}/protocols, /conversations/{id}.
• Each upload made unique (append a trailing tag) to avoid the duplicate-hash 409.
• Sweep executor: per-vu-iterations, increasing VUs, sampling backend container CPU/mem per level.
• Test file: sample-files/geotag_demo.pcap (538 KB, 1,890 packets), full pipeline (nDPI + Suricata + extraction).
• Caveat: run on a shared/contended dev box (backend got ~6 cores), not representative prod hardware, and a tiny pcap. Numbers below are indicative of shape/relative cost, not absolute capacity.

Findings

1. Concurrency sweep (per-analysis latency vs. concurrent uploads)

Concurrent uploads	Completed within 4 min	Analysis time (avg → p95)	HTTP errors
5	100%	66s → 76s	0%
10	100%	1m55s → 2m27s	0%
15	100%	2m43s → 3m41s	0%
20	42%*	3m20s → 4m+	0%
30	22%*	3m43s → 4m+	0%

* not failures — these exceeded the test's 4-minute poll cap; verified zero FAILED files. Backend CPU pegged ~6 cores throughout; memory (2–4 GB) was never the limiter. Failure mode is graceful degradation via queueing, not errors — until the queue overflows (see #451).

Breaking point on this hardware: ~20 simultaneous analyses before latency exceeds a 4-min budget. This scales with cores; it is not a fixed ceiling.

2. The dominant bottleneck: Suricata (controlled isolation test)

Pipeline config	Time per file
Full (nDPI + Suricata + extraction)	~53s
Suricata OFF	~3s
nDPI + Suricata OFF	~3s
Suricata OFF + extraction OFF	~2s

Suricata alone is ~94% of per-file analysis time, because it reloads the full Emerging Threats ruleset on every suricata -r invocation — a fixed cost independent of pcap size. Disabling it is an ~18× throughput gain.

3. Per-stage breakdown (from AnalysisService.analyzeFile timing logs, 538 KB file)

Stage	Time
1 Download (MinIO)	~2 ms
2 PCAP parse (tshark)	~0.8 s
3 Enrichment (nDPI + tshark + Suricata)	55–75 s
4 Signatures + classification + GeoIP	~0.8 s
5 Save analysis result	~1 ms
6 DB inserts	~0.13 s
7 File extraction	~0.9 s

Bottlenecks (ranked)

1. Suricata per-file ruleset reload — ~94% of analysis time, fixed per file. (Integrate Suricata IDS alerts into network viz, change detection & intelligence; make it configurable + document cost #452 covers config/cost.)
2. Async pool sizing — max-pool-size=10 caps concurrent analyses regardless of available cores (backend/.../config/AsyncConfig.java). On a many-core host this leaves CPU idle.
3. In-process, non-durable queue — bounded (100) and drops on overflow; lost on restart (Async analysis queue silently drops jobs on overflow; files stuck in PROCESSING forever #451).
4. Single-node monolith — one backend container owns the executor; no horizontal scale-out of ingestion.
5. Per-file fixed overhead generally — small files pay disproportionate process-startup cost (tshark/Suricata spawns); the 538 KB test is startup-dominated.

How to scale

Software (cheapest, biggest wins)

• Don't run Suricata per file on the bulk/ingest path. Make full IDS an on-demand action for files/windows under investigation. (~18×.)
• If kept on volume paths, run Suricata in persistent/socket mode or batch multiple pcaps per invocation to amortise the ruleset load.
• Right-size async.max-pool-size to physical cores (it's already @Value-backed) and document tuning.
• Make the queue durable + back-pressured (Async analysis queue silently drops jobs on overflow; files stuck in PROCESSING forever #451) before relying on high-rate, fire-and-forget ingestion.
• Externalise the work queue (DB work table or broker + N stateless workers) to scale ingestion horizontally beyond one node.
• Fix the duplicate-hash check (read-then-insert isn't atomic; needs a DB unique constraint) to avoid redundant work under concurrent uploads.

Hardware

• Ingestion is CPU-bound → more cores ≈ proportionally more analysis throughput (with the pool sized to match).
• Storage: put Postgres + MinIO hot set on SSD/NVMe; spinning disks hurt at scale. For large corpora, prefer windowed ingest + eviction over storing everything in MinIO.
• GPU is irrelevant to the analysis hot path (only the optional LLM-insights feature uses it).

Optimizations identified (summary)

• Suricata: on-demand / socket-mode / batched (biggest lever).
• Pool + durable queue tuning (Async analysis queue silently drops jobs on overflow; files stuck in PROCESSING forever #451).
• Atomic dedup (DB unique constraint on file hash).
• Finish Suricata integration so its cost actually produces value in network-viz/intelligence/change-detection (Integrate Suricata IDS alerts into network viz, change detection & intelligence; make it configurable + document cost #452).

Open questions / follow-up benchmarks

• Re-run on representative hardware (not a contended dev box).
• Benchmark with realistic large files (the 538 KB test under-measures size-dependent stages: parse, Suricata detection, extraction).
• Model a realistic mixed workload (users browsing between uploads) vs. the worst-case continuous-upload sweep here.
• Measure throughput with Suricata off and with pool sized to cores, to establish a real files/sec ceiling.

Related

• Async analysis queue silently drops jobs on overflow; files stuck in PROCESSING forever #451 — async queue silently drops jobs on overflow
• Integrate Suricata IDS alerts into network viz, change detection & intelligence; make it configurable + document cost #452 — Suricata integration + configurability + cost docs

---

🤖 Filed with Claude Code