Skip to content

scenarios: add otap-rust collector scenario - #19

Merged
bwplotka merged 3 commits into
mainfrom
bwplotka/add-otap-rust-scenario
Nov 7, 2026
Merged

bwplotka merged 3 commits into
mainfrom
bwplotka/add-otap-rust-scenario

Conversation

@bwplotka

@bwplotka bwplotka commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Adds the otap-rust scenario under manifests/scenarios/otap-rust to benchmark the Rust OTAP Dataflow Engine (df_engine) scraping avalanche with urn:otel:receiver:prometheus (prometheus/scrape.rs, open-telemetry/otel-arrow#4630) and exporting via urn:otel:exporter:otlp_grpc + urn:otel:extension:gcp_auth (open-telemetry/otel-arrow#4650) to Google Cloud Telemetry (https://telemetry.googleapis.com:443).

@bwplotka
bwplotka force-pushed the bwplotka/add-otap-rust-scenario branch from 0dec0c2 to 98a483b Compare November 4, 2026 15:32
@bwplotka bwplotka changed the title scenarios: add otap-rust and gmp-otap-rust collector scenarios scenarios: add otap-rust collector scenario Nov 4, 2026
@bwplotka
bwplotka force-pushed the bwplotka/add-otap-rust-scenario branch 2 times, most recently from 8bd066f to b0b124f Compare November 4, 2026 17:10
Adds the manifests/scenarios/otap-rust benchmark scenario for the Rust OTAP Dataflow Engine (df_engine) scraping the 16-pod avalanche deployment via urn:otel:receiver:prometheus (backed by prometheus/expfmt-rs) -> exporter:noop to measure HTTP scrape and Arrow decoding throughput, CPU, and memory in isolation.

Also adds a headless Kubernetes Service for avalanche so DNS service discovery resolves all 16 avalanche pod IPs, and updates core Prometheus/Parca configs to scrape df_engine's /api/v1/metrics endpoint and profile the otap-collector container.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
@bwplotka
bwplotka force-pushed the bwplotka/add-otap-rust-scenario branch from b0b124f to 87eb88e Compare November 4, 2026 17:14
…y.googleapis.com

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…tch limit

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
@bwplotka
bwplotka merged commit 36de1f2 into main Nov 7, 2026
1 check passed
@bwplotka

bwplotka commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner Author

Results

Initial results are spectacular. I couldn't believe when I saw it. Scalable metrics performance of like ~30x in memory usage. Logging, tracing and profiling pipelines could be improved as well (at least 10x).

image

If true, this changes a lot.

We don't need some caches, we can deprioritize various scalability work (e.g. complex VPA for DaemonSets), we can mitigate stockout problems, we effectively will save hundreds of millions of USD per year at Google, on collection alone around my space (yes, I calculated), easily $4+B across the world.

This is the evidence I and others in community needed that we should literally stop investing in Go implementations (Rust slow dev AI velocity is effectively solved) and row-based design for collections and backends (you will see some fun proposals in Prometheus and Thanos space soon!).

I am consulting those results with otel-arrow SIG. Huge thanks to otel-arrow folks and especially @lquerel for help. Thanks to @nicolastakashi for the collab on prometheus/scrape.rs#31 so far, and Saswata, Michael, Frederic, Matthias, Filip, and Jesus for inspiration for the columnar work.

Let's keep the good work 💪🏽

Disclaimer

Currently benchmarked version does NOT have a full parity. I am sending all metrics 1:1 across those scenarios, but:

On the other hand, we are converting Otel Arrow to inefficient OTLP (row based), so with OTAP this pipeline should be even better (assuming backends adopt OTAP).

WHY SO GOOD?

  • Go has GC (GOGC=100 2x heap multiplier) vs. Rust deterministic RAII deallocation in Rust
  • We don't have/need scrape cache, fully stateless pipeline.
  • We stream EVERYTHING into 200-point Arrow batches (20-30 KiB), including parsing (!). Scrapes are also sequential (lol):
image
  • Zero-copy Arrow spans, everything reuses memory buffers as much as possible.
  • Advanced interning (StringInterning).
  • No WAL (Prometheus has persistent WAL). Again, often not needed (at Google we don't need it).
  • We don't hold memory for rebuilding labels (labels.Builder) because allocs are cheap, no need for memory pooling (sync.Pool sucks so much in Go, I am so happy it's gone)

@lquerel

lquerel commented Nov 11, 2026

Copy link
Copy Markdown

Thanks, @bwplotka , for working on this integration. It's really exciting to see the impact otap/otel-arrow can have on this type of workload.

@bwplotka

Copy link
Copy Markdown
Owner Author

For completion, here are full parity results: #20 (comment)

@lomolabs

lomolabs commented Nov 18, 2026 •

Copy link
Copy Markdown

Independent reproduction (self-hosted, not GKE)

Tried to reproduce on a qualification lab: 4 vCPU / 16 GiB worker VMs, one collector at a time on a dedicated worker, VictoriaMetrics sink over OTLP/HTTP and remote_write, cAdvisor metrics from a separate Prometheus. Builds: bwplotka/otel-arrow@84984294 (otap-bench-v4, your manifest config incl. relabel set) and upstream 53e1a0c4, plus Prometheus v3.3.0 --agent and otelcol-contrib 0.165.0 (batch as in gmp-otel).

Memory: reproduced.

Load otap v4 otap main prom agent otelcol
your flags, 1 h churn (159k series), p50 33 MiB 24 MiB 840 MiB 615 MiB
2× series-count, no churn (318k), p50 38 MiB 33 MiB 1,161 MiB 1,044 MiB

Rust stays flat through churn; the agent gains ~260 MiB per churn hour.

CPU: not reproduced here. At 318k series: otap v4 0.68 / main 0.58 cores vs agent 0.19 / otelcol 0.32. Not the exporter: with http.compression: zstd TX drops 80× (6.3 MiB/s → 77 KiB/s) and CPU stays at 0.235 (gzip 0.249, none 0.241); main without the relabel set is also ~2× the agent at 159k. Remaining suspects: my -C target-cpu=x86-64-v2 build and VM vs GKE nodes; I haven't profiled yet. Still worth making gzip/zstd the otlp_http default.

Scale edges (1 core):

  • main scrapes targets sequentially and skips missed ticks: at 636k series (16 × 17.6 MiB bodies) it completed ~35 % of rounds, with no WARN and no overrun counter. v4's concurrent scheduling + scrapes_overrun look important to upstream.
  • v4's default max_body_bytes (16 MiB) rejects every 17.6 MiB body, so 0 output while the pod stays Ready (WARN + scrapes_failed do fire). With max_body_bytes: 64 MiB the same load gives 51 % delivery at 0.99 cores, CPU-bound on the single core, with 387 scrapes_overrun counted. That's the visible failure mode main lacks. At 1.27M series (35 MiB bodies) main dropped to 12 %: ~8 s per scrape × 16 sequential ≈ 130 s per 15 s round.

Questions: (1) With your flags, public avalanche v0.7.0 gives exactly 9,940 series per target, i.e. 159k / ~5.5k OTLP pts/s for 16 targets, yet the chart says 344,960 series and the soak 11,563 pts/s (~2.1×). Does v0.6.0-main-bw6 emit more, or is that counting churn generations? (2) How was the 0.103 cores computed, and with which target-cpu / exporter compression?

Small fix: cross-arch-build.sh installs gcc-x86-64-linux-gnu but never sets CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc, so arm64→amd64 builds fail at link (cc: unrecognized option '-m64').

Full tables, method and threats to validity are in the attached write-up (also happy to put it in a gist). Happy to contribute the self-hosted variant (kustomize, no GCP dependency, sink + watchdog) as a scenario if you want it.

scrape-rs-lab-reproduction.pdf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants