MEDIUM: otel: fast path for non-recording (sampled-out) streams

[ygkat]

Summary

When the sampler decides not to record a trace, the root span is a non-recording span and everything attached to it is discarded by the SDK. The filter nevertheless keeps doing all of its per-scope work for the rest of the stream: it evaluates attribute/event/baggage/link/status samples, creates and finishes every child span and records exceptions. Only the export is saved, so a sampled-out stream costs almost as much as a fully recorded one.

rate-limit is the documented way to bound the filter's cost, and it works well for the first hop. It does not cover the sampler path, though: a stream left out by rate-limit emits no traceparent at all (downstream services start their own traces), and every downstream HAProxy that extracts a parent context with sampled=0 goes through the non-recording path regardless of its own rate-limit. This PR makes that path cheap by using is_recording, which the C wrapper already exposes for exactly this purpose.

Measurements

HAProxy 3.4.2, filter v2.2.0, wrapper v3.3.0, SDK 1.28.0, nbthread 4, h2load --h1 -c 64 -n 100000, loopback origin (http-request return), 9-span scope set with header injection.

configuration	max req/s	vs no filter
no filter	108k	100%
ratio 0.0 (nothing exported)	25.1k	23%
ratio 0.1	22.4k	21%
ratio 1.0	17.0k	16%
patched, ratio 0.0	52.3k	48% (2.08x)
patched, ratio 0.1	38.2k	35% (1.7x)
patched, ratio 0.5	24.7k	23% (1.3x)
patched, ratio 1.0	16.2k	unchanged (fast path not taken)

Changes

• include/scope.h: flt_otel_runtime_context gains flag_norec and root_span.
• src/event.c
  • flt_otel_scope_run_span(): after a root span is created, query is_recording; if it is not recording, set flag_norec and remember the root.
  • flt_otel_scope_run(): with flag_norec set, spans that do not inject a context are skipped entirely, and no attribute/event/baggage/link/status sample is evaluated for the spans that are still created; exceptions are skipped as well.
• src/scope.c flt_otel_scope_span_init(): with flag_norec set, a parent that was skipped resolves to the root span instead of failing.

Context propagation is preserved: injecting spans are still created (non-recording, sampled flag cleared), so traceparent keeps reaching downstream services and they follow the sampling decision. finish of a skipped span only triggers the existing debug warning. rate-limit, otel-stop and require-context are unaffected.

Verification

• ratio 1.0: span trees identical to before (two HAProxy tiers, SPOAs and instrumented Go/Java applications in one trace).
• ratio 0.0: requests unaffected (including http-after-response otel-group scopes), downstream receives traceparent with sampled=0, the collector receives no spans.

Remaining cost and possible follow-ups

A filter that is attached but creates no span runs at the no-filter baseline (101k), while a single non-recording root span costs about 7 µs per request (60.7k). That root span is unavoidable as long as the sampling decision is made inside StartSpan; perf attributes most of its cost to the C wrapper (span-handle table lookups, shared_ptr copies of Tracer/Span/Context, a make_shared<Context> per StartSpan) rather than to the SDK. Two follow-ups would close the remaining gap, both out of scope here:

• deciding the sampling outcome before the root span is created (a wrapper API exposing the sampler), so that sampled-out streams skip the wrapper entirely;
• injecting the root context directly instead of creating the injecting span (changes the propagated parent span id).

[zaga00]

Hello @ygkat,

Thanks for taking the time to open this PR and address this issue! I reviewed the approach and decided to implement it a bit differently, so I’ve added my own commit to address the same problem.

I really appreciate you taking the initiative to look into this and put together a solution. Even though I went with a different implementation, your PR helped highlight the issue and get it on my radar.

I’ve also pushed my alternative implementation to the pr-8 branch b3a8ae5 if you’d like to take a look at it. Feel free to compare the approaches and let me know what you think!

Thanks again for the contribution and for taking the time to improve the project!