feat(telemetry): USDT probes for tracing spans end - #257
Open
jsitnicki wants to merge 2 commits into
Open
Conversation
Our code bases are already instrumented with tracing spans, but there is no easy way to attach a BPF tracer to a specific span's start/end and measure its duration: one would have to craft custom uprobes at specific code offsets, which shift with every build, and then match span start and end events to each other across await points and thread hops, since an async span can be suspended and resumed on a different thread. This macro emits a per-span USDT probe at span end which receives the span duration as an argument, so a tracer can attach to a single span by name while all other spans pay no cost. The probe is armed via a per-span semaphore only while a tracer is attached and works regardless of span sampling.
Runs two spans instrumented with span_with_probe! in a loop, with distinct delay ranges so their durations can be told apart. Comes with a bpftrace script that attaches to both span end probes and prints duration histograms.
TheJokr
reviewed
Aug 24, 2026
| /// semaphore in `.probes`). The two `{}` operands are the semaphore symbol | ||
| /// and the duration argument. | ||
| /// | ||
| /// `usdt_provider` must not contain `:` (bpftrace attach points are |
Collaborator
There was a problem hiding this comment.
please add a check for this when expanding the macro
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Code bases which use
foundationsare already instrumented with tracing spans, but there is no easy way to measure the duration of a specific span with a BPF-based tracer, likebpftraceorebpf_exporter. One would have to craft custom uprobes at code offsets that shift with every build, and match span start/end events across await points and thread hops.Solution
span_with_probe!("span::name")— a drop-in replacement fortracing::span("...")that additionally emits a per-span USDT probe at span end,span_end_<sanitized name>, receiving the span duration (ns) as an argument.Key features:
tracing::span. aarch64 support to follow.examples/span_with_probe, to show the intended use.