Skip to content

Add span events serialization benchmark and dd-trace-core performance rules - #12876

Draft
dougqh wants to merge 2 commits into
masterfrom
dougqh/span-events-mapper-benchmark
Draft

dougqh wants to merge 2 commits into
masterfrom
dougqh/span-events-mapper-benchmark

Conversation

@dougqh

@dougqh dougqh commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

  • Adds SpanEventsMapperBenchmark (dd-trace-core JMH). It maps a finished single-span trace with span events and links through the v0.4, v0.5 and v1 mappers. Events are recorded through AgentSpan.addEvent when it exists, otherwise as the pre-encoded events tag the OTel shim used to set, so the same source runs before and after Improve span link encoding performance #12768.
  • Adds a dd-trace-core performance rules section to AGENTS.md, with matching stricter posture in the perf-review skill for dd-trace-core only:
    • performance claims need JMH before/after numbers (time and -Pjmh.profilers=gc allocation)
    • no Collections.unmodifiable* views or defensive copies returned from internal accessors

Rules for other modules are unchanged: still advice, not findings.

Motivation

#12768 was titled "Improve span link encoding performance" but shipped no benchmark; only the unrelated startup SLOs ran. This measures it.

Results: 100acde503~1 (before) vs 100acde503 (#12768), links=0, -f2 -wi5 -i5 -prof gc, JDK 21.0.4, Apple M1 Max:

mapper events ns before ns after Δ B/op before B/op after Δ
v0.4 small 460 874 +90% 336 1024 +205%
v0.4 small×10 499 2629 +427% 1688 8632 +411%
v0.4 exception 509 4270 +740% 1984 8744 +341%
v0.5 small 1229 1754 +43% 2104 2800 +33%
v0.5 small×10 1320 4686 +255% 3376 10256 +204%
v0.5 exception 1350 6482 +380% 3672 10368 +182%
v1 small 2686 1069 −60% 7112 2064 −71%
v1 small×10 17256 1922 −89% 32457 2568 −92%
v1 exception 7387 972 −87% 15056 3640 −76%

Spans without events are unchanged within error for every mapper.

How to read this:

  • v1 is a large, measured win. The old path parsed the JSON events tag back before re-encoding it natively.
  • v0.4/v0.5 writer-side cost went up, by design. Before Improve span link encoding performance #12768 the OTel shim built the events JSON on the application thread at end(), so the "before" column excludes that work; after, the writer thread does it. Whether this is a net win for v0.4 (the default protocol) depends on the application-thread cost, which this benchmark does not measure, since the old encoder lived in the OTel shim. A follow-up benchmark through the OTel API is needed to settle that.
  • The exception case on v0.4 (+3.8 µs per span) is driven by DDSpanContext.appendJsonString escaping the stack trace character by character; worth a look regardless.
  • With links=2, the links add 24 B/op more after than before for v0.4 and v1, and 8 B/op for v0.5. That's consistent with the per-call unmodifiableList wrapper in DDSpan.getLinks(), but not attributed with a profiler.

Raw results: results.txt for both runs are available on request.

Additional Notes

Run with:

./gradlew :dd-trace-core:jmh -Pjmh.includes=SpanEventsMapperBenchmark -Pjmh.profilers=gc -PtestJvm=21

To reproduce the before column, copy the benchmark onto 100acde503~1.

Jira ticket: APMLP-2039

🤖 Generated with Claude Code

dougqh and others added 2 commits October 14, 2026 13:33
Maps a finished single-span trace with events and links through the
v0.4, v0.5 and v1 mappers. Events are recorded through
AgentSpan.addEvent when it exists, otherwise as the pre-encoded events
tag, so the same source runs before and after #12768. The Javadoc
records the #12768 before/after results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AGENTS.md now requires JMH before/after numbers for performance claims
and forbids unmodifiable wrappers on internal accessors in dd-trace-core.
The perf-review skill applies these as findings for dd-trace-core and
keeps its advisory posture elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added comp: core Tracer core type: documentation tag: performance Performance related changes tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM labels Oct 14, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.62 s 14.57 s [-0.4%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent 13.41 s 13.53 s [-1.8%; +0.0%] (no difference)
startup:petclinic:appsec:Agent 17.03 s 17.02 s [-0.8%; +0.9%] (no difference)
startup:petclinic:iast:Agent 17.01 s 17.08 s [-1.1%; +0.3%] (no difference)
startup:petclinic:profiling:Agent 16.89 s 16.55 s [+1.1%; +3.0%] (significantly worse)
startup:petclinic:sca:Agent 17.08 s 16.90 s [+0.3%; +1.9%] (maybe worse)
startup:petclinic:tracing:Agent 16.14 s 16.18 s [-1.2%; +0.8%] (no difference)

Commit: 8666f8f0 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes tag: performance Performance related changes type: documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant