Skip to content

Defer parsing the inherited OpenTelemetry tracestate member - #13028

Draft
dougqh wants to merge 5 commits into
dougqh/merge-aware-w3c-extractfrom
dougqh/lazy-inherited-ot
Draft

dougqh wants to merge 5 commits into
dougqh/merge-aware-w3c-extractfrom
dougqh/lazy-inherited-ot

Conversation

@dougqh

@dougqh dougqh commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Extraction no longer parses the inherited OpenTelemetry ot tracestate member on the request thread, and injection doesn't parse it either when the trace is relayed unchanged.

  • Deferred parse. The member stays raw in an InheritedSamplingState, which is ported from Defer parsing the extracted OpenTelemetry tracestate member #12987. The consistency checks made against it before the first read are recorded as two pending bits:

    • the compound merge's check;
    • the traceparent priority override;
    • external-override and unknown-mechanism reconciles.

    The first read parses the member, applies the pending checks, and publishes the result with a CAS. Local decisions, such as a manual or AppSec keep, still parse and rewrite the member as before.

  • No-op guard. A deferred decision that changes nothing installs no new state. Start extracted Datadog propagation tags from the extracted priority #13010 made the root span's reconcile of the extracted priority a no-op, and the deferred path keeps it that way.

  • Raw relay. SamplingState.getOtelTraceStateForWrite() returns the raw member when both of these hold:

    • the member is in the form OtelTraceState.parse keeps byte-for-byte;
    • rv >= th agrees with every pending check (OtelTraceState.isUnchangedByChecks, which allocates nothing).

    In that case the parsed member would equal the raw one, so the four W3C inject sites write it without parsing. Anything else is parsed as before.

Motivation

TraceSegment work (APMLP-2049): the per-member lazy tracestate described in the design comments there, starting with ot.

Since #12397, every extracted trace has parsed the ot member. Its readers are W3C injection, OTLP export and a local decision change. A leaf exporting to the agent never reads it. An intermediate relaying unchanged only needs to know that the member is unchanged.

Throughput and allocation: PropagationScenariosBenchmark, JDK 17, @Threads(8), 5 forks × 5 iterations, -prof gc, default sampler, base = #13015:

scenario upstream base B/op this PR B/op Δ B/op
leaf sameVersion 2001.8 ± 10.2 1943.7 ± 8.1 −58.2
leaf withOtelMember 2172.7 ± 43.2 2151.8 ± 22.0 −20.9
intermediate sameVersion 3714.2 ± 138.0 3638.1 ± 62.1 −76.0
intermediate withOtelMember 3776.6 ± 23.2 3728.0 ± 28.2 −48.6
  • Leaf skips the parse because nothing reads ot.
  • Intermediate skips it at inject through the raw relay. Without the relay, intermediate was flat (+5 to +8 B), because injection parsed the member anyway.
  • Leaf expectation: the deferred OtelTraceState is ~64 B and two snapshots are each ~16 B larger, so about −32 B. Both leaf arms are at least that.

Throughput is not settled yet. In the same runs the throughput was:

scenario upstream base ops/µs this PR ops/µs
leaf sameVersion 3.384 ± 0.122 3.327 ± 0.116
leaf withOtelMember 3.020 ± 0.056 3.134 ± 0.113
intermediate sameVersion 1.797 ± 0.148 2.035 ± 0.098
intermediate withOtelMember 1.754 ± 0.087 1.818 ± 0.089

The host drifted during these runs, so this doesn't support a claim either way:

  • The same base jar measured 2.182 ops/µs on intermediate / sameVersion two hours earlier, about 18% faster.
  • The load average fell from about 18 to 10 during this PR's run.

An interleaved A/B run will follow: alternating single-fork runs of the two jars, -t 4, load recorded per fork. A deferral-only run (without the raw relay) on the same host earlier showed leaf about level and intermediate about −2%, both within error. That is what motivated adding the relay.

Additional Notes

Output is unchanged.

  • Golden corpus. CompoundExtractionGoldenTest (650 curated cases, from Merge a trace context without building its propagation tags #13015) passes, and the 5110-case cartesian corpus is identical to the pre-Merge a trace context without building its propagation tags #13015 dump.
  • Check bits. Dropping either pending-check bit fails the golden (12 and 24 cases).
  • No-op guard. Extraction never reaches it with a needed check, so it is pinned by DeferredOtelCheckTest, which fails if the guard ignores the check bits.
  • Raw relay. OtelMemberForWriteTest asserts that, for 16 member forms (including ones the parser normalizes or rejects) across all four check combinations, the written content equals the parsed member's.
    • Mutating the fast path's consistency logic fails both this test and the golden.
    • Mutating its canonical-form checks (accepting a duplicate rv or a malformed field) fails this test.
    • Dropping the original-size adjustment in estimateHeaderSize changes nothing observable. That value only sizes the StringBuilder.
  • From Defer parsing the extracted OpenTelemetry tracestate member #12987. InheritedSamplingStateTest and InheritedOtelExtractionTest are ported unchanged and pass.

Tests: the full dd-trace-core suite passes locally on JDK 17, except for 2 Docker-dependent TracerConnectionReliabilityTest cases and two tests that fail only under full-suite load and pass on their own (PendingTraceBufferTest.testingTracerFlareDumpWithMultipleTraces, OtlpMetricsServiceTest.collectionAndTransportExceptionsAreRecordedAsFailures).

Supersedes #12987, which stays a reference.

Stacked: master ← #13008 ← #13000 ← #13009 ← #13010 ← #13015 ← this PR.

🤖 Generated with Claude Code

@dougqh dougqh added comp: core Tracer core tag: performance Performance related changes tag: no release notes Changes to exclude from release notes type: refactoring comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM labels Oct 23, 2026
@datadog-datadog-us1-prod

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.02 s 13.94 s [+0.1%; +1.1%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.87 s 12.95 s [-1.4%; +0.0%] (no difference)
startup:petclinic:appsec:Agent 17.09 s 16.97 s [-0.1%; +1.6%] (no difference)
startup:petclinic:iast:Agent 16.26 s 16.59 s [-9.1%; +5.1%] (unstable)
startup:petclinic:profiling:Agent 16.48 s 16.89 s [-3.4%; -1.4%] (significantly better)
startup:petclinic:sca:Agent 17.05 s 16.92 s [-0.2%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.19 s 16.18 s [-0.9%; +1.0%] (no difference)

Commit: f75723e8 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

dougqh and others added 5 commits October 23, 2026 16:14
Extraction parsed the inherited ot member and checked it against the
inherited decision on the request thread, though only W3C injection,
OTLP export and a local decision change read it. The member is now kept
raw in an InheritedSamplingState and parsed on first read, with the
consistency checks made before then recorded as pending. The compound
merge and external-override or unknown-mechanism reconciles record a
check instead of parsing; a deferred decision that changes nothing
installs no new state.

Ported from #12987 onto merge-mode extraction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Includes the equivalence and extraction tests from #12987 and a test
that a deferred decision repeating the priority still records a new
consistency check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
W3C injection read the inherited ot member through its parsed form, so
an intermediate service relaying a trace still parsed it at inject. When
the unparsed member is in the form the parser keeps as is and its
pending consistency checks all hold, the parsed member would equal it,
so injection now writes the raw member. Anything else is parsed as
before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also drops an unused accessor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: context propagation Trace context propagation comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant