Drop APM traces when APM tracing is disabled and keep LLM Observability working

[ncybul]

What Does This Do

Two fixes for running LLM Observability without APM tracing (solves #10051):

1. DD_APM_TRACING_ENABLED=false now drops APM traces. The standalone sampler was only selected when ASM/IAST/SCA was enabled; every other configuration — including LLM Observability — fell through to the default sampler, which keeps traces, so APM traces kept flowing. On distributed traces the drop was defeated a second way: an extracted sampling priority is locked onto the span context at construction, so a request arriving with x-datadog-sampling-priority: 1 silenced the drop before it was ever decided locally.
2. DD_TRACE_ENABLED=false with DD_LLMOBS_ENABLED=true no longer throws. LLMObsSystem.start() installed the real SDK implementation even with no tracer installed, and AgentTracer.get().buildSpan() returns null on the no-op tracer, so every LLMObs.start*Span() call threw a NullPointerException out of application code. The SDK now stays no-op unless a tracer is installed — including the CI Visibility case, which installs one even with DD_TRACE_ENABLED=false.

Motivation

DD_APM_TRACING_ENABLED=false is documented as working in all SDKs. LLM Observability users who don't want APM tracing had no working way to turn it off: the APM flag was a no-op, and DD_TRACE_ENABLED=false broke the LLMObs SDK.

Additional Notes

Approach. This is a reduced port of #10989, which had gone stale. That PR introduced a ProductTraceSource.LLMOBS bit plus a StandaloneProduct enum and a unified StandaloneSampler replacing AsmStandaloneSampler. Neither is needed: 0x20 is now taken by AI_GUARD, no other tracer allocates an LLMObs _dd.p.ts bit (dd-trace-js and dd-trace-py both have exactly APM/ASM/DSM/DJM/DBM/AI_GUARD), and LLMObs spans ship to the LLM Observability intake (/evp_proxy/v2/api/v2/llmobs), which is independent of APM sampling priority. That leaves the selection logic other tracers already have — compare getProductRateLimiter in dd-trace-js and LLMObsProcessor.process_trace in dd-trace-py:

• ASM/IAST/SCA enabled → unchanged AsmStandaloneSampler: 1 APM trace per minute for the service catalog.
• otherwise → new ApmTracingDisabledSampler: drops every APM trace, but still hands it to the writers, which is what keeps the LLM Observability spans riding on it flowing to their own intake. It re-reads ActiveSubsystems.APPSEC_ACTIVE per trace, because Remote Configuration can activate AppSec after the sampler was built, and falls back to the ASM trickle when that happens.

Sampling priority. Three things could outrank the drop, and each is now handled narrowly:

• Extracted priority. DDSpanContext locks an extracted priority at construction, so the drop-all vote was silently refused on exactly the distributed traces the flag exists to suppress. ContextInterpreter.samplingPriorityOrDefault now leaves it UNSET when APM tracing is disabled and no product marked the trace (no _dd.p.ts), letting the local sampler decide — the same shape as dd-trace-js' standalone onSpanExtract hook. It sits in ContextInterpreter because every propagation style funnels through it, including Haystack, whose defaultSamplingPriority() override manufactures a SAMPLER_KEEP out of headers that carry no priority at all.
• manual.keep. It force-keeps the span before the sampler ever votes, which left the trace indexed and billed as APM. The drop is now exempt from the priority lock in SamplingMechanism.canAvoidSamplingPriorityLock. The exemption is limited to a drop — a DEFAULT keep still cannot override a priority someone else set — and matches dd-trace-js, whose standalone sampler honors manual.keep only alongside a product trace source.
• Data Jobs. Excluded from that exemption: it keeps its traces with a DATA_JOBS priority and has no _dd.p.ts bit set to protect it, so the drop would silently discard Spark traces.

ASM and AI Guard are unaffected throughout: TraceCollector.setSamplingPriorityIfNecessary already skips the sampler for _dd.p.ts-marked traces, so asm.keep and ai_guard.keep keep working exactly as before, which AIGuardSamplingTest covers.

Flush. TraceProcessingWorker now sweeps the secondary queue before completing a flush. The flush marker is only ever offered to the primary queue, so a trace published to the secondary queue could still be sitting there when flush() returned and close() interrupted the serializer thread. That was harmless for a sampled-out APM trace, but a dropped trace now carries LLM Observability spans that have to reach their intake regardless.

Testing.

• SamplerTest: APM traces are dropped when APM tracing is disabled, with and without LLM Observability enabled, and the dropped trace is still handed to the writers; manual.keep is overridden, but honored on ASM-marked traces; a Data Jobs keep survives; the ASM trickle resumes when AppSec is activated at runtime.
• LLMObsSystemTest: the SDK stays no-op when tracing is disabled.
• DatadogHttpExtractorTest.ApmTracingDisabled: an upstream SAMPLER_KEEP is dropped when no product marked the trace, and preserved when _dd.p.ts is set.
• TraceProcessingWorkerTest: a trace sitting in the secondary queue is dispatched by a flush.
• Existing AIGuardSamplingTest, ApmTracingDisabledChunkMarkerTest and the apm-tracing-disabled smoke tests pass unchanged.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
• Update the CODEOWNERS file on source file addition, migration, or deletion — not needed, all touched paths are already covered
• Update public documentation with any new configuration flags or behaviors

Claude session: ba8ff7f0-b73e-41d6-a806-3603ab1c7329
Resume: claude --resume ba8ff7f0-b73e-41d6-a806-3603ab1c7329

🤖 Generated with Claude Code

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.00 s	13.96 s	[-0.5%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent	12.99 s	13.02 s	[-1.1%; +0.8%] (no difference)
startup:petclinic:appsec:Agent	17.07 s	16.99 s	[-0.5%; +1.5%] (no difference)
startup:petclinic:iast:Agent	16.95 s	16.95 s	[-0.7%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	16.36 s	16.88 s	[-4.1%; -2.1%] (significantly better)
startup:petclinic:sca:Agent	17.08 s	16.97 s	[-0.2%; +1.5%] (no difference)
startup:petclinic:tracing:Agent	16.24 s	16.05 s	[+0.3%; +2.1%] (maybe worse)

Commit: a946fca1 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[sabrenner]

makes sense for agent obs, probably just needs java tracer signoff!

[datadog-datadog-prod-us1]

A sampled parent can lock a keep priority before the new sampler runs. The sampler then fails to drop that normal APM trace when APM tracing is off.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 5923093 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 592309351a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[mhlidd]

One comment about expected behavior

[mhlidd]

LGTM following addressing Codex and Copilot comments 👍

[datadog-datadog-prod-us1]

The new drop sampler cannot replace manual keep decisions. APM traces can still leave the process when APM tracing is off. Remote AppSec activation and shutdown after single-span sampling also have trace-loss paths.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit fe764e7 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[ncybul]

@codex

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a153f2dacf

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[jandro996]

LGTM! thanks for the changes!

[datadog-datadog-prod-us1]

With APM tracing disabled, configured span sampling can leave LLMObs spans undelivered at shutdown. Mixed propagation formats can also lose product keep decisions, and repeated ASM sampling can overwrite the service-catalog keep.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 56f612a}