Strip _datadog trace context injected for Eventbridge, SNS, SQS

[eunskin]

What Does This Do

Strips the _datadog trace propagation carrier that Datadog injects into AWS Lambda event payloads (EventBridge, SQS, SNS) before the payload reaches the customer's handler. Disabled by default and gated by a new config flag DD_LAMBDA_STRIP_INJECTED_CONTEXT.

• Adds DD_LAMBDA_STRIP_INJECTED_CONTEXT (default false) to GeneralConfig / Config / supported-configurations.json.
• Adds StripInjectedContext as a helper in aws-java-lambda-handler-1.2. It uses datadog.json (already on the bootstrap classpath), so there are no Moshi or CoreTracer / AgentTracer API changes.
• LambdaHandlerInstrumentation calls notifyLambdaStart with the original stream first, so the Datadog Extension still extracts trace context from the unmodified event. When the flag is enabled, the advice then replaces the handler's InputStream argument with the stripped payload.

How stripping works

• Single pass over the JSON string tokens; the payload is never parsed into a tree or re-serialized, so numbers, key order and formatting of customer data are kept byte for byte.
• A "_datadog" key with an object value is cut out together with one adjacent comma, so the result stays valid JSON.
• A string value containing _datadog that looks like a JSON object (starts with {, ends with }) is decoded, stripped recursively up to 3 levels (SNS Message, SQS body, SNS-to-SQS fan-out) and re-encoded.
• Only object-shaped carriers are removed; a _datadog key with any other value is treated as customer data and kept.
  Fail-open: the original payload is used whenever stripping could alter customer data
• Object keys are never rewritten, even if they look like encoded JSON.
• Only ByteArrayInputStream is handled, read with mark/reset like LambdaAppSecHandler / LambdaEventParser. Other stream types are passed through unread.
• Payloads that contain U+FFFD (for example invalid UTF-8) are returned unchanged.
• Unclosed carriers stop the scan instead of rescanning, which keeps the work linear on malformed input.
• Any RuntimeException falls back to the original bytes.

Motivation

When Lambda tracing injects _datadog propagation context into an event payload, the extra key leaks into the customer's business payload and can break strict schema validation or unexpected-field checks on the consumer side. This strips the injected context before the handler sees it, matching the Go implementation in dd-trace-go#5054.

Testing

• StripInjectedContextTest: unit tests for comma placement, whitespace, nested and string-encoded carriers, non-object values, object keys, unsupported streams, invalid UTF-8, free text, and a parameterized suite with the 24 test cases from the Go PR (23 identical results, see the difference below).
• StripInjectedContextFuzzTest: Go-style fuzz test with a seed corpus, a regression corpus, and three randomized properties (never throws and always removes carriers; valid JSON stays valid and stripping is idempotent; arbitrary carriers are removed and sibling values are unchanged). The seed is logged and can be replayed with FUZZ_SEED, and FUZZ_ITERATIONS scales the run. Locally passed 30 runs x 100k iterations with different seeds.
• LambdaHandlerInstrumentationTest: end-to-end check that the handler receives the stripped EventBridge payload.
• ConfigTest: default value and enabling the flag.

Benchmark Results (JMH, avgt, us/op)

Payload	Carrier present	Before (Moshi)	After
EventBridge	yes	0.916	0.227
SQS	yes	0.599	0.147
SNS	yes	1.244	1.154
EventBridge	no	0.018	0.012
SQS	no	0.015	0.005
SNS	no	0.018	0.009

The no-carrier path, which is the common case, returns after a single byte scan for _datadog.

Additional Notes

• Disabled by default; no behavior or performance change unless DD_LAMBDA_STRIP_INJECTED_CONTEXT=true.
• Supported setup is dd-trace-java with the Datadog Extension. Code that reads _datadog from the event inside the handler (for example DDLambda(SQSEvent, Context) from datadog-lambda-java used without the Extension) will no longer see it when the flag is enabled.
• Differences from the Go implementation, both intentional:
  • Whitespace before the colon ("_datadog" : {...}) is also stripped.
  • String-encoded carriers are handled in any string value up to 3 levels deep, not only in top-level fields. Key order inside re-encoded strings is kept; non-ASCII characters and / are escaped as \uXXXX and \/ (semantically identical JSON).
• Kinesis is not covered: its record data is base64 encoded, so the carrier is not visible in the event JSON.
• Test-only dependency on jackson-databind for strict JSON validity checks in the fuzz test.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [APMSVLS-313]

[datadog-prod-us1-6]

Common EventBridge-to-SQS/SNS envelopes retain _datadog because encoded payloads beneath Records are never visited. Legal whitespace can also evade stripping, while non-object customer _datadog values can be deleted.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 832808b · @DataDog review to ask questions}

[PerfectSlayer]

❔ question: ‏About design, why is StripInjectedContext in dd-trace-core when is seems only used in lambda instrumentation?

[eunskin]

@PerfectSlayer Thank you for taking a look at it. Yes, it could be moved. The main reason it lives in dd-trace-core is that the instrumentation module cannot directly import it, so it has to be accessed via TracerAPI anyway, and CoreTracer (which implements TracerAPI) is in dd-trace-core. Moving it to a separate shared module is possible. Happy to refactor if the team prefers a different boundary. Additionally, I will address the code review comments as well.

[PerfectSlayer]

Why can’t it be imported if declared as an helper?
Related doc: https://github.com/DataDog/dd-trace-java/blob/master/docs/add_new_instrumentation.md#add-helper-class-names

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.74 s	14.64 s	[-0.1%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent	13.52 s	13.63 s	[-1.4%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent	17.01 s	16.88 s	[-0.1%; +1.7%] (no difference)
startup:petclinic:iast:Agent	16.90 s	16.95 s	[-1.1%; +0.5%] (no difference)
startup:petclinic:profiling:Agent	16.73 s	16.67 s	[-0.8%; +1.5%] (no difference)
startup:petclinic:sca:Agent	16.56 s	16.87 s	[-6.2%; +2.5%] (no difference)
startup:petclinic:tracing:Agent	16.08 s	15.33 s	[-1.0%; +10.8%] (unstable)

Commit: 3347b6b3 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[eunskin]

@PerfectSlayer Thanks! I moved it to helper! :) Could you please take a look at it again?

[mhlidd]

Config portion looks good to me. Leaving the rest of the review for the Serverless team

[datadog-prod-us1-6]

When context stripping is enabled, the scanner can rename customer property names, and a stream-read failure can leave the handler with truncated input. Both violate the intended preservation of customer payloads.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit d0cb94b}

[ValentinZakharov]

The option DD_LAMBDA_STRIP_INJECTED_CONTEXT matches dd-trace-go exactly, which is great for cross-language consistency 👍

But it's a bit inconsistent with the other AWS integration flags in this tracer, which all follow the DD_TRACE_<service>_..._ENABLED pattern and live in TraceInstrumentationConfig

• DD_TRACE_EVENTBRIDGE_INJECT_DATADOG_ATTRIBUTE_ENABLED
• DD_TRACE_SNS_INJECT_DATADOG_ATTRIBUTE_ENABLED
• DD_TRACE_SQS_INJECT_DATADOG_ATTRIBUTE_ENABLED
• DD_TRACE_SFN_INJECT_DATADOG_ATTRIBUTE_ENABLED

Each of those also ships a non-TRACE alias (e.g. DD_SNS_INJECT_DATADOG_ATTRIBUTE_ENABLED)

Can we keep DD_LAMBDA_STRIP_INJECTED_CONTEXT as the primary name, but add DD_TRACE_LAMBDA_STRIP_INJECTED_CONTEXT as an alias in supported-configurations.json so it also matches the Java-side convention. Low effort, and it keeps both audiences happy.

[eunskin]

Hello @ValentinZakharov,
Thank you for the review! :)
I added DD_TRACE_LAMBDA_STRIP_INJECTED_CONTEXT as an alias (DD_LAMBDA_STRIP_INJECTED_CONTEXT stays primary) in metadata/supported-configurations.json and Config, with a ConfigTest case.
I'm waiting for the feature parity config update since I don't have update permission.