[SLES-3059] feat(logs): keep HTTP library log lines out of Datadog

[lym953]

Problem

What is true today. When a flush to Datadog stalls, the extension's debug logs end at the line Attempting request. They cannot show whether DNS, the TCP connect, TLS or the response stalled. A customer's invocations ran to their 25 s timeout this way, and the logs could not tell us where. The HTTP libraries that make these flushes (h2, hyper, reqwest and rustls) log each of these steps, but the extension turns their logs off.

What #1413 adds. A new environment variable, DD_LOG_LEVEL_BY_TARGET, sets the log level for one target. It can turn the four HTTP libraries on.

The problem. The extension sends its own stdout to Datadog as logs. These libraries log every request, including the requests that send those logs. So their lines would go into the next flush, and each flush would log lines for the next one to send.

This PR

• Lines from the four HTTP libraries go to CloudWatch Logs only. The formatter starts each of them with {"target":"<target>",, and the Lambda logs processor drops extension records that start that way. In the same way, the Go extension keeps its failover message out of the logs that it sends.
• The formatter also writes the other fields of these lines as name=value. h2 logs each frame in a field, so its lines show send frame=Headers { stream_id: StreamId(1), … }. h2 leaves header values and body bytes out of this output.
• The extension installs its logger with init(). This call of tracing-subscriber, the library that writes the extension's logs, also forwards records from the log crate to the logger.
• The default-off filter for these libraries is built from the same list.
• The logger README, which [SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target #1413 adds, says that these lines appear only in CloudWatch Logs.

This PR is stacked on #1413. Merge #1413 first and this PR right after it, because #1413 alone lets an entry turn these libraries on.

Impact

• Records from crates that use the log crate reach the logger at the configured level. Before, nothing received them. For example, the crate that obfuscates trace data, libdd-trace-obfuscation, logs Failed to parse DD_APM_REPLACE_TAGS with log.
• The extension's own lines do not change. Only lines from the four libraries get the target key and the extra fields.

Testing

Automated tests

Steps

1. cargo test --lib -- logger:: logs::
2. cargo clippy --all-targets and cargo fmt --check

Result

87 tests pass, including the 5 new logger tests and the new processor test. Clippy and cargo fmt --check pass.

Only these unit tests cover an extension record that arrives as a JSON object.

End-to-end tests in Lambda

Steps

1. Built a test layer from this change on top of [SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target #1413, with serverless-components main. The build used the script that publishes a layer to the sandbox account, scripts/publish_bottlecap_sandbox.sh (x86_64, not FIPS).
2. Ran Python 3.13 functions with that layer, DD_LOG_LEVEL=debug and DD_LOG_LEVEL_BY_TARGET=hyper_util=debug,h2=debug:
   • Capture: the logs URL points at a server in the handler that answers 202 and counts the lines that it receives.
   • Stall: the metrics and trace URLs point at a listener in the handler that accepts TCP and never answers.
   • Dropped packets: the function runs in a VPC subnet with no route to the internet.
3. Ran the stall and dropped-packet functions with the released extension v99 and DD_LOG_LEVEL=debug.

Result

Last lines before the stall, for the same function with the released layer and with the test layer:

Case	Released v99	Test layer
Stall: TCP accepted, no reply	Attempting request, then no other line for that request until the 25 s timeout	connected to 127.0.0.1:9443, then no other line for that connection until the 25 s timeout
Dropped packets	Attempting request, then Request failed with error after 15.5 s	connecting to <ip>:443 for 3 intake IPs, and no connected to

Checks on the test layer only:

Check	Result
HTTP library lines in the flushes to the capture server	0 in 4 flushes of 29–35 extension lines each
HTTP library lines in CloudWatch Logs, same function	270
h2 lines	send frame=Headers { … }, send frame=Data { … END_STREAM }, received frame=Headers { … }
API key in HTTP library lines	0 of 38 lines

These cases did not get lines from all libraries. rustls wrote no lines, because release builds have no rustls logging feature (only a dev-dependency turns it on). The trace flush showed only the connect lines, because it uses HTTP/1 through hyper 1.x, which is built without tracing.

🤖 Generated with Claude Code

[datadog-datadog-prod-us1-2]

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64)

View more details · View in GitLab

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b0ed4d7 | Docs | View more details | Give us feedback!}

[Copilot]

🟢 Approval recommended

The scoped formatting, filtering, and initialization changes have regression coverage and no identified blocking issues.

0 open findings

What changed in this PR

Keeps HTTP-library diagnostics in CloudWatch without forwarding them to Datadog, preventing log-flush feedback loops when target-specific logging is enabled.

Changes:

• Marks HTTP-library logs and excludes them from Datadog ingestion.
• Includes diagnostic fields and forwards log crate records through tracing.
• Adds regression tests and documents CloudWatch-only behavior.

File	Description
bottlecap/​src/​logs/​lambda/​processor.rs	Drops marked HTTP-library records; adds coverage.
bottlecap/​src/​logger/​README.md	Documents CloudWatch-only diagnostics.
bottlecap/​src/​logger/​mod.rs	Adds target markers, diagnostic fields, detection, and tests.
bottlecap/​src/​bin/​bottlecap/​main.rs	Shares default-off targets and initializes log forwarding.
bottlecap/​Cargo.toml	Adds the tracing-log dependency.
bottlecap/​Cargo.lock	Records the direct dependency.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.