[NO COMMIT][SLES-3059] debug build: log TCP connects and HTTP/2 frames at debug level

[lym953]

Do not merge. This branch is extension v99 plus one commit, for a custom layer for one support ticket. The reusable change for main is #1413 and #1429.

Problem

A customer's invocations run until their 25 s timeout after the handler returns. In the extension's debug logs, the flush to Datadog ends at the line Attempting request, and nothing follows until the timeout. The logs cannot show whether the TCP connect, the TLS handshake or the reply stalled.

The customer runs extension v99. A custom build from main would include 41 more commits, some of which change the flush path, such as the libdatadog crate that sends traces. So this build starts from v99, and its only change is the extra logs.

This PR

When DD_LOG_LEVEL is debug, the extension also logs at debug level for two HTTP libraries:

• hyper_util, the library that opens the TCP connections, logs connecting to <ip>:<port> and connected to <ip>:<port>.
• h2, the HTTP/2 library, logs each frame that the extension sends or receives, for example send frame=Headers { … }.

The extension sends its own logs to Datadog, and these libraries also log the requests that send those logs. So their lines go to CloudWatch Logs only:

• The formatter starts each of these lines with {"target":"<target>", and writes the event's other fields, such as the h2 frame, as name=value.
• The Lambda logs processor does not send extension records that start that way.

The customer changes only the extension layer. The function already has DD_LOG_LEVEL=debug.

Testing

Automated tests

Steps

1. cargo test --locked --lib -- logger:: logs::
2. cargo clippy --all-targets --locked

Result

81 tests pass, including the 3 new logger tests and the new processor test. Clippy passes.

End-to-end tests in Lambda

Steps

1. Built a layer from this branch with the script that publishes a layer to the sandbox account, scripts/publish_bottlecap_sandbox.sh (x86_64, not FIPS, us-east-2).
2. Ran Python 3.13 functions with that layer and DD_LOG_LEVEL=debug, and no other new variable:
   • Stall: the metrics and trace URLs point at a listener in the handler that accepts TCP and never answers.
   • Dropped packets: the function runs in a VPC subnet with no route to the internet.
   • Capture: the logs URL points at a server in the handler that answers 202 and counts the lines that it receives.

Result

Case	Last HTTP library lines before the stall, in CloudWatch Logs
Stall	connected to 127.0.0.1:9443, then no other line for that connection until the 25 s timeout
Dropped packets	connecting to <ip>:443 for 3 intake IPs, and no connected to

Check	Result
HTTP library lines in the flush to the capture server	0, with 30 extension lines
h2 lines	send frame=Headers, send frame=Data, received frame=Headers, received frame=Data
API key in HTTP library lines	0 lines

The trace flush uses HTTP/1, so it shows only the hyper_util connect lines.

🤖 Generated with Claude Code

[datadog-datadog-prod-us1]

 

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

DataDog/datadog-lambda-extension | publish layer e2e sandbox (amd64)

View more details · View in GitLab

DataDog/datadog-lambda-extension | publish layer e2e sandbox (amd64, fips)

View more details · View in GitLab

ℹ️ Info

🔄 Datadog auto-retried 2 jobs - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9ccd537 | Docs | View more details | Give us feedback!}

[Copilot]

🔵 Needs a closer look

This customer-specific diagnostic branch is explicitly marked “Do not merge” and requires human release approval.

0 open findings

What changed in this PR

Adds customer-specific diagnostics to extension v99 to help locate stalled flushes without introducing unrelated changes from main.

Changes:

• Enables TCP connection and HTTP/2 frame debug logs.
• Marks HTTP library logs and excludes them from Datadog forwarding, retaining CloudWatch visibility.
• Adds formatter and log-filtering tests.

File	Description
bottlecap/​src/​logs/​lambda/​processor.rs	Drops HTTP library records and tests exclusion.
bottlecap/​src/​logger.rs	Marks HTTP library logs, includes event fields, and tests formatting.
bottlecap/​src/​bin/​bottlecap/​main.rs	Enables hyper_util and h2 diagnostics at debug/trace settings.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.