feat: Add `datadog-mock-intake` crate and serve `trace-agent` tests with it by lucaspimentel · Pull Request #174 · DataDog/serverless-components

lucaspimentel · 2026-09-25T16:42:02Z

What does this PR do?

Adds a shared datadog-mock-intake crate: an in-process mock Datadog APM intake served with hyper that accepts the APM endpoints serverless components flush to, decodes msgpack/protobuf payloads on arrival, and exposes typed query methods plus raw request capture for test assertions.

POST /api/v0.2/traces - APM traces and spans
POST /api/v0.2/stats - APM trace stats
POST /api/v0.1/pipeline_stats - DSM

Rewires the trace-agent integration tests onto the crate, replacing the test-local MockServer.

Motivation

The mock intake was originally built in datadog-lambda-extension (#1385) as a bottlecap test fixture and then promoted to a feature-gated library module plus a standalone mock-intake binary. It has already proven useful there:

It is the payload-level harness behind bottlecap's APM/DSM integration tests (19 APM + 1 DSM), which exercise the full flush path against a real HTTP intake instead of asserting on in-memory buffers.
Its stats failure injection (MOCK_INTAKE_FAIL_STATS_FIRST_N) is what lets those tests cover tracer retry behavior against 500 responses, including the guarantee that exactly the first N attempts are rejected.
As a standalone binary, it has served for local debugging against a live tracer and the test-mode trace processor: one summary line per request with hits grouped by the full stats aggregation key, plus JSON dumps of every decoded payload, make it possible to see exactly what a tracer flushed without wiring up a real backend.

The APM/DSM integration tests in datadog-lambda-extension (bottlecap) and datadog-trace-agent each carried their own intake fixture. Extracting the bottlecap mock intake into this shared crate lets both test suites use one implementation. A follow-up PR in datadog-lambda-extension (#1392) consumes this crate (stacked on this PR's revision), deletes the bottlecap-local copy, and supersedes #1385.

Additional Notes

Ported from the bottlecap mock intake (axum 0.8) to hyper 1.6 to match this workspace's existing HTTP stack; no axum dependency is introduced.
Behavior preserved from the original: endpoint set, gzip/zstd decoding, the 2 MiB wire-body limit (413 before decode), 404/405 routing, stats failure injection, request summaries, and JSON dumps.
Additions beyond the original: a 64 MiB cap on decompressed bodies (guards against decompression bombs on top of the wire-body limit), a pipeline_stats_url() accessor, and raw request capture (CapturedRequest, requests_for_path()). Typed query methods return only accepted, successfully decoded payloads; raw capture records completed POST attempts including rejected ones.
Robustness for test reliability: request decoding runs off the async runtime, raw and typed captures are recorded atomically under a single named mutex, non-UTF-8 header values are captured lossily instead of dropped, transient listener accept errors are logged with a brief backoff instead of silently stopping the server, and in-flight connections are aborted when MockIntake is dropped.
Behavior change for the trace-agent tests: the old test-local MockServer answered /api/v0.2/traces with 200 {"ok":true}; the mock intake answers every accepted payload with 202 and an empty body. libdd-trace-utils treats any non-4xx/5xx response as success, so trace flushing is unaffected.
New workspace dependencies (rmp-serde, flate2, zstd) are already covered in LICENSE-3rdparty.csv; dd-rust-license-tool check passes.

Describe how to test/QA your changes

cargo test -p datadog-mock-intake: includes ports of the original bottlecap intake's unit tests plus new transport regression tests (body limit, 404/405, raw-capture semantics, zstd decoding).
cargo test --workspace: passes.
cargo clippy --workspace --all-targets: clean. dd-rust-license-tool check: passes.
Trace-agent integration tests now assert on the intake's decoded capture instead of re-decoding captured bytes; the DSM forwarding test sends a valid msgpack payload and verifies both unchanged byte forwarding and intake acceptance.

lucaspimentel · 2026-09-25T22:55:09Z

@DataDog review

datadog-datadog-prod-us1-2

Two of the most critical issues are in datadog-mock-intake: accepted connection tasks are not cancelled on MockIntake drop, allowing traffic to leak across tests; and typed payload captures can record requests out of arrival order due to async decoding. Additionally, the no-stats integration test asserts against decoded stats_payloads() rather than raw captures, meaning a malformed stats POST would silently pass the assertion.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit a074193 · @DataDog review to ask questions}

lucaspimentel · 2026-09-29T14:52:46Z

@DataDog review

datadog-datadog-prod-us1-2

The mock intake can leave an accepted keep-alive connection running after teardown due to a registration race. The patch also fails the repository’s required rustfmt check in two locations.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit de90579 · @DataDog review to ask questions}

lucaspimentel · 2026-09-29T15:38:56Z

@DataDog review

datadog-datadog-prod-us1-2

Concurrent handlers append captures after independent blocking decodes complete, so the documented arrival order can be reversed and order-sensitive integration assertions can inspect the wrong request.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 5d7b1e2 · @DataDog review to ask questions}

lucaspimentel · 2026-09-29T18:30:22Z

@DataDog review

datadog-datadog-prod-us1-2

A client disconnect during blocking payload processing can permanently strand the capture sequence, leaving every later accepted payload invisible to query methods.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 463cde2 · @DataDog review to ask questions}

chatgpt-codex-connector · 2026-09-30T16:37:08Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-30T16:40:59.769786Z	`463cde2`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot

Copilot review overview

🟡 Changes recommended

A disconnected request can permanently wedge ordered capture, and connection errors and shutdown-test races need handling.

Review effort: Balanced
Findings: 1 · 2

Open (3)

Finalize request IDs even when clients disconnect · New
Handle and report connection errors · New
Allow expected write failures after server shutdown · New

What changed in this PR

Adds a reusable Hyper-based mock APM intake and migrates trace-agent integration tests to its typed payload capture APIs.

Changes:

Supports trace, stats, and DSM intake endpoints with decoding, limits, diagnostics, and failure injection.
Replaces the trace-agent’s local mock server.
Updates workspace dependencies and lockfile.

File	Description
`crates/datadog-mock-intake/src/lib.rs`	Implements the mock intake and tests.
`crates/datadog-mock-intake/Cargo.toml`	Defines crate dependencies.
`crates/datadog-trace-agent/tests/integration_test.rs`	Migrates integration tests.
`crates/datadog-trace-agent/tests/common/mod.rs`	Removes the old mock module.
`crates/datadog-trace-agent/tests/common/mock_server.rs`	Deletes the local mock server.
`crates/datadog-trace-agent/tests/common/helpers.rs`	Removes redundant decoding logic.
`crates/datadog-trace-agent/Cargo.toml`	Adds the mock intake dev dependency.
`Cargo.lock`	Records dependency resolution changes.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 463cde2ee8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lucaspimentel · 2026-09-30T17:51:50Z

@DataDog review

datadog-datadog-prod-us1-2

The lockfile downgrade makes default workspace builds resolve rustls 0.23.44, which the repository marks as affected by RUSTSEC-2026-0285; restoring 0.23.45 avoids extending the FIPS-only exception to default artifacts.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 6a78169 · @DataDog review to ask questions}

lucaspimentel · 2026-10-05T15:44:49Z

@DataDog review

datadog-datadog-prod-us1-2

The lockfile downgrades rustls from 0.23.45 to vulnerable 0.23.44, reintroducing RUSTSEC-2026-0285 for TLS 1.3 users.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit d54fc76 · @DataDog review to ask questions}

In-process mock Datadog intake that accepts the APM endpoints serverless components flush to (/api/v0.2/stats, /api/v0.2/traces, /api/v0.1/pipeline_stats), decodes msgpack/protobuf payloads on arrival, and exposes typed query methods plus raw request capture for test assertions. Ported from the bottlecap fake intake and served with hyper; preserves the 2 MiB wire-body limit, gzip/zstd decoding, stats failure injection, request summaries, and JSON dumps.

Replace the test-local MockServer with the shared datadog-mock-intake crate. Payload assertions now use the intake's decoded capture instead of re-decoding captured bytes, positive assertions wait on decoded payload arrival, and the DSM forwarding test sends a valid msgpack pipeline-stats payload so it verifies both unchanged byte forwarding and intake acceptance.

…-intake Guards decompress() against decompression-bomb-style memory blow-up by capping gzip/zstd output at 64 MiB. Also skips building the diagnostic JSON dump when no dump_dir is configured (the default), avoiding a wasted allocation on every request, and marks DSM pipeline-stats payload fields #[serde(default)] so a payload missing an optional field still decodes. 🤖

Unknown Content-Encoding values were silently decoded as identity, masking mismatches with the real intake, which rejects unsupported encodings. Rejected attempts are still raw-captured but do not consume the stats failure-injection budget; explicit identity encoding is still accepted.

Request ids were assigned inside the blocking handlers, so under concurrency dump and summary ids could interleave out of arrival order relative to the captured requests.

Summaries reported a fixed payload count of 1 per pipeline-stats request; report the number of decoded points like the other endpoints.

The dump serializer mirrors pb::AgentPayload fields by hand; an exhaustive struct literal in the test forces it to be updated whenever the protobuf struct gains a field, instead of silently dropping it from dumps.

Indexing decoded payload collections panicked with bare out-of-bounds errors; use first() with expects that name the failed assertion.

Drop the inline datadog_mock_intake qualifications in favor of the top-level import used for the other helper types.

Captured requests were appended after the blocking handler completed, so under concurrent clients a later-arriving request that decompressed or decoded faster could be recorded before an earlier one, breaking the arrival-order contract of the query APIs. Completed requests are now buffered by request id and flushed as a contiguous prefix in one critical section.

A request canceled by a client disconnect could leave its request id unrecorded, holding back every later capture from the query APIs. Captures are now recorded inside the blocking handler task, which always runs to completion, with handler panics recorded as rejected requests.

Connection failures such as malformed HTTP or requests cut off mid-message were silently discarded, making a test timeout look like a missing payload. They are now logged to stderr.

The keep-alive shutdown test required writes after Drop to succeed, but the server may already have closed the socket. Broken pipe and reset errors are now accepted as evidence of shutdown; the capture-count assertion is unchanged.

Request ids were assigned after the body was fully read, so a slow upload could be ordered after a later request. Ids are now assigned when the request head arrives. Requests that end without a capture (oversized or unreadable body, client disconnect) release their id so they cannot hold back later captures.

Drop the HandledRequest request_id field, which always echoed the handler's request_id parameter, and use the parameter directly at summary and dump call sites. Replace the three then_some/flatten accepted-payload constructs with Option::filter, and remove the always-false inject_failure parameter from decompress_failure_status. No behavior change.

Replace hand-written Display impl with thiserror, reuse the existing content-encoding helper, inline the one-use request-id helper, drop a redundant state clone, and only compute payload counts when summaries are enabled. Extract a shared helper for the duplicated marker-stats assertion in the trace-agent tests.

Captures were only exposed once every earlier request finished, so a 2xx did not guarantee the capture was visible, and a stalled request hid all later captures indefinitely. Captures are now visible as soon as the intake responds, still ordered by arrival.

test_internal_span_kind_does_not_produce_stats and test_peer_tags_in_flushed_stats read the intake capture immediately after shutdown; wait for it like the other tests to avoid timing-dependent failures.

The trace-agent verify helpers waited for decoded payloads, so a payload the intake rejected surfaced as a 60s timeout claiming no request arrived. Wait for the raw request, then assert it was accepted.

A Content-Encoding value that was not valid UTF-8 was treated as absent, so the body was silently decoded as identity. It is now rejected with 415 like any other unsupported encoding.

The dual-transport test byte-searched raw request bodies for the service names and checked only after the first trace POST arrived. It now waits until spans from both services appear in the decoded trace payloads.

This was referenced Sep 25, 2026

Use shared datadog-mock-intake crate and add standalone mock-intake binary DataDog/datadog-lambda-extension#1392

Draft

APMSVLS-494: add a standalone fake intake for local APM debugging DataDog/datadog-lambda-extension#1385

Closed

lucaspimentel changed the title ~~Add datadog-mock-intake crate and serve trace-agent tests with it~~ Add datadog-mock-intake crate and serve trace-agent tests with it Sep 25, 2026

datadog-datadog-prod-us1-2 Bot reviewed Sep 25, 2026

View reviewed changes

Comment thread crates/datadog-mock-intake/src/lib.rs Outdated

Comment thread crates/datadog-trace-agent/tests/integration_test.rs Outdated

datadog-datadog-prod-us1-2 Bot reviewed Sep 29, 2026

View reviewed changes

Comment thread crates/datadog-mock-intake/src/lib.rs Outdated

Comment thread crates/datadog-mock-intake/src/lib.rs

lucaspimentel changed the title ~~Add datadog-mock-intake crate and serve trace-agent tests with it~~ feat: Add datadog-mock-intake crate and serve trace-agent tests with it Sep 29, 2026

datadog-datadog-prod-us1-2 Bot reviewed Sep 29, 2026

View reviewed changes

Comment thread crates/datadog-mock-intake/src/lib.rs Outdated

datadog-datadog-prod-us1-2 Bot reviewed Sep 29, 2026

View reviewed changes

Comment thread crates/datadog-mock-intake/src/lib.rs Outdated

lucaspimentel marked this pull request as ready for review September 30, 2026 16:36

Copilot AI balanced review requested due to automatic review settings September 30, 2026 16:36

lucaspimentel requested review from a team as code owners September 30, 2026 16:36

lucaspimentel requested review from kathiehuang and lym953 and removed request for a team September 30, 2026 16:36

Copilot started reviewing on behalf of lucaspimentel September 30, 2026 16:37

View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

Comment thread crates/datadog-mock-intake/src/lib.rs Outdated

Comment thread crates/datadog-mock-intake/src/lib.rs Outdated

Comment thread crates/datadog-mock-intake/src/lib.rs Outdated

chatgpt-codex-connector Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread crates/datadog-mock-intake/src/lib.rs Outdated

datadog-datadog-prod-us1-2 Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread Cargo.lock Outdated

datadog-datadog-prod-us1-2 Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread Cargo.lock Outdated

DataDog deleted a comment from datadog-datadog-us1-prod Bot Oct 5, 2026

lucaspimentel added 2 commits October 19, 2026 09:42

lucaspimentel added 29 commits October 19, 2026 09:42

Abort accepted connections when mock intake is dropped

35e1043

Assert no raw stats requests in no-stats integration test

4c2ee5c

Keep mock intake serving after accept errors

c89bc92

Record raw and typed intake captures atomically

b311294

Decode intake requests off the async runtime

0a66d33

Keep non-UTF-8 header values in captured requests

2d9882a

Name the single capture mutex in panic messages

04ac689

Describe only this crate's API in mock-intake docs

40f7c1a

Apply rustfmt to mock-intake

9c24dd0

Abort connections racing with MockIntake shutdown

664fb6c

fix(mock-intake): assign request ids at arrival time

a7191dd

Request ids were assigned inside the blocking handlers, so under concurrency dump and summary ids could interleave out of arrival order relative to the captured requests.

fix(mock-intake): count pipeline stats points in summaries

eff6ff4

Summaries reported a fixed payload count of 1 per pipeline-stats request; report the number of decoded points like the other endpoints.

test(mock-intake): pin AgentPayload JSON dump fields

c92ba9b

The dump serializer mirrors pb::AgentPayload fields by hand; an exhaustive struct literal in the test forces it to be updated whenever the protobuf struct gains a field, instead of silently dropping it from dumps.

test(trace-agent): readable failures for empty decoded captures

c9a0ebd

Indexing decoded payload collections panicked with bare out-of-bounds errors; use first() with expects that name the failed assertion.

test(trace-agent): import PipelineStats helper types

af94d98

Drop the inline datadog_mock_intake qualifications in favor of the top-level import used for the other helper types.

fix(mock-intake): log connection errors

b882fe5

Connection failures such as malformed HTTP or requests cut off mid-message were silently discarded, making a test timeout look like a missing payload. They are now logged to stderr.

test(mock-intake): accept closed socket after Drop

e70d1ac

The keep-alive shutdown test required writes after Drop to succeed, but the server may already have closed the socket. Broken pipe and reset errors are now accepted as evidence of shutdown; the capture-count assertion is unchanged.

Wait for stats captures in trace-agent tests

695d446

test_internal_span_kind_does_not_produce_stats and test_peer_tags_in_flushed_stats read the intake capture immediately after shutdown; wait for it like the other tests to avoid timing-dependent failures.

Fail fast when the mock intake rejects a payload

098d30d

The trace-agent verify helpers waited for decoded payloads, so a payload the intake rejected surfaced as a 60s timeout claiming no request arrived. Wait for the raw request, then assert it was accepted.

Reject non-UTF-8 Content-Encoding in mock intake

45afb6c

A Content-Encoding value that was not valid UTF-8 was treated as absent, so the body was silently decoded as identity. It is now rejected with 415 like any other unsupported encoding.

Assert decoded services in dual-transport test

d107012

The dual-transport test byte-searched raw request bodies for the service names and checked only after the first trace POST arrived. It now waits until spans from both services appear in the decoded trace payloads.

lucaspimentel force-pushed the lpimentel/add-mock-intake branch from d54fc76 to d107012 Compare October 19, 2026 13:48

Conversation

lucaspimentel commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Describe how to test/QA your changes

Uh oh!

lucaspimentel commented Sep 25, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lucaspimentel commented Sep 29, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lucaspimentel commented Sep 29, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lucaspimentel commented Sep 29, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

lucaspimentel commented Sep 30, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lucaspimentel commented Oct 5, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lucaspimentel commented Sep 25, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 30, 2026 •

edited

Loading