Add support for the v3 metrics intake payload#7144
Conversation
The agent's /api/intake/metrics/v3/series intake uses a columnar,
dictionary-encoded protobuf format that is very different from the v2
MetricPayload, and streams its body as several concatenated zstd frames.
This adds end-to-end support for it in the proxy:
- core.py: decode multi-frame zstd bodies (mitmproxy's content only
decodes the first frame, silently truncating the payload).
- _decoders/metrics_v3.{proto,descriptor,py}: a cherry-picked proto (built
via update_protobuf.sh) parses the envelope and packed columns; the
decoder undoes the dictionary indirection, delta-encoding and tagset
framing layered on top, producing a v2-like {"series": [...]} dict.
- _deserializer.py: route the v3 path to the decoder.
- _agent.py: get_metrics() now also reads the v3 series path.
Adds characterization tests (synthetic branch coverage plus a real
286-series payload snapshot captured from RUNTIME_METRICS_ENABLED) under
the TEST_THE_TEST scenario.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f132cec6ef
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
get_metrics() now reads the v3 path, but the decoder dropped the per-point
columns, so callers that sum points (e.g. tests/ffe/test_flag_eval_metrics.py
::test_ffe_eval_metric_count) saw 0 for v3 payloads.
Decode the per-point columns into a v2-shaped `points` field: timestamps is a
single delta-encoded stream consumed sequentially across series, and each
value is read from the vals* column selected by the series' value type
(type & 0x30 -> sint64 / float32 / float64 / zero), with an independent cursor
per column. Emitted as [{"value": <float>, "timestamp": <str>}], matching the
v2 MessageToDict(MetricPayload) shape, and only when non-empty.
Adds a synthetic test covering all four value types and re-freezes the real
286-series snapshot (now with points).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2 participants
What
Adds end-to-end proxy support for the agent's
/api/intake/metrics/v3/seriesintake, which uses a columnar, dictionary-encoded protobuf format (very different from the v2MetricPayload) and streams its body as several concatenated zstd frames. This was changed here: DataDog/datadog-agent#52059The diff numbers look scary, but most of that is just a sample expected output file I used to validate the decoder via the TEST_THE_TESTS scenario
Changes
utils/proxy/core.py—get_decoded_content()decodes multi-frame zstd bodies with a frame-aware reader. mitmproxy'smessage.contentonly decodes the first zstd frame, silently truncating the v3 payload.utils/proxy/_decoders/metrics_v3.proto/.descriptor— a cherry-picked proto (Payload/Metadata/MetricData) from the agent'sdogstatsdhttp/payload.proto, built through the canonicalupdate_protobuf.shpipeline (protoc 3.21.12), like the other descriptors.metricDatais declaredrepeated(wire-compatible with the upstream singular field) so multipleMetricDatamessages per request yield a list instead of protobuf merging them.utils/proxy/_decoders/metrics_v3.py— protobuf parses the envelope and the packed scalar columns; the decoder undoes only what protobuf can't express — the string-dictionary unpacking, the delta-encoding of reference columns, and the self-framed tagset reconstruction — producing a v2-like{"series": [...]}dict soget_metricskeeps working unchanged.utils/proxy/_decoders/protobuf_schemas.py— exposesMetricsV3Payload.utils/proxy/_deserializer.py— routes the v3 path to the decoder.utils/interfaces/_agent.py—get_metrics()now also reads the v3 series path.Tests
tests/test_the_test/test_metrics_v3_decoder.py(runs underTEST_THE_TEST, no infra):tagsRef=0, multipleMetricDatablobs).RUNTIME_METRICS_ENABLED.These were written before the proto-based decoder rewrite and pass identically before and after, so they double as a refactor guard.
Verified: guard tests green,
./format.shgreen.🤖 Generated with Claude Code but also reviewed and edited by Andrew