Add CSS error sampler coverage (TS021) by ichinaski · Pull Request #7171 · DataDog/system-tests

ichinaski · 2026-06-18T07:34:01Z

Motivation

Phase C (APMSP-3049) of the Client-Side Stats v1.2.0 coverage: test the interaction between CSS and the agent error sampler. The agent keeps a portion of error traces, so a tracer with CSS enabled must still send all traces containing an error even when sampling would otherwise drop them.

Changes

Adds Test_Error_Sampler in tests/stats/test_stats.py and a new TRACE_STATS_COMPUTATION_ERROR_SAMPLER scenario (DD_TRACE_SAMPLE_RATE=0 with CSS enabled). The test asserts stats are still computed and error traces are still sent to the agent. Active for golang, python, dotnet and java; marked missing_feature for ruby (no stats computed when sampling drops all traces), php (errors counted in stats but error traces dropped client-side) and cpp (no client-side stats).

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

Add Test_Error_Sampler end-to-end test asserting that traces containing errors are still sent to the agent when sampling would drop them, with a new TRACE_STATS_COMPUTATION_ERROR_SAMPLER scenario. Active for golang, python, dotnet and java; marked missing_feature for ruby and php (with reasons) and cpp (no client-side stats).

github-actions · 2026-06-18T07:34:36Z

CODEOWNERS have been resolved as:

.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/schemas/test_schemas.py                                           @DataDog/system-tests-core
tests/stats/test_stats.py                                               @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core

datadog-prod-us1-3 · 2026-06-18T07:37:46Z

Tests

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 6 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.1-zts 1
🧪 1 Test failed
tests.ffe.test_exposures.Test_FFE_Exposure_Events.test_ffe_multiple_remote_config_files[apache-mod-7.1-zts] from system_tests_suite
AssertionError: Timed out waiting for exposure event for flags [&#39;test-flag-1&#39;, &#39;test-flag-2&#39;] and subject &#39;test-user-multi&#39;
assert False
 &#43;  where False = &lt;bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator(&#39;agent&#39;)&gt;(&lt;function wait_for_exposure_event.&lt;locals&gt;.&lt;lambda&gt; at 0x7f5b07ad6de0&gt;, timeout=30)
 &#43;    where &lt;bound method ProxyBasedInterfaceValidator.wait_for of AgentInterfaceValidator(&#39;agent&#39;)&gt; = AgentInterfaceValidator(&#39;agent&#39;).wait_for
 &#43;      where AgentInterfaceValidator(&#39;agent&#39;) = interfaces.agent

self = &lt;tests.ffe.test_exposures.Test_FFE_Exposure_Events object at 0x7f5b30c8ed50&gt;

    def test_ffe_multiple_remote_config_files(self):
        &#34;&#34;&#34;Test that FFE correctly handles multiple remote config files with different flags.&#34;&#34;&#34;
...
DataDog/system-tests | ruby-app-deployment-mode.amd64.DOA9: [public.ecr.aws/lts/ubuntu:22.04, linux/amd64, 3.1.7]

DataDog/system-tests | ruby-app-deployment-mode.arm64.DOA9: [public.ecr.aws/lts/ubuntu:22.04, linux/arm64, 3.1.7]
View all 6 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 66d900a | Docs | Datadog PR Page | Give us feedback!}

vertx does not flag 500 responses as errors, so no error span is produced and the error-sampler assertion cannot hold on those weblogs.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 496bf72a1c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-18T12:30:21Z

+        # Droppable P0 traffic: with sample rate 0 these are dropped from traces but still counted in stats.
+        for _ in range(5):
+            weblog.get("/")


Assert that non-error sampled traces are dropped

Because this loop discards the successful request handles, the test below only proves that an error trace was emitted; it never proves that DD_TRACE_SAMPLE_RATE=0 actually dropped non-error P0 traces. In contexts like Node.js, where manifests/nodejs.yml already records that DD_TRACE_SAMPLE_RATE=0 still emits P0 traces, this scenario can pass while the prerequisite sampling behavior is broken, giving a false activation signal. Please retain these requests and assert they have no emitted spans, or mark unsupported libraries accordingly.

Useful? React with 👍 / 👎.

The false-activation risk doesn't apply to the libraries this test runs on.

chatgpt-codex-connector · 2026-06-18T12:30:21Z

+    def test_error_traces_always_sent(self):
+        """Test that error traces are sent and stats are computed when sample rate is 0."""
+        # Client-side stats are computed despite sampling being disabled.
+        stats_requests = list(interfaces.library.get_data("/v0.6/stats"))


Add the new stats scenario to the schema allow-list

This new scenario reads /v0.6/stats payloads, but the auxiliary schema test's APMSP-2158 allow-list in tests/schemas/test_schemas.py only includes the existing trace-stats scenarios, not trace_stats_computation_error_sampler. When this runs for libraries covered by that known schema bug, Test_DdtraceSchemas.test_library will treat the same stats schema errors as new failures for this scenario. Please add this scenario to the existing known-bug condition or fix the schema before enabling it.

Useful? React with 👍 / 👎.

…P-2158) The new TRACE_STATS_COMPUTATION_ERROR_SAMPLER scenario reads /v0.6/stats, so it is subject to the same known schema bug as the other trace-stats scenarios; add it to the existing allow-list.

tabgok

approved from IDM's standpoint

Mark Test_Error_Sampler missing_feature for java vertx3/vertx4

496bf72

vertx does not flag 500 responses as errors, so no error span is produced and the error-sampler assertion cannot hold on those weblogs.

ichinaski marked this pull request as ready for review June 18, 2026 12:27

ichinaski requested review from a team as code owners June 18, 2026 12:27

ichinaski requested review from claponcet, jandro996, tabgok and ygree and removed request for a team June 18, 2026 12:27

chatgpt-codex-connector Bot reviewed Jun 18, 2026

View reviewed changes

Add error sampler scenario to /v0.6/stats schema known-bug list (APMS…

66d900a

…P-2158) The new TRACE_STATS_COMPUTATION_ERROR_SAMPLER scenario reads /v0.6/stats, so it is subject to the same known schema bug as the other trace-stats scenarios; add it to the existing allow-list.

tabgok approved these changes Jun 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CSS error sampler coverage (TS021)#7171

Add CSS error sampler coverage (TS021)#7171
ichinaski wants to merge 3 commits into
mainfrom
ichinaski/css-tests-phase-c

ichinaski commented Jun 18, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-3 Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Uh oh!

ichinaski Jun 18, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Uh oh!

tabgok left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ichinaski commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-3 Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

ichinaski Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

tabgok left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ichinaski commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

datadog-prod-us1-3 Bot commented Jun 18, 2026 •

edited

Loading

ichinaski Jun 18, 2026 •

edited

Loading