Skip to content

Add provisional per-subject evaluation telemetry opt-out tests #8080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leoromanovsky wants to merge 3 commits into
base: main
Choose a base branch
from
+433 −32
Open

Add provisional per-subject evaluation telemetry opt-out tests #8080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
89aa7bc
test(ffe): cover per-subject evaluation telemetry opt-out
leoromanovsky Oct 18, 2026
35686a7
test(ffe): associate telemetry opt-out coverage with feature 649
leoromanovsky Oct 18, 2026
20ce91e
test(ffe): address telemetry opt-out review feedback
leoromanovsky Oct 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions manifests/c.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ manifest:
tests/ai_guard/: irrelevant (dd-trace-c does not implement AI Guard)
tests/appsec/: irrelevant (dd-trace-c does not implement AppSec or IAST)
tests/cws/test_thread_context_sharing.py::Test_ThreadContextSharing: missing_feature (missing /security/thread_context_sharing endpoint on weblog)
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/integrations/test_dbm.py: incomplete_test_app (The Perl Mojolicious workload does not provide database endpoints)
tests/parametric/otel_env_vars/test_otel_exporter_otlp_compression.py::Test_OTEL_EXPORTER_OTLP_COMPRESSION: missing_feature
tests/parametric/otel_env_vars/test_otel_exporter_otlp_headers.py::Test_OTEL_EXPORTER_OTLP_HEADERS: missing_feature
Expand Down
1 change: 1 addition & 0 deletions manifests/cpp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_flag_eval_evp.py: irrelevant (only parametric tests are run for cpp)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/cpp_httpd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ manifest:
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_agentless_configuration.py: missing_feature (C++ is not in scope for v1 agentless server SDK coverage; FFL-2731 tracks the system-tests contract)
tests/ffe/test_dynamic_evaluation.py: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (C++ does not implement per-subject evaluation telemetry opt-out)
tests/ffe/test_exposure_egress.py: missing_feature (Not yet implemented)
tests/ffe/test_exposures_datadog_agent.py: missing_feature
tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446)
Expand Down
1 change: 1 addition & 0 deletions manifests/cpp_nginx.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ manifest:
tests/docker_ssi/test_docker_ssi_appsec.py::TestDockerSSIAppsecFeatures::test_telemetry_source_ssi: missing_feature
tests/ffe/test_agentless_configuration.py: missing_feature (C++ is not in scope for v1 agentless server SDK coverage; FFL-2731 tracks the system-tests contract)
tests/ffe/test_dynamic_evaluation.py: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (C++ does not implement per-subject evaluation telemetry opt-out)
tests/ffe/test_exposure_egress.py: missing_feature (Not yet implemented)
tests/ffe/test_exposures_datadog_agent.py: missing_feature
tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446)
Expand Down
2 changes: 2 additions & 0 deletions manifests/dotnet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -822,6 +822,8 @@ manifest:
weblog: [poc, uds]
- declaration: bug (FFL-2184)
excluded_weblog: [poc, uds]
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_evaluation_telemetry_opt_out.py::Test_FFE_Evaluation_Telemetry_Opt_Out_EVP: missing_feature (.NET has no EVP evaluation producer; FFL-2446)
tests/ffe/test_exposure_egress.py: v3.36.0
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct: missing_feature (Not yet implemented)
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct_Shutdown: missing_feature (Not yet implemented)
Expand Down
1 change: 1 addition & 0 deletions manifests/golang.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1728,6 +1728,7 @@ manifest:
weblog: [echo, gin, chi, net-http-orchestrion, net-http, uds-echo]
- declaration: bug (FFL-2182)
excluded_weblog: [echo, gin, chi, net-http-orchestrion, net-http, uds-echo]
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py: v2.6.0-dev # Easy win for chi, echo, gin, net-http, net-http-orchestrion, uds-echo and version 2.5.0
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct: v2.12.0-dev
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct_Shutdown:
Expand Down
1 change: 1 addition & 0 deletions manifests/java.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3476,6 +3476,7 @@ manifest:
weblog: [spring-boot]
- declaration: bug (FFL-2184)
excluded_weblog: [spring-boot]
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py:
- weblog_declaration:
"*": irrelevant
Expand Down
1 change: 1 addition & 0 deletions manifests/java_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ manifest:
tests/appsec/waf/test_blocking.py::Test_Blocking: missing_feature
tests/appsec/waf/test_blocking.py::Test_Blocking_strip_response_headers: missing_feature
tests/appsec/waf/test_blocking.py::Test_CustomBlockingResponse: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (This server weblog scenario does not apply to Lambda)
tests/ffe/test_flag_eval_evp.py: irrelevant (server-side FFE EVP weblog scenario is not applicable to lambda)
tests/test_resource_renaming.py: missing_feature
tests/test_rum_injection.py: irrelevant (RUM injection only supported for Java)
Expand Down
1 change: 1 addition & 0 deletions manifests/java_otel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (OpenTelemetry test library does not implement Datadog feature flag providers)
tests/ffe/test_flag_eval_evp.py: irrelevant (OpenTelemetry test library does not implement server-side FFE EVP)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/nodejs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1871,6 +1871,7 @@ manifest:
weblog: [express5]
- declaration: bug (FFL-2182)
excluded_weblog: [express5]
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py:
- weblog_declaration:
"*": incomplete_test_app
Expand Down
1 change: 1 addition & 0 deletions manifests/nodejs_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ manifest:
tests/appsec/waf/test_blocking.py::Test_Blocking: missing_feature
tests/appsec/waf/test_blocking.py::Test_Blocking_strip_response_headers: missing_feature
tests/appsec/waf/test_blocking.py::Test_CustomBlockingResponse: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (This server weblog scenario does not apply to Lambda)
tests/ffe/test_flag_eval_evp.py: irrelevant (server-side FFE EVP weblog scenario is not applicable to lambda)
tests/test_resource_renaming.py: missing_feature
tests/test_rum_injection.py: irrelevant
Expand Down
1 change: 1 addition & 0 deletions manifests/nodejs_otel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (OpenTelemetry test library does not implement Datadog feature flag providers)
tests/ffe/test_flag_eval_evp.py: irrelevant (OpenTelemetry test library does not implement server-side FFE EVP)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/php.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1240,6 +1240,7 @@ manifest:
"*": v1.21.0-dev
laravel11x: incomplete_test_app
symfony7x: incomplete_test_app
tests/ffe/test_evaluation_telemetry_opt_out.py: incomplete_test_app (Deferred until the PHP OpenFeature test-app foundation is available)
tests/ffe/test_exposure_egress.py:
- weblog_declaration:
"*": v1.21.0-dev
Expand Down
1 change: 1 addition & 0 deletions manifests/python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1499,6 +1499,7 @@ manifest:
flask-poc: v4.14.0
tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Down_From_Start: v4.0.0
tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Unavailable: flaky (FFL-1622)
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py: v4.2.0-dev
tests/ffe/test_exposure_egress.py::Test_FFE_Exposure_Egress_Agentless_Direct:
- weblog_declaration:
Expand Down
1 change: 1 addition & 0 deletions manifests/python_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (This server weblog scenario does not apply to Lambda)
tests/ffe/test_flag_eval_evp.py: irrelevant (server-side FFE EVP weblog scenario is not applicable to lambda)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/python_otel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ manifest:
tests/debugger/test_debugger_exception_replay.py::Test_Debugger_Exception_Replay::test_exception_replay_stackoverflow: missing_feature (Implemented only for dotnet)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot: missing_feature (Not yet implemented)
tests/debugger/test_debugger_probe_snapshot.py::Test_Debugger_Line_Probe_Snaphots::test_process_tags_snapshot_svc: missing_feature (Not yet implemented)
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (OpenTelemetry test library does not implement Datadog feature flag providers)
tests/ffe/test_flag_eval_evp.py: irrelevant (OpenTelemetry test library does not implement server-side FFE EVP)
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_sqs.py::Test_SQS_PROPAGATION_VIA_AWS_XRAY_HEADERS: irrelevant (Localstack SQS does not support AWS Xray Header parsing)
Expand Down
1 change: 1 addition & 0 deletions manifests/ruby.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2137,6 +2137,7 @@ manifest:
- weblog_declaration:
"*": irrelevant
rails72: v2.23.0-dev
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py:
- weblog_declaration:
"*": irrelevant
Expand Down
1 change: 1 addition & 0 deletions manifests/ruby_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,4 +85,5 @@ manifest:
tests/appsec/waf/test_blocking.py::Test_Blocking::test_html_template_v2: missing_feature
tests/appsec/waf/test_blocking.py::Test_Blocking_strip_response_headers: v3.30.0
tests/appsec/waf/test_blocking.py::Test_CustomBlockingResponse: v3.29.0
tests/ffe/test_evaluation_telemetry_opt_out.py: irrelevant (This server weblog scenario does not apply to Lambda)
tests/ffe/test_flag_eval_evp.py: irrelevant (server-side FFE EVP weblog scenario is not applicable to lambda)
1 change: 1 addition & 0 deletions manifests/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ manifest:
tests/docker_ssi/test_docker_ssi_appsec.py::TestDockerSSIAppsecFeatures::test_telemetry_source_ssi: missing_feature
tests/ffe/test_agentless_configuration.py: missing_feature (Rust is not in scope for v1 agentless server SDK coverage; FFL-2731 tracks the system-tests contract)
tests/ffe/test_dynamic_evaluation.py: missing_feature
tests/ffe/test_evaluation_telemetry_opt_out.py: missing_feature (Provisional per-subject evaluation telemetry opt-out coverage; enable after the SDK release)
tests/ffe/test_exposure_egress.py: missing_feature (Not yet implemented)
tests/ffe/test_exposures_datadog_agent.py: missing_feature
tests/ffe/test_flag_eval_evp.py: missing_feature (FFL-2446)
Expand Down
39 changes: 37 additions & 2 deletions tests/ffe/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,56 @@ This directory contains system tests for the Feature Flags & Experimentation (FF
| `test_exposures_datadog_agent.py` | Flag exposure tracking and reporting through the Datadog Agent |
| `test_flag_eval_metrics.py` | Evaluation metrics (OTel counter) |
| `test_flag_eval_evp.py` | Server-side EVP flagevaluation payloads, aggregation, and bounds |
| `test_evaluation_telemetry_opt_out.py` | Per-subject evaluation telemetry opt-out, default-off behavior, and unchanged exposures |

## Running FFE Tests

```bash
./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION --library <language>
```

## Per-subject evaluation telemetry opt-out

All four test classes report under [#649: Per-subject evaluation telemetry opt-out](https://feature-parity.us1.prod.dog/#/?feature=649).

Candidate SDK implementations must pass the applicable local checks before delivery.
Released-version CI activation is a separate step. Manifests keep the tests disabled until supported SDK versions are known.
Use a local candidate SDK and force these tests to validate unreleased changes:

```bash
./run.sh FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT tests/ffe/test_evaluation_telemetry_opt_out.py \
-F tests/ffe/test_evaluation_telemetry_opt_out.py
./run.sh FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT_DISABLED tests/ffe/test_evaluation_telemetry_opt_out.py \
-F tests/ffe/test_evaluation_telemetry_opt_out.py
```

The enabled scenario sets `DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED=true`.
The disabled scenario leaves the setting absent. Both use the same requests.

| Context value | Enabled | Disabled |
| --- | --- | --- |
| Boolean `true` | No evaluation telemetry | Evaluation telemetry remains |
| Boolean `false`, missing, string `"true"`, or number `1` | Evaluation telemetry remains | Evaluation telemetry remains |

Separate tests check EVP counts, evaluation metrics, and automatic trace annotations.
Each test includes allowed subjects as positive controls.
Mixed subjects evaluate the same flag to detect a flag-scoped opt-out decision.
One rule targets the opt-out attribute to verify that it remains available for evaluation.
All requests must return the same flag value. Exposures must retain their ordinary deduplication and context attributes.
Validation reads final captures after the scenario stops its application and drains the Agent.

PHP coverage is deferred until its OpenFeature test-app foundation is available.
.NET has no EVP evaluation producer. Exclude the EVP class when forcing its local runs with `-k 'not EVP'`.
Browser and mobile SDKs are outside this server harness and retain their repository integration tests.

## Test-contract self-tests

The self-tests in `testing_tests/` validate the FFE assertions using synthetic
The self-tests validate the FFE assertions using synthetic
captures. They do not run SDKs or contact an intake. With the Python 3.12 runner
environment installed, run them explicitly:

```bash
./run.sh FEATURE_FLAGGING_CONTRACT_TESTS tests/ffe/testing_tests
./run.sh FEATURE_FLAGGING_CONTRACT_TESTS tests/ffe/test_tests.py tests/ffe/test_evaluation_telemetry_opt_out_contract.py
```

This no-container scenario has no CI workflow or scenario-group membership.
Expand Down
Loading
Loading