ci: install ddtest 1.7.0 and riot 0.23.0 in testrunner image - #19960
Conversation
Codeowners resolved asResolved from the full PR diff against No remaining files require a CODEOWNERS review. |
Circular import analysis
|
Dependency direction analysis
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 0f12a37 | Docs | View more details | Give us feedback! |
89b354e to
ed4a821
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed4a8214a5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Update the testrunner Dockerfile to: - Bump riot from 0.22.0 to 0.23.0 - Install ddtest 1.7.0 (pre-built binary from GitHub releases) Update ci/requirements/ci.txt to pin riot==0.23.0 (with correct hash). This is a prerequisite for the ddtest dogfooding PR (#19870): once the testrunner image is rebuilt with these changes, ddtest is available in PATH and the .ddtest_build CI job can be removed. NOTE: the testrunner image hash in .gitlab/testrunner.yml must be updated after the image is rebuilt.
ed4a821 to
0f12a37
Compare
BenchmarksBenchmark execution time: 2026-08-31 12:15:26 Comparing candidate commit 0f12a37 in PR branch Found 0 performance improvements and 6 performance regressions! Performance is the same for 574 metrics, 10 unstable metrics, 2 known flaky benchmarks, 16 flaky benchmarks without significant changes.
|
## Description Updates known flaky benchmarks with `httppropagationinject-ids_only`, `iastaspectsospath-ospathbasename_aspect`, `iastaspectssplit-(rsplit_aspect|rstrip_aspect)`, `telemetryaddmetric-1-count-metric-1-times` and `tracer-small`. This prevents flaky benchmarks from breaching SLOs and polluting PR comments. ## Testing None. ## Risks None. ## Additional Notes I investigated 49 recent PRs that change code that shouldn't impact benchmarks: docs, release notes, tests, `.riot/` lockfiles, CI config. 33 reported performance changes on scenarios outside our known flaky benchmarks. Adding the ones flagged on more than half of the 33: - `httppropagationinject-ids_only` — 33/33, median +12.7% - `iastaspectsospath-ospathbasename_aspect` — 33/33, median +31.2% - `telemetryaddmetric-1-count-metric-1-times` — 33/33, median +16.3% - `tracer-small` — 33/33, median +10.0% - `iastaspectssplit-rsplit_aspect` — 23/33, median +14.6% Some scenarios are already in the known flaky benchmarks regex, corroborated by this data: - `span-start` -- 29/33 - `iastaspects-ljust_noaspect` -- 10/33 - `iastaspects-casefold_noaspect` -- 7/33 - `iastaspects-title_noaspect` -- 3/33 Flagged on half of the 33 or fewer, left out as weak evidence: - `iastaspects-add_aspect` — 15/33 - `iastaspects-title_aspect` — 14/33 - `iastaspects-join_aspect` — 11/33 - `httppropagationextract-tracecontext_headers` — 10/33 - `httppropagationextract-wsgi_valid_headers_all` — 6/33 - `iastaspects-upper_aspect` — 4/33 - `iastaspects-repr_aspect` — 3/33 - `iastaspects-modulo_noaspect` — 2/33 - `iastaspects-rstrip_aspect` — 2/33 - `iastaspects-stringio_noaspect` — 2/33 - `iastaspects-stringio_aspect` — 2/33 - `iastaspects-lstrip_aspect` — 2/33 - `iastaspects-ljust_aspect` — 1/33 - `iastaspects-lower_aspect` — 1/33 - `iastaspects-swapcase_aspect` — 1/33 - `iastaspects-translate_noaspect` — 1/33 - `samplingrules-high_match` — 1/33 <details> <summary>See investigated PRs (49)</summary> - [#19545](#19545) — ci: itr rollout configuration - [#19566](#19566) — docs: add benchmarking CI documentation - [#19786](#19786) — chore: handle potential after-import exceptions - [#19826](#19826) — chore: update django latest version to 6.1 _(no benchmark report)_ - [#19828](#19828) — chore(tests): bump gevent to 26.8.0 to stop the flaky django IAST shutdown test - [#19831](#19831) — test(profiling): try to make uwsgi less flaky _(no benchmark report)_ - [#19835](#19835) — ci(hooks): block partially staged Python files from committing unformatted (PROF-15836) - [#19836](#19836) — ci: reuse test installs and lock documentation builds - [#19838](#19838) — ci(sg): fail commits on double backticks in ddtrace comments/docstrings (PROF-15837) - [#19839](#19839) — test(ci): harden Django and pytest subprocess tests - [#19842](#19842) — ci(commenter): safely serialize message from PR commenter bot - [#19843](#19843) — chore(py-315): raise SSI exclusive max to 3.16 and pre-stage 3.15 injection (PROF-15812) - [#19846](#19846) — test(appsec): bound and raise the test server startup budget - [#19847](#19847) — test(iast): flaky test_packages_patched pyjwt _(no benchmark report)_ - [#19853](#19853) — test(appsec): enable appsec_integrations_fastapi on Python 3.15 - [#19862](#19862) — test(opentracing): remove stale Riot environments - [#19866](#19866) — ci(appsec): isolate Django server from gevent cleanup _(no benchmark report)_ - [#19871](#19871) — test(aiguard): cover multi-turn sensitive data redaction scenarios - [#19872](#19872) — chore(kafka): remove stale _dd.svc_src from service override snapshot _(no benchmark report)_ - [#19874](#19874) — test(kafka): update service source snapshots [Part 2] _(no benchmark report)_ - [#19878](#19878) — refactor(tests): standardize test runner interface _(no benchmark report)_ - [#19879](#19879) — test(dramatiq): enable tests for Python 3.14 - [#19880](#19880) — ci: withhold cp315 wheels from PyPI and the prerelease index (PROF-15857) - [#19885](#19885) — ci: automatically retry os tests - [#19897](#19897) — chore(ci): djangosimple-tracer-dont-create-db-spans slo bump - [#19904](#19904) — ci(build_base_venvs): let riot install the dev package on Python 3.15 (PROF-14439) - [#19907](#19907) — ci(testrunner): rebuild 3.15-dev and pin Cython<3.3 on the 3.15 cache (PROF-14439) - [#19914](#19914) — test(profiling): unflake `gevent_greenlet_switch_not_blocked_by_profiler` - [#19923](#19923) — chore: exclude integration manifests yaml from source distributions _(no benchmark report)_ - [#19926](#19926) — ci: auto-retry `build windows` jobs - [#19936](#19936) — ci: bump wheel-builder IMAGE_TAGs after images (PROF-15844) - [#19951](#19951) — test(profiling): unflake gc tests - [#19954](#19954) — test(llmobs): fix flaky test _(no benchmark report)_ - [#19960](#19960) — ci: install ddtest 1.7.0 and riot 0.23.0 in testrunner image - [#19961](#19961) — test(iast): fix cloned venv poisoning [APPSEC-69907] _(no benchmark report)_ - [#19962](#19962) — test(iast): give appsec test servers their own port [APPSEC-69907] _(no benchmark report)_ - [#19964](#19964) — test(iast): assert MODULES_TO_UNPATCH deltas [APPSEC-69907] _(no benchmark report)_ - [#19965](#19965) — test(iast): unwrap weak hash before each test patches it [APPSEC-69907] _(no benchmark report)_ - [#19966](#19966) — test(iast): stop the global vulnerability limit leaking between tests [APPSEC-69907] _(no benchmark report)_ - [#19967](#19967) — chore(ci): pin numeric UID for testrunner image's bits user _(no benchmark report)_ - [#19968](#19968) — test(appsec): stop waiting 0.5s per api10 test teardown [APPSEC-69907] _(no benchmark report)_ - [#19973](#19973) — chore(tests): declare tracer environments in suitespec - [#19974](#19974) — test(telemetry): support PathResolutionError in wrapt 2.4.0 - [#19977](#19977) — ci(testrunner): install pip-tools for Python 3.15 - [#19984](#19984) — ci: update testrunner image - [#19985](#19985) — test(appsec): appsec_threats 55% faster, ~12h of CI saved per pipeline [APPSEC-69907] - [#19994](#19994) — ci(benchmarks): bump codeprovenancefork-fork-10 SLO - [#19997](#19997) — chore(tests): execute tracer suites with uv - [#20025](#20025) — test(appsec): run appsec_threats suites in parallel with pytest-xdist </details> [APPSEC-69907]: https://datadoghq.atlassian.net/browse/APPSEC-69907?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: augusto.deoliveira <augusto.deoliveira@datadoghq.com>
Update the testrunner Dockerfile to:
Update ci/requirements/ci.txt to pin riot==0.23.0 (with correct hash).
This is a prerequisite for the ddtest dogfooding PR (#19870): once the testrunner image is rebuilt with these changes, ddtest is available in PATH and the .ddtest_build CI job can be removed.
NOTE: the testrunner image hash in .gitlab/testrunner.yml must be updated after the image is rebuilt.
Description
Testing
Risks
Additional Notes