feat(requirements): add uv pip compile backend with --exclude-newer support - #267
Merged
Merged
Conversation
juanjux
requested review from
Yun-Kim and
r1viollet
and removed request for
a team
May 20, 2026 08:19
This comment has been minimized.
This comment has been minimized.
juanjux
added a commit
to DataDog/dd-trace-py
that referenced
this pull request
May 20, 2026
Closes the transitive-dependency gap on TEST-CD (APMLP-1362). The previous commit applied the 48h cooldown at the "outdated detection" level only; this commit adds two complementary mechanisms so freshly published transitive dependencies cannot make it into a regenerated .riot/requirements/*.txt either. 1. uv pip compile backend (defense-in-front): Sets RIOT_PIP_COMPILE_BACKEND=uv and RIOT_PIP_COMPILE_EXCLUDE_NEWER (a cutoff 48h in the past) on the `Run regenerate-riot-latest` step of `generate-package-versions.yml`. With the riot change in DataDog/riot#267 these tell riot to resolve dependencies via `uv pip compile --exclude-newer=<cutoff>`. Older riot versions ignore both variables, so it is safe to set them before the riot bump lands. 2. Post-compile validator (defense-in-depth): Adds scripts/check_lockfile_cooldown.py, which walks the regenerated lockfiles, queries PyPI for the upload time of every `name==version` pin, and exits non-zero if any release is younger than COOLDOWN_DAYS (= 2). Wired into regenerate-riot-latest.sh immediately after compile-and-prune-test-requirements so the workflow fails before opening an update PR if a transitive bypass ever slips through. Tests cover the lockfile parser, the cooldown decision logic, and both pass / fail paths via `tests/internal/test_check_lockfile_cooldown.py`. Refs APMLP-1362, depends on DataDog/riot#267. Co-authored-by: Cursor <cursoragent@cursor.com>
juanjux
force-pushed
the
juanjux/uv-pip-compile-exclude-newer
branch
2 times, most recently
from
May 20, 2026 08:25
dbcc181 to
6b0a5f8
Compare
…upport Add an opt-in ``uv pip compile`` backend for ``VenvInstance.requirements`` selectable via two new environment variables: - ``RIOT_PIP_COMPILE_BACKEND``: ``piptools`` (default) or ``uv``. - ``RIOT_PIP_COMPILE_EXCLUDE_NEWER``: forwarded to ``uv pip compile`` as ``--exclude-newer=<value>``. Ignored (with a warning) when the backend is ``piptools`` because pip-tools has no equivalent option. The historical behaviour is preserved exactly when neither variable is set. When the ``uv`` backend is selected the ``uv`` executable must already be on PATH; a clear ``RuntimeError`` is raised if it isn't. This is motivated by the cross-language supply-chain hardening work in ``dd-trace-py`` (APMLP-1343 / TEST-CD): downstream projects need a way to keep ``uv pip compile`` from pulling transitive dependencies that were published less than 48h ago, and ``--exclude-newer`` is the ergonomic primitive for that. ``pip-tools`` has no equivalent flag, so projects that want the cooldown must opt in to the ``uv`` backend. Tests cover the new code paths (uv backend invocation, exclude-newer pass-through, missing-uv error, unknown-backend error, and the warning emitted when ``RIOT_PIP_COMPILE_EXCLUDE_NEWER`` is set alongside the pip-tools backend). Co-authored-by: Cursor <cursoragent@cursor.com>
juanjux
force-pushed
the
juanjux/uv-pip-compile-exclude-newer
branch
from
May 20, 2026 08:41
6b0a5f8 to
09a512d
Compare
brettlangdon
left a comment
brettlangdon
left a comment
Member
There was a problem hiding this comment.
suggestion, but in general lgtm
brettlangdon
approved these changes
May 20, 2026
brettlangdon
left a comment
brettlangdon
left a comment
Member
There was a problem hiding this comment.
I am ok with this as-is, if we find no issues during the transition, then migrating to be only uv compile based makes sense as a follow-up
e.g.
- this PR
- next PR adds
uvas project dep and change default touv - remove backend selection and only use
uv
given we are primarily the only users, we can probably move a little quicker than that, but all in all seems good
juanjux
added a commit
to DataDog/dd-trace-py
that referenced
this pull request
May 20, 2026
…release) The TEST-CD layer-2 (uv pip compile --exclude-newer) only takes effect once we are on a riot version that ships the new RIOT_PIP_COMPILE_BACKEND / RIOT_PIP_COMPILE_EXCLUDE_NEWER env vars (DataDog/riot#267, merged into master as c04e0aa). The next riot release will be 0.22.0 (semver minor bump for a feat: commit on top of 0.21.0). This commit: - Bumps RIOT_VERSION in docker/Dockerfile from 0.21.0 to 0.22.0. - Bumps riot in ci/requirements/ci.in from 0.20.1 to 0.22.0. ci/requirements/ci.txt is intentionally NOT regenerated yet because riot 0.22.0 is not on PyPI yet (uv pip compile --generate-hashes fails with "No solution found"). Before merging this PR a maintainer must: 1. Wait for riot 0.22.0 to be cut and uploaded to PyPI. 2. Run scripts/update-ci-dependencies to regenerate ci.txt with the new pin and hashes. 3. Commit the regenerated ci.txt. The dd-trace-py CI does not enforce ci.in/ci.txt sync (the check-ci-dependencies job only validates that ci.txt is internally consistent under --require-hashes), so this commit is safe to push while we wait for the release. Co-authored-by: Cursor <cursoragent@cursor.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/dd-trace-py
that referenced
this pull request
Jun 16, 2026
…#18182) ## Note Depends on the next riot version (0.22.0) so not mergeable until that one has been released and `scripts/update-ci-dependencies` has been run! ## Summary Implements the cross-language supply-chain hardening "cooldown" controls documented in epic APMLP-1343 for `dd-trace-py`. The 48h cooldown is now enforced everywhere Dependabot or our daily test-lockfile job would otherwise pull in a freshly published release, including transitive dependencies pulled in by lockfile recompilation. Tickets addressed: - APMLP-1359 — GHA-CD: added a `cooldown` block to the existing `github-actions` entry in `.github/dependabot.yml`. GitHub Actions are already SHA-pinned (the existing `GHA-PIN` work); this adds the missing 48h delay before proposing an update. - APMLP-1360 — DOCKER-CD: added new `docker` and `docker-compose` ecosystems to `.github/dependabot.yml` for `docker/`, `benchmarks/`, `lib-injection/`, the root `docker-compose*.yml` files, with the same 48h cooldown. This replaces a fully manual digest-rotation flow with cooldown-aware automation. (Tightening DOCKER-PIN on the unpinned `docker-compose.yml` images, e.g. `mysql:5.7`, `redis:4.0-alpine`, etc., is intentionally out of scope here and would be a follow-up.) - APMLP-1362 — TEST-CD: three layers of cooldown enforcement on riot test lockfile updates: 1. **Trigger** — `scripts/freshvenvs.py` ignores PyPI releases that are less than `COOLDOWN_DAYS` (= 2) old when deciding whether the lockfiles are outdated. Falls back to the absolute latest if every candidate is too fresh so detection is never silently disabled. 2. Transitive Deps — the `Run regenerate-riot-latest` workflow step sets `RIOT_PIP_COMPILE_BACKEND=uv` and `RIOT_PIP_COMPILE_EXCLUDE_NEWER=<cutoff>` (computed as `now - 48h`). With the riot change in [DataDog/riot#267](DataDog/riot#267) this tells riot to resolve the lockfile via `uv pip compile --exclude-newer=<cutoff>` so transitive dependencies younger than the cooldown cannot enter the lockfile in the first place. 3. Validator — `scripts/check_lockfile_cooldown.py` walks the regenerated lockfiles after compile, queries PyPI for the upload time of every `name==version` pin, and exits non-zero if any release is younger than `COOLDOWN_DAYS`. It is invoked from `scripts/regenerate-riot-latest.sh` immediately after `compile-and-prune-test-requirements`, so the workflow fails before opening an update PR if a fresh transitive pin ever slips through (e.g. because we are still on an older riot, or because someone disables the uv backend). Co-authored-by: juanjo.alvarezmartinez <juanjo.alvarezmartinez@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
uv pip compilebackend toVenvInstance.requirements, selectable via two new environment variables:RIOT_PIP_COMPILE_BACKEND:piptools(default) oruv.RIOT_PIP_COMPILE_EXCLUDE_NEWER: forwarded touv pip compileas--exclude-newer=<value>. Ignored (with a warning) when the backend ispiptoolsbecause pip-tools has no equivalent option.The historical behaviour is preserved exactly when neither variable is set. When the
uvbackend is selected theuvexecutable must already be onPATH; a clearRuntimeErroris raised if it isn't.Why
Cross-language supply-chain hardening work in
dd-trace-py(tracked in DataDog/dd-trace-py#18182) needs a 48h "cooldown" on every release that ends up in a compiled lockfile, including transitive dependencies.--exclude-newer=<date>is the ergonomic primitive for that onuv pip compile;pip-toolshas nothing equivalent. Rather than fork riot or post-process its output, projects can now flip a single env var and get the cooldown for free.