Skip to content

build: replace venv with uv - #339

Draft
nikbpetrov wants to merge 6 commits into
forecastingresearch:mainfrom
nikbpetrov:uv-tooling
Draft

build: replace venv with uv#339
nikbpetrov wants to merge 6 commits into
forecastingresearch:mainfrom
nikbpetrov:uv-tooling

Conversation

@nikbpetrov

@nikbpetrov nikbpetrov commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

on top of #337

  • first two commits are the basic blocks, with marginal benefit
  • the third+fourth commits start getting some uv magic going: venv replaced with uv in makefiles ++ uv dep groups for reqs management; deps are no longer painful/slow, provided that it's acceptable to move from per-job-folder-requirements.txt to a centralized store
    • technically we can preserve a per-job-folder-.toml file w/ deps but i prefer the centralized solution tbh; reqs are touched once and checked via CI, I don't need them dangling around in files left and right)
  • (optional) fifth commit now uses uv in the Dockerfiles instead of translating to requirements.txt on deploy (strong time benefits but this is practically negligible as deploys are not that often)

@nikbpetrov
build: replace black, isort, flake8 and pydocstyle with ruff
5ed45f3 
Swap the four lint/format tools for a single exact-pinned `ruff`, configured in `pyproject.toml`
with a ruleset that mirrors the previous setup: `E`/`W`/`F`/`B`/`I`/`D` with `E203`, `E701` and
bugbear's opt-in `B9xx` ignored, pycodestyle `max-line-length = 110` standing in for `B950`,
`pep257` docstrings, and tests exempt from `D` as they were under `pydocstyle`. `setup.cfg` is gone.
Markdown files are excluded from `ruff format` so fenced code blocks stay untouched, as before.

`make lint` keeps its name and its `MAKE_LINT_ERROR_OUT=1` check mode, so the CI workflow is
unchanged. `ruff check --fix` runs before `ruff format`, as ruff recommends.

Code changes are mechanical:

* `# noqa: B950` becomes `# noqa: E501`, or is dropped where ruff reports the directive unused. The
  46 in `sources/_metadata.py` were leftovers on closing parentheses of lines black had already
  wrapped. `# flake8: noqa: B950` used to exempt the whole file from flake8; `# ruff: noqa: E501`
  only exempts line length, and those files are otherwise clean.
* `# noqa: E402` after `sys.path` edits is dropped because ruff allows imports there.
* The `ruff format` pass, including the PEP 758 `except A, B:` form from `target-version = "py314"`.

Closes forecastingresearch#121
@nikbpetrov
build: manage the local virtual environment with uv
d7254f6 
`make setup-python-env` now creates `.venv` with `uv venv`, which honours `.python-version` instead
of whichever `python3` is first on `PATH`, and installs each requirements file with
`uv pip install`. The install order is unchanged: the `fri-utils` pin from
`requirements.runtime.txt` first, then the root dev requirements, then every job's
`requirements.txt` under `src/`, now in sorted order so the result is deterministic. Deployment is
untouched: Cloud Run images still `pip3 install` the staged `requirements.txt`.

CI installs uv with `astral-sh/setup-uv` and caches on the requirements files. The Makefile fails
with a pointer to the installer when `uv` is missing. `test_runtime_requirements.py` keeps asserting
the install order and the absence of editable installs without naming the installer.
@nikbpetrov
build: unify the gcsfs pin at 2025.7.0
59df4d0 
The naive forecaster and the nightly manager pinned `gcsfs==2024.3.1` while the resolve and
leaderboard jobs pinned `2025.7.0`, the only conflicting pin across the per-job requirements. A single
lockfile needs one version; 2025.7.0 is the one already running in production for the other two jobs.
@nikbpetrov
build: declare dependencies in pyproject.toml and lock them with uv
b0f5ed0 
`pyproject.toml` gains a `[project]` table and PEP 735 dependency groups: `dev` for the lint and
test tools, `runtime` for the shared `fri-utils` package, and one group per Cloud Run job named
after its directory under `src/`. Each job group is a verbatim copy of that job's `requirements.txt`
plus the `runtime` group, and `fri-utils` is a `[tool.uv.sources]` git entry pinned to the same
revision as `requirements.runtime.txt`. Tests keep the groups and the files equal until deployment
reads the lockfile. `uv.lock` resolves all 183 packages once, so every job shares one version of
`pandas`, `pandera` and friends.

`make setup-python-env` is now `uv sync`, which creates the environment on the Python version in
`.python-version` and installs every group; `make lint` and `make test` run through `uv run`, so
nothing needs activating. Setting `UV_PROJECT_ENVIRONMENT` relocates the environment, which is how
automated agents keep off the repository `.venv`. CI adds `uv lock --check` so a stale lockfile fails
the build. `required-version` is the oldest uv verified to read this lockfile.
@nikbpetrov
build: deploy jobs from uv.lock
e1c733c 
Deploy recipes export a job's dependency group from `uv.lock` into the staged `requirements.txt`
(`export-requirements` in `deploy_config.mk`) instead of concatenating `requirements.runtime.txt`
with the job's `requirements.txt`, so every Cloud Run image installs the locked versions rather than
whatever resolves on the day it is built. The Dockerfile template is unchanged: pip installs the
exported file, with the `fri-utils` git revision as a direct reference. Each job Makefile names its
group with `DEPENDENCY_GROUP`, and the 35 requirements files are gone; `pyproject.toml` is the only
place dependencies are declared.

Tests check the deploy contract at that level: every job Makefile declares the group its directory
maps to, every group belongs to a Makefile and includes `runtime`, `make -n deploy` exports that
group after clearing the upload directory, and no `requirements*.txt` is tracked, so a rebased
branch that re-adds one fails loudly instead of being ignored.
@nikbpetrov
build: install job dependencies with uv inside the Cloud Run images
d888f5b 
Stage `pyproject.toml` and `uv.lock` next to each job's code instead of exporting a flat
`requirements.txt`, and have the Dockerfile templates run `uv sync --locked --only-group <job>` with
the image's Python. The build context now carries the lock, `--locked` fails the build on a stale
lock, and the install step is several times faster than pip.

`deploy_config.mk` renders `UV_VERSION` and `DEPENDENCY_GROUP` into the Dockerfile. The uv release
is pinned once in the root Makefile and must match the `setup-uv` pin in CI, which a test enforces.
`COPY` now chowns `/app` to `www-data` so uv can create `/app/.venv`; the venv goes on `PATH` so
`CMD` is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nikbpetrov