bug: cog build fails with externally-managed-environment error on 0.17.x

[markphelps]

Summary

cog build on 0.17.0–0.17.2 intermittently fails during the Python dependency installation step with:

error: externally-managed-environment

× This environment is externally managed
╰─> This Python installation is managed by uv and should not be modified.

The failure is intermittent because Docker layer caching masks it — when the pip install layer is cached, the command is skipped. A clean build (or any cache invalidation) triggers the error deterministically.

Environment

• Cog version: 0.17.0, 0.17.1, 0.17.2 (all affected)
• Introduced in: PR Use uv and uv run pip for Dockerfile generation #2747 ("Use uv and uv run pip for Dockerfile generation"), merged 2026-02-19
• Not present in: 0.16.x and earlier

Root cause

PR #2747 changed the generated Dockerfile to use uv for package management. The generated install command is:

RUN --mount=type=cache,target=/root/.cache/pip uv run pip install --cache-dir /root/.cache/pip -r /tmp/requirements.txt

uv run executes pip inside a uv-managed Python environment. As of uv 0.5+ and PEP 668, uv-managed Python installations place an EXTERNALLY-MANAGED marker file in the stdlib directory, which causes pip to refuse to install packages.

Reproduction

Minimal cog.yaml

build:
  gpu: false
  python_version: "3.12"
  python_packages:
    - "requests==2.32.3"
predict: "predict.py:Predictor"

Minimal predict.py

from cog import BasePredictor

class Predictor(BasePredictor):
    def setup(self):
        pass

    def predict(self, text: str = "hello") -> str:
        import requests
        return f"ok: {text}"

Steps to reproduce

mkdir /tmp/cog-repro && cd /tmp/cog-repro
# Create cog.yaml and predict.py as above
cog build --no-cache

Expected

Build completes successfully.

Actual

#15 [stage-0  7/12] RUN --mount=type=cache,target=/root/.cache/pip uv run pip install --cache-dir /root/.cache/pip -r /tmp/requirements.txt
#15 1.077 error: externally-managed-environment
#15 1.077
#15 1.077 × This environment is externally managed
#15 1.077 ╰─> This Python installation is managed by uv and should not be modified.
#15 ERROR: process "/bin/sh -c uv run pip install ..." did not complete successfully: exit code: 1

Why it appears intermittent

Docker layer caching. If requirements haven't changed and prior layers are identical, Docker reuses the cached layer and never executes the command. The error surfaces when:

• Building with --no-cache
• Requirements have changed
• Building on a different machine / CI runner
• Docker cache has been pruned

Suggested fix

Replace uv run pip install with uv pip install --system in the Dockerfile generator:

# Before (broken):
RUN --mount=type=cache,target=/root/.cache/pip uv run pip install --cache-dir /root/.cache/pip -r /tmp/requirements.txt

# After (fixed):
RUN --mount=type=cache,target=/root/.cache/uv uv pip install --system --cache-dir /root/.cache/uv -r /tmp/requirements.txt

This is faster (uv's resolver vs pip's), avoids the PEP 668 issue entirely, and is the idiomatic way to install packages in a uv-managed environment.

Workarounds

• Custom Dockerfile: Run cog debug to get the generated Dockerfile, fix the offending line, and build with cog build --dockerfile Dockerfile
• Downgrade to 0.16.12: Last release before the uv migration

References

• PR #2747: Use uv and uv run pip for Dockerfile generation
• Issue #2167: Use uv instead of pip/pyenv
• PEP 668 – Marking Python base environments as "externally managed"
• uv #11729 – externally-managed-environment

[rehan243]

Debugging ML systems can be tricky! A few approaches that help:

• Trace logging: We log the full input/output chain for every request, so
  we can reproduce any issue by replaying the exact inputs
• Deterministic mode: Setting random seeds and using deterministic algorithms
  during debugging makes issues reproducible
• Divide and conquer: When a pipeline fails, test each component in isolation
  with the same inputs to narrow down where things go wrong

Happy to help debug further if you can share a minimal reproduction!