Skip to content

fix: auto-detect CUDA_HOME from bundled nvidia headers on Linux - #2291

Open
abendrothj wants to merge 1 commit into
exo-explore:mainfrom
abendrothj:fix/cuda-home-autodetect
Open

fix: auto-detect CUDA_HOME from bundled nvidia headers on Linux#2291
abendrothj wants to merge 1 commit into
exo-explore:mainfrom
abendrothj:fix/cuda-home-autodetect

Conversation

@abendrothj

Copy link
Copy Markdown

Problem

MLX's CUDA backend JIT-compiles kernels with NVRTC at runtime, and NVRTC needs the CUDA headers. If neither CUDA_HOME nor CUDA_PATH is set, compilation fails with:

Can not find locations of CUDA headers

On a pip-installed Linux CUDA setup the headers are present — nvidia-cuda-runtime ships them under the nvidia namespace package — but nothing exports their location, so MLX can't find them.

The failure is badly timed: it doesn't happen at model load, but at the first NVRTC-compiled kernel. In my case that was the distributed send in ring prefill, so a node came up healthy, loaded weights, and only died once inference actually started.

Fix

In the runner bootstrap, when neither variable is set and we're on Linux, resolve the bundled headers from the nvidia namespace package and set CUDA_HOME.

specification = importlib.util.find_spec("nvidia")
...
candidate = Path(location) / "cuda_runtime"
if (candidate / "include").is_dir():
    os.environ["CUDA_HOME"] = str(candidate)

Deliberately conservative:

  • No-op unless Linux. Returns immediately on darwin.
  • Never overrides an existing setting. A system CUDA install or an explicit CUDA_HOME/CUDA_PATH wins.
  • Never guesses. Only sets the variable if the include/ directory actually exists.

So a machine with a working CUDA toolchain is unaffected; this only fills in the gap for pip-provided CUDA.

Tests

src/exo/worker/tests/unittests/test_runner/test_cuda_home.py covers the detection, the non-Linux skip, the existing-value skip, and the missing-headers case.

Note on overlap

#2103 and #2129 also touch bootstrap.py, but only the MLX_METAL_FAST_SYNCH block — neither sets CUDA_HOME. Different hunk, no functional overlap; happy to rebase behind either if they land first.

MLX's CUDA backend JIT-compiles kernels with NVRTC at runtime (first
triggered by the distributed send in ring prefill) and fails with 'Can not
find locations of CUDA headers' unless CUDA_HOME/CUDA_PATH is set. The pip
nvidia-cuda-runtime package ships the headers; resolve them from the nvidia
namespace package in the runner bootstrap when nothing is configured.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant