[core][sandbox] Add GPU support to Ray Sandboxes via CDI

[klueska]

Description

Ray sandboxes currently have no GPU support. This PR adds it by leveraging CDI (Container Device Interface) to inject GPU device nodes and their supported user-mode driver files into a gVisor sandbox at runtime. The change is implemented as follows.

• Generic CDI framework: We added a vendor-agnostic CDI spec module that can be used to parse CDI specs and merge devices from them with a standard OCI image spec. The implementation of this module is kept free of any Ray imports so it stays a candidate for extraction into a standalone library at some point in the future. As part of this, theAcceleratorManager class gains two new optional methods, get_cdi_kind()/generate_cdi_spec(). These methods are implemented by an AcceleratorManager to both generate a spec with the full set of CDI devices associated with the accelerator, as well as advertise what kind of CDI devices are associated with it (e.g. nvidia.com/gpu).

• NVIDIA specific implementation : We extended the NvidiaGPUAcceleratorManager to be CDI aware and implement the get_cdi_kind()/generate_cdi_spec() methods mentioned above. At present, we generate CDI devices by shelling out to nvidia-ctk cdi generate, but the preferred long-term path is to build (or leverage) a canonical python library for CDI maintained upstream. Since gVisor only supports NVIDIA GPUs at the moment, the GVisorSandboxBackend gates GPU passthrough to CDI kinds it's actually supports (i.e. nvidia.com/gpu ), rejecting anything else rather than attempting unverified passthrough.

• num_gpus and gpu_ids: When creating a Sandbox actor, you can now pass a num_gpus field in addition to the existing cpu and memory fields (similar to what is possible when creating a non-sandboxed actor). When doing so, the sandbox auto-inherits the full Ray GPU assignment derived from num_gpus (with no way to limit it further when calling remote() later on, unlike for cpus and memory). When creating a sandbox via SandboxRuntime.create() (i.e. within a pre-existing actor), there is no num_gpus field supported. Instead, we allow passing a gpu_ids field as an explicit parameter to specify which of the GPUs that have been granted to the surrounding actor should be isolated inside the sandbox. This allows a single actor with multiple GPUs to manage several sandboxes, each pinned to a different GPU.

• 39 new unit tests and 6 real-GPU CI tests: We added new unit test coverage under ray/_common/tests/ and extended the existing suites under ray/experimental/sandbox/tests/ and ray/tests/accelerators/. The real-GPU tests run via two new Buildkite jobs (core: sandbox gpu tests, 1 GPU; core: sandbox multi gpu tests, 4 GPUs), gated behind the same manual-trigger block the existing multi-GPU core jobs use.

Validated end-to-end on an NVIDIA L4 GPU and a machine with 4 NVIDIA GB200 GPUs (ARM64/Grace CPU).

AI assistance

Developed with Claude Code (Sonnet 5), including driving the hardware validation above. I reviewed every changed line and take responsibility for the change end-to-end.

Related issues

No tracking issue. Sibling of other active sandbox work: image cache LRU eviction (#65748), per-sandbox netns (#65820), HTTP API (#65633), gRPC facade (#65839), multi-uid netns (#65823); none overlap with this change.

Duplicate check

Searched open/closed issues and PRs for sandbox gpu, CDI, gvisor, nvproxy, nvidia-ctk, and container device interface — nothing addresses GPU or CDI support for Ray Sandboxes.

Testing

• test_gvisor_gpu.py (real nvidia-ctk generation, device injection, --nvproxy boot, nvidia-smi inside the sandbox, and a real CUDA kernel launch via NVIDIA's vectorAdd sample) — passes on an NVIDIA L4 GPU (x86_64). On a machine with 4 NVIDIA GB200 GPUs (ARM64/Grace CPU), everything passes except the new CUDA kernel test: test_sandbox_gpu_cuda_vectoradd_runs_a_real_kernel fails with cudaErrorSystemNotReady (802) — real CUDA context creation is currently broken inside a sandbox on GB200/NVSwitch-fabric hardware. Traced this to gVisor's nvproxy: every ioctl in the CUDA/UVM init sequence succeeds at the transport layer, but the fabric-readiness state one of those ioctls' responses carries never resolves to "ready" — an upstream gVisor gap on NVSwitch systems, not something fixable from this PR's CDI code. nvidia-smi (NVML-only, no CUDA context) still works fine on GB200; only real CUDA workloads are affected. Left as a genuinely failing test rather than xfail, since it's real, disclosed, unresolved behavior on this hardware class.

• test_gvisor_multi_gpu.py (both tests, needing 4 GPUs) — passes on the GB200 machine, each sandbox pinned to a distinct GPU.

• python/ray/experimental/sandbox/tests/ (full directory, including the above) + python/ray/_common/tests/test_cdi.py + test_cdi_lib.py + python/ray/tests/accelerators/test_nvidia_gpu.py — on the L4: 131 passed, 1 skipped. On the GB200 machine: 130 passed, 1 known failure (test_sandbox_gpu_cuda_vectoradd_runs_a_real_kernel, see above), 1 skipped.

• pre-commit run — passed (SKIP=vale; its Go/cgo toolchain doesn't build in this environment).

• ARM64 hosts booting with 64K pages (e.g. NVIDIA Grace/GB200) need runsc built from source — see "Testing manually" below.

Testing manually

Building the published ray image needs a wheel via ./build-wheel.sh, which needs Anyscale's internal cr.ray.io/Bazel-remote-cache access to build in reasonable time. Instead, overlay this branch's changed .py files onto a stock published rayproject/ray:2.58.0 image — same site-packages layout, no core rebuild needed since nothing outside pure Python changed. Needs a machine with an NVIDIA GPU, Docker, and the nvidia container runtime.

Dockerfile: https://gist.github.com/klueska/9f1297a5d7f5a7b7cf9b31b3fb393feb

Save it locally, then build from a checkout of this branch (the repo itself is the build context):

curl -fsSL -o Dockerfile.sandbox-gpu-test https://gist.githubusercontent.com/klueska/9f1297a5d7f5a7b7cf9b31b3fb393feb/raw/f78740b83836b38c79ea4d57e1057fed269a3af2/Dockerfile.sandbox-gpu-test
docker build -t ray-cdi-gpu-test:latest -f Dockerfile.sandbox-gpu-test .

Run it. --gpus all for GPU access; --privileged --user root are needed for gVisor in this specific nested-Docker test rig — a KubeRay/bare-metal node shouldn't need the non-root workaround:

# Pure-logic CDI tests + real-nvidia-ctk accelerator tests, no gVisor/root needed:
docker run --rm --gpus all ray-cdi-gpu-test:latest python3 -m pytest -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/_common/tests/test_cdi.py \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/_common/tests/test_cdi_lib.py \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/tests/accelerators/test_nvidia_gpu.py

# The real end-to-end GPU sandbox test: exercises real nvidia-ctk generation, device injection,
# and gVisor --nvproxy passthrough. No host setup needed -- nvidia-persistenced can be running
# or not, both are handled.
docker run --rm --privileged --user root --gpus all ray-cdi-gpu-test:latest \
  python3 -m pytest -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/test_gvisor_gpu.py

# The two 4-GPU tests: one with 4 separate Sandbox actors, one with a single actor managing 4
# sandboxes via SandboxRuntime.create(gpu_ids=...). Both check each sandbox gets a distinct GPU.
# Needs a machine with 4 GPUs; `--gpus all` must expose at least 4.
docker run --rm --privileged --user root --gpus all ray-cdi-gpu-test:latest \
  python3 -m pytest -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/test_gvisor_multi_gpu.py

# Full sandbox suite (mocked-GPU config/scheduling tests + real-gVisor CPU and GPU tests):
docker run --rm --privileged --user root --gpus all -e TEST_SANDBOX=1 -e RAY_SANDBOX_IGNORE_CGROUPS=1 \
  ray-cdi-gpu-test:latest python3 -m pytest -v \
  /home/ray/anaconda3/lib/python3.10/site-packages/ray/experimental/sandbox/tests/

[gemini-code-assist]

Code Review

This pull request introduces GPU support for Ray Sandboxes using the Container Device Interface (CDI) to safely expose GPUs inside isolated gVisor containers. It adds vendor-agnostic CDI spec parsing and OCI merging utilities, integrates CDI spec generation into the NVIDIA GPU accelerator manager, and updates the sandbox configuration and gVisor backend to validate and pass through scheduled GPUs. The review feedback highlights several robust improvements, including resolving inconsistent default type handling and potential None values in CDI parsing, failing fast in the gVisor backend to prevent unnecessary image pulls, elevating the log level for nvidia-ctk failures to aid troubleshooting, and replacing os.mknod with safer socket and FIFO creation methods to ensure compatibility in unprivileged environments.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 7d62005. Configure here.}

[dstrodtman]

Docs-side review from the Anyscale docs team (Douglas Strodtman). Claude Code assisted; I read every comment and stand behind each one.

Scope: style, grammar, and docs conventions on doc/source/ray-core/sandboxes.md only, against the Ray documentation style guide. I'm not reviewing the GPU/CDI code or the CI changes; those belong to the Ray core maintainers, and the bot threads already cover them. Nothing here is a merge blocker from the docs side, so this is a COMMENT, not a change request.

The new GPU section is clear and well-organized, and the cross-links between Requirements and the GPU section read well. The suggestions below are all one thing: the style guide asks you to avoid em dashes, semicolons, and parentheticals in prose and restructure into separate sentences instead. The new prose introduced a few of each (Vale independently flags the semicolons and parens; it has no em-dash rule, which is why they need a human eye). Each suggestion preserves your technical claims exactly and only reshapes the punctuation, plus a couple of small word-choice items the guide calls out (actually as filler, since → because for causation).

Take or leave any of them; they're all optional polish. Once you're happy with the prose, the docs side has no objection.

[dstrodtman]

The guide asks to avoid semicolons and parentheticals in prose and split into separate sentences. This also lifts the trailing no other configuration required out of the comma-splice into its own sentence.

Suggested change

	To give sandboxes GPU access (see [GPU access](#gpu-access) below), additionally install `nvidia-container-toolkit-base` (which provides `nvidia-ctk`) on GPU worker nodes; Ray uses it to generate a [CDI](https://github.com/cncf-tags/container-device-interface) spec each time a sandbox actor requests a GPU, no other configuration required.
	To give sandboxes GPU access, also install `nvidia-container-toolkit-base` on GPU worker nodes. This package provides `nvidia-ctk`, which Ray uses to generate a [CDI](https://github.com/cncf-tags/container-device-interface) spec each time a sandbox actor requests a GPU. No other configuration is required. See [GPU access](#gpu-access).

[dstrodtman]

Splits the semicolon into two sentences. I also matched the parallel sentence under Schedule a Sandbox actor with custom resources, which opens with Because Sandbox is a standard Ray actor, and swapped like for a plain comparison since the guide reserves like for genuine comparisons.

Suggested change

	Since `Sandbox` is a standard Ray actor, request GPUs with `num_gpus` like any other actor; the sandbox automatically exposes exactly those GPUs inside the isolated container, the same way it inherits `num_cpus` and `memory`. This requires `nvidia-container-toolkit-base` on the node (see [Requirements](#requirements)) and a gVisor build with `--nvproxy` support.
	Because `Sandbox` is a standard Ray actor, request GPUs with `num_gpus` the same way you would for any other actor. The sandbox automatically exposes exactly those GPUs inside the isolated container, the same way it inherits `num_cpus` and `memory`. This requires `nvidia-container-toolkit-base` on the node (see [Requirements](#requirements)) and a gVisor build with `--nvproxy` support.

[dstrodtman]

This line has two em dashes and a parenthetical, which the guide asks you to restructure into separate sentences. I also dropped actually (the guide lists it as filler) and changed since to because for the causal clause. Same technical content, just reshaped.

Suggested change

GPU sandbox creation requires `ray.get_gpu_ids()` to return at least one GPU for the calling actor or task — Ray populates it when `num_gpus` is requested, which is how the sandbox knows which GPUs are actually its to use. If it's empty (for example, a custom actor that never requested `num_gpus`, or driver code running outside any Ray task or actor), creating a GPU sandbox fails with a clear error rather than guessing. There's no way to opt in without going through Ray's `num_gpus` scheduling — setting `CUDA_VISIBLE_DEVICES` yourself has no effect here, since this checks Ray's own resource assignment, not that environment variable.

GPU sandbox creation requires `ray.get_gpu_ids()` to return at least one GPU for the calling actor or task. Ray populates it when `num_gpus` is requested, which is how the sandbox knows which GPUs are its to use. If it's empty, creating a GPU sandbox fails with a clear error rather than guessing. This happens, for example, for a custom actor that never requested `num_gpus`, or for driver code running outside any Ray task or actor. There's no way to opt in without going through Ray's `num_gpus` scheduling. Setting `CUDA_VISIBLE_DEVICES` yourself has no effect here, because this checks Ray's own resource assignment, not that environment variable.

[dstrodtman]

Em dash → sentence break, per the guide.

Suggested change

	To give several sandboxes managed by one actor each their own GPU, use `SandboxRuntime` directly instead of `Sandbox` — a `Sandbox` actor is always exactly one actor to one sandbox, so it can't do this. Schedule the actor with `num_gpus=N`, then pass an explicit `gpu_ids=[id]` to each `SandboxRuntime.create()` call, one call per GPU the actor was assigned:
	To give several sandboxes managed by one actor each their own GPU, use `SandboxRuntime` directly instead of `Sandbox`. A `Sandbox` actor is always exactly one actor to one sandbox, so it can't do this. Schedule the actor with `num_gpus=N`, then pass an explicit `gpu_ids=[id]` to each `SandboxRuntime.create()` call, one call per GPU the actor was assigned: