CUDA initialization fails when an Ada GPU and a Blackwell GPU are visible together

[3ricj]

NVIDIA Open GPU Kernel Modules Version

595.71.05

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 24.04.4 LTS

Kernel Release

.8.0-111-generic NVIDIA#111-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 11 23:16:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-c799a796-c1e5-9ccb-f458-80042d45a6e3) GPU 1: NVIDIA GeForce RTX 4090 (UUID: GPU-65f9d914-5893-3e19-ec99-edce2d01ae58) GPU 2: NVIDIA GeForce RTX 5090 (UUID: GPU-6638f5a7-d57d-b4fc-2cf5-73e5661f3591) GPU 3: NVIDIA GeForce RTX 5090 (UUID: GPU-37ae5dd4-ab38-5c80-fb72-789a0505da91)

Describe the bug

CUDA initialization fails when an Ada GPU and a Blackwell GPU are visible together

Summary

CUDA initialization fails with CUDA_ERROR_NOT_INITIALIZED when an RTX 4090 (Ada, AD102) and an RTX 5090 (Blackwell, GB202/GB200-family reporting) are visible in the same process. Each GPU works by itself, and two RTX 5090 GPUs work together.

The failure is caused by UVM rejecting a mixed coherent/non-coherent VA space during UVM_REGISTER_GPU. The 5090 is treated as coherent, while the 4090 is not, so uvm_va_space_register_gpu() returns NV_ERR_INVALID_DEVICE.

Environment

• Driver branch: NVIDIA open GPU kernel modules, 595.71.05-based patched tree
• GPUs:
  • RTX 4090, Ada, PCI 0000:06:00.0
  • RTX 5090, Blackwell, PCI 0000:09:00.0
  • RTX 5090, Blackwell, PCI 0000:0D:00.0
• Kernel: Linux 6.8.0-111-generic

Reproducer

Minimal CUDA Driver API program:

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 ~/cuda-init-debug/driver_probe
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=2 ~/cuda-init-debug/driver_probe
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1,2 ~/cuda-init-debug/driver_probe
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=2,1 ~/cuda-init-debug/driver_probe
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=2,3 ~/cuda-init-debug/driver_probe

Observed before the fix:

CUDA_VISIBLE_DEVICES=1     # RTX 4090 alone: success
CUDA_VISIBLE_DEVICES=2     # RTX 5090 alone: success
CUDA_VISIBLE_DEVICES=1,2   # RTX 4090 + RTX 5090: cuInit -> CUDA_ERROR_NOT_INITIALIZED
CUDA_VISIBLE_DEVICES=2,1   # RTX 5090 + RTX 4090: cuInit -> CUDA_ERROR_NOT_INITIALIZED
CUDA_VISIBLE_DEVICES=2,3   # RTX 5090 + RTX 5090: success

Root Cause

uvm_va_space_register_gpu() rejects mixing GPUs whose parent coherency classification differs:

// Mixing coherent and non-coherent GPUs is not supported
for_each_va_space_gpu(other_gpu, va_space) {
    if (uvm_parent_gpu_is_coherent(gpu->parent) != uvm_parent_gpu_is_coherent(other_gpu->parent)) {
        status = NV_ERR_INVALID_DEVICE;
        goto done;
    }
}

Instrumentation showed the failing mixed case hit exactly this path:

register_gpu reject mixed coherency
gpuCoherent=1 otherCoherent=0
gpuArch=0x1b0 otherArch=0x190
rmStatus=0x26 (NV_ERR_INVALID_DEVICE)

This NV_ERR_INVALID_DEVICE was returned through the UVM_REGISTER_GPU ioctl payload, while the Linux ioctl syscall itself returned 0. libcuda surfaced the UVM payload failure as CUDA_ERROR_NOT_INITIALIZED.

Minimal Fix

Treat Ada and newer GPUs as coherent in uvm_parent_gpu_is_coherent() when using the BAR1/SYS_COH P2P path:

diff --git a/kernel-open/nvidia-uvm/uvm_gpu.h b/kernel-open/nvidia-uvm/uvm_gpu.h
@@
-    if (parent_gpu->rm_info.gpuArch >= NV2080_CTRL_MC_ARCH_INFO_ARCHITECTURE_GB100)
+    if (parent_gpu->rm_info.gpuArch >= NV2080_CTRL_MC_ARCH_INFO_ARCHITECTURE_AD100)
         return true;

Result After Fix

After rebuilding and reloading with the change above:

CUDA_VISIBLE_DEVICES=1     # success
CUDA_VISIBLE_DEVICES=2     # success
CUDA_VISIBLE_DEVICES=1,2   # success
CUDA_VISIBLE_DEVICES=2,1   # success
CUDA_VISIBLE_DEVICES=2,3   # success

The mixed Ada/Blackwell pair still reports asymmetric BAR1 atomics (1/0 or 0/1), but that was not the direct CUDA initialization failure. Forcing BAR1 atomics to 0/0 did not fix the failure. The decisive failure was UVM's mixed coherency rejection.

To Reproduce

well, anything which calls cuda init -- I was using the below test code:

#include <cuda.h>

#include <cstdio>

static const char *name_for(CUresult result) {
    const char *name = nullptr;
    if (cuGetErrorName(result, &name) != CUDA_SUCCESS || !name) {
        return "<unknown>";
    }
    return name;
}

static const char *string_for(CUresult result) {
    const char *text = nullptr;
    if (cuGetErrorString(result, &text) != CUDA_SUCCESS || !text) {
        return "<unknown>";
    }
    return text;
}

static void print_result(const char *call, CUresult result) {
    std::printf("%s -> %d (%s): %s\n", call, static_cast<int>(result),
                name_for(result), string_for(result));
}

int main() {
    CUresult result = cuInit(0);
    print_result("cuInit", result);
    if (result != CUDA_SUCCESS) {
        return 1;
    }

    int count = -1;
    result = cuDeviceGetCount(&count);
    print_result("cuDeviceGetCount", result);
    std::printf("visible_count=%d\n", count);
    if (result != CUDA_SUCCESS) {
        return 1;
    }

    for (int i = 0; i < count; ++i) {
        CUdevice device{};
        result = cuDeviceGet(&device, i);
        std::printf("device=%d ", i);
        print_result("cuDeviceGet", result);
        if (result != CUDA_SUCCESS) {
            continue;
        }

        char name[256]{};
        result = cuDeviceGetName(name, sizeof(name), device);
        std::printf("device=%d ", i);
        print_result("cuDeviceGetName", result);

        char pci[32]{};
        result = cuDeviceGetPCIBusId(pci, sizeof(pci), device);
        std::printf("device=%d ", i);
        print_result("cuDeviceGetPCIBusId", result);

        CUuuid uuid{};
        result = cuDeviceGetUuid(&uuid, device);
        std::printf("device=%d ", i);
        print_result("cuDeviceGetUuid", result);

        int major = -1;
        int minor = -1;
        cuDeviceGetAttribute(&major, CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR, device);
        cuDeviceGetAttribute(&minor, CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR, device);

        std::printf("device=%d name=%s pci=%s uuid=", i, name, pci);
        for (int j = 0; j < 16; ++j) {
            std::printf("%02x", static_cast<unsigned char>(uuid.bytes[j]));
        }
        std::printf(" cc=%d.%d\n", major, minor);

        CUcontext context = nullptr;
        result = cuCtxCreate(&context, nullptr, 0, device);
        std::printf("device=%d ", i);
        print_result("cuCtxCreate", result);
        if (result == CUDA_SUCCESS) {
            result = cuCtxDestroy(context);
            std::printf("device=%d ", i);
            print_result("cuCtxDestroy", result);
        }
    }

    return 0;
}

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response

[aikitoria]

Interesting. I don't think changing the earlier GPUs to coherent is correct though?