[Bug][CUDA] A tensor with a leading zero extent compiles, and the generated kernel faults with cudaErrorIllegalAddress at the first StreamSync

[lackjava-creator]

Environment

• TVM version: main @ 48242ec33403f2b6e4fac6e763ca7a683fb9d5df (2026-09-03 21:20:55 -0400, reports 0.26.dev0); also present on release 0.26.0, 0.19.0 and 0.12.0.

• Build / install: source build (/home/lxx/tvm-main), USE_CUDA=ON, nvcc 12.2, LLVM 15

• OS / Python: Ubuntu 24.04.3 LTS, x86_64 (Xeon E5-2698 v4), Python 3.10

• GPU / driver: Tesla V100-DGXS-32GB, driver 535.309.01, CUDA_VISIBLE_DEVICES=0, sm_70

• Target: cuda. opt_level 3 and 0 both fail. llvm is clean.

• Every run is one case per fresh process — cudaErrorIllegalAddress is sticky for the life of a CUDA context.

• Pipeline coverage (relax.get_pipeline("default_build") performs no operator fusion, so all three are reported separately). Verified 2026-09-06 on main @ 48242ec, CUDA, one pipeline per fresh process (a cudaErrorIllegalAddress poisons the context for the life of the process):

  • get_pipeline("default_build") (no FuseOps/FuseTIR) — fails, 2/2 fresh processes
  • get_pipeline("zero") (fuses) — fails, fresh process
  • forced FuseOps + FuseTIR appended to the legalize/fold sequence — fails, fresh process

Minimal reproducer

Single Neg node with input X : float32[0]. model.onnx + feed.npz + run.py attached (a MaxPool variant, model_maxpool_alt.onnx + feed_maxpool_alt.npz, is included to show the trigger is the shape, not the operator).

import numpy as np, onnx, tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx
model = onnx.load("model.onnx"); feed = dict(np.load("feed.npz"))     # X: float32[0]
mod = from_onnx(model, shape_dict={k: list(v.shape) for k, v in feed.items()}, keep_params_in_input=False)
order = [i.name for i in model.graph.input]
with tvm.transform.PassContext(opt_level=3):
    ex = tvm.compile(relax.get_pipeline("default_build")(mod), target="cuda")   # succeeds
dev = tvm.cuda(0)
out = relax.VirtualMachine(ex, dev)["main"](*[tvm.runtime.tensor(np.ascontiguousarray(feed[n]), dev) for n in order])
print(np.asarray(out.numpy()))                                                  # <-- faults here

Expected vs actual

• Expected: an empty float32 tensor of shape (0,). numpy.negative(numpy.zeros((0,), numpy.float32)) is an empty array; onnxruntime returns the same; the identical program on TVM's llvm target returns the same.
• Actual: tvm.compile succeeds and the VM call succeeds; the failure surfaces at the first synchronisation:

CUDA Runtime Error: cudaErrorIllegalAddress (700)
  cuda_device_api.cc:279  CUDADeviceAPI::StreamSync
  <- runtime/tensor.cc:104 Tensor::CopyToBytes

After this the whole CUDA context is poisoned; every later allocation in the same process fails.

• Because the fault is asynchronous, a validator that never copies the empty result back to the host will report a pass. (This is why an earlier bisect of ours wrongly recorded a "(0.12, 0.19] regression" — the 0.12 run never triggered the sync. Corrected: the defect is present on 0.12, 0.19, 0.26.0 and main alike.)

Root cause (if known)

The scheduled PrimFunc for shape (0,) carries no thread binding at all — the dlight/GPU schedule gives up on the zero-extent loop instead of binding it or guarding it. The launched kernel therefore addresses memory it does not own.

Relevant prior art in the tree: merged PR #7273 deliberately guaranteed at least one block for empty tensors on the (then) GPU path. That guarantee does not hold on the current Relax/dlight CUDA pipeline.

llvm is clean at both opt levels, so the defect is in the CUDA scheduling path, not in Neg.

Why this is a bug (not tolerance / not undefined behaviour)

A zero-extent tensor is legal — ONNX permits it, numpy defines every operation on it, and TVM's own llvm target handles it. Compiling to a kernel that performs an illegal memory access is a memory-safety defect: it corrupts the CUDA context of the whole process, and because the fault is asynchronous it can be attributed to whatever unrelated code happens to synchronise next. Even if one argued the empty case should be rejected, silently emitting a faulting kernel is not an acceptable rejection.

How found

Found by EquiAutomaton (equivalence-graph differential testing against onnxruntime, TVM CUDA vs TVM llvm). Depth-0 — a single Neg node. 15 observations in the original 0.12 campaign; re-verified on main @ 48242ec in a fresh process, 2/2.

Reproducer archive

TVM-C2-reproducer.zip

Triage

• Needs triage