Improve error when cuobjdump is missing or fails

[monperrus]

Problem

KernelRuntime finds a cubin's kernel symbol by shelling out to
$CUDA_HOME/bin/cuobjdump -symbols kernel.cubin and then asserting
DG_HOST_ASSERT(exit_code == 0).

cuobjdump is part of the CUDA toolkit but is packaged separately from
nvcc — e.g. conda's cuda-cuobjdump (or cuda-command-line-tools).
A JIT-only CUDA install that has nvcc but not cuobjdump compiles kernels
fine and then dies here with:

Assertion error (.../csrc/jit/kernel_runtime.hpp:NN): exit_code == 0

which gives the user nothing to go on. call_external_command already
captures the command's combined stdout/stderr (it appends 2>&1), but that
output — which contains the actual cuobjdump: not found / error text — is
discarded on the failure path.

This is easy to hit from the kernels/Hugging Face flow and from minimal
CUDA containers.

Change

Diagnostics only, no behavior change on the success path:

• Check cuobjdump exists up front and, if not, report its expected path and
  how to fix it (install cuda-cuobjdump / cuda-command-line-tools, or point
  CUDA_HOME at a full toolkit).
• On a non-zero exit, surface the command, exit code, and the captured output
  instead of the bare exit_code == 0 assertion.

Reproduce

A CUDA env with nvcc but no cuobjdump (e.g.
conda create -c nvidia cuda-nvcc without cuda-cuobjdump) — any JIT kernel
load reaches this path.

🤖 Generated with Claude Code

[ds-review-bot]

🔵 suggestion: std::filesystem::exists(cuobjdump_path) uses the throwing overload; on exotic failures (e.g. permission-denied on a parent directory) it can throw std::filesystem::filesystem_error instead of the friendly DGException. Consider the noexcept overload std::filesystem::exists(cuobjdump_path, ec) so any filesystem hiccup still funnels into the actionable message. Low impact since the codebase already uses the throwing overload elsewhere (e.g. check_validity).

_{🤖 v5}

[ds-review-bot]

🔵 suggestion: Pre-existing (not introduced here): the command interpolates cuobjdump_path and cubin_path unquoted, so a CUDA_HOME or DG_JIT_CACHE_DIR containing spaces breaks the popen'd shell command. The new error message that echoes the full command actually makes this easier to diagnose, but wrapping both paths in quotes would fix it outright.

_{🤖 v5}

[ds-review-bot]

🔵 suggestion: The existence pre-check only tests that a file exists at $CUDA_HOME/bin/cuobjdump; a present-but-non-executable or mismatched-architecture binary still falls through to the (now well-diagnosed) exit-code path. If you want the very first failure to be maximally actionable, consider also checking std::filesystem::status(...).permissions() for the execute bit in the same pre-flight block — optional, since the improved non-zero-exit reporting already surfaces the underlying error text.

_{🤖 v4f}

[ds-review-bot]

🤖 ds-review-bot Code Review

v6

变更仅改进 cuobjdump 缺失或执行失败时的诊断信息，成功路径未受影响，未发现会破坏现有行为的缺陷。

v5

Commit b75f054 ('jit: actionable error when cuobjdump is missing or fails') changes exactly one file, csrc/jit/kernel_runtime.hpp, and fully matches the issue's intent: (1) an up-front std::filesystem::exists check on $CUDA_HOME/bin/cuobjdump that throws an actionable message naming the expected path and the fix (install cuda-cuobjdump / cuda-command-line-tools, or point CUDA_HOME at a full toolkit), and (2) on non-zero exit, the command, exit code, and the combined stdout+stderr captured by call_external_command (which appends 2>&1, verified in csrc/utils/system.hpp:35-50) are surfaced via DG_HOST_UNREACHABLE instead of the opaque DG_HOST_ASSERT(exit_code == 0). Verified: the success path is behaviorally unchanged (same command string, same symbol parsing, same load_kernel call); DG_HOST_UNREACHABLE(reason) exists in csrc/utils/exception.hpp:39 and accepts the std::string produced by fmt::format; the check is correctly placed inside the #else (non-DG_JIT_USE_LIBRARY_ENUM_KERNELS) branch, so the enum-kernels build path is untouched. Note that exists() does not imply executable — a present-but-non-executable cuobjdump falls through to the exit_code != 0 path, which now prints the shell's error output, so that case remains diagnosable; the up-front check being best-effort is acceptable. Embedding the full captured cuobjdump output in the exception message is fine for diagnostics since that output is typically short. Approve; only non-blocking suggestions below.

v4f

The MR improves the diagnostics around KernelRuntime's use of $CUDA_HOME/bin/cuobjdump -symbols <cubin> without changing success-path behavior. Previously, a CUDA install that had nvcc but not cuobjdump (cuobjdump is packaged separately, e.g. conda cuda-cuobjdump / cuda-command-line-tools) compiled kernels fine and then died with an opaque DG_HOST_ASSERT(exit_code == 0), discarding the captured command output. The change (1) checks cuobjdump existence up front and raises an actionable DG_HOST_UNREACHABLE message that names the expected path and the fix (install the package or point CUDA_HOME at a full toolkit), and (2) on a non-zero exit surfaces the full command, the exit code, and the combined stdout+stderr already captured by call_external_command (which appends 2>&1). The fix is confined to csrc/jit/kernel_runtime.hpp (1 file, +19/-2), is consistent with the existing helper macros (DG_HOST_UNREACHABLE in csrc/utils/exception.hpp) and helper signature (call_external_command in csrc/utils/system.hpp returns tuple<int, string>), and leaves the symbol-parsing success path untouched. Verified: the working tree at HEAD (b75f054) already contains exactly this change, the failure messages correctly include path/command/exit-code/output, and no correctness or compile issues were found; no further edits were required.

Files reviewed: 1
Issues found: 🔵 5 suggestion
Inline comments posted: 3
General comments (无法定位到 diff): 2

---

📍 未定位到 diff 的评论

🔵 suggestion csrc/jit/compiler.hpp:L153: Compiler::disassemble has the same failure shape (cuobjdump invocation, printf of output, then bare DG_HOST_ASSERT(false and "cuobjdump failed")) and no existence check on cuobjdump_path; it is also reachable in JIT-only installs when SASS dumping is enabled. Out of scope for this single-file MR, but a follow-up applying the same existence check / DG_HOST_UNREACHABLE-with-output pattern (or a shared helper) would keep diagnostics consistent. _{🤖 v5}

🔵 suggestion csrc/jit/compiler.hpp:L153: Out of scope for this MR (1-file change), but the sibling --dump-sass path in Compiler still ends in a bare DG_HOST_ASSERT(false and "cuobjdump failed") after printing output to stdout. For consistency with this MR's better diagnostics, consider converting that to the same DG_HOST_UNREACHABLE pattern (command + exit code + captured output) in a follow-up, since that path also uses cuobjdump and would benefit from an exception carrying the details instead of relying on prior printf output. _{🤖 v4f}