Conversation
The first `torch.profiler.profile(activities=[CUDA])` session in a process
records no device activity: `key_averages()` comes back empty, no line matches
any kernel name, and `bench_kineto` silently returns 0.
Because the first `bench_kineto` call in each test file feeds a division, six
of our own test files abort on a `ZeroDivisionError` four frames away from the
cause:
File "tests/test_bf16.py", line 49, in test_gemm
f'{2 * m * n * k / t / 1e12:4.0f} TFLOPS | '
~~~~~~~~~~~~~~^~~
ZeroDivisionError: division by zero
test_bf16, test_fp8_fp4, test_einsum, test_attention, test_hyperconnection and
test_legacy all fail this way. test_layout survives only because it happens to
guard its division, and instead silently reports its first shape as
`0 us | 0 GB/s`.
This is a property of the profiler, not of DeepGEMM. It reproduces with no
DeepGEMM code at all, and only the first session in the process is affected:
def run(i):
sch = torch.profiler.schedule(wait=0, warmup=1, active=1, repeat=1)
p = torch.profiler.profile(
activities=[torch.profiler.ProfilerActivity.CUDA],
schedule=sch, acc_events=True)
with p:
for _ in range(2):
for _ in range(30): a @ b
torch.cuda.synchronize(); p.step()
print(i, len(p.key_averages().table().split('\n')))
for i in range(4): run(i) # -> 0 1 / 1 10 / 2 10 / 3 10
Adding profiler warmup steps does not help, and neither does dropping the
schedule; opening and closing one throwaway (empty) session does. So this is
CUPTI lazy initialization rather than a schedule problem. Adding
`ProfilerActivity.CPU` also fixes it, but that puts CPU op rows in the table
where they can match kernel-name substrings and trip the existing uniqueness
assertion, so this uses the throwaway session instead. It runs no GPU work at
all and leaves every measured number unchanged.
Also assert that the table is non-empty. An absent kernel name legitimately
means "this kernel did not run" (e.g. the `reduce` name in test_bf16 when there
is no split-K), so that case must keep returning 0; but a table with no device
activity at all is always a profiling failure and should say so rather than
report every kernel as infinitely fast. test_layout already wraps its call in
`try/except AssertionError`, so it is unaffected.
Measured on GB200 (sm_100a, aarch64, cubins built for sm_100), CUDA 13.1,
PyTorch 2.9.0+cu130, third-party/cutlass at the pinned f3fde58.
- unpatched tests/test_bf16.py fails 3/3 runs with the identical traceback;
patched, it passes and reports 272 perf rows, none of them 0.0 us, and an
aggregate "Average speedup over cuBLASLt: 0.894x" that previously could not
be computed at all because the run aborted on the first shape.
- tests/test_layout.py first row: `0 us | 0 GB/s` -> `3 us | 49 GB/s`, which
is what the same shape measures later in the same run; every later row is
unchanged.
- paired first-call check on a real sm100 kernel, both at m=1,n=2112,k=7168
and m=n=k=4096: pre-fix call 0 records nothing while calls 1 and 2 are
fine; post-fix call 0 agrees with calls 1 and 2 to within run-to-run noise.
DiegoCao
changed the title
|
@LyricZhao @soundOfDestiny Could you help review? Thanks. |
|
|
||
|
|
||
| _is_kineto_warmed_up = False | ||
|
|
There was a problem hiding this comment.
🔵 suggestion: _is_kineto_warmed_up is a module-level flag with a check-then-set pattern; it is not thread-safe. Fine for the current single-threaded test usage, but if bench_kineto is ever called from multiple threads, two profiler sessions could be opened concurrently (torch.profiler does not support nested/concurrent sessions). Consider a threading.Lock or a once-style guard if that ever becomes a concern.
🤖 v5
| # NOTES: an empty table means no device activity was recorded at all; without this | ||
| # check every kernel below would silently parse as a zero time | ||
| assert any(line.strip() for line in prof_lines), 'The profiler recorded no device activity' | ||
| kernel_names = (kernel_names, ) if isinstance(kernel_names, str) else kernel_names |
There was a problem hiding this comment.
🔵 suggestion: The assertion any(line.strip() for line in prof_lines) catches the fully-empty-table case, but it would also be satisfied by a table containing only header rows with no kernel rows (if a future torch version emits headers for empty results). A slightly stronger check (e.g. that at least one line contains a time unit such as 'us' or 'ms') would be more robust, though the current check is adequate for the observed failure mode.
🤖 v5
🤖 ds-review-bot Code Reviewv6变更以一次性空 profiler 会话初始化 CUPTI，并在设备活动完全缺失时明确失败；未发现会破坏现有行为的缺陷。 v5This MR fixes a real, well-diagnosed bug: the first torch.profiler CUDA-only session in a process records no device activity (CUPTI lazy initialization), so bench_kineto's parser fell through to its v4p该 MR 修复 bench_kineto 在同一进程中首次调用时，因 CUPTI 惰性初始化导致首个 CUDA profiler 会话不记录任何设备活动、从而所有 kernel 时间被解析为 0 的问题：通过进程级一次性空 profiler 预热，并在解析前增加表非空断言。整体实现简洁，符合既有代码模式，未发现明确的功能或正确性问题。 Files reviewed: 1 |
2 participants
Summary
bench_kinetoreturns0for every kernel on the first call in a process. The firsttorch.profiler.profile(activities=[CUDA])session records no device activity at all —key_averages()comes back empty, nothing matches any kernel name, and the parser falls through to itselse 0branch.Since the first
bench_kinetocall in each test file feeds a division, six of our own test files abort four frames away from the cause:test_bf16,test_fp8_fp4,test_einsum,test_attention,test_hyperconnectionandtest_legacyall fail this way.test_layoutsurvives only because it happens to guard its division — it silently reports its first shape as0 us | 0 GB/sinstead.Root cause
This is a property of the profiler, not of DeepGEMM. It reproduces with no DeepGEMM code at all, and only the first session in the process is affected:
I tried each candidate fix in a fresh process:
wait=0, warmup=1, active=1)wait=1, warmup=2, active=1, 4 iters)ProfilerActivity.CPUMore warmup steps do not help, so this is CUPTI lazy initialization at session level rather than a schedule problem. Adding
ProfilerActivity.CPUalso works, but it puts CPU op rows in the table where they can match kernel-name substrings and trip the existing uniqueness assertion — so this PR uses the throwaway session. Itswithbody is empty, so it runs no GPU work and costs nothing.Changes
_warmup_kineto()— open and close one empty profiler session, once per process, before the measured session.reducename intest_bf16when there is no split-K), so that case must keep returning0; but a table with no device activity at all is always a profiling failure and should say so rather than report every kernel as infinitely fast.test_layoutalready wraps its call intry/except AssertionError, so it is unaffected.Verification
Measured end-to-end on GB200 / sm_100a: capability
(10, 0),get_arch_major() == 10, cubins built forsm_100(cuobjdump --list-elf->kernel.sm_100.cubin), aarch64, CUDA 13.1, PyTorch 2.9.0+cu130,third-party/cutlassat the pinnedf3fde58.Determinism. Unpatched
tests/test_bf16.pyfails 3/3 independent runs with the identical traceback. Patched, it passes.Recovered data point.
tests/test_layout.py, first row:3 us | 49 GB/sis what the same shape measures later in the same run, and every subsequent row is unchanged.Paired first-call check against a real sm100 kernel, at
test_bf16's actual first shape and at a large square:Post-fix call 0 agrees with calls 1 and 2 to within run-to-run noise: the warmup recovers the measurement without shifting it.
Rows produced. Perf rows emitted, and how many of them read
0.0 us:test_bf16test_fp8_fp4test_einsumtest_bf16also now reportsAverage speedup over cuBLASLt: 0.894x, an aggregate that previously could not be computed at all. Coverage spans the sm100 GEMM family:sm100_bf16_gemm,sm100_bf16_k_grouped_gemm,sm100_bf16_m_grouped_gemm_contiguous,sm100_bf16_m_grouped_gemm_masked,sm100_fp8_fp4_gemm_1d1d,sm100_tf32_hc_prenorm_gemm,sm100_bmn_bnk_mn_gemm.Out of scope
Two test files fail on this machine both before and after, for reasons unrelated to this bug:
test_mega_moeneeds at least 6 GPUs (ValueError: device_id cuda:5 is out of range); this box has 4.test_sanitizerhits a JIT assertion under compute-sanitizer's build flags:not std::regex_search(output, std::regex(R"(Local memory used)"))while compilingfp8_gemm_nt_skip_head_mid, i.e. that kernel spills local memory. Reported separately if useful.🤖 Generated with Claude Code