You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dce246e
Browse filesBrowse the repository at this point in the historyBrowse files
Add CUDA shared-memory targeting for cross-arch inference (pytorch#21984)
This PR enables lowering models to AOTI-cuda with specific shared-memory
constraint. With this PR, exported pte on one GPU (e.g. A100) can be
infered on other GPUs by runtime cubin rebuilt from ptx file as long as:
1. the GPU running on is in the same or newer series compare with the
GPU exported from
2. the GPU running on has the same or lower shared-memory compared with
what we set.
Below is the performance comparsion. We can see the
shared-memory-controlled version works on different architecture, with
less than 5% performance sacrifice, which is good enough as a general
and fallback solution.
Next step we will support to bundle the native export cubins into a
single pte to support specific optimization across different hardwares
in single pte.
### A100: native export vs. shared-memory-controlled export
Both artifacts were exported on A100 and executed on A100. `Native A100
export` uses the A100's normal 166,912-byte opt-in shared-memory limit;
`A100 export, target SMEM ≤ 101,376 B` constrains kernel selection to
the RTX 5090 limit.
| Prompt tokens | Native A100 export prefill tok/s | A100 export, target
SMEM ≤ 101,376 B prefill tok/s | Prefill delta | Native A100 export
decode tok/s | A100 export, target SMEM ≤ 101,376 B decode tok/s |
Decode delta |
|---:|---:|---:|---:|---:|---:|---:|
| 2,048 | 3,145.9 | 3,061.3 | -2.69% | 62.7 | 61.6 | -1.73%† |
| 8,192 | 3,185.1 | 3,119.6 | -2.06% | 60.4 | 59.5 | -1.43% |
| 32,768 | 2,710.3 | 2,720.9 | +0.39% | 59.8 | 58.7 | -1.79% |
| 130,048 | 1,754.4 | 1,735.6 | -1.07% | 58.3 | 57.7 | -0.97% |
**Result:** On A100, constraining shared memory to the RTX 5090 budget
costs approximately 0–2% steady-state decode throughput and -2.69% to
+0.39% prefill throughput across the tested context lengths.
### RTX 5090: native export vs. the same shared-memory-controlled A100
export
Both artifacts were executed on RTX 5090. `Native RTX 5090 export` was
exported natively on RTX 5090; **`A100 export, target SMEM ≤ 101,376 B`
is byte-for-byte the same A100-exported PTE used in the A100 table
above**.
| Prompt tokens | Native RTX 5090 export prefill tok/s | A100 export,
target SMEM ≤ 101,376 B prefill tok/s | Prefill delta | Native RTX 5090
export decode tok/s | A100 export, target SMEM ≤ 101,376 B decode tok/s
| Decode delta |
|---:|---:|---:|---:|---:|---:|---:|
| 2,048 | 3,544.49 ± 6.73 | 3,468.85 ± 6.70 | -2.13% | 66.662 ± 0.146 |
63.674 ± 0.196 | -4.48% |
| 8,192 | 3,639.93 ± 5.18 | 3,602.15 ± 5.17 | -1.04% | 65.738 ± 0.197 |
62.687 ± 0.041 | -4.64% |
| 32,768 | 3,276.02 ± 1.68 | 3,240.44 ± 1.12 | -1.09% | 65.736 ± 0.222 |
63.046 ± 0.670 | -4.09% |
| 130,048 | 2,282.54 ± 99.50 | 2,296.52 ± 1.74 | +0.61% | 62.047 ± 0.502
| 61.091 ± 0.235 | -1.54% |
**Result:** With exactly 512 generated tokens in every run, the portable
A100 artifact remains within 2.13% of native prefill throughput and
within 4.64% of native decode throughput at every tested length.
0 commit comments