Skip to content

Commit dce246e

Browse files
authored
Add CUDA shared-memory targeting for cross-arch inference (pytorch#21984)
This PR enables lowering models to AOTI-cuda with specific shared-memory constraint. With this PR, exported pte on one GPU (e.g. A100) can be infered on other GPUs by runtime cubin rebuilt from ptx file as long as: 1. the GPU running on is in the same or newer series compare with the GPU exported from 2. the GPU running on has the same or lower shared-memory compared with what we set. Below is the performance comparsion. We can see the shared-memory-controlled version works on different architecture, with less than 5% performance sacrifice, which is good enough as a general and fallback solution. Next step we will support to bundle the native export cubins into a single pte to support specific optimization across different hardwares in single pte. ### A100: native export vs. shared-memory-controlled export Both artifacts were exported on A100 and executed on A100. `Native A100 export` uses the A100's normal 166,912-byte opt-in shared-memory limit; `A100 export, target SMEM ≤ 101,376 B` constrains kernel selection to the RTX 5090 limit. | Prompt tokens | Native A100 export prefill tok/s | A100 export, target SMEM ≤ 101,376 B prefill tok/s | Prefill delta | Native A100 export decode tok/s | A100 export, target SMEM ≤ 101,376 B decode tok/s | Decode delta | |---:|---:|---:|---:|---:|---:|---:| | 2,048 | 3,145.9 | 3,061.3 | -2.69% | 62.7 | 61.6 | -1.73%† | | 8,192 | 3,185.1 | 3,119.6 | -2.06% | 60.4 | 59.5 | -1.43% | | 32,768 | 2,710.3 | 2,720.9 | +0.39% | 59.8 | 58.7 | -1.79% | | 130,048 | 1,754.4 | 1,735.6 | -1.07% | 58.3 | 57.7 | -0.97% | **Result:** On A100, constraining shared memory to the RTX 5090 budget costs approximately 0–2% steady-state decode throughput and -2.69% to +0.39% prefill throughput across the tested context lengths. ### RTX 5090: native export vs. the same shared-memory-controlled A100 export Both artifacts were executed on RTX 5090. `Native RTX 5090 export` was exported natively on RTX 5090; **`A100 export, target SMEM ≤ 101,376 B` is byte-for-byte the same A100-exported PTE used in the A100 table above**. | Prompt tokens | Native RTX 5090 export prefill tok/s | A100 export, target SMEM ≤ 101,376 B prefill tok/s | Prefill delta | Native RTX 5090 export decode tok/s | A100 export, target SMEM ≤ 101,376 B decode tok/s | Decode delta | |---:|---:|---:|---:|---:|---:|---:| | 2,048 | 3,544.49 ± 6.73 | 3,468.85 ± 6.70 | -2.13% | 66.662 ± 0.146 | 63.674 ± 0.196 | -4.48% | | 8,192 | 3,639.93 ± 5.18 | 3,602.15 ± 5.17 | -1.04% | 65.738 ± 0.197 | 62.687 ± 0.041 | -4.64% | | 32,768 | 3,276.02 ± 1.68 | 3,240.44 ± 1.12 | -1.09% | 65.736 ± 0.222 | 63.046 ± 0.670 | -4.09% | | 130,048 | 2,282.54 ± 99.50 | 2,296.52 ± 1.74 | +0.61% | 62.047 ± 0.502 | 61.091 ± 0.235 | -1.54% | **Result:** With exactly 512 generated tokens in every run, the portable A100 artifact remains within 2.13% of native prefill throughput and within 4.64% of native decode throughput at every tested length.
1 parent 20ad5ee commit dce246e

5 files changed

Lines changed: 197 additions & 0 deletions

File tree

‎.github/workflows/cuda.yml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,104 @@ jobs:
159159
./cmake-out/executor_runner --model_path "./${model}.pte" --data_path "./aoti_cuda_blob.ptd"
160160
done
161161
162+
export-cuda-target-smem-cross-arch:
163+
name: export-cuda-target-smem-cross-arch-a100
164+
needs: [changed-files, run-decision]
165+
if: |
166+
contains(needs.changed-files.outputs.changed-files, 'backends/cuda') ||
167+
contains(needs.changed-files.outputs.changed-files, 'backends/aoti') ||
168+
contains(needs.changed-files.outputs.changed-files, '.github/workflows/cuda.yml') ||
169+
needs.run-decision.outputs.is-full-run == 'true'
170+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
171+
permissions:
172+
id-token: write
173+
contents: read
174+
with:
175+
timeout: 90
176+
runner: mt-l-x86iavx512-11-125-a100
177+
gpu-arch-type: cuda
178+
gpu-arch-version: "13.0"
179+
use-custom-docker-registry: false
180+
submodules: recursive
181+
upload-artifact: cuda-target-smem-cross-arch
182+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
183+
script: |
184+
set -eux
185+
186+
PYTHON_EXECUTABLE=python ./install_executorch.sh
187+
export LD_LIBRARY_PATH="/opt/conda/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
188+
# A10G (sm_86) supports 99 KiB of opt-in shared memory per block.
189+
export ET_CUDA_TARGET_SMEM_BYTES=101376
190+
# The A100 exporter has an AVX-512 host CPU while the A10G runner has
191+
# AVX2. Keep the generated AOTI host wrapper compatible with both.
192+
export ATEN_CPU_CAPABILITY=avx2
193+
export TORCHINDUCTOR_CPP_MARCH=x86-64-v3
194+
195+
python - <<'PY'
196+
import torch
197+
198+
capability = torch.cuda.get_device_capability()
199+
assert capability == (8, 0), f"Expected A100 sm_80 exporter, got {capability}"
200+
PY
201+
202+
for model in linear sdpa; do
203+
model_dir="${RUNNER_ARTIFACT_DIR}/${model}"
204+
mkdir -p "${model_dir}"
205+
python -m examples.cuda.scripts.export \
206+
--model_name="${model}" \
207+
--output_dir "${model_dir}"
208+
done
209+
210+
test-cuda-target-smem-cross-arch:
211+
name: test-cuda-target-smem-cross-arch-a10g
212+
needs: [export-cuda-target-smem-cross-arch]
213+
if: needs.export-cuda-target-smem-cross-arch.result == 'success'
214+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
215+
permissions:
216+
id-token: write
217+
contents: read
218+
with:
219+
timeout: 90
220+
runner: mt-l-x86aavx2-29-113-a10g
221+
gpu-arch-type: cuda
222+
gpu-arch-version: "13.0"
223+
use-custom-docker-registry: false
224+
submodules: recursive
225+
download-artifact: cuda-target-smem-cross-arch
226+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
227+
script: |
228+
set -eux
229+
230+
PYTHON_EXECUTABLE=python ./install_executorch.sh
231+
export LD_LIBRARY_PATH="/opt/conda/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
232+
233+
python - <<'PY'
234+
import torch
235+
236+
capability = torch.cuda.get_device_capability()
237+
assert capability == (8, 6), f"Expected A10G sm_86 runner, got {capability}"
238+
properties = torch.cuda.get_device_properties(0)
239+
target = 101376
240+
assert properties.shared_memory_per_block_optin >= target, (
241+
properties.shared_memory_per_block_optin,
242+
target,
243+
)
244+
PY
245+
246+
cmake -DCMAKE_BUILD_TYPE=Release \
247+
-DEXECUTORCH_BUILD_CUDA=ON \
248+
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
249+
-DPYTHON_EXECUTABLE=python \
250+
-Bcmake-out .
251+
cmake --build cmake-out --target executor_runner -j4
252+
253+
for model in linear sdpa; do
254+
model_dir="${RUNNER_ARTIFACT_DIR}/${model}"
255+
./cmake-out/executor_runner \
256+
--model_path "${model_dir}/${model}.pte" \
257+
--data_path "${model_dir}/aoti_cuda_blob.ptd"
258+
done
259+
162260
unittest-cuda:
163261
name: unittest-cuda
164262
needs: [changed-files, run-decision]

‎backends/cuda/BUCK‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ fbcode_target(
9393
name = "cuda_backend",
9494
srcs = [
9595
"cuda_backend.py",
96+
"target_smem.py",
9697
"cuda_weight_collector.py",
9798
],
9899
visibility = ["PUBLIC"],

‎backends/cuda/cuda_backend.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
from executorch.backends.cuda.passes.replace_int64_floordiv import (
3131
ReplaceInt64FloorDivWithFloatPass,
3232
)
33+
from executorch.backends.cuda.target_smem import target_smem_context
3334
from executorch.backends.cuda.triton.replacement_pass import (
3435
ReplaceEdgeOpWithTritonOpPass,
3536
)
@@ -855,6 +856,7 @@ def get_extra_aoti_compile_context_manager(
855856
compilation for the CUDA backend. Each manager is documented at
856857
its own `enter_context` call site below.
857858
859+
The optional shared-memory target applies to every CUDA compilation.
858860
The low-memory export monkey-patch (CPU clones for mutated buffers)
859861
is gated on the ``low_memory_mode`` compile spec — only models that
860862
explicitly opt in (currently Qwen3.5 MoE) get it. Other models go
@@ -882,6 +884,7 @@ def _combined():
882884
# `ReplaceEdgeOpWithTritonOpPass` are unaffected; this is
883885
# only the fallback for the `triton_kernel_mode="OFF"` path.
884886
stack.enter_context(torch.nn.attention.sdpa_kernel([SDPBackend.MATH]))
887+
stack.enter_context(target_smem_context())
885888
if low_memory_mode == "ON":
886889
# Force AOTI's mutated-buffer clones onto CPU during
887890
# compile so we stay under tight GPU memory caps (e.g.

‎backends/cuda/target_smem.py‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
"""Export-time CUDA shared-memory targeting for cross-architecture artifacts.
8+
9+
This experiment-only helper is intentionally inactive unless
10+
``ET_CUDA_TARGET_SMEM_BYTES`` is set to a positive integer.
11+
"""
12+
13+
import os
14+
from contextlib import contextmanager
15+
from typing import Iterator
16+
17+
18+
_TARGET_SMEM_ENV = "ET_CUDA_TARGET_SMEM_BYTES"
19+
20+
21+
def _target_smem_bytes() -> int | None:
22+
raw = os.environ.get(_TARGET_SMEM_ENV)
23+
if raw is None:
24+
return None
25+
try:
26+
value = int(raw)
27+
except ValueError as exc:
28+
raise ValueError(f"{_TARGET_SMEM_ENV} must be an integer, got {raw!r}") from exc
29+
if value <= 0:
30+
raise ValueError(f"{_TARGET_SMEM_ENV} must be positive, got {value}")
31+
return value
32+
33+
34+
@contextmanager
35+
def target_smem_context() -> Iterator[int | None]:
36+
"""Constrain generated CUDA kernels to a target dynamic-smem budget.
37+
38+
Triton kernels are checked after each candidate is compiled, using the
39+
exact ``metadata.shared`` value while retaining the local GPU limit as an
40+
upper bound.
41+
"""
42+
43+
target = _target_smem_bytes()
44+
if target is None:
45+
yield None
46+
return
47+
48+
from triton.compiler import compiler as triton_compiler
49+
50+
original_max_shared_mem = triton_compiler.max_shared_mem
51+
52+
def target_max_shared_mem(device):
53+
return min(int(original_max_shared_mem(device)), target)
54+
55+
triton_compiler.max_shared_mem = target_max_shared_mem
56+
print(
57+
f"CUDA export target shared-memory budget: {target} bytes "
58+
"(exact Triton metadata filter)"
59+
)
60+
try:
61+
yield target
62+
finally:
63+
triton_compiler.max_shared_mem = original_max_shared_mem

‎backends/cuda/tests/test_cuda_export.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,12 @@
44
# This source code is licensed under the BSD-style license found in the
55
# LICENSE file in the root directory of this source tree.
66

7+
import contextlib
8+
import os
79
import unittest
810
from typing import Tuple
911
from unittest import mock
12+
from unittest.mock import patch
1013

1114
import torch
1215
from executorch.backends.cuda.cuda_backend import CudaBackend
@@ -101,6 +104,35 @@ def test_invalid_autotune_at_compile_time_compile_spec(self):
101104
[CompileSpec(key="autotune_at_compile_time", value=b"MAYBE")]
102105
)
103106

107+
def test_target_smem_context_is_applied(self):
108+
with patch(
109+
"executorch.backends.cuda.cuda_backend.target_smem_context",
110+
return_value=contextlib.nullcontext(),
111+
) as target_smem_context:
112+
with CudaBackend.get_extra_aoti_compile_context_manager([]):
113+
pass
114+
115+
target_smem_context.assert_called_once_with()
116+
117+
def test_target_smem_context_only_patches_exact_triton_limit(self):
118+
from executorch.backends.cuda.target_smem import target_smem_context
119+
from torch._inductor.template_heuristics.triton import BaseConfigHeuristic
120+
from triton.compiler import compiler as triton_compiler
121+
122+
original_checker = BaseConfigHeuristic._get_exceeding_shared_memory_checker
123+
with mock.patch.dict(os.environ, {"ET_CUDA_TARGET_SMEM_BYTES": "4096"}):
124+
with mock.patch.object(
125+
triton_compiler, "max_shared_mem", return_value=8192
126+
) as local_max_shared_mem:
127+
with target_smem_context():
128+
self.assertIs(
129+
BaseConfigHeuristic._get_exceeding_shared_memory_checker,
130+
original_checker,
131+
)
132+
self.assertEqual(triton_compiler.max_shared_mem(0), 4096)
133+
134+
self.assertIs(triton_compiler.max_shared_mem, local_max_shared_mem)
135+
104136

105137
class TestCudaExport(unittest.TestCase):
106138
"""Test CUDA export functionality for various operations using to_edge_transform_and_lower."""

0 commit comments

Comments
 (0)