Check the shipped device code covers the GPUs a wheel claims

[shoumikhin]

A GPU wheel can install cleanly, load all its libraries, and still fail
the moment a model runs. GPU code is compiled per hardware generation, so a wheel built for one
generation works there and is broken everywhere else the release promised. The failure surfaces
late and looks like a model problem rather than a packaging one.

Nothing checks for this today. This adds a release check that compares what the wheel claims
against what it actually contains:

✓ device code covers every claimed GPU architecture ['80', '90', '100', '120']

And when it does not:

libaoti_cuda_shims.so claims GPU architectures ['80', '90'] but only carries
device code for ['80']; a model would fail on hardware needing ['90']

It fails closed on a GPU wheel

Every reason the check cannot do its job is a reason to stop, not to pass: no architecture list
reached the build, the inspection tool is missing, no library is present, or an inspection
failed. Each of those leaves the shipped GPU code unverified. A CPU wheel says so and moves on.

Each library is inspected separately, so one library's GPU code cannot stand in for another's
that is missing.

Tested

Ran against real GPU wheels on Linux x86_64 and on two Jetson devices covering different GPU
generations. Confirmed it passes when the wheel matches its claim, and blocks in each failure
case above, including the one where the architecture list never reached the build.

[shoumikhin]

Stack from ghstack (oldest at bottom):

• -> Check the shipped device code covers the GPUs a wheel claims #21567
• Publish CUDA wheels so a GPU user does not build from source #21569
• Ship the CUDA delegate as its own library #21531
• Ship the XNNPACK delegate as its own library #21526
• Ship the merged CPU kernels as their own library #21524
• Ship the thread pool as its own library so a process has one pool #21522
• Ship a linkable shared runtime and make the Python bindings use it #21514

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21567

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 29 New Failures, 168 Pending, 2 Unrelated Failures

As of commit 6adc652 with merge base 6bbb75a ():

NEW FAILURES - The following jobs have failed:

• Apple / build-frameworks-ios / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / windows (pybind) / build (gh)
• Build Presets / windows (windows) / build (gh)
• pull / test-arm-backend-public-api-backward-compatibility / linux-job (gh)
  RuntimeError: Command docker exec -t b4632316bf352a7b2931e126177c81a40dc6d3dbdb62d51410e6fcb71ba595db /exec failed with exit code 1
• pull / test-lora-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-lora-multimethod-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 1f71abefab7f682e4a9954799e4479194e70e6b7963eec35e05d210679ef97b8 /exec failed with exit code 1
• pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t b6de46e0133a3119edc8c248bae5b7624272d82ac37c9b2178011be672f2c82a /exec failed with exit code 1
• pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
  RuntimeError: Command docker exec -t 6b584c5e6d8be39d333a0766566fa91a67616c053fdd4be0deabbedb5ac11f2f /exec failed with exit code 1
• pull / test-phi-3-mini-runner-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / unittest / linux / linux-job (gh)
  backends/xnnpack/test/passes/test_channels_last_tagged_reshape.py::TestChannelsLastTaggedReshapePass::test_conv_linear_dim_order_swaps
• pull / unittest-editable / linux / linux-job (gh)
  backends/xnnpack/test/passes/test_channels_last_tagged_reshape.py::TestChannelsLastTaggedReshapePass::test_conv_linear_dim_order_swaps
• Test WebGPU Native (Dawn) / test-webgpu-native / linux-job (gh)
  [ FAILED ] WebGPUNative.SdpaAllFamiliesRanWhenDirSet
• trunk / test-arm-backend-ethos-u (test_deit_e2e_ethos_u) / linux-job (gh)
  RuntimeError: Command docker exec -t 4bc622f52b5dad263508bcfa10fde15b05a62a9b76de735104258596950308d1 /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_model_smollm2_135M_ethos_u85) / linux-job (gh)
  RuntimeError: Command docker exec -t 60e72be5dab79628f054cae8204f84606e08bc16a284edbb37b753d0d2dbc5a7 /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_ootb_tests_ethos_u) / linux-job (gh)
  RuntimeError: Command docker exec -t 00368d13d589f150c625604b58f38b0d20358e8613b22b9ab0e1ceacc03fbf4d /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_ootb_tests_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t 4350f4a0ca9982fe21fc1ba469ea63a18ea95eb41557f1b0f59ceb32c3be6e87 /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_pytest_models_ethos_u55) / linux-job (gh)
  RuntimeError: Command docker exec -t 141ac4c6e791336f5c596c22104fd9143313ba4ec8c77135694454d79903188b /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_pytest_models_ethos_u85) / linux-job (gh)
  RuntimeError: Command docker exec -t 03dbdcd248c0995b14f192e253098e571256185660c57d24b01c270348f71cf9 /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_pytest_ops_ethos_u55) / linux-job (gh)
  RuntimeError: Command docker exec -t 3062413def41d287ce710a9bd7cdbbd8852c25bc89199f012f2070edda41d534 /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_pytest_ops_ethos_u85) / linux-job (gh)
  RuntimeError: Command docker exec -t 74bce9a75fac0778c5f4b332ae034dc8a7986e1cf71681dbf3f9f45519cf7243 /exec failed with exit code 1
• trunk / test-arm-backend-vkml (test_pytest_models_vkml) / linux-job (gh)
  RuntimeError: Command docker exec -t ec79eb3c73883dd0c7f0b9e292681a75cdfff8094edbdcbb6b0c963de3827ef7 /exec failed with exit code 1
• trunk / test-huggingface-transformers-xnnpack (gemma3-1b|xnnpack|--quantize) / linux-job (gh)
  RuntimeError: Command docker exec -t adccc684c98c2f80c0578ad05625af75f1056e424576f1102ddb32046b45a3c1 /exec failed with exit code 1
• trunk / test-huggingface-transformers-xnnpack (llama3.2-1b|xnnpack|--quantize) / linux-job (gh)
  RuntimeError: Command docker exec -t cef072df0e3be9b39bcfffdc7f6b495a9e989dd83c4568f4324e572abd07e0d9 /exec failed with exit code 1
• trunk / test-huggingface-transformers-xnnpack (qwen3-0.6b|xnnpack|--quantize) / linux-job (gh)
  RuntimeError: Command docker exec -t 0452ee0d5fdba2a4bb1153786857855ec1b04507ec5d7f3bdd5484a1493e3da8 /exec failed with exit code 1
• trunk / test-huggingface-transformers-xnnpack (qwen3-1.7b|xnnpack|--quantize) / linux-job (gh)
  RuntimeError: Command docker exec -t 50a48b4a72b1a598ed8ac6dadda62bd938cabb23a0a184b21ec9cc8859ddbc7f /exec failed with exit code 1
• trunk / test-huggingface-transformers-xnnpack (smollm2-135m|xnnpack|--quantize) / linux-job (gh)
  RuntimeError: Command docker exec -t 3e16692a9e6ecfa6bb38a6c09626ae8675ae230bb62710d4d64822f9a7526cc7 /exec failed with exit code 1
• trunk / test-huggingface-transformers-xnnpack (smollm3-3b|xnnpack|--quantize) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d34d3a9b26096d0ed9c2c851802f89c4dabe95921a8709fd81c1db9fe6e7b1e /exec failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh)
  backends/xnnpack/test/passes/test_channels_last_tagged_reshape.py::TestChannelsLastTaggedReshapePass::test_conv_linear_dim_order_swaps

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / unittest-editable / windows / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-arm-backend-vkml (test_pytest_ops_vkml) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.