[ROCm] Test ROCm failures on PR#1554 - #1642
Open
akashveramd wants to merge 62 commits into
Open
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1642
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 25ce9e6 with merge base d249ad5 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
August 18, 2026 23:21
35e74f6 to
61016ce
Compare
librocjpeg links against libva.so.2 and libva-drm.so.2 (VA-API, used by rocJPEG's HYBRID GPU-JPEG backend). auditwheel was pulling both into the wheel because they weren't excluded, and repair_wheel.py's check_bundling() then rejected them as unexpected. libva/libva-drm are system-provided display-stack libraries, present on any ROCm install alongside libdrm (which is already excluded). Add "libva*" to the same exclude list so auditwheel leaves them on the system rather than bundling them. Co-authored-by: Cursor <cursoragent@cursor.com>
ROCm >= 7.14 distributes the full ROCm stack (including rocJPEG) as pip wheels (_rocm_sdk_core / _rocm_sdk_devel site-packages) rather than system RPMs. The rocjpeg-devel, libva-amdgpu and mesa-amdgpu-va-drivers DNF packages therefore don't exist on the ROCm 7.14 builder image and the install was failing with "No package rocjpeg available". Check for librocjpeg.so under /opt/conda (pip-wheel install path) and /opt/rocm (classic RPM install path) before attempting dnf install. If already present, skip the install entirely. The existing dnf path is preserved for ROCm 7.2 and earlier where RPMs are the only source. Co-authored-by: Cursor <cursoragent@cursor.com>
librocjpeg links libva.so.2 as a DT_NEEDED entry and needs it at dlopen time even when only the HARDWARE backend is used. The previous fix skipped all dnf installs when librocjpeg was found via ROCm 7.14 pip wheels, which left libva absent on the test machine and caused libtorchcodec_image.so to fail to load with OSError. libva ships in AlmaLinux standard repos so install it unconditionally. Co-authored-by: Cursor <cursoragent@cursor.com>
_find_rocjpeg_lib() only searched ROCM_HOME/lib, torch's ROCM_HOME, and /opt/rocm/lib. For ROCm >= 7.14, librocjpeg lives inside the _rocm_sdk_core pip-wheel's site-packages (e.g. /opt/conda/lib/python3.11/site-packages/_rocm_sdk_core/lib/librocjpeg.so), none of which were on those search paths. The function therefore returned None, printed a WARNING, and auditwheel never bundled librocjpeg. At test time libtorchcodec_image.so had DT_NEEDED: librocjpeg.so but no copy was on the dynamic-linker search path → OSError: Could not load this library. Fix: add a site-packages-first fallback that checks _rocm_sdk_core/lib and _rocm_sdk_devel/lib in the current interpreter's site-packages, then falls back to a broad /opt/conda/** glob (mirrors install_rocjpeg.sh). Also add ldd diagnostics before pytest in the install-and-test workflow so that any future missing transitive dep is visible in the log rather than hidden behind a bare OSError, and update the stale comment that said the wheel does not bundle librocjpeg. Co-authored-by: Cursor <cursoragent@cursor.com>
…yout PyTorch ROCm wheels may bundle the ROCm runtime (libamdhip64, libhsa-runtime64 etc.) inside torch/lib/ rather than depending on a separate _rocm_sdk_core pip wheel. Add \$ORIGIN/../../torch/lib to the RPATH patched onto the bundled librocjpeg-HASH.so so it can find those deps in either layout: - torch/lib/ (torch bundles ROCm runtime, like it does for CUDA) - _rocm_sdk_core/lib/ (TheRock separate pip-wheel layout) - /opt/rocm/lib (ROCm <= 7.2 system install) Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
August 19, 2026 16:50
7435c2f to
2efd33f
Compare
…nostics The bundled librocjpeg-HASH.so has DT_NEEDED entries for librocm_sysdeps_va.so.2, librocm_sysdeps_drm_amdgpu.so.1, librocprofiler-register.so.0 etc. These live in _rocm_sdk_core/lib/ and _rocm_sdk_core/lib/rocm_sysdeps/lib/ but are not on the system search path and are not loaded by torch at import time. The RPATH on the bundled lib uses $ORIGIN which ldd cannot expand without execute permission (ldd shows all $ORIGIN-based entries as "not found"). Whether $ORIGIN resolves correctly at actual dlopen time is unclear, so set LD_LIBRARY_PATH explicitly before pytest as belt-and-suspenders. Also fix the rocjpeg_so find path (torchcodec.libs/ is a sibling of torchcodec/, not a child) and add objdump RPATH output so we can verify the RPATH was patched correctly on the bundled librocjpeg. Co-authored-by: Cursor <cursoragent@cursor.com>
Bundling librocjpeg moves it out of _rocm_sdk_core/lib, breaking AMD's own relative RPATH inside it that points to librocm_sysdeps_* transitive deps. We then have to re-patch those paths ourselves, which is fragile and was causing the missing librocm_sysdeps_va.so.2 / libamdhip64.so.7 failures at runtime. The cleaner approach: exclude librocjpeg from auditwheel bundling and instead append _rocm_sdk_core/lib (ROCm >= 7.14) and /opt/rocm/lib (ROCm <= 7.2) to the RPATH of libtorchcodec_image.so itself. At runtime the dynamic linker finds librocjpeg in its original location; AMD's own RPATH on that library then correctly resolves all transitive deps without any intervention from us. _find_rocjpeg_lib() is kept: auditwheel still needs librocjpeg in LD_LIBRARY_PATH to resolve the DT_NEEDED during its analysis pass, even though we exclude it from bundling. Co-authored-by: Cursor <cursoragent@cursor.com>
…ness Co-authored-by: Cursor <cursoragent@cursor.com>
check_bundling() previously required librocjpeg to be bundled in every ROCm wheel, which is now intentionally wrong. Update it to: - Treat "not bundled" as correct for ROCm wheels - Instead verify that libtorchcodec_image.so has _rocm_sdk_core/lib in its RPATH (confirming _patch_image_so_rpath_in_wheel ran correctly) - Raise if librocjpeg IS bundled (that would be a regression) - Remove _is_rocjpeg from the bundled-lib allowlist for the same reason - Add top-level `import tempfile` (used by the new RPATH check) Co-authored-by: Cursor <cursoragent@cursor.com>
… rocjpeg - bundle_third_party_licenses: warn (not raise) if rocjpeg LICENSE is missing, since librocjpeg is no longer redistributed in the wheel - check_bundling license check: only enforce rocjpeg keyword if the license file was actually bundled (it's optional, not mandatory) - Fix stale "ROCm wheels bundle librocjpeg" comment - Fix stale _find_rocjpeg_license docstring Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
$ORIGIN/../../_rocm_sdk_core/lib resolves to python3.x/_rocm_sdk_core/lib which does not exist. The correct path is $ORIGIN/../_rocm_sdk_core/lib: libtorchcodec_image.so lives in site-packages/torchcodec/, so one ../ brings us to site-packages/ where _rocm_sdk_core/ lives. This matches auditwheel's own convention: it sets $ORIGIN/../torchcodec.libs for the same reason. Co-authored-by: Cursor <cursoragent@cursor.com>
rocJPEG's HYBRID backend calls vaInitialize() to initialize the AMD GPU video decoder. libva.so.2 alone is not enough; it also needs the AMD VA-API backend driver (mesa-amdgpu-va-drivers) which provides the radeonsi/amdgpu DRI plugin that vaInitialize() dlopen-s at runtime. Without it, vaInitialize() fails and rocJPEG returns ROCJPEG_STATUS_NOT_INITIALIZED / ROCJPEG_STATUS_NOT_IMPLEMENTED, causing all jpeg_cuda tests to fail with "Failed to initialize rocJPEG with the hybrid backend". Install mesa-amdgpu-va-drivers alongside libva in the ROCm 7.14 pip-wheel path; fall back to libva-only if the AMD graphics repo is not available (build-only runners). Co-authored-by: Cursor <cursoragent@cursor.com>
Since ROCm 7.14, mesa (including the amdgpu/radeonsi VA-API backend driver) ships inside the _rocm_sdk_core pip wheel. There is no need to install mesa-amdgpu-va-drivers via dnf. However, libva's driver discovery (vaInitialize) needs to know where to find the DRI driver plugin. Set LIBVA_DRIVERS_PATH to _rocm_sdk_core/lib/dri so the vendored librocm_sysdeps_va.so.2 can find the bundled mesa backend, fixing the ROCJPEG_STATUS_NOT_INITIALIZED / ROCJPEG_STATUS_NOT_IMPLEMENTED failures. Also add a fallback diagnostic that lists _rocm_sdk_core/lib/ if the dri subdirectory is not found, to help debug future layout changes. Reverts the mesa-amdgpu-va-drivers dnf install added in the previous commit (442ac6c) as it is not needed. Co-authored-by: Cursor <cursoragent@cursor.com>
The multi-line python block inside $() in the YAML run: | block had lines starting at column 0, which breaks the YAML scalar parser (same issue as before). Collapse to a single line. Also improve the *_drv_video.so search: use rglob instead of checking a hardcoded lib/dri/ path so the mesa VA-API backend driver is found regardless of the exact subdirectory AMD chose inside _rocm_sdk_core. The fallback diagnostic now greps the full recursive listing for dri/va/video/mesa/gallium/radeon keywords to pinpoint the layout if the driver isn't found. Co-authored-by: Cursor <cursoragent@cursor.com>
…iver libva auto-detects the DRM driver name as "radeonsi" from the kernel and looks for radeonsi_drv_video.so. AMD vendored it as librocm_sysdeps_gallium_drv_video.so, so LIBVA_DRIVER_NAME must be set to "librocm_sysdeps_gallium" to override that. Verified locally: vaInitialize returns 0 with both LIBVA_DRIVERS_PATH and LIBVA_DRIVER_NAME set. Co-authored-by: Cursor <cursoragent@cursor.com>
1. test/utils.py: assert_frames_equal crashes with TypeError when cuda_version_used_for_building_torch() returns None on ROCm. Handle None explicitly, treating it as >= CUDA 13 (use atol=3). 2. DecodeJpegRocm: ROCJPEG_OUTPUT_RGB_PLANAR via the HW (VCN) backend produces ~51% correct pixels for colour (YCbCr) JPEG sources on MI350X: the hardware returns unconverted YCbCr planes instead of RGB. Fix: set force_hybrid=true in make_plan() for colour JPEGs requesting RGB output, routing them to the HYBRID backend which handles YCbCr->RGB in software. 3. DecodeJpegRocm: rocJpegDecodeBatched writes with an internally-aligned pitch that does not match our tensor row stride, producing completely wrong output when the batch mixes images of different dimensions. Fix: replace rocJpegDecodeBatched with individual rocJpegDecode calls in decode_batched_hardware(). 4. test_decoders.py: test_cuda_jpeg_errors expects corrupt JPEG input to raise RuntimeError. rocJPEG silently "succeeds" on this corrupt JPEG (nvJPEG raises, rocJPEG does not). Skip the corrupt-JPEG assertion on ROCm; the CPU-tensor-on-GPU error check still runs. Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
commented
Aug 19, 2026
akashveramd
commented
Aug 19, 2026
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
August 25, 2026 19:28
86646fa to
99d3266
Compare
Set ROCJPEG_LOG_LEVEL=3 as an inline env var for the pytest invocation so that librocjpeg emits detailed logs (including VA-API initialization) to stderr during CI test runs. This helps diagnose why rocJpegCreate() returns ROCJPEG_STATUS_NOT_INITIALIZED on the HARDWARE backend. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
List the full contents of ROCM_PATH/lib/rocm_sysdeps/lib/ so we can confirm exactly which VA-API libraries are present on the CI runner. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
…lure Add three more diagnostic probes to narrow down why vaInitialize fails: - vainfo with LIBVA_DRIVERS_PATH pointing to rocm_sysdeps/lib (the correct location of the AMD VA-API driver, not lib/dri) - vainfo --display drm --device /dev/dri/renderD128 to test VA-API initialization directly against the DRM node - /sys/class/drm/renderD128/device/unique_id to expose the GPU UUID registered in the DRM subsystem, for comparison against the HIP UUID Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
We collected the VA-API and UUID diagnostic data we needed. The vainfo commands and the hardcoded renderD128 sysfs path were temporary probes that are no longer needed and caused failures due to the hardcoded device node not existing in all environments. Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>
…ents Updates the CI to use ROCm 10.0 instead of 7.14. Replaces version-specific comments (ROCm >= 7.14, ROCm <= 7.2) in packaging scripts with layout-based descriptions (TheRock/pip-wheel layout vs legacy ROCm system install) since the _rocm_sdk_core layout applies regardless of the specific ROCm version. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
…heel layout Removes all legacy ROCm system install (pre-pip-wheel) support from the packaging scripts since we only target ROCm 10.0 which ships via the TheRock/pip-wheel layout (_rocm_sdk_core). Removes _get_rocm_search_roots() and the ROCM_HOME/ROCM_PATH search paths from _find_rocjpeg_lib() and _find_rocjpeg_license(). The RPATH patch now only injects $ORIGIN/../_rocm_sdk_core/lib. Updates _find_rocjpeg_license() to search _rocm_sdk_core/lib directly. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
…r ROCm 10.0 install_rocjpeg.sh was only needed to install rocJPEG from DNF repos for legacy ROCm system installs. Since we now only support ROCm 10.0 with the TheRock/pip-wheel layout where rocJPEG is bundled in _rocm_sdk_core, this script is no longer needed. Removes the install_rocjpeg.sh invocations from linux_rocm.yaml and pre_build_script.sh. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the /opt/rocm fallback in CMakeLists.txt: for the TheRock/pip-wheel layout, ROCM_HOME is set via torch.utils.cpp_extension and points to _rocm_sdk_core in site-packages. If it is not set, fail with a clear message instead of silently probing /opt/rocm. Update the rocJPEG-not-found fatal error to mention _rocm_sdk_core / _rocm_sdk_devel pip packages instead of legacy DNF packages. Remove :-/opt/rocm fallbacks from the diagnostic block in linux_rocm.yaml: ROCM_PATH is already set from _rocm_sdk_core above, so falling back to /opt/rocm (which does not exist in a pip-wheel environment) only produces misleading output. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
site.getsitepackages() already returns the conda site-packages directory where _rocm_sdk_core is installed, so the broad /opt/conda glob was redundant. If _find_rocjpeg_lib() returns None, the caller already handles it gracefully with a warning. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
The upstream generate_binary_build_matrix.yml only knows about ROCm 7.2 and 7.14, so delegating matrix generation to it caused the build job to produce rocm7.2 and rocm7.14 wheels while the install-and-test job tried to download a rocm10.0 artifact that was never built. Replace the upstream matrix generation with a hardcoded matrix that only includes the single ROCm 10.0 / Python 3.10 entry we actually need. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
pytorch/test-infra#8670 updates the nightly ROCm matrix from 7.2/7.14 to 7.14/10.0. Once that lands, the upstream matrix generator will emit ROCm 10.0 entries natively, so there is no need to hardcode the matrix here. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
The upstream generate_binary_build_matrix (pytorch/test-infra#8670) will include both 7.14 and 10.0 once merged. torchcodec only supports the TheRock/pip-wheel layout (ROCm 10.0+), so add a filter-matrix job that strips any entries with gpu_arch_version != "10.0" before passing the matrix to build_wheels_linux.yml. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
… 10.0 Excluding by version name is more future-proof: when ROCm 11.0 is added to the upstream matrix it will pass through automatically without any code change here. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
Using xargs -I{} to write to GITHUB_OUTPUT stripped the double quotes
from the JSON keys, making the output invalid for fromJSON(). Use a
variable assignment instead.
Co-authored-by: AI assistant
Co-authored-by: Cursor <cursoragent@cursor.com>
For the TheRock/pip-wheel layout (ROCm 10.0), libva is bundled inside _rocm_sdk_core, not provided by the system. The exclusion from auditwheel bundling is still correct since _rocm_sdk_core is a required dependency. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
…ding bundled libva The _rocm_sdk_core pip wheel ships librocm_sysdeps_gallium_drv_video.so but may be missing the radeonsi_drv_video.so symlink that VA-API needs to locate the AMD GPU driver. Without it, vaInitialize fails with "unknown libva error" and rocJPEG falls back to the HYBRID backend, which is unsupported on gfx950. The fix has three parts: 1. Create the radeonsi_drv_video.so -> librocm_sysdeps_gallium_drv_video.so symlink at runtime if the wheel did not include it. 2. LD_PRELOAD the bundled librocm_sysdeps_va.so.2 and librocm_sysdeps_va-drm.so.2 so rocJPEG uses _rocm_sdk_core's VA-API implementation instead of any system-installed libva. 3. Set LIBVA_DRIVERS_PATH and LIBVA_DRIVER_NAME=radeonsi so VA-API finds the bundled driver. Root cause confirmed by the rocJPEG team: the pip wheel installation was missing the symlinks and radeonsi_drv_video.so present in the TheRock tarball reference installation. Co-authored-by: Claude (Sonnet 4.6) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The _rocm_sdk_core pip wheel is missing several symlinks present in the reference TheRock tarball installation. rocJPEG's HARDWARE backend uses VA-API to initialize, and libva needs radeonsi_drv_video.so (plus the canonical libva/libva-drm sonames) in the sysdeps directory. Without them vaInitialize() fails, causing ROCJPEG_STATUS_NOT_INITIALIZED on MI350X VF GPU environments. The fix creates the four missing symlinks at CI runtime (guarded so they are no-ops if the wheel is already correct in a future release): radeonsi_drv_video.so -> librocm_sysdeps_gallium_drv_video.so libgallium_drv_video.so -> librocm_sysdeps_gallium_drv_video.so libva.so -> librocm_sysdeps_va.so.2 libva-drm.so -> librocm_sysdeps_va-drm.so.2 Also removes LIBVA_DRIVERS_PATH and LIBVA_DRIVER_NAME: per AMD reviewer guidance (AryanSalmanpour), ROCM_PATH is the only environment variable required once the symlinks are present. A before/after ls -la of rocm_sysdeps/lib is added to diagnostics so the CI log can be compared directly against the reviewer's reference tarball output. This PR was authored with an AI assistant. Test Plan: CI on #1642 Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
August 31, 2026 20:56
f2d1271 to
36c2e0d
Compare
Remove temporary debugging instrumentation that was added while investigating the rocJPEG VA-API initialization failures: - ls -la of rocm_sysdeps/lib before symlink fixup - VA-API / DRM diagnostics group (device nodes, ldd, find, rocm-smi) - ROCJPEG_LOG_LEVEL=3 verbose logging The root cause (missing symlinks in the pip wheel) is fixed. The production workflow is now clean. This PR was authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>
Per AMD reviewer guidance (AryanSalmanpour): the correct workflow is to run `rocm-sdk init` after installing TheRock, which expands the development package and creates all required symlinks (including VA-API driver symlinks for rocJPEG) automatically. ROCM_PATH should then point to _rocm_sdk_devel, not _rocm_sdk_core. Remove the manual symlink creation block (radeonsi_drv_video.so etc.) since rocm-sdk init handles this correctly. This PR was authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
September 2, 2026 23:55
9762cbe to
b5b5aff
Compare
rocm-sdk-devel==10.0.0 is only available for Python 3.11+ but the test conda env uses Python 3.10. Use `conda run -n base` to install rocm[devel] and run rocm-sdk init in the base env which has the matching Python version. Also probe the base env when detecting ROCM_PATH. This PR was authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>
akashveramd
force-pushed
the
rocm_decoder_analysis
branch
from
September 3, 2026 04:07
b5b5aff to
9313e7a
Compare
The previous fallback used || which only fires on non-zero exit. When _rocm_sdk_devel is missing from the Python 3.10 test env, the python invocation exits 0 with empty output, so the base-env fallback never ran, leaving ROCM_PATH unset. Without ROCM_PATH, rocJPEG can not find its HIP compute kernels and returns NOT_IMPLEMENTED even for the HYBRID backend. Fix: always probe the base env when the test env returns empty string. Co-authored-by: Cursor <cursoragent@cursor.com>
LakshmiKumar23
added a commit
to LakshmiKumar23/torchcodec
that referenced
this pull request
Sep 8, 2026
These scripts are no longer needed with the TheRock/pip-wheel layout introduced in PR meta-pytorch#1642. rocDecode and RPP are now provided by the _rocm_sdk_core and _rocm_sdk_devel pip packages. Added ROCM_ROCDECODE_RPP_NOTES.md documenting the repair_wheel.py changes needed after PR meta-pytorch#1642 lands (adding librocdecode* and librpp* to the auditwheel exclude list). Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Creating this PR to-