Repository navigation
Add AMD/ROCm CI - #5918
Add AMD/ROCm CI#5918
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1dfddd6afb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
This PR introduces an initial AMD/ROCm CI path for TRL, centered around a scheduled/manual GitHub Actions workflow on self-hosted ROCm hardware plus local helper scripts for running and summarizing the test suite with pytest --report-log.
Changes:
- Add a scheduled/manual AMD ROCm workflow (
tests-amd.yml) using therocm/pytorchcontainer, validating HIP PyTorch, running a Flash Attention smoke test, and executing the full test suite via a new runner script. - Add
scripts/run_amd_ci.shto run tests split per file and generate a summarized report from report-log JSON lines. - Add a ROCm Flash Attention smoke-check script and update
.gitignoreto ignore generated AMD CI reports.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
scripts/run_amd_ci.sh |
New local/host AMD CI runner that executes pytest per-file and summarizes --report-log results. |
scripts/check_amd_flash_attn.py |
New ROCm Flash Attention smoke test using the kernels-community/aiter-flash-attn Hub kernel. |
.github/workflows/tests-amd.yml |
New scheduled/manual workflow to run AMD/ROCm CI on a self-hosted runner group inside rocm/pytorch. |
.gitignore |
Ignore amd_ci_reports/ and fix the /.claude/skills ignore entry formatting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
cc8563c to
f99cb8a
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Abdennacer-Badaoui
left a comment
Abdennacer-Badaoui
left a comment
There was a problem hiding this comment.
Thanks @kashif for this.
This fits into our ongoing partnership with AMD, as TRL is one of the libraries the AMD team asked us to make sure we support on AMD GPUs.
The one open item before this can actually run in CI: the runs-on group (amd-mi325) is a placeholder and needs wiring to a real provisioned AMD runner. I'll reach out once we're able to allocate some runners for TRL to enable the CI.
LGTM!
qgallouedec
left a comment
qgallouedec
left a comment
There was a problem hiding this comment.
thanks! can you add a report to slack?
|
what is the status of this PR? Is it still a required feature? |
- Bump base image to rocm7.2.4 to match the AMD CI image used by triton and transformers. - Switch GPU isolation from --env ROCR_VISIBLE_DEVICES to --env-file /etc/podinfo/gha-gpu-isolation-settings, which forwards both HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES from the runner. Same pattern as transformers' AMD CI. - Add PIP_BREAK_SYSTEM_PACKAGES so pip can install into the PEP 668 image directly. - Add continue-on-error: true so a scheduled AMD failure doesn't block the caller workflow, matching triton. - Add a run_all_tests_multi_gpu job that mirrors the multi-GPU job in slow-tests.yml. Both jobs have their `runs-on` commented out with a placeholder runner group, since the runner groups (`amd-mi325` and `amd-mi325-2gpu`) are not yet provisioned. Leaving `runs-on` active without a backing runner would let the job sit for the 24h runner timeout on every schedule, same hazard transformers' AMD CI caller warns about.
- pytest wasn't given --report-log, so Generate Report always found nothing (also caught by Bugbot on the PR) - SLACK_API_TOKEN was set but never used anywhere; switched to the same CI_SLACK_CHANNEL + post-slack action every other workflow uses - pin actions/checkout to the SHA used elsewhere instead of @v6 - retry the flash-attn wheel install, it's a single external host - add RCCL env vars to the multi-gpu job for known AMD hang issues - give slow_tests the same --reruns as the fast test target
Abdennacer-Badaoui
left a comment
Abdennacer-Badaoui
left a comment
There was a problem hiding this comment.
LGTM!
One more thing on the install step: ROCm torch depends on triton-rocm, but liger-kernel (in .[dev]) requires triton. pip doesn't count triton-rocm as triton, so it installs the CUDA triton into the same triton/ directory and overwrites the ROCm build. That breaks liger, torch.compile and the Triton kernels.
A dry run on MI300X still adds triton==3.8.0, even with triton-rocm in the constraints, and even with liger pre-installed with --no-deps, since .[dev] resolves it again.
Could you add a check after pip install that fails loudly if the CUDA triton was installed, or reinstalls triton-rocm to repair it?
…ncy installs them
…next step checks it
…rom PyTorch's ROCm wheels
…th the same job names
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8b5441d. Configure here.
qgallouedec
left a comment
qgallouedec
left a comment
There was a problem hiding this comment.
Thanks, i made a few changes:
- The AMD jobs now live in
tests.ymlnext to the CUDA ones, with the same triggers and path filters (PRs and pushes, not nightly), and every job and Slack title says its hardware. - They're set up like the CUDA jobs: Python 3.13 with uv, and torch/torchvision from PyTorch's ROCm 7.2 wheels in
rocm/dev-ubuntu-24.04. The image's system Python had Debian packages pip can't upgrade, and a preinstalled pytest that ran outside the venv. - Stable tests only (
make test), no experimental ones. - 19 pytest workers with
OMP_NUM_THREADS=1: the pod sees 160 CPUs but its quota is 19, so-n autogot the job OOM-killed, and 24 workers made the trainer tests 10-17x slower. Now 2834 passed in 9m40s, same as CUDA.
Resolves #7735.
Adds two AMD MI300 jobs to
tests.yml, set up like the CUDA ones: Python 3.13 with uv,make testonhfc-amd-mi300-1gpu(19 pytest workers, the pod's CPU quota, one thread each) andtests/distributed/test_distributed.pyonhfc-amd-mi300-2gpu. torch and torchvision come from PyTorch's ROCm 7.2 wheels in arocm/dev-ubuntu-24.04container, and a first step fails if CUDA torch got in or Triton can't compile.kernels-community/aiter-flash-attn@v2and tiny Qwen3 (head dim 128), and the GRPO and RLOO continuous-batching tests are no longer skipped.