For anyone running the fork on multi-GPU consumer RDNA4 rigs (I've seen at least one 4x R9700 box in the community threads) — two things from our dual-R9700 work, one immediately usable and one measured-and-published.
1. The RCCL deadlock on gfx1201, and the one-line workaround
If you do tensor-parallel work through RCCL (torch/vLLM-style — not llama.cpp's own split modes) on 2+ RDNA4 cards, stock RCCL deadlocks: both GPUs spin at 100%, step 0, no progress, no error. Root cause is that RCCL's tuning-index lookup (rcclGetTuningIndexForArch) has no gfx1201 entry, so it silently falls back to gfx906/MI50 PCIe tuning and the AllReduce wedges. Confirmed by multiple R9700 owners (vLLM #40980).
Workaround: NCCL_PROTO=Simple. That's the whole cheat code — it reverts to the pre-regression protocol and dual-GPU work runs.
Status upstream: the real fix merged in ROCm/rocm-systems ggml-org#6966 + ggml-org#7183 but has not reached a public RCCL release as of the rocm-7.2.4 rebuild; TheRock-built RCCL >= 2.28.3 carries real gfx1201 objects and needs no workaround.
2. INT6 compressed all-reduce for the no-P2P case — measured, published
The nastier problem on consumer boards: hipDeviceCanAccessPeer returns 0 between cards on different root complexes (verified two ways on our box — the HIP probe and AMD's own RVS pbqt both report peers:false), so every inter-GPU byte goes device -> host -> QPI -> host -> device. On that path, compressing the payload is nearly free speedup.
We built a QuickReduce-style two-shot all-reduce with a quality-first INT6 codec (block-32, fp16 scale — integer reduce is exact in-domain given a shared scale, drift 4.77e-7). Measured A/B against an uncompressed fp16 two-shot with the identical protocol shape on the dual-R9700:
| message |
speedup |
| 256K |
1.18x |
| 1M |
1.72x |
| 4M |
1.98x |
| 16M |
1.89x |
Code, benches, numpy reference and the full caveat list are now public: https://github.com/The-Monk/int6-allreduce (MIT).
The caveats are real and in the README, the biggest being ~2.16% cross-rank asymmetry with independent per-rank scales (the two ranks' outputs are not bit-identical after the compressed reduce) — any TP integration has to either tolerate that or add a scale exchange. It is standalone kernels today, not integrated into this fork's split-mode path; if there's interest in compressed transport for row-split on no-P2P boxes, that's the natural next conversation, and the dual-R9700 box is available for validation as always.
For anyone running the fork on multi-GPU consumer RDNA4 rigs (I've seen at least one 4x R9700 box in the community threads) — two things from our dual-R9700 work, one immediately usable and one measured-and-published.
1. The RCCL deadlock on gfx1201, and the one-line workaround
If you do tensor-parallel work through RCCL (torch/vLLM-style — not llama.cpp's own split modes) on 2+ RDNA4 cards, stock RCCL deadlocks: both GPUs spin at 100%, step 0, no progress, no error. Root cause is that RCCL's tuning-index lookup (
rcclGetTuningIndexForArch) has no gfx1201 entry, so it silently falls back to gfx906/MI50 PCIe tuning and the AllReduce wedges. Confirmed by multiple R9700 owners (vLLM #40980).Workaround:
NCCL_PROTO=Simple. That's the whole cheat code — it reverts to the pre-regression protocol and dual-GPU work runs.Status upstream: the real fix merged in ROCm/rocm-systems ggml-org#6966 + ggml-org#7183 but has not reached a public RCCL release as of the rocm-7.2.4 rebuild; TheRock-built RCCL >= 2.28.3 carries real gfx1201 objects and needs no workaround.
2. INT6 compressed all-reduce for the no-P2P case — measured, published
The nastier problem on consumer boards:
hipDeviceCanAccessPeerreturns 0 between cards on different root complexes (verified two ways on our box — the HIP probe and AMD's own RVSpbqtboth report peers:false), so every inter-GPU byte goes device -> host -> QPI -> host -> device. On that path, compressing the payload is nearly free speedup.We built a QuickReduce-style two-shot all-reduce with a quality-first INT6 codec (block-32, fp16 scale — integer reduce is exact in-domain given a shared scale, drift 4.77e-7). Measured A/B against an uncompressed fp16 two-shot with the identical protocol shape on the dual-R9700:
Code, benches, numpy reference and the full caveat list are now public: https://github.com/The-Monk/int6-allreduce (MIT).
The caveats are real and in the README, the biggest being ~2.16% cross-rank asymmetry with independent per-rank scales (the two ranks' outputs are not bit-identical after the compressed reduce) — any TP integration has to either tolerate that or add a scale exchange. It is standalone kernels today, not integrated into this fork's split-mode path; if there's interest in compressed transport for row-split on no-P2P boxes, that's the natural next conversation, and the dual-R9700 box is available for validation as always.