feat: CPU proxy RDMA for EP on rail-isolated clusters (v3-clean) - #7
Open
itej89 wants to merge 1 commit into
Open
feat: CPU proxy RDMA for EP on rail-isolated clusters (v3-clean)#7itej89 wants to merge 1 commit into
itej89 wants to merge 1 commit into
Conversation
Add CPU proxy transport for IBGDA operations on clusters where GPU doorbells don't work. Per-NIC proxy threads poll a host-pinned ring buffer and execute ibv_post_send on behalf of the GPU. - Per-NIC proxy threads with targeted quiet - Agreed-rail QP routing: max(localGpu, peerLocalGpu) % numNics - Signal delivery: RDMA_WRITE for data, SEND_WITH_IMM for barriers - Per-NIC MR registration for rail-isolated fabric - Gated behind MORI_USE_IBGDA_PROXY=1 Verified: 31-41 GB/s dispatch, 66-73 GB/s combine on AINIC cluster. Co-Authored-By: Claude <noreply@anthropic.com>
itej89
pushed a commit
that referenced
this pull request
Aug 27, 2026
- Check wc.status first in DrainCq — on error, opcode may be garbage - Null-check atomic target address from SEND_WITH_IMM payload - Replace fprintf with MORI_LOG_ERROR in proxy_thread.cpp - Replace fprintf with MORI_APP_INFO in init.cpp proxy log line Addresses PR ROCm#558 review comments #1, #3, and #7. Co-Authored-By: Claude <noreply@anthropic.com>
itej89
pushed a commit
that referenced
this pull request
Aug 27, 2026
- Check wc.status first in DrainCq — on error, opcode may be garbage - Null-check atomic target address from SEND_WITH_IMM payload - Replace fprintf with MORI_LOG_ERROR in proxy_thread.cpp - Replace fprintf with MORI_APP_INFO in init.cpp proxy log line Addresses PR ROCm#558 review comments #1, #3, and #7. Co-Authored-By: Claude <noreply@anthropic.com>
itej89
pushed a commit
that referenced
this pull request
Aug 27, 2026
- Check wc.status first in DrainCq — on error, opcode may be garbage - Null-check atomic target address from SEND_WITH_IMM payload - Replace fprintf with MORI_LOG_ERROR in proxy_thread.cpp - Replace fprintf with MORI_APP_INFO in init.cpp proxy log line Addresses PR ROCm#558 review comments #1, #3, and #7. Co-Authored-By: Claude <noreply@anthropic.com>
2 tasks
itej89
pushed a commit
that referenced
this pull request
Aug 27, 2026
- Check wc.status first in DrainCq — on error, opcode may be garbage - Null-check atomic target address from SEND_WITH_IMM payload - Replace fprintf with MORI_LOG_ERROR in proxy_thread.cpp - Replace fprintf with MORI_APP_INFO in init.cpp proxy log line Addresses PR ROCm#558 review comments #1, #3, and #7. Co-Authored-By: Claude <noreply@anthropic.com>
itej89
pushed a commit
that referenced
this pull request
Aug 27, 2026
- Check wc.status first in DrainCq — on error, opcode may be garbage - Null-check atomic target address from SEND_WITH_IMM payload - Replace fprintf with MORI_LOG_ERROR in proxy_thread.cpp - Replace fprintf with MORI_APP_INFO in init.cpp proxy log line Addresses PR ROCm#558 review comments #1, #3, and #7. Co-Authored-By: Claude <noreply@anthropic.com>
itej89
pushed a commit
that referenced
this pull request
Aug 27, 2026
- Check wc.status first in DrainCq — on error, opcode may be garbage - Null-check atomic target address from SEND_WITH_IMM payload - Replace fprintf with MORI_LOG_ERROR in proxy_thread.cpp - Replace fprintf with MORI_APP_INFO in init.cpp proxy log line Addresses PR ROCm#558 review comments #1, #3, and #7. Co-Authored-By: Claude <noreply@anthropic.com>
itej89
added a commit
that referenced
this pull request
Aug 27, 2026
* fix: null-check atomic target address in proxy DrainCq Guard against dereferencing a null pointer from the SEND_WITH_IMM payload. The address comes from the wire — skip the atomic if zero. Addresses PR ROCm#558 review comments #1 and #7 (wc.status check was already implemented in the current code). Co-Authored-By: Claude <noreply@anthropic.com> * fix: check wc.status before opcode, use MORI logger in proxy - Check wc.status first in DrainCq — on error, opcode may be garbage - Null-check atomic target address from SEND_WITH_IMM payload - Replace fprintf with MORI_LOG_ERROR in proxy_thread.cpp - Replace fprintf with MORI_APP_INFO in init.cpp proxy log line Addresses PR ROCm#558 review comments #1, #3, and #7. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Tej Kiran <kiran.tej@amd.com> Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MORI_USE_IBGDA_PROXY=1— zero behavior change without the flagFiles (22 files, proxy only)
proxy/{types,thread,device_primitives}.hpp,proxy_thread.cppshmem_ibgda_kernels.hpp,internal.hpp,init.cpp,context.cpp,symmetric_memory.cpp,ionic.cppFor side-by-side comparison with v6 (PR #5)
🤖 Generated with Claude Code