Improve multi-NIC performance in OFI netmod#7874
Open
mthyoung-amzn wants to merge 4 commits into
Open
Conversation
New systems have tens of NICs per host. Increase the maximum NICs to 128 to future proof.
It is important to know a rank's position relative to other ranks that coexist on the same NUMA node, to distribute attached resources and maximize locality. This adds a rudimentary shmem communication step at init to negotiate package_rank among all local ranks.
It is sometimes useful to distribute IO resources across PCI bridges in a system with deep PCIe topologies. This adds a function for determining a device's parent node in the PCIe topology. This allows for assigning IO resources by round robin across multiple host bridges, for instance.
mthyoung-amzn
force-pushed
the
many-nic-2
branch
2 times, most recently
from
July 13, 2026 17:36
a36bb42 to
e5a8f4a
Compare
This enhances the NIC assignment to consider both NUMA locality, and PCIe bus topology. The new approach attempts to assign NICs that are attached directly to the rank's NUMA node, and across ranks that share a NUMA node, does round-robin assignment of NICs across host bridges to better distribute IO throughput.
mthyoung-amzn
force-pushed
the
many-nic-2
branch
from
July 13, 2026 17:56
e5a8f4a to
2fadc99
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Description
This increases the maximum supported number of OFI NICs, and adds NUMA and bus topology awareness to NIC mapping across local ranks. The goal of the new NIC assignment logic is to ensure that only NICs attached to the rank's bound NUMA node is assigned, and to distribute load across connected busses by striping NICs by host bridge. This avoids cross-NUMA bottlenecks, and spreads load across PCIe busses attached to a given node.
Author Checklist
Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
Commits are self-contained and do not do two things at once.
Commit message is of the form:
module: short descriptionCommit message explains what's in the commit.
Whitespace checker. Warnings test. Additional tests via comments.
For non-Argonne authors, check contribution agreement.
If necessary, request an explicit comment from your companies PR approval manager.