Display-aware BAR1 P2P on GB206 and transactional mailbox P2P setup - #31
Open
ducphuc wants to merge 3 commits into
Open
Display-aware BAR1 P2P on GB206 and transactional mailbox P2P setup#31ducphuc wants to merge 3 commits into
ducphuc wants to merge 3 commits into
Conversation
On display-attached GB206 boards the console mapping occupies low BAR1 VA, so requiring the static BAR1 window to cover all of client FB made the auto-enable check fail and disabled BAR1 P2P. Place the static window after a 512MB-aligned console/mailbox reservation and clip it to the remaining BAR1 VA. Ranges spanning the static/dynamic boundary now fall back to dynamic mappings, and the BAR1 P2P address encoding validates each address against the DMA window instead of assuming it fits.
Mailbox P2P setup used to assert-and-continue, so a failure while programming the HSHUB peer masks or setting up the mailboxes left half-programmed peer state behind. Convert the setup path to return status and unwind on failure: tear down partially set up mailboxes, invalidate peer masks programmed for a fresh mapping, and restore the peer bookkeeping. When taking an additional reference on an existing mapping, a failure only drops the newly taken references and leaves the mapping intact.
There was a problem hiding this comment.
Pull request overview
This PR enables BAR1 P2P on GB206 even with a display attached by making the static BAR1 window placement “display-aware” (post console/mailbox reservation) and hardens PCIe mailbox P2P setup by making it transactional with explicit failure unwinding.
Changes:
- Add DMA-window bounds checking when encoding BAR1 P2P IOVA addresses in
nv_gpu_ops(returning status instead of assuming all offsets fit). - Adjust TU102 static BAR1 eligibility/mapping sizing to allow GB206 static BAR1 to start after a 512MB-aligned console/mailbox reservation and clip to remaining BAR1 VA; spanning allocations fall back to dynamic mappings.
- Make GM200 PCIe mailbox P2P setup transactional: return status, unwind partially programmed mailbox / peer-mask state on failure, and validate mailbox BAR1 area presence.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/nvidia/src/kernel/rmapi/nv_gpu_ops.c | Adds BAR1 P2P address encoding validation against the DMA window and propagates errors to callers. |
| src/nvidia/src/kernel/gpu/bus/arch/turing/kern_bus_tu102.c | Makes static BAR1 sizing/placement display-aware on GB206 and adjusts static/dynamic boundary handling. |
| src/nvidia/src/kernel/gpu/bus/arch/maxwell/kern_bus_gm200.c | Refactors PCIe mailbox P2P setup into a transactional flow with rollback on failure. |
| README.md | Documents GB206 (RTX 5060 Ti / 5060) BAR1 P2P support with display attached. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
452
to
+454
| NV_PRINTF(LEVEL_ERROR, "BAR1 size %lld is not large enough to map FB size" | ||
| "%lld to force static BAR1\n", | ||
| bar1VASizeAligned, bar1MapSize); | ||
| "%lld at offset %lld to force static BAR1\n", | ||
| bar1VASizeAligned, bar1MapSize, staticBar1Offset); |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This enables BAR1 P2P on GB206 (RTX 5060 Ti / 5060) including when a display is attached, and hardens the PCIe mailbox P2P setup path.
Enable display-aware BAR1 P2P on GB206. On display-attached boards the console mapping occupies low BAR1 VA, so the static BAR1 auto-enable check — which required the window to cover all of client FB — always failed and disabled BAR1 P2P. The static window is now placed after a 512MB-aligned console/mailbox reservation and clipped to the remaining BAR1 VA. Allocations spanning the static/dynamic boundary fall back to dynamic mappings (the unmap path already classifies by BAR1 VA range, so it stays symmetric).
_nvGpuOpsEncodeBar1P2PAddrsnow validates each encoded address against the DMA window instead of assuming it fits, which matters once the static window no longer covers all of FB.Make PCIe mailbox P2P setup transactional. Mailbox setup used to assert-and-continue, so a failure while programming the HSHUB peer masks or setting up the mailboxes left half-programmed peer state behind. The setup path now returns status and unwinds on failure: partially set up mailboxes are torn down, peer masks programmed for a fresh mapping are invalidated, and peer bookkeeping is restored. Taking an additional reference on an existing mapping only drops the newly taken references on failure and leaves the mapping intact.
Non-GB206 behavior is unchanged, with one edge-case fix: the auto-size check previously added a spurious full 512MB of required BAR1 when
consoleSize + mailboxSizewas an exact 512MB multiple.Testing