gve: fix AF_XDP pool teardown ordering - #96
Open
m4r1k wants to merge 1 commit into
Open
Conversation
Quiesce the datapath before unmapping an XSK pool. DQO ring reconfiguration already unregisters the old pool and starts replacement queues, so return after that operation instead of falling through and disabling NAPI on the newly created rings. Use the locked NAPI helpers in the remaining path because XSK pool setup runs under the netdev operation lock on current kernels. Fixes: 8c783e0 ("gve: implement DQO RX datapath and control path for AF_XDP zero-copy") Signed-off-by: Federico Iezzi <fiezzi@google.com>
m4r1k
marked this pull request as ready for review
August 10, 2026 21:06
Author
|
Full reproducer. Run on both GCE guests deployed on export IFACE=ens3
export QUEUES=8
sudo ethtool -K "$IFACE" rx-gro-hw off
sudo ethtool -G "$IFACE" rx-buf-len 2048 tcp-data-split off
sudo ethtool -L "$IFACE" rx "$QUEUES" tx "$QUEUES"
ethtool -i "$IFACE"
ethtool -l "$IFACE"
ethtool -g "$IFACE"
sudo dmesg | grep -E 'DQO|GVE version|queue format' | tail -20The channel output should show: Pre-set maximums: Current hardware settings: Install Wireblast v0.2.4 on both: curl -fL \
https://github.com/atoonk/wireblast/releases/download/v0.2.4/wireblast_linux_amd64.tar.gz \
-o /tmp/wireblast.tar.gz
tar -xzf /tmp/wireblast.tar.gz -C /tmp
sudo install -m 0755 /tmp/wireblast /usr/local/bin/wireblast
wireblast --versionOn the receiver, obtain its private address: export IFACE=ens3
export RX_IP=$(ip -4 -o addr show dev "$IFACE" |
awk '{split($4, address, "/"); print address[1]}')
printf 'Receiver address: %s\n' "$RX_IP"Then start Wireblast on the receiver first: sudo prlimit --memlock=unlimited -- \
/usr/local/bin/wireblast \
--no-tui \
--yes \
--interface ens3 \
--mode receive \
--rx-mode udp-port \
--rx-port 9000 \
--queues 8 \
--duration 42sOn the sender, substitute the receiver’s private IP: export RX_IP="<RECEIVER_PRIVATE_IP>"
sudo prlimit --memlock=unlimited -- \
/usr/local/bin/wireblast \
--no-tui \
--yes \
--interface ens3 \
--mode udp \
--packet-size 64 \
--dst-ip "$RX_IP" \
--dst-port 9000 \
--flows 65535 \
--flow-order sequential \
--bps unlimited \
--queues 8 \
--duration 30sBoth processes should initially report something equivalent to: The important part of the reproducer is process termination. With the unpatched driver, AF_XDP socket teardown can cause either: or a Wireblast task stuck in uninterruptible sleep during the After Wireblast exits, inspect: sudo dmesg -T |
grep -E -A40 -B10 \
'NULL pointer|gve_rx_xsk_dqo|blocked for more than|wireblast'If the guest has already become unreachable, retrieve the serial-console output instead: gcloud compute instances get-serial-port-output <RECEIVER_VM> \
--zone <ZONE> \
--port 1Repeating the receiver-then-sender sequence three to five times makes the teardown race easier to reproduce. The UDP-port receive filter only redirects benchmark traffic, so SSH remains outside the XDP redirect rule. |
Collaborator
|
CC-ing @josh8551021 to take a look. |
Collaborator
|
Ah, I had a fix for this in the works, and was planning to send it up soon, along with a few other XDP-related fixes, once the current GVE fixes in the process of being upstramed were accepted. For reference, these are the fixes that will be going up soon: |
Author
|
Thanks @josh8551021! Looking forward for the new release then. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix two ordering problems while disabling an AF_XDP zero-copy pool:
Root cause
gve_xsk_pool_disable()currently callsxsk_pool_dma_unmap()before stopping DQO RX. An in-flightgve_rx_xsk_dqo()can consequently reachxsk_buff_dma_sync_for_cpu()after the pool's DMA device has been cleared. In the observed fault, this produced a NULL-device access at0x31c:On DQO,
gve_configure_rings_xdp()already tears down the old queues and creates replacements without the pool after its bitmap bit is cleared. Falling through afterward operates on the replacement rings. On kernels where XSK pool setup is serialized by the netdev operation lock, the subsequent plainnapi_disable()can also self-deadlock:Reproducer
The failure was reproduced on two Compute Engine
c4n-highcpu-16VMs using the DQO RDA queue format and Ubuntu 26.04 kernel7.0.0-1008-gcp. The in-tree driver reports version1.0.0; the affected teardown sequence is also present in official GVE v1.4.11.Configure the gVNIC for native XDP, repeatedly bind AF_XDP zero-copy sockets to all queues, send traffic, then close the sockets and detach XDP:
The baseline produced either the
gve_rx_xsk_dqo()NULL dereference above or tasks stuck ingve_xsk_pool_disable()during socket release.Validation
The patched v1.4.11 module was built against the exact
7.0.0-1008-gcpheaders and loaded on both VMs. Module SHA-256:Results:
The patch is intentionally limited to XSK pool teardown and does not add device-specific behavior to the AF_XDP application.