vulkan: implement MatMul (fixes #6858)#6865
Conversation
MatMul had no Vulkan implementation, so every MatMul in a Vulkan graph ran
on the CPU. That forces a download of both operands and a re-upload of the
result in the middle of the graph, which is the dominant cost for
attention-based models -- on a 22-model YOLO zoo, yolo12n (16 MatMul
layers) was the only model where Vulkan was *slower* than the CPU.
The shader indexes elements one at a time and keeps the unpacked layout
(support_vulkan_packing stays false), which makes batch slicing a simple
stride instead of re-deriving pack4/pack8 addressing for every rank.
Output shape and broadcasting follow MatMul::forward exactly:
* A 1-D operand contributes no dimension of its own. The CPU path
computes into a degenerate (N,1) or (1,M) blob and reshapes the extra
axis away, so the final shape is created directly here and described to
the shader with explicit strides -- a reshaped blob does not carry the
strides the general case would imply.
* With mixed ranks, MatMul::forward lifts a 3-D operand with
reshape(w, h, c, 1), which turns its CHANNEL axis into the DEPTH axis;
depth then strides by the original cstep rather than by w*h.
* Dispatch is over the logical (N, M, outd, outc) grid rather than over
the output blob, because a reshaped blob no longer carries M and outd
separately and the shader decodes depth out of y as gy / M.
Correctness: tests/test_matmul passes (it exercises the GPU path through
test_layer, covering 1-D through 4-D, transB and broadcasting). Full ctest
is 177/178 both with and without this commit -- identical failure sets; the
one failure is a pre-existing atan2 special-value mismatch on RADV.
Raw forward pass on a Radeon 8060S (Strix Halo, RADV GFX1151), 640x640,
4 threads, median of 3 alternating sessions, same build both sides:
model MatMuls vulkan before vulkan after
yolo12n 16 62.64 ms 8.93 ms 7.0x
yolo11n 2 9.77 ms 6.02 ms 1.6x
yolov8n 0 5.99 ms 5.93 ms 1.01x
yolov8n has no MatMul and is included as a control: it does not move.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fe0817fbae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| else if (outc == 1 && outd == 1) | ||
| { | ||
| top_blob.create(N, M, elemsize, opt.blob_vkallocator); | ||
| } | ||
| else if (outd == 1) |
There was a problem hiding this comment.
When neither operand is 1-D, these extent-based branches collapse singleton batch axes: a rank-3 multiplication with batch size 1 returns a 2-D blob, while a rank-4 multiplication with outd == 1 returns a 2-D or 3-D blob. The CPU implementation instead unconditionally creates rank 3 for max_ABdims == 3 and rank 4 for max_ABdims == 4, even when those extents equal one, so enabling Vulkan changes the output shape and can make downstream layers interpret its axes differently. Select the output rank from max_ABdims, not solely from outc/outd values.
Useful? React with 👍 / 👎.
2 participants
Implements the Vulkan
MatMulrequested in #6858.MatMulhad no Vulkan implementation — there is nosrc/layer/vulkan/matmul_vulkan.cpp— so everyMatMulin a Vulkan graph ran on the CPU, forcing a download of both operands and a re-upload of the result mid-graph. On a 22-model YOLO zoo,yolo12n(16MatMullayers) was the only model where Vulkan was slower than the CPU.Results
Raw forward pass, Radeon 8060S (Strix Halo, RADV GFX1151), 640×640, 4 threads, median of 3 alternating sessions, same build on both sides:
yolov8ncontains noMatMuland is included as a control — it does not move, which is what attributes the other two gains to this layer rather than to build or machine differences.Two notes on honesty of measurement:
MatMulalone. With both applied the numbers improve further, but that would confound the two changes.1.81x -> 4.19xI quoted in Vulkan: MatMul has no Vulkan implementation — attention models fall back to CPU mid-graph (yolo12n 1.81x vs 5-8x for CNN peers) #6858; that earlier run was on a differently-loaded machine and a build that also carried the download fix. The before/after ratio here is measured in one sitting with alternating sessions.Correctness
tests/test_matmulpasses. It runs the GPU path throughtest_layer, covering 1-D through 4-D operands,transB, and broadcasting.Full
ctest: 177/178 both with and without this commit, identical failure sets. The single failure istest_binaryop_3→test_binaryop_atan2_special op_type=10, a pre-existing atan2 special-value mismatch on RADV, byte-identical on pristinea4d2ea1.Worth flagging: my first draft failed
test_matmul, and the failures were instructive enough to be worth recording here.Degenerate axes. A 1-D operand contributes no dimension of its own.
MatMul::forwardcomputes into a(N,1)/(1,M)blob and reshapes the extra axis away. Creating(N, M, …)unconditionally gavedims=2where the CPU givesdims=1. The final shape is now created directly and described to the shader with explicit strides, because a reshaped blob does not carry the strides the general case implies.Rank promotion. With mixed ranks,
MatMul::forwardlifts a 3-D operand withreshape(w, h, c, 1)— its channel axis becomes the depth axis, and depth then strides by the originalcstep, notw*h. ReadingA.cas channels rejected valid broadcasts:test_matmul_13(A(14,23,10)× B(5,14,1,16)) hit the broadcast guard, returned −1, and the never-created blob then segfaulted inrecord_download.Dispatch grid. Dispatch is over the logical
(N, M, outd, outc)grid rather than over the output blob, since a reshaped blob no longer carriesMandoutdseparately while the shader decodes depth out of y asgy / M. TheMatdispatcher overload supplies the group counts without allocating.Implementation notes
support_vulkan_packingstays at its defaultfalse: the shader indexes elements one at a time, which keeps batch slicing a simple stride instead of re-deriving pack4/pack8 addressing for every rank.ncnn_add_layer(MatMul)already picks the Vulkan source up; no CMake change needed.This is a straightforward scalar implementation — correct and a large improvement over the CPU round trip, but not a tuned GEMM. Tiling and cooperative matrix would go considerably further on the batched attention shapes from #6858. Happy to iterate if you would rather land something closer to that first.
🤖 Generated with Claude Code