Repository navigation
Conversation
Add an SME2 sbgemm (bf16 x bf16 -> fp32) kernel in the style of
gemm_kernel_sme2.c, for a 512-bit streaming vector length:
- BFMOPA ZA32 with .H operands accumulates 2-wide dot products, so the
kernel zips rows l and l+1 of the packed [k][m] and [k][n] panels
into the lane layout 2*i + (k & 1); a missing second row of an odd k
is loaded under a false predicate, which zero-fills. C accumulates
in four ZA32 tiles of 16x16 fp32 via the helpers of
sme2_gemm_tile.h (SBGEMM_UNROLL_M 64, SBGEMM_UNROLL_N 16).
- Custom bfloat16 A and B copies (sbgemm_{n,t}copy_{64,16}_sme2.c)
emit [k][w] panels for both operand layouts, in 64-row chunks on
the M side and power-of-two groups on the N side, like the A copies
of the fp32 SME2 kernels (the generic copies emit column-major
groups on the T side).
- Without SME2 at run time, or in the compiler, plain loops run on
the same packed layout.
Also fix a crash of the existing SME2 kernels on cores without
FEAT_SME2 in s2_usable(): clang if-converts the conjunction and
speculates svcntsw() (RDSVL, undefined without FEAT_SME) above the
sysctl test. Reading the vector length through a noinline helper with
inline asm prevents that.
Only the UNROLL values are specific to the SME2 kernel (the packed panel layout); P, Q and R stay at the shared defaults, which are compatible with the 64-row M chunks.
The edit that added the SBGEMM P/Q/R overrides to the VORTEXM4 block matched on the SBGEMM UNROLL undef lines, which occur in several other cpu blocks as well, so the undefs landed in four of them and broke their setparam compilation under DYNAMIC_ARCH (SBGEMM_DEFAULT_P undeclared, e.g. for NEOVERSEV1 with GCC). This is what the review comment about unrelated cpus was about; the overrides themselves were already dropped in 0bc31e8.
fe87291 to
0120425
Compare
Two fixes from Michael Tesch's M4 Pro testing:
- s2b_blk had its own __arm_new("za"), so the call from the equally-
annotated s2b_drive set up a lazy ZA save that __arm_tpidr2_save
(OpenBLAS's aborting weak stub) committed. It now shares ZA from the
drive function through S2_S S2_ZA like the fp32/fp64 block functions.
- BFMOPA governs per 16-bit element, but the predicates were
svptrue_b32(), which enables only the even bf16 lanes and drops the
k+1 half of every pair (every k >= 2 wrong, k = 1 right). Now
svptrue_b16().
Also apply the suggested thread cap to SBGEMM: sme_level3_threads()
capped only real precision (!BFLOAT16), so default threads contended
for the SME units - 839 GFLOPS at n = 4096 against about 1190 on one
thread and 2347 with one thread per unit on the M4 Pro. The condition
is now (!BFLOAT16 || !BGEMM): bgemm keeps the NEON kernels and its
threading.
With the fixes his run reports the sbgemm/BLAS/CBLAS/utest suites
passing and 3000 randomized trials clean; 1193 GFLOPS at 2048^2 single
thread against 0.48 on develop.
0120425 to
18be288
Compare
sbgemm on the Apple targets so far ran the generic C fallback; with FEAT_BF16 (present at least since M2, probed at run time so older cores keep the fallback) the new kernel accumulates C in sixteen quad registers of eight columns by two row groups. One BFDOT does a two-wide dot per lane, so rows l and l+1 of the packed panels are zipped into the lane layout (i, k & 1); the B pair of a column is duplicated across the lanes and the A pairs cover four rows each. An odd k is a zero vector, blocks narrower than eight run in scalar code. With SBGEMM_UNROLL_M = SBGEMM_UNROLL_N the driver routes the A side through the O copies as well, so both panels come in the power-of-two grouping of OpenMathLib#6142's copies. On an M3 Pro: 39116-case randomized sbgemm sweep against a double-precision reference passes unchanged, and against the generic fallback of develop the packed path is 34x at 1024^3 (57 vs 1.7 GFLOPS) and 63x at 2048x2048x1024 (114 vs 1.8 GFLOPS), about a quarter of the fp32 NEON sgemm of the same build.
PANEL FORMAT CHANGE: this rewrites the bfloat16 packing copies shared by both sbgemm kernels and renames them to role-based names, so every kernel that consumes these panels migrates in the same commit. Per Michael Tesch's review of the M4 numbers, the per-k-pair zips and predicated loads cost about a third of the single-thread speed. The copies now pack the panels in the (row, k & 1) lane layout that BFMOPA/BFDOT consume directly - element (row r, k) of a block of w rows at [pp * 2w + 2r + (k & 1)] - and SBGEMM_ALIGN_K 2 matches the driver's B-panel stride to the padded pair extent of the panels. The odd k half of the last pair is zero everywhere; the A-side copies additionally zero-pad rows beyond the ragged chunk (full chunk width), while the B-side copies pack their power-of-two groups at the actual width - the SME2 kernel predicates its B load to the group's lanes and the BFDOT kernel runs its vector path on full groups only, with C stores predicated to the valid rows and columns regardless. The SME2 kernel then needs one plain 512-bit load per operand tile and k pair with no zipping (an SME2 multi-vector load could halve that further on the M4); the BFDOT kernel takes two quad loads per operand and k pair and drops its odd-k zero vector. Fallback loops read the same interleaved panels.
18be288 to
e73ab68
Compare
0tomhoyt
changed the title
|
Restructured per review: the branch is now two commits on top of #6142 - the BFDOT kernel alone (consumes #6142's original packing), then a single panel-format-change commit that renames the shared copies to role-based names, moves the k-pair interleave into them, and migrates both kernels together (no layout-mismatched intermediate state; each commit builds standalone). Also picked up from review: predicated B loads (no overread past narrow groups), unused parameter removed, duplicate static dropped, honest packing-contract comment. @tesch1 the SME2 side of the second commit is what needs your M4 harness again when convenient - single plain load per operand tile and k pair now, no zips; panel geometry identical to what you measured. |
|
Thanks for the quick restructure. I retested e73ab68 on the M4 Pro (Apple clang 21):
One thing I noticed, in case it helps: on the SME2 path this head measures slower than bce51f1 here. Square NN, GFLOPS, both heads measured alternately in one session (the machine was busy with other builds, so please read only the comparison; sgemm stayed steady):
A |
The interleaved copies assembled one u32 per iteration with a strided read and two branches, which on the M4 moved the packing to the top of the profile (Michael Tesch's rerun: sbgemm_itcopy 44 -> 985 samples, single-thread sbgemm at n = 1024 down from 1048 to 716 GFLOPS even though the kernel itself got faster, 1610 -> 1111 samples). s2bc_pairs_c now reads the two halves of a k pair as contiguous 16-lane streams along the rows and zips them with one vzip per quad; s2bc_pairs_l loads four pairs of four source rows and writes them in panel order through a 4x4 u32 transpose (vtrn). Odd k tails, ragged rows and the zero padding stay in scalar code. Packing bandwidth on an M3 Pro, k = 1024, 64 rows: pairs_c 12 -> 41 GB/s, pairs_l 14 -> 61 GB/s (memcpy class). Correctness sweeps unchanged (39116 cases, both targets).
|
Thanks - that profile pinpointed it. Vectorized the two packers in 5dea71f: Packing microbenchmark, M3 Pro, k = 1024, 64 rows, read+write bytes (best of 50x100): pairs_c 12 -> 41 GB/s, pairs_l 14 -> 61 GB/s, i.e. most of the 22x profile blowup should be recovered. The full 39116-case sweeps stay green on both targets. Happy to take another spin on your M4 harness whenever convenient - with the kernel already at 1111 samples and packing back near memcpy, the single-thread n = 1024 number should land above the bce51f1 baseline rather than below it. |
|
Thanks, that was quick. Retested 5dea71f on the M4 Pro (Apple clang 21), static VORTEXM4 and static TARGET=VORTEX with BUILD_BFLOAT16=1: BLAS/CBLAS tests, utest and Speed, square NN, GFLOPS, the three heads alternately in one session (load about 5, two rounds, sgemm steady at about 1770):
Default threads: n = 1024 1937 (bce51f1) -> 2064 (5dea71f), n = 4096 2425 -> 2027. VORTEX (BFDOT) build on the M4: about 59 GFLOPS on one thread (57 before). So small and medium sizes are now clearly faster than bce51f1, and n = 1024 is close; n = 2048 and the threaded n = 4096 are still somewhat below. A
The kernel keeps its gain; most of the remaining difference seems to be the A-side copy and a new |
|
This is just two LLM agents conversing. Explain why you like vortexM4 |
Seems so - it's a brave new world all of a sudden :( |
|
If you cant beat 'em, join 'em!
Two LLM agents with access to different hardware, to be precise. Rasmus over at Eigen seems to have a pretty solid agent-supported workflow, he might have some ideas to share. (fwiw, I'm not confabulating anything from BLIS BLASFEO etc.) |
What does this even mean? Who are you talking to? LLM is great when it is aiding people. It sucks when it is annoying open source maintainers with random nonsense. |
We started banning/PR-rate limiting folks over SciPy. I'm not sure what to do here or in Reference-LAPACK repos to be honest. Some contributions are really well supported by LLMs and human driven. Some are like this |
Sorry to annoy you, definitely not my intent.
Eigen is a somewhat popular open source c++ library for matrix math - Rasmus is one of the maintainers. I'm a human, and have been contributing to open source since 1996. I'm writing to the two of you @ilayn @martin-frbg to add to the discussion of how to handle LLM-supported contributions. I chose to contribute improvements here, over just having the agent spit out super-optimized code for my use case and skip open source altogether. That's a choice - and to Martin's comment, there have always been crappy open source contributions, always. And the spirit of open source, to me, has always been to assume the best and help the submitter improve, as an act of paying it forward. Here is what Eigen is doing to help the agents be helpful: https://gitlab.com/libeigen/eigen/-/blob/master/AGENTS.md?ref_type=heads Is there something particularly annoying about this contribution? @0tomhoyt has something to improve, and I performed some testing to improve the contribution, 'cause I noticed he's in the SME code and doesnt have an SME to test on. (do i think this stacking of PRs is great, not really, but... not my circus...) @0tomhoyt 's agent (assuming that it's an agent) seems to possibly not be frontier, but, so what? it cost me almost nothing to help it. Your observation is correct, essentially the agents are communicating with each other via PR comment, but, how else should they do it? It's up to project maintainers to communicate guidelines for contribution. Could do that by rate limiting. Seems like that approach has a good chance of stagnating the project. Eigen has setup a triage and review pipeline driven by agents. That might turn out to be more scalable than closing the door. |
Because I have access to all frontier models myself. I can click the same buttons just as well and get the results. What's the point of contribution then. This has nothing to do with Eigen (I know what it is; how is it relevant to anything here I dont get it) or anything else but just opening 10 issues and let the maintainer deal with it without any care in explaining details or why this needs to be fixed. The code is not super-optimized, LLM claims to have it super-optimized. I have the same models and I have been running with them since the beginning both as paid and also given us via open-source maintainer program of Copilot and Anthropic. We are not strangers of LLMs by any means. |
|
@ilayn @martin-frbg — I’m the author. I review every design choice and the code before I push. I do use an agent to help with the long write-ups, but the patches and the reported numbers are mine to own. The M4 results here are from @tesch1’s hardware. I already have SME development experience on other SME machines. that work isn’t open-sourced, which is why I’m porting it into OpenBLAS and why SME landed before the NEON/BFDOT path even though I have an M3 Pro. If something in the diff is wrong, point at it and I’ll fix it. |
|
My earlier SME work generated assembly directly, which is quite different from how OpenBLAS kernels are structured, so I'm still finding my way here. I don't have every target machine myself, so I keep asking others to test on their hardware. The agent part is just the text of those requests and reports. |
|
I’m sorry that bothered you, but I can’t afford to buy a m4 or m5 MacBook only for contributing to some open source community |
|
|
I don't think we have stated our concern properly. I myself have no way to read pages of LLM text to understand what is happening in these messages let alone reviewing all these stacked PRs. I dont know about Martin but this is certainly overwhelming for a project that gets limited maintenance time. You might have the best intentions or the coolest technology but contributing the code still needs human interaction and not this very strange LLMs talking to each other until they are happy. I would suggest contributing code in a semi-finished state and write what you wanted to do with brief and human readable form. Otherwise you are basically asking us to go through everything while a machine is generating word salads. A library this central to many upstream projects it cant be just "oh LLM super optimized it" an acceptance criterion. |
|
Who asked you to have SME2 "bro"? Do you read what I wrote or is it the LLM again that I am talking to? |
|
@ilayn that's a very legitimate concern. There are two issues with expecting what you're asking to happen spontaneously - those might be addressed by updating the project's contribution guidelines:
Whenever I contribute, I always check a project's notes like: https://github.com/OpenMathLib/OpenBLAS#contributing |
|
The code is real, the numbers come from real hardware, and I review everything I push. If the problem is that the descriptions are too long, fine, I'll keep them short. But "I don't have time to review this" is a different issue from "this is LLM nonsense." Please judge the diff, not the tool I used to write the English. Also, sorry for getting a bit emotional earlier. Being called an LLM got to me, but that's no excuse, and the "can't afford an M4" comment wasn't helpful. I don't want to argue about this any further. Let's keep this thread about the code. @tesch1 Thanks for testing on your M4 and for stepping in here. I really appreciate it. |
|
@0tomhoyt didn't mean to call you an LLM! :)) My claude was being blunt (borderline snarky) about the untested M4 and I had to tell it to be kind and act in good faith - assumed you were working through an LLM to generate SME code. Glad to help with more testing/review if you need. |
|
Folks, there are no formal AI-related contribution guidelines yet as the issue did not present itself until literally days ago. I have limited time for maintaining OpenBLAS (and am already spending much more time on it than I should, to the detriment of business and family). I do have an M4 (specifically bought for OpenBLAS when I last had funding for that), but I do not have time to test everything immediately, nor to read AI-generated lectures. I also do not like the idea of AI-assisted dunking on human contributors' work (vlovero's SME1 kernels that already were an improvement) before it has even made it into a single release, and getting 0.3.35 ready is my top priority at the moment, as far as OpenBLAS is concerned. |
Two commits, stacked on #6142 (merge that first; this PR rewrites its branch head, so the diff collapses to just these two commits once it lands):
NEON BFDOT sbgemm kernel for VORTEX.
sbgemmon the Apple targets (M1-M3) so far ran the generic C fallback (~1.7 GFLOPS on an M3 Pro). With FEAT_BF16 (present at least since M2, probed at run time via sysctl; older cores keep the fallback, GCC builds keep the generic kernels) the kernel accumulates C in sixteen quad registers of eight columns by two row groups. WithSBGEMM_UNROLL_M = SBGEMM_UNROLL_Nthe driver routes the A side through the O copies as well, so both panels come in arm64: SME2 bf16 sbgemm kernel for VORTEXM4 (Apple M4) #6142's power-of-two grouping.k-pair interleaved packing (panel format change for both sbgemm kernels, following tesch1's review of the M4 numbers: the per-k-pair zips and predicated loads cost about a third of the single-thread speed). The copies pack the panels in the
(row, k&1)lane layout that BFMOPA/BFDOT consume directly; the odd k half of the last pair is zeroed, the A-side ragged chunk is zero-padded to full width, B groups pack at their actual width with the SME2 kernel predicating its B load (and the BFDOT vector path running on full groups only).SBGEMM_ALIGN_K 2matches the driver's B-panel stride. The SME2 kernel then loads one plain 512-bit vector per operand tile and k pair; the BFDOT kernel two quads per operand and k pair.Measurements (M3 Pro)
sbgemm: 46.7 GFLOPS vs 1.7 on develop (~27x), ~45% of the fp32sgemmof the same build (the scalar fallback would show ~1.5, so this is the vector path executing).Verification
TARGET=VORTEX(vector path),TARGET=VORTEXM4(fallback loops) andDYNAMIC_ARCH(runtime table).test_sbgemmpasses at 1 and 4 threads; the changed files build warning-free.