Repository navigation
Conversation
zgemm_kernel_sme2.c is the CGEMM/ZGEMM and TRMM kernel of the level-3 driver. The packed panels keep the generic interleaved layout; FMOPA on the interleaved vectors accumulates re*re, im*re, re*im and im*im in separate ZA rows, and FCADD/FCMLA form the complex result and apply alpha on the store. All four conjugation variants come from the same code. CGEMM unroll is 16x16, ZGEMM 16x8 (ZA full, B one streaming vector); the copies are the generic ones, with new generic zgemm_tcopy_16.c and zlaswp_ncopy_16.c. ztrsm_kernel_sme2.c does the TRSM update in the GEMM kernel and the solve in plain or NEON code. SYMM, HEMM, SYRK, HERK, SYR2K, HER2K, TRMM, TRSM and the complex LAPACK routines on the level-3 driver use these kernels. Complex GEMM itself still goes to the SME1 direct kernels from interface/gemm.c.
…EXM4 On VORTEXM4, CGEMM/ZGEMM now skip the SME1 direct kernels and run on the level-3 driver with the SME2 complex kernel, which gives them the driver's threads and removes the static packing buffers of the SME1 kernels from this path. ARMV9SME keeps the SME1 path. The one-thread-per-SME-unit cap of num_cpu_avail() now covers complex level-3 BLAS and LAPACK too.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #6151.
Before: on VORTEXM4, complex SYMM/HEMM/SYRK/HERK/SYR2K/HER2K/TRMM/TRSM and the complex LAPACK level-3 paths ran on the NEON kernels of NEOVERSEN1 (8x4 for C, 4x4 for Z), at about 100 (C) and 50 (Z) GFLOPS on one thread. CGEMM/ZGEMM went from
interface/gemm.cto the SME1 direct kernels, which run on one thread only and pack into static buffers (see #6157).After: a new SME2 complex kernel serves all of them through the level-3 driver. One thread, n = 1024, CSYMM goes from 101 to 1554 GFLOPS (Accelerate 1181) and ZSYMM from 51 to 398 (342).
How:
kernel/arm64/zgemm_kernel_sme2.c(GEMM kernel, and TRMM kernel withTRMMKERNEL): the packed panels keep the generic interleaved complex layout, so all generic complex copies and the driver work unchanged. FMOPA on the interleaved vectors puts re(b)·A and im(b)·A in neighbouring ZA rows; on the store, FCADD (rotation 90 or 270) forms the complex product and two FCMLA apply alpha, so the four conjugation groups differ only in the rotations. CGEMM unroll 16 x 16, ZGEMM 16 x 8 (ZA full, B one streaming vector), with the generic power-of-two tails. Plain loops without SME2.kernel/arm64/ztrsm_kernel_sme2.c: LN/LT/RN/RT (and CONJ), the update with the GEMM kernel, the solve on NEON (FCMLA on C columns on the right side, real/imaginary planes on the left side).kernel/generic/zgemm_tcopy_16.c,kernel/generic/zlaswp_ncopy_16.c: the 16-column versions that did not exist yet (Makefile.LAand CMake takezlaswp_ncopy_$(UNROLL_N)).KERNEL.VORTEXM4(non-GCC block), the clang-only VORTEXM4 section ofparam.h(unroll, P = 512/256, Q = 512, R = 4096/2048 within the 32 MB buffer), and the VORTEXM4 branch ofcmake/prebuild.cmake.interface/gemm.cno longer sends complex GEMM on VORTEXM4 to the SME1 kernels (ARMV9SME keeps them), so CGEMM/ZGEMM get the driver's threads;common_thread.hextends the one-thread-per-SME-unit cap ofsme_level3_threads()to complex.Results
M4 Pro, GFLOPS, develop 791b02a -> this PR (Accelerate). Best of two runs while other builds ran on the machine; spread about 5% at n >= 1024 on one thread, 10-30% at n = 256 and with threads.
LAPACK, n = 2048: cgetrf 97 -> 564 GFLOPS on one thread (Accelerate 158), 397 -> 752 with threads; zgetrf 49 -> 246 (92).
Slower than before: SYMM/TRMM etc. at n <= 12 (about 60 ns of streaming-mode entry per kernel call; CSYMM n = 4: 19 -> 8 GFLOPS), ZGEMM at n = 8-24 against the SME1 kernel (n = 8: 28 -> 6), and one-thread ZGEMM at large n about 3% below the SME1 kernel. TRSM is about half the speed of the other routines (NEON solve per 16 x 16 block).
Tests
make: BLAS/CBLAS tests, utest 144/144 and 1564/1564.make lapack-test: REAL, DOUBLE and COMPLEX 0 errors; COMPLEX16 one error, ZHS innep.in(n = 16, type 18/20, inverse-iteration ratio). The same ZHS case fails on develop withTARGET=VORTEX(which takes the standard complex level-3 path plus the complex GEMM -> GEMV forwarding), and it passes with that forwarding switched off. Every forwarded GEMV call of the test, recomputed in long double, is within 3.5e-16 of the elementwise bound, so this looks like a rounding-sensitive test case rather than a wrong result. I can open a separate issue for it.-DTARGET=VORTEXM4 -DNOFORTRAN=1 -DBUILD_TESTING=ON, ctest 14/14),DYNAMIC_ARCH=1 NO_FORTRAN=1(randomized check, 0 failures) and GCC 16 (TARGET=VORTEXM4, NEON kernels as before; BLAS tests and utest pass).AI assistance
AI coding tools (Claude Code) helped to prepare this PR. The changes were reviewed, tested and benchmarked as described above.