Skip to content

arm64: VORTEXM4 level-3 panel copies in streaming mode - #6154

Open
tesch1 wants to merge 2 commits into
OpenMathLib:developfrom
tesch1:sme2-copies
Open

tesch1 wants to merge 2 commits into
OpenMathLib:developfrom
tesch1:sme2-copies

Conversation

@tesch1

@tesch1 tesch1 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Addresses the copy part of #6146.

Before: on VORTEXM4 the B-side GEMM copies were the generic C copies, and the A-side copies of the SME2 kernels used NEON. At n <= 512 these copies took most of the level-3 run time (about 70-80% of SSYRK and STRMM at n = 256, one thread). NEON versions of the B-side copies were tried first: they gain little, because the copies are limited by the bandwidth that one core gets from L2.

After: the copies run in streaming mode. The SME unit moves the same data 2-4x faster (fp32, one thread, data in L2: oncopy 50 -> 165 GB/s, otcopy 75 -> 220 GB/s at m = n = 256).

How:

  • sme2_gemm_copy.h: s2c_rows() (contiguous rows) uses predicated streaming vector loads and stores; s2c_cols() (contiguous columns) loads columns into horizontal ZA slices and stores rows from vertical slices. Pieces smaller than 512 elements, and CPUs without SME2 (s2_usable()), keep the NEON code.
  • New gemm_[nt]copy_b_sme2.c for S/DGEMMONCOPY and S/DGEMMOTCOPY: the same helpers, with the layout of generic/gemm_[nt]copy_16.c (UNROLL_N columns, then 8/4/2/1), in one streaming-mode entry per call.
  • TRMM/TRSM A-side copies: the diagonal block goes through the panel copy, then each row gets its zero side and its unit or inverted diagonal (no per-element tests).

Only VORTEXM4 files change. GCC builds keep the generic copies (the KERNEL.VORTEXM4 section for non-GCC compilers).

Not in this PR: the OUTER TRMM/TRSM/SYMM copies (right-side B panels) are fixed to the generic files in kernel/Makefile.L3; changing them needs a shared-file change.

Results

M4 Pro, one thread, GFLOPS, develop 791b02a -> this PR:

fp32 n = 64 n = 256 n = 1024
SSYMM (L, L) 181 -> 215 764 -> 1124 1285 -> 1648
SSYRK (L, N) 91 -> 102 423 -> 518 1016 -> 1144
SSYRK (L, T) 76 -> 100 355 -> 508 915 -> 1157
STRMM (L, L, N) 50 -> 85 285 -> 421 805 -> 1010
STRMM (R, L, T) 78 -> 90 314 -> 374 812 -> 967
STRSM (L, L, N) 42 -> 68 221 -> 321 677 -> 837
STRSM (R, U, N) 63 -> 71 273 -> 329 810 -> 837
fp64 n = 64 n = 256 n = 1024
DSYMM 97 -> 129 255 -> 381 340 -> 423
DSYRK (L, T) 41 -> 60 149 -> 229 285 -> 369
DTRMM (L, L, N) 32 -> 44 119 -> 171 257 -> 321
DTRSM (L, L, N) 29 -> 38 104 -> 146 242 -> 296

SGEMM/DGEMM do not change. Default threads: also faster, no case slower (SSYMM n = 1024 2540 -> 2955, STRMM 1753 -> 1967, at n = 4096 5-11%).

Tests

  • make: BLAS/CBLAS tests, utest 144/144 and 1564/1564.
  • make lapack-test: 0 numerical errors.
  • Randomized GEMM/SYMM/SYRK/SYR2K/TRMM/TRSM check against naive references, s and d, 1000 trials each: 0 failures.
  • New copies against the generic ones for all m, n <= 70: identical output, with and without SME2 (NEON path).
  • CMake (-DTARGET=VORTEXM4 -DNOFORTRAN=1 -DBUILD_TESTING=ON, ctest 14/14) and DYNAMIC_ARCH=1 NO_FORTRAN=1 builds.

AI assistance

AI coding tools (Claude Code) helped to prepare this PR. The changes were reviewed, tested and benchmarked as described above.

The SME unit moves data 2-4x faster than NEON or the generic copies.
In sme2_gemm_copy.h, s2c_rows() uses predicated streaming vectors
and s2c_cols() transposes through ZA, for pieces of 512 elements or
more; smaller pieces and CPUs without SME2 stay on NEON. The new
B-side copies (S/DGEMMONCOPY, S/DGEMMOTCOPY) use the same helpers
with the layout of the generic copies (UNROLL_N, then 8/4/2/1).

Part of OpenMathLib#6146
…ement tests

The diagonal block goes through the panel copy; then each row gets
its zero side and its (unit or inverted) diagonal element.

Part of OpenMathLib#6146

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant