Skip to content

Commit bce51f1

Browse files
committed
arm64: fix the SME2 sbgemm kernel's ZA handoff and BFMOPA predicates
Two fixes from Michael Tesch's M4 Pro testing: - s2b_blk had its own __arm_new("za"), so the call from the equally- annotated s2b_drive set up a lazy ZA save that __arm_tpidr2_save (OpenBLAS's aborting weak stub) committed. It now shares ZA from the drive function through S2_S S2_ZA like the fp32/fp64 block functions. - BFMOPA governs per 16-bit element, but the predicates were svptrue_b32(), which enables only the even bf16 lanes and drops the k+1 half of every pair (every k >= 2 wrong, k = 1 right). Now svptrue_b16(). Also apply the suggested thread cap to SBGEMM: sme_level3_threads() capped only real precision (!BFLOAT16), so default threads contended for the SME units - 839 GFLOPS at n = 4096 against about 1190 on one thread and 2347 with one thread per unit on the M4 Pro. The condition is now (!BFLOAT16 || !BGEMM): bgemm keeps the NEON kernels and its threading. With the fixes his run reports the sbgemm/BLAS/CBLAS/utest suites passing and 3000 randomized trials clean; 1193 GFLOPS at 2048^2 single thread against 0.48 on develop.
1 parent c151d85 commit bce51f1

2 files changed

Lines changed: 4 additions & 5 deletions

File tree

‎common_thread.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ extern int blas_omp_threads_local;
157157
extern int blas_is_num_threads_set_explicitly;
158158

159159
#if defined(ARCH_ARM64) && defined(OS_DARWIN) && defined(__clang__) && (defined(VORTEXM4) || defined(DYNAMIC_ARCH)) && \
160-
!defined(COMPLEX) && !defined(BFLOAT16) && !defined(HFLOAT16)
160+
!defined(COMPLEX) && (!defined(BFLOAT16) || !defined(BGEMM)) && !defined(HFLOAT16)
161161
#include <sys/sysctl.h>
162162
#ifdef DYNAMIC_ARCH
163163
extern char *gotoblas_corename(void);

‎kernel/arm64/sbgemm_kernel_sme2.c‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ S2_INL void s2b_op(const int SUB, int t, svbool_t p, svbfloat16_t b, svbfloat16_
9292
/* ZA += or -= the A panel (wm rows) times the B panel (wn columns) over k steps of the packed layout. */
9393
S2_NOINL void s2b_mac(const int SUB, BLASLONG k, const IFLOAT *a, BLASLONG wm, const IFLOAT *b, BLASLONG wn) S2_S
9494
S2_ZA {
95-
const svbool_t pt = svptrue_b32();
95+
const svbool_t pt = svptrue_b16(); /* BFMOPA governs per 16-bit element */
9696
const svbool_t pm0 = svwhilelt_b16_s32(0, (int32_t)wm);
9797
const svbool_t pm1 = svwhilelt_b16_s32(S2B_UN * 2, (int32_t)wm);
9898
const svbool_t pn = svwhilelt_b16_s32(0, (int32_t)wn);
@@ -116,9 +116,8 @@ S2_NOINL void s2b_mac(const int SUB, BLASLONG k, const IFLOAT *a, BLASLONG wm, c
116116

117117
/* Block of wm rows and wn columns of C; sgn 1 for C += alpha AB with alpha = +-1 (the AB part
118118
added or subtracted in ZA), sgn 0 for a general alpha scaled on the way out of ZA. */
119-
__arm_new("za") __arm_locally_streaming static void s2b_blk(BLASLONG k, const IFLOAT *a, BLASLONG wm, const IFLOAT *b,
120-
BLASLONG wn, FLOAT *C, BLASLONG ldc, FLOAT alpha,
121-
int sgn) {
119+
S2_NOINL static void s2b_blk(BLASLONG k, const IFLOAT *a, BLASLONG wm, const IFLOAT *b, BLASLONG wn, FLOAT *C,
120+
BLASLONG ldc, FLOAT alpha, int sgn) S2_S S2_ZA {
122121
if (sgn) {
123122
s2_c_load(1, S2_NT, C, ldc, (int)wn, (int)wm, S2_LOAD, (FLOAT)1);
124123
s2b_mac(alpha == (FLOAT)-1, k, a, wm, b, wn);

0 commit comments

Comments
 (0)