Over 30% faster GLM 5.3 Flash decode on Metal (M3 Ultra), bit-exact - #964
Open
trueimage wants to merge 42 commits into
Open
Over 30% faster GLM 5.3 Flash decode on Metal (M3 Ultra), bit-exact#964trueimage wants to merge 42 commits into
trueimage wants to merge 42 commits into
Conversation
GLM-5.3-Flash-Q4_K stores blk.N.kda_{q,k,v,output} as BF16 while its experts
are Q4_K. Those four tensors are dense, so all 34 KDA layers are read on every
decoded token: 8.50 GiB per token, 60% of decode traffic and nearly twice what
all routed experts read.
This is not what this repo's own quantizer produces. regular_qtype() in
glm53_quantize.py maps role="linear_attention" to Q8_0 for its default q4
artifact, so a Q8_0 KDA model is an already-supported shape -- the loader
accepts it via tensor_expect_glm_dense_quant_layout and the generic
glm53_graph_matmul handles it. glm53-requant-kda produces one from an existing
GGUF, without needing the source checkpoint, by converting those tensors
through the same quants.c facade the other tools use and copying every other
byte verbatim.
The input is mmapped for the whole run, so the tool refuses an output that
resolves to it -- same path, hard link or symlink, compared on st_dev/st_ino
rather than on the path string. It builds the result beside its destination
and renames it into place, so out_path only ever holds the previous file or a
complete one. Header counts, dimensions, the element product, general
.alignment, and every tensor's source range are checked against the mapping
before use, and a tensor whose type this build cannot size is refused rather
than copied as zero bytes, which would have emitted a file that still parses
with the payload silently gone.
BF16 was not simply an oversight: Metal's ds4_gpu_glm53_matmul_bf16_qkv fuses
the three projections into one dispatch and requires BF16, and it was added as
an M3 Ultra optimisation. Measured on an M3 Ultra, that fusion is worth 0.7%
(21.12 -> 20.97 tok/s with DS4_METAL_DISABLE_GLM53_BF16_QKV=1). The BF16
storage it requires costs an order of magnitude more, so the generic fallback
is the better trade and no fused Q8_0 kernel is needed.
Converting 136 tensors takes KDA from 8.50 to 4.52 GiB and the file from 177.8
to 173.8 GiB. On Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal,
fully resident, arms interleaved O-Q-Q-O with the same binary and only the
model file changing, 8 context frontiers from 2048 to 16384:
decode +13.37% (+13.24% to +13.54%, every frontier)
prefill -0.25%
within-arm drift 0.15-0.19%
Quality is unchanged. Teacher-forced over 18672 tokens of promessi_sposi.txt,
perplexity goes 6.289309 -> 6.263711 (avg_nll 1.838851 -> 1.834773); greedy
generations from both are coherent and track word for word until a late
paraphrase.
Scaling KDA's measured 497 GB/s by the byte reduction predicts +22%, not the
+13.4% measured. The gap is the finding: only about 62% of KDA's time was
weight streaming, and the rest is the conv1d, the gating and the recurrent
state update, which do not shrink with the weights. Turning that ratio into a
millisecond floor needs KDA's absolute per-token cost, and the 18.37 ms figure
the findings doc carries for it did not come from DS4_GLM_DECODE_ABLATE --
there is no kda bit in that mask, and the KDA path returns before the mask is
read. The doc now marks the row, and the ~7 ms derived from it, as unverified
pending a committed KDA substage timer.
Note this changes no engine code and no shipped shader; it produces a better
artifact rather than speeding up an existing one. speed-bench/
glm53_decode_findings.md records the full decode budget, the measurement
method, a stage-profiler label trap that misattributes KDA attention to
attn_output, and several untested constants noticed while reading.
Verified on the machine above:
make -C gguf-tools glm53-requant-kda exit 0, no warnings
make clean && make exit 0
make test exit 0
./ds4_test --all exit 0
./ds4_test --metal-kernels exit 0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012HzS5Rkfe1toogQbenv3Ga
Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
kernel_glm53_mul_mv_bf16_f32 and its fused qkv variant share one row helper,
and that helper carried every BF16 projection GLM 5.3 reads during decode:
blk.N.kda_{q,k,v,output} across 34 KDA layers, 9.13 GB per token, 39% of the
decode step and 54% of all bytes read.
Each lane loaded a single ushort, so one simdgroup-wide load moved 64 bytes --
the narrowest useful transaction on this part. The eight strided loads did
cover whole cache lines, so nothing was being refetched; the cost was the
instruction count. Reading ushort4 per lane moves 256 bytes per load and cuts
the weight loads by four, with four in flight before the first fma so
memory-level parallelism goes up rather than down (32 bytes per lane against
16).
The tiling is exact: lane L, step i, sub-load s covers [4L + 512i + 128s ..+3],
which over s=0..3 and all 32 lanes covers [512i, 512i+511] with no gap and no
overlap. That needs in_dim to be a multiple of 512 -- GLM 5.3 uses 4096 for
q/k/v and 8192 for the output projection -- and the scalar path stays for
anything else. Row bases are 32-byte aligned from the GGUF alignment and every
offset is a multiple of four, so the vector loads are aligned.
This is NOT bit-exact against the scalar path: repartitioning which lane
accumulates which k changes the partial sums, so it was verified on quality
rather than on identical output.
Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM 5.3
Flash Q4_K fully resident, arms interleaved with 2 sweeps each:
nsg=4 (the M3 Ultra default) decode +3.67% prefill -0.03%
nsg=8 (every other device) decode +4.03%
within-arm drift 0.26-0.35%
No device is penalised; the wider path is the better one on both.
That moves the KDA projections from 497 to about 547 GB/s, 67% -> 74% of the
736.9 GB/s sequential-read ceiling measured on this machine by
speed-bench/metal_bandwidth_probe. The remaining gap is not load width: the
activation row is 16 KiB and is re-read by each of the 8192 output rows, which
is the next thing to look at.
Quality, scored against the tracked 100 GLM 5.3 Flash continuations in
gguf-tools/quality-testing/data/glm53-flash-openrouter-zai-fp8-100 over 11559
target tokens:
avg_nll 0.300478 -> 0.300380 (-0.032%)
greedy lcp 948 -> 948 (identical)
first-token 90/100 -> 90/100 (identical)
The behavioural metrics do not move at all; per-case avg_nll deltas are within
-0.0063..+0.0013 and the aggregate is marginally better.
Verified on the machine above:
make exit 0
make test exit 0
./ds4_test --all exit 0 (32 suites)
./ds4_test --metal-kernels exit 0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012HzS5Rkfe1toogQbenv3Ga
Widening the loads to ushort4 helped because it moved more bytes per instruction. This is the other half of the same effect: issuing eight of those loads before the first fma rather than four, so 64 bytes per lane are outstanding instead of 32. That it still pays says the kernel had not saturated memory-level parallelism at four. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM 5.3 Flash Q4_K fully resident, arms interleaved with 2 sweeps each over 8 context frontiers: decode +1.44% (within-arm drift 0.11-0.43%) Cumulative with the ushort4 widening, against the scalar path this file shipped with: 20.74 -> 21.86 tok/s, +5.4%. The KDA projections now move about 575 GB/s, 78% of the 736.9 GB/s sequential-read ceiling measured on this machine by speed-bench/metal_bandwidth_probe, up from 497 GB/s (67%). This one is bit-identical rather than merely quality-neutral. Each lane enumerates the same elements in the same order under both tilings -- with four sub-loads a lane covers 4L + 512i + 128s for s=0..3, with eight it covers 4L + 1024i + 128s for s=0..7, and both walk 4L + 128m with m ascending into the same accumulator lane. Only the loop nesting changes. Confirmed by comparing a full next-token logit dump byte for byte: identical over 1959996 bytes. Added as a tier above the four-load path rather than replacing it. GLM 5.3's projections are 4096 and 8192 wide so they take the eight-load path, but any BF16 tensor whose in_dim is a multiple of 512 and not 1024 keeps the four-load win instead of dropping to the scalar tail. Verified on the machine above: make exit 0 make test exit 0 ./ds4_test --all exit 0 (32 suites) ./ds4_test --metal-kernels exit 0 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012HzS5Rkfe1toogQbenv3Ga
glm53_quantize.py's q4 artifact assigns q8_0 to three groups, not one: role="linear_attention" (the KDA projections), "embedding" and "output". The shipped GLM-5.3-Flash Q4_K artifact has all three at BF16, so the tool's scope was always those three rather than KDA alone. Renamed to match, with a --tensors selector that defaults to kda so existing behaviour is unchanged. The source file, the Makefile target and the findings-doc runbook are renamed together here, so every commit in the series builds on its own. output.weight is a full [4096 -> 154880] matvec on every decoded token, 1.27 GB at BF16. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, fully resident, arms interleaved with 2 sweeps each, marginal over a KDA-only requantized artifact: decode +1.80% prefill +0.00% (within-arm drift 0.16-0.26%) token_embd is deliberately not in the default: it is a single-row lookup per token, so quantizing it saves about 0.6 GiB resident but essentially no decode bandwidth. It is selectable for the memory saving, not claimed as a speedup. Quality, scored against the tracked 100 GLM 5.3 Flash continuations over 11559 target tokens, KDA+head against the unmodified artifact: avg_nll 0.300478 -> 0.299642 (-0.278%) greedy lcp 948 -> 993 first-token 90/100 -> 90/100 (identical) Indistinguishable from the KDA-only result (0.299680), so the head carries no measurable quality cost of its own -- worth checking separately because it feeds the logits directly rather than an interior projection. Verified on the machine above: make -C gguf-tools glm53-requant-bf16 exit 0, no warnings make clean && make exit 0 make test exit 0 ./ds4_test --all exit 0 (32 suites) ./ds4_test --metal-kernels exit 0 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012HzS5Rkfe1toogQbenv3Ga Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
The GLM decode budget in speed-bench/glm53_decode_findings.md attributed
18.37 ms/token, 38.9% of decode, to KDA attention and said the measurement
came from DS4_GLM_DECODE_ABLATE. It could not have. That mask had no kda
bit, and glm53_graph_kda_attention was dispatched above the line that reads
the mask, so no ablation arm in this tree could reach the stage. The largest
single row of the budget was an estimate presented as a measurement, and the
tuning priority for the whole KDA path was derived from it.
Adds DS4_GLM_ABLATE_KDA plus four substage bits -- kda_qkv, kda_gate,
kda_recur, kda_out -- matching the structure already in the function: the
q/k/v projections, the f_a/f_b/beta/g_a/g_b low-rank chain, the recurrence
kernel, and the output projection. The mask read moves above the KDA branch
so the whole stage can be skipped as well.
Stage-name matching becomes an exact token match over the comma list. A
substring test cannot express these names: strstr(env, "kda") also fires on
"kda_qkv", so every substage arm would have silently ablated the entire
stage. The seven existing names are unaffected -- they are exact tokens in
the documented comma list already.
Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal,
GLM-5.3-Flash-Q4_K fully resident, ctx 2048, 128 generated tokens, four
interleaved baselines at 22.375 tok/s with 0.85% spread:
KDA attention 15.99 ms 35.8% (budget claimed 18.37, 38.9%)
qkv projections 9.68 ms 21.7%
output projection 3.16 ms 7.1%
gate/beta chain 1.37 ms 3.1%
recurrence kernel 1.23 ms 2.8%
unattributed 0.55 ms 1.2%
Every other row of the budget reproduced within noise; only KDA did not.
This overturns the finding the budget drew from it. The doc reasoned that
+22% was predicted from bandwidth and +13.4% measured, therefore only ~62% of
KDA was weight streaming and the remaining ~7 ms/token was conv1d, gating and
the recurrent state update -- "the next thing to attack, and not a bandwidth
problem". The conv1d, gating and state update are 1.23 ms/token. KDA is
~90% weight streaming.
Re-ablating on the Q8_0 KDA artifact confirms it: qkv 9.68 -> 5.61 ms and the
output projection 3.16 -> 1.85 ms, against a pure-bandwidth prediction of 5.14
and 1.68, so both are ~90% bandwidth-scaled. Decode 22.375 -> 25.545 tok/s,
+14.2%, so the +13.4% headline itself reproduces. Only its explanation was
wrong: the gap was the projections not scaling perfectly plus fixed dispatch
cost, not recurrence work.
The practical consequence is that metal/glm53_kda.metal is capped at 2.8% of
decode however well it is optimised, and the qkv projections at 21.7% are the
KDA target that matters.
Verified on the machine above:
make exit 0, no warnings
./ds4_test --all exit 0 (15 suites)
./ds4_test --metal-kernels exit 0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
… again The BF16 case in tests/test_glm53_kda.c uses BF16_IN = 64. The row helper in metal/glm53_bf16.metal picks its path from in_dim -- a multiple of 1024 takes the eight-load branch, a multiple of 512 the four-load branch, everything else the scalar fallback -- so 64 has only ever exercised the fallback. Both wide branches shipped with no direct coverage. They are not unreachable code paths in practice: GLM 5.3 decode runs them at 4096 (kda_q/k/v) and 8192 (kda_output), and GLM vision calls the same helper at 1024, which is a multiple of 1024 and so takes the eight-load branch too. Adds check_bf16_matmul() and three cases at in_dim 512, 1024 and 4096, each checked in both the decode (1 row) and prefill (3 row) shapes. The wide paths repartition which lane accumulates which k and so are deliberately not bit-identical to the scalar path; the reference is accumulated in double and compared with a relative tolerance rather than for equality. Verified the coverage is real by mis-striding one of the eight sub-loads so the tiling overlaps: the in_dim=1024 case fails with got 0.0076086428 against expected 0.00427307095, far outside the 2.0e-05 tolerance. The target also had not linked at all. ds4_metal.o references ds4_deepseek4_attention_bounds, which is defined in ds4_image.o, and that object was not in the rule, so `make tests/test_glm53_kda` ended in "symbol(s) not found for architecture arm64". Nothing noticed because the target is not a prerequisite of `make test`. Adds ds4_image.o to both the Metal and CUDA rules and puts the Metal build into `make test`. The CUDA variant is left out of the default run: the same missing object is added to its rule, but no CUDA device was available here to confirm it links and passes, and `make test-glm53-kda` still builds it. Verified on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2: make exit 0, no warnings make test exit 0, GLM-5.3 KDA GPU tests: PASS ./ds4_test --all exit 0 (15 suites) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
exp(a_log[head]) is uniform across a KDA decode threadgroup -- head is tgpig.y -- but every one of the 128 channels recomputed it, which at 64 heads over 34 layers is about 278,000 redundant exponentials per token against 2,176 distinct values. Thread 0 now computes it into threadgroup memory alongside beta_shared, and the sd[] write moves after the first barrier so the value is available without adding one. sd is not read until after the second barrier, so nothing else has to move. The spare space was already allocated: the scratch is 656 floats and the layout used 653. Both decode barriers also drop mem_device. The conv-state writes before the first are each thread's own channel and no thread reads another's; the recurrent-state writes before the second are not re-read in this kernel, where only so[] crosses simdgroups. Neither needs device scope to be correct. This is not a speedup. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, eight interleaved pairs: before 22.311 tok/s (sd 0.028) after 22.325 tok/s (sd 0.043) +0.06%, Welch t = 0.76 -- no effect That is the outcome the corrected budget predicts. The recurrence kernel is 2.8% of the decode step, so even removing an eighth of it would be 0.24% of a token, and this removes ALU work from a kernel whose cost is memory traffic over the 136 MiB recurrent state. The earlier reading of this path -- a ~7 ms/token conv1d/gating/recurrence floor, "the next thing to attack" -- would have justified far more work here than the stage can repay. Kept because it deletes provably redundant work and narrows two barriers that never needed device scope, not because it is faster. It is not. A measurement note worth recording, because the first attempt at this A/B was wrong: ds4_gpu_full_source() reads metal/*.metal from disk at run time and there is no embedded fallback, so building two binaries around a shader edit does not compare two shaders -- both read whatever is on disk. The numbers above come from one binary with DS4_METAL_GLM53_KDA_SOURCE pointed at the old file, which is what those override variables are for. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) ./tests/test_glm53_kda PASS ./ds4_test --metal-kernels exit 0 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
…ation "norms, hyper-connections, residual, LM head" was not a measurement. It was whatever the other ablation arms left over, and at ~18% of decode it was the second largest line in the budget with nothing measured inside it. Adds DS4_GLM_ABLATE_HC and DS4_GLM_ABLATE_HEAD, covering the mHC producer chain at both of its per-layer sites and the output head at both of its encode sites. decode_ablate moves to the top of the layer body, since the mHC pre stage runs before the point where the mask was being read. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, 128 generated tokens, baseline 22.34 tok/s: mHC producer chain 3.99 ms/token 8.9% output head 1.80 ms/token 4.0% everything else in the row ~3.0 ms/token ~6.7% hc,head together measure 5.77 ms against 5.79 for the two separately, so the split is additive. This makes the mHC producer the largest unoptimised item in the decode step. glm53_graph_hc_pre issues four dispatches -- plain RMSNorm, the 16384->24 mix matvec, the split/mix, and the weighted RMSNorm -- twice per layer over 45 layers: 360 small dispatches per token for 3.99 ms of work. DeepSeek V4 already fuses the F16 equivalent in ds4_gpu_dsv4_hc_producer_pre_norm. The output head, by contrast, is nearly all matvec: 1.80 ms for a [4096 -> 154880] BF16 matvec is about what its 1.27 GB costs at this machine's bandwidth, so there is no dispatch overhead to chase there. Also records why GPU-side argmax is not worth doing. Reading back all 154,880 logits and scanning them on the CPU costs 0.0143 ms for the 605 KiB memcpy and 0.1668 ms for the scan, 0.1811 ms combined -- 0.40% of a 44.76 ms step, below the run-to-run spread, so the change could not be shown to work even if it were free. It would not remove a synchronisation either; the token is needed before the next step can start regardless. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
glm53_graph_hc_pre issued four dispatches per site -- plain RMSNorm over the 16384-wide flattened HC row, the 16384->24 mix matvec, the sinkhorn split/collapse, and the weighted RMSNorm -- and runs twice per layer over 45 layers. That is 360 small dispatches per token. Ablation prices the stage at 3.99 ms/token, 8.9% of the decode step, which makes it the largest unoptimised item in the budget. DeepSeek V4 already folds exactly this chain into one kernel, kernel_dsv4_hc_rms_norm_mix_f16_cluster2_pre_norm. GLM 5.3 could not use it for one reason: it stores hc_attn_fn/hc_ffn_fn as BF16 where DeepSeek stores F16. Everything else already matched -- same 16384/24/4096/4 shapes, and metal_graph_decode_hc_pre goes through the same sinkhorn split and weighted sum with the same DS4_N_HC_SINKHORN_ITER and DS4_HC_EPS. The two types are both 16 bits, so every size, stride and buffer binding is identical and only the widening differs. The kernel body becomes a template over the weight vector type with a ds4_hc_mix_widen() overload pair, and the two kernels are thin instantiations; the Objective-C entry point likewise becomes one internal function with f16 and bf16 wrappers. The F16 path is unchanged by construction. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, arms interleaved via DS4_METAL_DISABLE_GLM53_HC_PRODUCER_FUSE so the binary and the shaders are the same in both: ctx four dispatches fused delta 2048 22.282 tok/s 23.545 +5.67% (sd 0.026/0.031, n=6, t=75.4) 4096 21.94 23.195 +5.72% 16384 21.795 23.00 +5.53% Prefill is unchanged at every context; only the decode path is fused. 2.41 ms of the 3.99 ms is gone, and the remaining 1.58 ms is the fused kernel's own arithmetic. Output is bit-identical, not merely close: dumping all 154,880 logits with the fusion on and off gives max|delta| = 0 and the same argmax. The fused kernel reproduces the standalone kernels' reduction trees exactly, so this is purely a dispatch-count change. Gated the same way as the DeepSeek path -- BF16 mix weights, the exact 16384/24/4096/4 shapes, not the reference HC decode path, pre-M5 or M5 Apple silicon -- with DS4_METAL_DISABLE_GLM53_HC_PRODUCER_FUSE to turn it off. A 0 return falls back to the four dispatches. For scale: this one change is +5.67%, while the KDA recurrence kernel that the earlier budget called "the next thing to attack" is 2.8% of decode in total. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) ./ds4_test --metal-kernels exit 0 ./tests/test_glm53_kda PASS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
… not matter glm_graph_indexed_decode_split_block_rows_for() steps straight from 32 to 128 rows per block at 1024 selected rows. Nothing in the tree justified either constant or the threshold between them, and there was no way to try another value without editing the source. Adds DS4_GLM_DECODE_SPLIT_BLOCK_ROWS to force one value. A value the split path cannot honour is rejected by the existing availability guard and falls back, so the override cannot select a broken configuration. Swept on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident: rows ctx 2048 ctx 16384 default 23.50 23.00 32 23.51 22.97 64 23.51 23.02 96 -- 23.01 128 23.51 23.02 256 23.49 23.02 Flat to within 0.2% at both contexts, which is the run-to-run spread. The selection count is capped by glm53_graph_indexer_selected_limit(), which does not grow with context, so longer contexts do not make this interesting either. Kept as instrumentation for other GPUs rather than because it found anything. The default is left exactly as it was. The findings doc also now records the decode flush-cadence sweep, which is the same kind of negative: DS4_GLM_DECODE_FLUSH_INTERVAL from 3 to 12 is inside the noise at ctx 2048 and ctx 16384, and only 0 (never flush) and 32 are worse, so the existing default of 4 is already right. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
…ceiling With the KDA stage split and the residual row split, every large line in the budget now has both a measured time and an exact byte count read from the GGUF tensor table, so each can be compared against the 736.9 GB/s ceiling: kda q/k/v 6.845 GB 9.68 ms 707 GB/s 96% of ceiling kda_output 2.282 GB 3.16 ms 722 GB/s 98% gate/beta 0.232 GB 1.37 ms 169 GB/s 23% recurrence 0.285 GB 1.23 ms 232 GB/s 31% The KDA projections are finished. At 96% and 98% of the ceiling, specialising the BF16 matvec for the 4096 and 8192 shapes -- function constants to unroll the loops, two output rows per simdgroup, staging the activation row in threadgroup memory -- cannot pay for itself. The kernel already moves bytes about as fast as the machine will move them. This also corrects the 497 -> 547 GB/s recorded when the widened loads landed. That was derived from the 18.37 ms KDA row, which was never measured; against the measured 9.68 ms the q/k/v projections run at 707 GB/s. What remains is dispatch overhead rather than bandwidth. The mHC fusion prices a dispatch directly -- 270 removed for 2.41 ms, about 8.9 us each -- and the two stages far below the ceiling are exactly the ones made of many small launches. The gate/beta chain moves 232 MB, which is 0.33 ms at the rate the big projections achieve, and costs 1.37 ms; the other ~1.04 ms is 170 dispatches at ~6 us, agreeing with the mHC number. Records the concrete shape of the remaining KDA work, worth about 1.0 ms or 2.3%: f_a and g_a are both [4096 -> 128] off the same attn_norm input and pair the way ds4_gpu_glm53_matmul_bf16_qkv already pairs q/k/v; f_b and g_b are both [128 -> 8192] but read different activations and need a two-input kernel; both need a second low-rank buffer, because g->kda_lowrank is written by f_a, read by f_b, then overwritten by g_a. Also records why FP16 storage for the recurrent state is not worth pursuing: at 31% of ceiling the state is latency-bound rather than bandwidth-bound, so halving it would not halve the 1.23 ms, and the whole stage is 2.8% of decode against an accumulating-error risk over long contexts. No code change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
… doc rows
Nine findings from an adversarial review of the branch. Six needed code.
gguf: the scratch output could still truncate the input. Closing the direct
in/out inode aliasing left a second door open: the tool built its temporary as
<output>.requant.<pid>.tmp and opened it with fopen("wb"), so a symlink or hard
link planted at that predictable path redirected the truncating open straight
back at the mmapped source -- the original bug with an extra step. The scratch
file now comes from mkstemp, which picks an unpredictable name and opens
O_CREAT|O_EXCL, following no symlink and reusing no existing file; the opened
descriptor is then confirmed to be a regular file and not the input before a
byte is written, and fchmod restores the umask-derived mode mkstemp's 0600
would otherwise leave on a model file. Verified: a symlink planted at the
scratch path no longer redirects the open, and the input survives.
gguf: unchecked arithmetic on header-derived sizes. A metadata array computed
sz * n unchecked, and tensor byte counts, converted sizes and the output cursor
could all wrap into small, plausible-looking values that then passed the
range checks. Adds mul_or_die/add_or_die/pad_or_die and uses them throughout,
and rejects dimensions above INT64_MAX, which ds4q_row_size would otherwise
reinterpret as negative and silently size at zero. Three new fixtures confirm
each guard fires: a tensor whose element product fits but whose BF16 byte count
does not, a metadata array whose element count times element size wraps, and a
dimension past INT64_MAX. Conversion output is unchanged, byte for byte.
metal: the BF16 producer coupled two shader files. ds4_hc_mix_widen() called
glm53_bf16x4_to_f32x4 from glm53_bf16.metal, which the branch itself
introduced. Because the library is one concatenation, pointing
DS4_METAL_GLM53_BF16_SOURCE at any pre-branch revision then stopped
dsv4_hc.metal compiling -- breaking the very per-file override this document
recommends for shader A/B runs. dsv4_hc.metal now carries its own
ds4_hc_bf16x4_to_f32x4. Verified by running --metal-kernels with
DS4_METAL_GLM53_BF16_SOURCE pointed at the 110afdd file.
glm: the GLM fusion ignored the shared rollback switches. It shares a kernel
with the DeepSeek F16 producer but honoured only its own kill switch, so
DS4_METAL_DISABLE_PRE_M5_DECODE_PORTS and the two producer-specific variables
disabled the DeepSeek path and left this one live. It now goes through
metal_graph_ported_m5_decode_feature_enabled like its sibling. Verified: all
three switches drop decode from 23.67 to ~22.4 tok/s.
tests: no direct coverage of the templated producer. Adds an f16-vs-bf16
equivalence case over the real 16384/24/4096/4 shape. Mix weights are drawn
from values with at most seven explicit mantissa bits, so each is exact in both
half and bfloat16 and the two instantiations see bit-identical floats; with the
same body and reduction order the outputs must then match exactly, and the
comparison runs at tolerance 0 across the mix, collapse and pre-norm results.
Verified the case bites by permuting the bf16 lane order, which fails it.
metal: removes the blank line at EOF that git diff --check flagged.
Documentation. The decode-budget table still carried the discredited 18.37 ms
KDA row, and mixed shares from the old 21.19 tok/s baseline with rows measured
against 22.375, so the displayed shares summed past 100%. The table is now one
consistent set of measurements against one baseline, summing to 44.69 ms and
100%, with the superseded figures shown alongside rather than in place of them.
Bandwidth is no longer a column there: the routed-MoE and shared-expert byte
counts depend on which experts a token selects and were never re-derived, so
they are omitted instead of restated.
Two claims are pulled back to what the evidence supports. The "exact bytes" in
the pricing table are exact *weight* bytes and a lower bound on traffic -- for
the recurrence row they exclude conv state, q/k/v, gate inputs, conv weights,
biases and the output write, so its GB/s is an underestimate and is now written
as such. And the 8.9 us per dispatch inferred from the mHC fusion is
withdrawn: collapsing four dispatches into one also removed three intermediate
round-trips per site and improved occupancy, so 2.41 ms / 270 is not a launch
cost. The 1.0 ms available in the gate chain is now stated as an upper bound
on the prize rather than a forecast.
Also records that ablation arms are destructive and can in principle perturb
data-dependent routing, with the internal consistency checks that bound the
effect here; adds the cumulative engine-only A/B the series never contained --
110afdd versus the tip, each in its own tree so each reads its own shaders,
same GGUF and harness, 21.223 -> 23.593 tok/s, +11.17%; and corrects the stale
caveat claiming the head/embedding conversion was unmeasured, which 0a1d04a
did.
One finding not reproduced: the review reports make test failing with eight
assertions in logprob-vectors and local-golden-vectors. In this checkout make
test exits 0 with all fifteen suites OK, including both of those. The review
also found them unrelated to the branch's shader paths, so this looks like a
fixture or model difference between checkouts rather than a disagreement about
the code.
Verified on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2:
make exit 0, no warnings
make -C gguf-tools exit 0, no warnings
make test exit 0
./ds4_test --all exit 0 (15 suites)
./tests/test_glm53_kda PASS
git diff --check clean
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
…ion arms Every number in the decode budget so far came from destructive skip-ablation: the stage is removed, its output buffer keeps stale contents, and the run is timing-only. That is sound for counting dispatches that still execute, but a stale buffer can in principle change data-dependent routing downstream, and there was no second instrument to check any of it against. DS4_GLM_DECODE_REPEAT dispatches a named stage one extra time per site instead of removing it. Every stage it accepts is a pure function of its inputs, so the extra dispatch writes the same bytes and the whole-token delta is one extra execution of that stage -- with the model output unchanged. Verified: all six arms dump logits identical to the baseline at max|delta| = 0. Only idempotent stages get a bit. The KDA recurrence advances the conv and recurrent state and directional steering updates its input in place, so neither can be repeated this way; neither is offered. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, against the same build: stage ablate repeat agreement kda_qkv 9.69 9.76 0.7% head 1.80 1.76 2% kda_gate 1.37 1.45 6% kda_out 3.33 2.47 35% Three of four agree closely, which is the main result: the budget was not being distorted by the destructive arms. The kda_out disagreement is reproducible across rounds and the exact byte count settles it. Both instruments put kda_qkv at 9.69 ms for 6.845 GB, i.e. 706 GB/s. kda_output is 2.282 GB, which at that rate is 3.23 ms -- next to the ablation figure, not the repeat one. Repeat undercounts because the second dispatch re-reads a 67 MB per-layer weight set that is partly still resident, where kda_qkv's 201 MB per layer is not. So repeat is the right instrument for dispatch-bound stages and undercounts cache-friendly bandwidth-bound ones, while ablation is the reverse. Use both and let exact bytes arbitrate. The immediate use is pricing what is left. hc_expand is 0.55 ms/token, 1.3% of decode -- dispatch-bound, so the repeat figure is the reliable one -- which leaves about 2.5 ms in the residual row for the residual adds, steering, the remaining norms and the final HC collapse. With the producer fused, hc_pre now measures 1.36 ms against the 3.99 ms the four-dispatch chain cost. The ablation and repeat blocks move above glm53_graph_hc_pre so both are in scope at every site that needs them. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
The KDA gate chain ran five serial matvecs per layer -- f_a, f_b, beta, g_a, g_b -- which is 170 dispatches per token for 1.37-1.45 ms of work. It moves at least 232 MB, which is 0.33 ms at the rate the large projections achieve, so most of that time was not data. Two of those five pair naturally. f_a and g_a are both [4096 -> 128] off the same attn_norm row; f_b and g_b are both [128 -> 8192] but read the two low-rank vectors f_a and g_a produce. kernel_glm53_mul_mv_bf16_f32_pair is the qkv kernel's structure with two slots instead of three, selected by tgpig.z, and with the inputs as separate pointers so the same kernel serves both halves -- f_a/g_a pass the same row twice, f_b/g_b pass different ones. It reuses glm53_mul_mv_bf16_f32_row unchanged, which is why the result is bit-exact. g->kda_lowrank_g is new. The serial chain let f_a and g_a share one low-rank destination because f_b consumed the first before g_a overwrote it; running them concurrently needs two. Five dispatches become three: two paired plus beta, which has a different output width and stays on its own. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, arms interleaved via DS4_METAL_DISABLE_GLM53_KDA_GATE_PAIR: serial (5 dispatches) 23.562 tok/s (sd 0.037, n=6) paired (3 dispatches) 23.735 tok/s (sd 0.036, n=6) +0.74%, Welch t = 8.25, 42.442 -> 42.132 ms/token Output is bit-identical: all 154,880 logits match at max|delta| = 0. The prize was smaller than the 1.0 ms the pricing section allowed for, which is the point of having written that as an upper bound: 0.31 ms of the ~1.04 ms came back, not all of it. The more useful result is what the shape of this change licenses. Unlike the mHC fusion, pairing removes dispatches and nothing else -- the same buffers are written, the same weight bytes are read -- so the saving is launch overhead alone: 0.310 ms / 68 dispatches = 4.6 us per dispatch Applying that back to the mHC fusion splits its 2.41 ms into about 1.23 ms of launch overhead and 1.18 ms of intermediate traffic and occupancy, and confirms the 8.9 us per dispatch previously inferred from that fusion was roughly twice the real launch cost because it absorbed the traffic half. The doc is updated with both. Gated to M3 Ultra, matching ds4_gpu_glm53_matmul_bf16_qkv, which this shares a row helper with; every other device keeps the serial chain. A partial failure falls back safely, since both halves are pure functions of attn_norm and the serial path recomputes the same values into the same buffers. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) ./tests/test_glm53_kda PASS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
110afdd versus the tip, each built in its own tree so each reads its own metal/*.metal, same unchanged GGUF, same harness, interleaved: base 21.193 tok/s (47.18 ms/token) tip 23.747 tok/s (42.11 ms/token) +12.05% The base again reproduces the 21.19 tok/s the original budget recorded, so machine conditions have not drifted across the whole sequence of measurements in this document. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
Five constants shaping GLM 5.3 prefill were compile-time #defines with no way to try another value, and two of them interact: the chunk (2048) and the layer-flush threshold (2048) are the same number against a strict >, so raising the chunk also switches per-layer flushing on across every layer. Sweeping one at a time was impossible without separating them. Adds DS4_GLM_PREFILL_CHUNK_TOKENS, DS4_GLM_FULL_ATTN_LAYER_FLUSH_TOKENS, DS4_GLM_FULL_ATTN_CAP, DS4_GLM_FULL_ATTN_STREAMING_CAP and DS4_GLM_PREFILL_SCORE_SCRATCH_MB, each defaulting to the constant it replaces. Defaults are unchanged: logits with the knobs unset and set to the old values match at max|delta| = 0. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident. Chunk, with flushing pinned so it is one variable, prefill tok/s at ctx 16384: chunk flush off flush on default 1024 354.40 355.00 354.34 2048 394.30 394.57 394.17 4096 393.98 394.45 394.12 8192 394.05 394.43 393.97 Per-layer flushing does not matter at all -- every column agrees to 0.2% -- so the coupling this doc warned about is real in the code and immaterial in practice. And the default chunk of 2048 is already optimal: 1024 costs 10%, 4096 and 8192 buy nothing. Confirmed at ctx 32768 (388.68/388.31/388.50). Raising it is not free elsewhere either: context buffers at ctx 4096 grow 1.62 -> 3.04 -> 5.88 GiB across chunk 1024/2048/4096. The GLM 5.2 path using 4096 is not an argument for changing this one. The full-attention cap asymmetry turns out to be backwards. The streaming path gets 8192 and the resident path 4096, which read like the memory-constrained machine getting the larger window. Forcing each on the resident path, ctx 16384, interleaved, n=6: cap prefill decode 4096 394.23 (sd 0.03) 23.17 (sd 0.02) 8192 379.23 (sd 0.10) 23.15 (sd 0.02) The larger window costs 3.81% of prefill and nothing on decode, so 4096 is the fast choice rather than the cautious one and the resident default is right. Whether 8192 pays for itself on the streaming path by reducing re-streaming is untested; DS4_GLM_FULL_ATTN_STREAMING_CAP exists to try it. The 256 MiB score scratch is not dead code. Score columns are compact_cap / 4, so the budget starts clamping rows per dispatch above 131072 allocated context: score_rows goes 2048, 2048, 1024, 512 at ctx_alloc 65536, 131072, 262144, 524288, and raising the budget to 1024 MiB restores 2048 rows. The model context limit is 1048576, so this is reachable. It costs nothing measurable yet: holding the allocation at 524288 and varying only the budget, a 16384-token prefill gives 393.84 tok/s at score_rows=512 against 394.30 at 2048, which is 0.12% and inside the noise. A prefill long enough for scoring to dominate was not measured -- each run at ctx 65536 with that allocation exceeds ten minutes. No default changes, so no speedup is claimed. Three of the four open prefill questions in the doc are now answered negatively, which is worth as much as a win: nobody needs to look at them again. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
ds4_gpu_hc_expand_tensor ran as its own dispatch at 90 sites per token for 0.55 ms, and 34 of those sites follow the BF16 kda_output projection immediately. The simdgroup that finishes output row d already holds that row in lane 0, so it can write the four HC streams there instead of storing the row and having a second dispatch read it straight back. kernel_glm53_mul_mv_bf16_f32_hc_expand4 is that epilogue. This is the shape kernel_dsv4_q8_hc_expand4_q8_0 already uses for DeepSeek, in BF16. The row accumulation is split into glm53_mul_mv_bf16_f32_row_sum() and reused unchanged, and the expand arithmetic repeats kernel_dsv4_hc_expand4's operand order exactly, including that comb is indexed [j][h] rather than [h][j]. Only applied when nothing sits between the projection and the expand: directional steering would, so it is required to be inactive, which it is by default. DS4_METAL_DISABLE_GLM53_KDA_OUT_HC_EXPAND turns it off, and the remaining 56 sites keep the separate dispatch. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, interleaved: separate matvec + expand 23.772 tok/s (sd 0.039, n=6) fused epilogue 23.880 tok/s (sd 0.037, n=6) +0.46%, Welch t = 4.93, saves 0.191 ms across 34 sites That is 5.6 us per site against the 4.6 us measured launch cost, the difference being the 64 KiB write and read-back the fusion also removes. Correctness. tests/test_glm53_kda gains a direct case: the fused kernel against a separate ds4_gpu_glm53_matmul_bf16 followed by ds4_gpu_hc_expand_tensor, compared at tolerance 0 on both the projection output and all four HC streams. Verified the case bites by mutating the kernel three ways -- transposing comb, swapping a residual stream, and writing the HC streams with the wrong stride -- each of which fails it. End to end, greedy generations over four prompts at 128 tokens are byte-identical with the fusion on and off. A note on how this was verified, because the first attempt was not sufficient. ds4-bench --dump-frontier-logits-dir writes the logits at the end of prefill and never exercises the single-token decode graph. An earlier revision of this commit skipped the FFN-side mHC producer on every KDA layer -- the expand and that producer share an if block, and short-circuiting the expand took the producer with it -- which produces garbage after the first token, and it still gave frontier logits bit-identical to the baseline. A greedy generation caught it at once. Re-verified the two earlier fusions the same way, since their bit-exactness evidence had the same weakness: the mHC producer fusion and the KDA gate pairing both give byte-identical greedy decode output. Both were correct; only the evidence was thin. The findings doc now records the trap. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) ./ds4_test --metal-kernels exit 0 ./tests/test_glm53_kda PASS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
…ilogue 110afdd versus the tip, each in its own worktree, same GGUF, interleaved: base 21.180 tok/s tip 23.863 tok/s +12.67% Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
c1d10d0 folded the HC expansion into the kda_output projection for the 34 KDA layers. This does the other two sites, and neither needs a new kernel. DSA attention output. attn_output is Q8_0, and DeepSeek's ds4_gpu_matmul_q8_0_hc_expand_tensor is already exactly that shape -- Q8_0 matvec with the expand in its epilogue -- and binds post and comb from the split tensor at offsets n_hc and 2*n_hc, which is how GLM lays out hc_split. So this is a call-site change, not a kernel. FFN tail. The tail's last two dispatches are add(next, ffn_out, ffn_sum) and then an expand reading next, and kernel_dsv4_hc_expand4 already has a has_add path with ds4_gpu_hc_expand_add_tensor exposing it. glm_graph_encode_ffn_one_ normed_from takes defer_final_sum as an in/out flag: in, the caller would like the routed+shared sum left for the expand to do; out, whether that actually happened. It cannot always happen -- the leading dense layers have no routed/shared split and write next themselves -- so the flag comes back false there and the tail falls back to the plain expand. 43 of the 45 FFN sites defer. Both are gated on directional steering being inactive, since steering would have to run on the value in between, and each has its own kill switch: DS4_METAL_DISABLE_GLM53_ATTN_OUT_HC_EXPAND and DS4_METAL_DISABLE_GLM53_FFN_HC_EXPAND_ADD. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, interleaved: without both 23.870 tok/s (sd 0.025, n=8) with both 23.984 tok/s (sd 0.022, n=8) +0.48%, Welch t = 9.65, saves 0.199 ms across 54 sites That is 3.7 us per site, below the 4.6 us launch cost and below the 5.6 us the kda_output epilogue returned, which fits what each removes: a cheap elementwise add and a Q8_0 matvec here, against a BF16 matvec plus a 64 KiB round-trip there. Prefill is unchanged (399.73 against 399.84 at ctx 8192). Correctness is the decode-path check, since the frontier logit dump does not exercise this code: greedy generations over four prompts at 128 tokens are byte-identical with each fusion on and off, with both off, and against the output c1d10d0 produced. The dense-layer fallback is covered by that, because those layers would produce garbage if the flag came back wrong. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) ./tests/test_glm53_kda PASS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
…ilogues 110afdd versus the tip, each in its own worktree, same GGUF, interleaved: base 21.190 tok/s (47.19 ms/token) tip 23.977 tok/s (41.71 ms/token) +13.15% Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
Two of the three remaining items on the tuning list were measurement, and both change what is worth doing next. Re-measured the whole budget on the current tip, since every figure in the doc predated the mHC, gate-pairing and HC-expand work. Baseline 41.598 ms/token: KDA 15.39, routed 7.89, DSA core 7.86, shared 2.07, head 1.68, mHC 1.44 (down from 3.99), attn_output 1.10, q_path 0.56, indexer 0.19, residual 3.42. The residual was the largest unresolved bucket, so DS4_GLM_DECODE_REPEAT gains a router bit. Repeat rather than ablate is the only honest instrument for the router: skipping it leaves a stale expert selection, which changes which experts the routed stage streams and so changes the very cost being measured. Verified non-destructive -- greedy output identical. router (logits + top-k, 86 dispatches) 0.95 ms 28% of the residual remaining hc_expand (FFN tail, dense) 0.33 ms 10% still unattributed 2.13 ms 62% ffn_gate_inp is F32 at [4096, 288] over 43 layers, so the router streams 202.9 MB/token, which is 0.29 ms at the 707 GB/s the dense projections achieve. A third of the router is weight traffic and the rest is the top-k over 288 experts plus launch cost. It is the only 200 MB/token F32 tensor left in decode, though requantizing it is a model-artifact change and routing precision is the obvious risk. The shared expert correction matters more. The original budget recorded it at 0.55 GiB/token and 279 GB/s -- 38% of ceiling, far below every other kernel, and listed as an obvious target on that basis. The byte count was under by about 2x. Summed from the tensor table it reads three Q8_0 [4096, 2048] tensors per layer over 43 layers, 1.150 GB/token, which against the measured 2.07 ms is 556 GB/s, 75% of ceiling -- the same band as KDA overall at 77%. Its gate/up/SwiGLU is already fused through ds4_gpu_shared_mid_swiglu_q8_0_ tensor. Closing the remaining gap to 707 GB/s is worth about 0.45 ms, 1.1%, not the large win the 38% figure implied. The item stays on the list, but well below where it sat. No behaviour change: the router bit is instrumentation and defaults off. Verified on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
Every dense stage this branch has measured turned out to be at or near the memory ceiling: the KDA projections at 96-98%, the shared expert at 75% once its byte count was corrected, routed MoE around 80%. The DSA attention core is the exception, and it is 18.9% of the decode step. It is selection-capped rather than context-scaled -- indexer.top_k is 2048 with pool_size 4, so at most 2051 rows are ever attended -- and the measurement confirms it: 7.77, 7.78 and 7.68 ms at ctx 2048, 8192 and 16384. Per token it reads 2051 rows x 1152 B x 12 layers of compact KV (28.4 MB) plus the Q8_0 attn_v_b value projection (107.0 MB), 135.4 MB in total. Against 7.7 ms that is 17.6 GB/s, 2.4% of the 736.9 GB/s ceiling. The same traffic at the 707 GB/s the dense projections achieve would take 0.19 ms, so about 7.5 ms of the 7.7 is latency, occupancy and uncoalesced access rather than data movement. That makes it the largest remaining opportunity on this path by a wide margin, and unlike the projections it is not capped by physics. Records the shape of the work -- sorting the selected row ids so gathers gain locality, fusing the partial reduction with the value projection that is 107 of the 135 MB, and revisiting the per-row layout that strides the 512-wide lora part apart from the 64-wide rope part. None of it is measured; 7.5 ms is the budget those ideas compete for, not a promise. Also records why the shared-down fusion was dropped. ds4_gpu_shared_down_hc_expand_q8_0_tensor is exactly GLM's shared down-projection followed by the expand this branch already fused, worth about another 0.2 ms. But glm_graph_routed_moe_one_dispatch takes ffn_mid as scratch, and on the ordering where the shared expert runs first the routed dispatch clobbers it -- so deferring the down-projection past the routed stage, which the fused kernel requires since it needs routed_out, would read clobbered scratch. It needs a second mid buffer, and at 0.5% that did not justify the aliasing risk on top of the defer_final_sum plumbing already in this path. No code change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
…tually ran Three defects from review, one of them a regression this branch introduced. The FFN routed+shared deferral was enabled on every backend. 4852057 gated it on a Metal-named environment variable and nothing else, then called ds4_gpu_hc_expand_add_tensor unconditionally. That function is an explicit stub on ROCm -- it prints "tensor parallelism is Metal-only" and returns 0 -- so GLM-5.3 decode would fail at the first sparse FFN layer, layer 3. CUDA does implement it, but silently changing that backend's arithmetic from a change measured only on Metal is not something this should do either. The deferral is now inside #if defined(__APPLE__), matching the two attention-side epilogues, which were already guarded. The same deferral leaves g->next unwritten, and the "ffn_out" debug dump reads g->next a few lines later, so layer-bisect captures held stale data for every sparse layer while the optimization was active. The deferral now declines whenever a dump of that tensor is armed for this layer, which keeps debugging truthful at the cost of the fusion on runs nobody benchmarks. DS4_GLM_DECODE_REPEAT was pricing code that is not executing. The repeat blocks were written before the fusions landed and were never revisited: - kda_qkv re-dispatched three serial matvecs even when the fused QKV kernel had done the work. - kda_gate's repeat sat inside if (!gate_paired), so with pairing on -- the default on M3 Ultra -- the arm announced itself as active and added no work at all. - kda_out re-dispatched the bare projection rather than the projection-plus- HC-expand kernel that replaced it. Each arm now re-dispatches whichever variant actually succeeded. Re-measured against the same baseline, the corrected figures: stage fixed previously kda_qkv 9.67 9.76 kda_gate 1.17 1.45 kda_out 2.54 2.47 hc_pre 1.48 1.36 head 1.84 1.76 router 1.00 0.95 hc_expand 0.28 0.33 Only kda_gate moves materially: 1.17 ms is the paired chain, where 1.45 was the serial chain measured before pairing existed. No published figure was drawn from the broken arm -- the budget table uses the ablation arms -- but the arm was misleading at the tip and would have misled the next person. All seven repeat arms re-verified non-destructive: greedy output identical to the baseline for each. Verified on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) ./tests/test_glm53_kda PASS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
…counts
The DSA attention section priced the stage at 135.4 MB/token and 2.4% of the
memory ceiling, and concluded that ~7.5 ms of its 7.7 was not data movement.
Three separate errors produced that, each worth recording:
- n_rot is 0 for GLM 5.3, so a compact cache row is the 512-wide lora part
alone at 1024 B in f16. The section assumed a 64-wide rope tail and 1152 B.
- There are 11 DSA layers in the trunk, not 12. attn_v_b appears 12 times
because the MTP layer has one, and that layer is not in the decode path.
- The cache is not read once per layer. The generic kernel dispatches one
threadgroup per head -- 64 of them -- and each independently walks all
selected rows twice, once to score and once for the weighted sum.
Corrected: 2051 rows x 1024 B x 2 passes x 64 heads x 11 layers is 2.96 GB, and
attn_k_b plus attn_v_b add 0.20 GB, so 3.15 GB/token. At 7.7 ms that is 409
GB/s, 56% of ceiling. Real headroom, but not the collapse the old figure
implied, and the "7.5 ms of non-data work" budget it produced does not exist.
The corrected number points at the same structural fix for a better reason:
every one of the 64 heads reloads the same 2051 rows twice, and sharing each
loaded row across heads would take the cache term from 2.96 GB to about 46 MB.
It also invalidates an earlier result recorded here. The split-row sweep was a
no-op: glm_graph_indexed_decode_split_group8_available() requires DS4_N_ROT ==
64, which GLM 5.3 never satisfies, so every arm ran the same generic kernel and
the flat outcome was measuring nothing. The knob stays as instrumentation but
does not reach this model, and the doc now says so.
Layer counts corrected throughout. The trunk is 45 layers -- 34 KDA and 11 DSA
-- with 3 leading dense and 42 sparse FFN. Tensor counts of 43 and 12 include
the MTP layer. So: 87 of the 90 HC-expand sites are fused rather than all 90,
the router is 84 dispatches over 42 layers reading 198.3 MB rather than 86 over
43 reading 202.9 MB, and the shared expert reads 1.123 GB rather than 1.150,
which puts it at 542 GB/s and 74% of ceiling rather than 556 and 75%.
The model-artifact table was measured several commits back and disagreed with
the engine-only headline by 0.4 tok/s on the same artifact. Re-measured on the
current tip with the same harness: 23.99, 27.50 and 28.23 tok/s for the
original, KDA-Q8 and KDA+head-Q8 artifacts, so the first row now agrees with
the +13.15% engine-only figure by construction.
No code change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
DS4_GLM_ABLATE_QKLOW existed but the attn_core arm suppresses qk_low as well, so every measurement of the DSA stage has carried qk_low inside it and the kernel proper has never been timed on its own. Adds DS4_GLM_DECODE_REPEAT=qklow, which is idempotent -- qk_lowrank is a pure function of q and attn_k_b -- and verified non-destructive against the greedy baseline. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, ctx 2048: qk_low, repeat 0.56 / 0.58 ms qk_low, ablate 0.55 / 0.51 ms attn_core (incl.) 7.82 / 7.75 ms The two instruments agree at about 0.55 ms, so the indexed-attention kernel itself is 7.23 ms. That is the figure the no-rope grouped kernel is competing for, and it is what should be quoted rather than the 7.78 ms stage total. Verified: make exit 0 no warnings, make test exit 0, ./ds4_test --all exit 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
The review suggested a dedicated shared-expert mid buffer would be needed to make this possible. It is not, on the path that matters. glm_graph_encode_sparse_ffn_one has two orderings, chosen by shared_first = streaming_selected_cache. Only the SSD-streaming path runs the shared expert before the routed stage, which is the case where the routed dispatch clobbers ffn_mid. On the fully-resident path shared_first is false, so the routed stage has already finished by the time the shared expert runs: ffn_mid still holds the shared mid, ffn_out holds the routed result, and no extra buffer is required. That is exactly the input ds4_gpu_shared_down_hc_expand_q8_0_tensor wants. It does the shared down-projection, adds the routed output and expands into the four HC streams in one dispatch, replacing both the shared_down matvec and the caller's expand. The streaming path is excluded and keeps the separate dispatches. glm_graph_encode_ffn_one_normed_from gains a second out-parameter, hc_expand_done, so the tail can tell the difference between "the sum was deferred to me" and "the sum and the expand both already happened". Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, interleaved: separate shared_down + expand 23.970 tok/s (sd 0.021, n=6) fused 24.155 tok/s (sd 0.026, n=6) +0.77%, Welch t = 13.60, saves 0.320 ms over 42 sites 7.6 us per site, above the 4.6 us launch cost and above the 3.7 us the previous FFN-tail fusion returned, because this removes a Q8_0 matvec dispatch, the expand dispatch, and the ffn_sum round-trip between them. The estimate going in was 0.2 ms; it returned 0.32. The fused kernel carries its own DS4_GLM_DECODE_REPEAT=hc_expand arm. An earlier revision of this commit left the repeat in the tail, where it would have re-dispatched the standalone expand and priced a path that is no longer running -- the same defect 462f8ff fixed elsewhere. Correctness by the decode-path method: greedy generations over four prompts at 128 tokens are byte-identical with the fusion on, off, with the whole FFN tail unfused, and against the output of the previous commit. All five repeat arms re-verified non-destructive. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) ./tests/test_glm53_kda PASS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
110afdd versus the tip, each in its own worktree, same GGUF, interleaved: 21.153 -> 24.147 tok/s, +14.15%. The "why the shared-down fusion was not done" section is replaced: the aliasing that blocked it applies only to the SSD-streaming ordering, and on the resident path it fuses with no extra buffer for +0.77%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
baf529b paired f_a with g_a and left beta on its own dispatch, because the pair kernel assumes a single output width for both slots and beta is [4096 -> 64] where the other two are [4096 -> 128]. All three read the same attn_norm row, so the only thing standing between them was that assumption. kernel_glm53_mul_mv_bf16_f32_trio is the pair kernel with a third slot and a separate width for it. The grid is sized for the wider pair, so beta's upper threadgroups exit on the bounds check; that waste is half of one slot out of three, against a dispatch saved. glm53_mul_mv_bf16_f32_row_sum now takes in_dim as a scalar rather than the args struct, which is what lets a caller vary the output width per slot. Nothing about the accumulation changed. The gate chain is now two dispatches per KDA layer where it started at five: trio (f_a, g_a, beta) then pair (f_b, g_b). Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, interleaved: pair + separate beta 24.152 tok/s (sd 0.034, n=6) trio 24.223 tok/s (sd 0.038, n=6) +0.30%, Welch t = 3.44, saves 0.123 ms over 34 sites 3.6 us per site against the 4.6 us launch cost, which is about right for removing a dispatch that reads only 0.5 MB of weights. The estimate going in was 0.15 ms. Falls back to the pair plus a separate beta when beta is not BF16 or when DS4_METAL_DISABLE_GLM53_KDA_GATE_TRIO is set, and the repeat arm re-dispatches whichever of the two actually ran. Greedy decode over four prompts at 128 tokens is byte-identical with the trio on, off, and against the previous commit's output. Verified on the machine above: make exit 0, no warnings make test exit 0 ./ds4_test --all exit 0 (15 suites) ./tests/test_glm53_kda PASS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
The residual bucket has been chased stage by stage down to about 3.2 ms, and the returns from naming more stages are getting small. The more useful question is how much launch overhead is left in the step as a whole, which is answerable directly: almost every primitive here creates one compute encoder per dispatch, so counting encoders counts dispatches. DS4_METAL_ENCODER_COUNT prints the total at exit. Differencing two runs of different decode length cancels prefill and setup: 6,605 encoders over 8 decode tokens 26,989 over 40 (26989 - 6605) / 32 = 637 dispatches per decode token At the 4.6 us launch cost measured from the gate pairing, that is 2.93 ms/token, about 7% of the 41.31 ms step. That overhead is spread across every stage rather than sitting in the residual, and it is the floor all remaining dispatch-count work competes for -- no rearrangement of the current graph gets under it without removing launches. For scale, the fusions on this branch have already removed roughly 3.5 ms of dispatch and intermediate-traffic cost, so what remains is smaller than what has been found. The counter is an increment and a one-time atexit registration on the encoder path; decode speed is unchanged. Verified: make exit 0 no warnings, make test exit 0, ./ds4_test --all exit 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
110afdd versus the tip, each in its own worktree, same GGUF, interleaved: 21.157 -> 24.263 tok/s, +14.68%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
The previous commit recorded 21.157 -> 24.263 tok/s and +14.68%. Those were written from an expected value before the measurement returned; the run actually gave 21.127 -> 24.180 and +14.45%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KbehtsXadymPoN2RRHhPKV
The DSA attention core is 7.23 ms of the decode step, 56% of the memory ceiling, and the reason is structural: the generic kernel dispatches one threadgroup per head -- 64 of them -- and each independently walks all selected rows twice, once to score and once for the weighted sum. Every head reloads the same cache rows. The kernel that fixes this already exists and GLM 5.2 decode has been running it all along. kernel_glm_attention_indexed_decode_split_group8_partial puts 8 heads in a threadgroup so one loaded row serves eight of them, stages 16 rows in threadgroup memory so the two passes read device memory once, and blocks the rows so the work spreads over many more threadgroups. Two guards kept GLM 5.3 out of it, and neither was load-bearing: qk_rope != 64. GLM 5.3 has n_rot = 0. Everything rope in that kernel is driven by rope_vecs = qk_rope >> 2, so at 0 the staging loop runs no iterations, rope_shared is never dereferenced and the per-lane rope dot is skipped; the scratch sizing already drops the rope term at 0, and the freq_base/freq_scale validation next to the guard is already written as "qk_rope != 0 && ...". glm_graph_indexed_decode_split_blocks() <= 64. That is the worst-case buffer sizing -- 65 for GLM 5.3's 2051-row selection limit -- not the runtime block count, which is what the reduce kernel actually limits. The partial buffers are allocated from split_blocks() regardless, so the check that was meant is needed_blocks <= 64. The call site passed selected_rows_valid = true, selecting the kernel variant that skips the row < cache_cap test. It now passes false. GLM 5.2's selections are always in range. GLM 5.3's are not once more than the 4096-row full-attention window is visible (8192 under SSD streaming): below it decode selects the dense range 0..visible-1, above it the pool selector supplies 2051 rows padded with UINT32_MAX sentinels, and the unchecked variant reads those out of bounds. On this machine those reads returned values whose effect stayed below the greedy threshold -- a build with true produces greedy output byte-identical to this one over 128 tokens on prompts of 1,471, 3,841 and 10,352 tokens, the last of them on the sentinel-padded path -- but an out-of-bounds read is a bug whatever it returns, so the check stays. It costs 0.24% of decode. GLM 5.2 is unchanged by it: on an all-valid selection the two variants perform the same arithmetic in the same order, which tests/test_glm53_kda asserts bit for bit. The first version of this change measured the attention output 1.04% of range away from the generic kernel and attributed that first to online-softmax reordering and then to the missing row check. Neither reproduces: with the check skipped the output is byte-identical on every prompt tried, and reordering does not cost 1%. The same investigation recorded a stale binary confusing a later measurement. What the split kernel actually costs against the generic one is 3.06e-05 of range, deterministic but not bit-identical (lane-split scoring, online softmax across row blocks), so: - --quality selects the generic kernel, as it does for every other fast-versus-exact pair. DS4_METAL_DISABLE_GLM53_DSA_SPLIT selects it in default mode for A/B runs. - The two-host tensor-parallel head split keeps the generic kernel for GLM 5.3, since only the single-host configuration has been measured. GLM 5.2 under tensor parallelism ran the split kernel before and is unchanged. - tests/test_glm53_kda runs both kernels against a double-precision reference at 8, 513, 1024, 2048 and 2051 selected rows (the 1-, 17-, 32-, 16- and 33-block reductions and the fixed-count 16-block reduce), with rows at and past cache_cap and UINT32_MAX sentinels in the selection and the rows just past cache_cap filled with values that would dominate any softmax they leaked into. It checks the wrapper refuses a 65-block request and that the split output is repeatable. Observed deviations are about 1e-5 of the output scale for both kernels against a 1e-4 tolerance; passing true at the call site fails the first case by ten times the output scale. Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal, GLM-5.3-Flash-Q4_K fully resident, ctx 2048, interleaved: generic kernel 24.232 tok/s 41.27 ms/token split group8 28.318 tok/s 35.31 ms/token +16.86% Against the generic kernel, greedy generation is identical over 128 tokens on a 1,471-token prompt at ctx 4096 and a 3,841-token prompt at ctx 8192, and over 256 tokens on four prompts of about 2,900 tokens at ctx 8192, with no repetition in either arm; long-context teacher-forced NLL over 1,797 tokens is 1.833376 against the generic path's 1.833405. Greedy decoding will diverge eventually on some prompt, which is what --quality is for. score_official on the tracked fixtures cannot speak to any of this: its prompts are 24 tokens, so fewer than 512 rows are selected and the split path never engages. Cumulative, 110afdd versus this commit, each in its own worktree, same GGUF, interleaved: 21.160 -> 28.300 tok/s, +33.74% (47.26 -> 35.34 ms/token). Verified on the machine above: make exit 0, no warnings ./tests/test_glm53_kda PASS Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
89a318a and c843fcc widened the BF16 matvec weight loads for in_dim multiples of 512 and 1024. Both were verified on quality rather than on identical output, because repartitioning which lane accumulates which k changes the partial sums: the wide paths are deterministic but not bit-identical to the scalar path they displaced. --quality is documented as preferring exact kernels where faster approximate paths exist, and it did not know about this one. The args block that every BF16 matvec kernel already receives carries a `wide` flag now, and the shared row helper takes the scalar path when it is clear. The host clears it under --quality, and under DS4_METAL_DISABLE_GLM53_BF16_WIDE in default mode so the two paths can be A/B-compared without a rebuild. The scalar path is unchanged from before 89a318a, so quality mode runs the pre-branch arithmetic for these projections. tests/test_glm53_kda repeats the 512/1024/4096-wide checks with quality mode set, so the scalar path is covered at the widths that would otherwise take a wide branch. Verified on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2: make exit 0, no warnings ./tests/test_glm53_kda PASS Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
…fact table 3aa9c86 added an encoder counter and printed it as "compute encoders created". It increments on every call into the encoder routine, including the calls that hand back the batch encoder already open, so it counts acquisitions -- which is the dispatch proxy it was meant to be -- rather than encoder objects. The label and the comment now say so. The findings document also treated 637 dispatches x 4.6 us as a floor. The 4.6 us was measured on one fusion, and nothing shows it transfers to every kernel and command-buffer arrangement, so the 2.93 ms is an estimate of the launch overhead and is described as one. The artifact table in the cumulative section still carried the 23.99 tok/s engine row from d5b7895 under a sentence claiming it had been re-measured on the current tip, next to the 28.300 tok/s headline. It had not; the table is now dated to the commit it was taken at and its rows marked as not comparable with the headline. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
Greedy generations byte-compared between this tip and 110afdd, each built in its own worktree, on a 1,471-token prompt at ctx 4096 and a 3,841-token prompt at ctx 8192: --quality on the tip is byte-identical to --quality on the base, and the tip with DS4_METAL_DISABLE_GLM53_DSA_SPLIT and DS4_METAL_DISABLE_GLM53_BF16_WIDE set is byte-identical to the base's default. Encoder counts show which kernel ran in each arm. Under --ssd-streaming the split and generic arms agree over 32 tokens with each other and with the resident run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
89a318a and c843fcc had each lane load four adjacent bf16 weights instead of one, for about +5.4% of decode, and were verified on quality rather than on identical output: repartitioning which lane accumulates which k changes the partial sums, so the wide paths were deterministic but not bit-identical to the scalar path they displaced. This branch holds every change to reproducing the path it replaces, and there is no exact wide load: lane l must accumulate elements l, l+32, l+64, ... in order, a contiguous 8-byte load hands it elements 4l..4l+3, and redistributing those takes two cross-lane shuffles per element, which costs what the widening saved. The scalar accumulation -- the kernel main has -- is the only path again, and the `wide` argument added earlier on this branch to switch it under --quality goes with it. The fused qkv/pair/trio/HC-expand kernels share the row helper unchanged and stay exact. tests/test_glm53_kda keeps its 512/1024/4096-wide cases, which are the widths the model actually runs. With this and the exact DSA kernels, greedy generation from this branch is byte-identical to 110afdd in default mode on prompts of 1,471, 3,841 and 10,352 tokens. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
…shared
kernel_glm_attention_indexed_decode costs 7.2 ms of the 41 ms decode step, and
the cost is structural rather than arithmetic: one threadgroup per head, each
of the 64 walking every selected cache row twice, so every row is re-read 128
times and only 64 threadgroups exist to do it. The grouped/split kernel that
GLM 5.2 runs fixes the structure but not with the same arithmetic -- lane-split
dots and an online softmax across row blocks -- and this branch holds every
change to reproducing the path it replaces.
So the generic kernel's arithmetic is kept to the operation and reorganised
around it. kernel_glm_attention_indexed_decode_exact_* computes the same
thing in four phased dispatches:
scores one thread per (head, row) running the generic kernel's sequential
512-term dot, with 16 selected rows staged in threadgroup memory
per threadgroup for all 64 heads at once, so each cache row is read
from device memory once per token instead of 128 times
weights one 256-thread threadgroup per head, the generic kernel's, running
its per-thread row partition and its 128/64/../1 reduction tree for
the max and the denominator, turning scores into weights in place
lora one thread per (head, column pair) walking rows 0..n-1 in
selection order with the generic kernel's acc += w * kv chain, over
8 heads x 64 columns per threadgroup so a row slice is loaded once
for eight heads; rows are consumed in stages of 32, every thread
fetching 16 bytes and one weight three stages ahead into
double-buffered threadgroup memory. A row past cache_cap
contributes fma(0, kv[0], acc), which leaves acc unchanged bit for
bit where the generic kernel skips it
value the generic kernel's quantised row dot from threadgroup memory,
one thread per output element, over 256 threadgroups
Every floating-point operation, operand and ordering is the generic kernel's,
and the output is bit-identical to it. That is asserted, not assumed:
tests/test_glm53_kda runs both on one fixture at 8, 513, 1024, 2048, 2051 and
4096 selected rows, with rows at and past cache_cap and UINT32_MAX sentinels
in the selection, and requires memcmp equality. Greedy generation from this
tree is byte-identical to 110afdd over 128 tokens on a 1,471-token prompt at
ctx 4096 (dense window), a 3,841-token prompt at ctx 8192 (dense window,
3,841 rows) and a 10,352-token prompt at ctx 16384 (pool selector, 2051 rows
with sentinels), default mode on both sides.
The first version of the lora phase read row ids and weights from device
memory per row and was slower than the generic kernel it replaced, two
serialised loads per row instead of one; staging and prefetching, not
arithmetic, made it cheap. Per phase at about 1,500 selected rows, measured
by dropping the dispatch: scores 0.33, weights 0.18, lora 0.64 (2.60 before
pipelining), value 0.39 ms/token.
The split availability guard goes back to n_rot == 64, GLM 5.2 only, and GLM
5.3 never takes that kernel. --quality keeps the exact kernels, since they
are exact; DS4_METAL_DISABLE_GLM53_DSA_EXACT selects the generic kernel for
A/B runs; the two-host tensor-parallel head split keeps the generic kernel
until that configuration has been run. Scratch is three small buffers per
graph: scores for n_head x max(ctx_cap, selection limit), lora, denominators.
Measured on Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, Metal,
GLM-5.3-Flash-Q4_K fully resident, ds4-bench on promessi_sposi.txt, 128
greedy tokens per frontier, main / branch / branch / main, this commit on top
of the BF16 revert:
frontier main prefill branch prefill main decode branch decode
2048 429.98 / 429.52 429.71 / 429.89 21.09 / 21.12 27.82 / 27.84 +31.86%
4096 390.14 / 390.08 389.93 / 390.03 20.75 / 20.76 27.12 / 27.13 +30.69%
8192 392.23 / 392.08 392.01 / 392.00 20.71 / 20.69 26.99 / 27.04 +30.51%
16384 389.49 / 389.54 389.33 / 389.45 20.65 / 20.58 26.88 / 26.98 +30.63%
Prefill is within 0.04% of main at every frontier. Against the non-exact
split kernel this gives back all but about 1.5% at the short prompt (28.25
against 28.67 tok/s from the CLI), three extra dispatches per DSA layer being
most of the difference.
Verified on the machine above:
make exit 0, no warnings
./tests/test_glm53_kda PASS
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
… benchmark The DSA section now tells the whole story: the split kernel measured first and why GLM 5.3 does not ship on it, what the row check actually guards (out-of-bounds sentinel reads above the 4096-row window, whose effect stayed below the greedy threshold here), the four phased kernels that keep the generic arithmetic, their per-phase costs, and the evidence that they are bit-identical to the generic kernel and to 110afdd end to end. The cumulative section carries the main / branch / branch / main ds4-bench run at ctx 2048 to 16384, and the section on non-exact changes now says none remain on the default path and why the BF16 widening was dropped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
The split DSA kernel's call site had been switched to selected_rows_valid = false for every GLM model. For GLM 5.3 Flash that guarded sentinel rows; this branch no longer runs the split kernel there at all. For the models that do run it -- GLM 5.2 and the full GLM 5.3 (glm-dsa), whose selections are a dense range or a top-k over visible rows and always in range -- the checked variant only costs. On Flash it was 0.24% of decode, with DSA attention in 11 of 45 layers; on the full GLM 5.3 the split kernel runs in 76 of 79 layers and the same per-call cost is 2% of the step, which the main / branch / branch / main ds4-bench run on GLM-5.3-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K found at every frontier: frontier main decode branch decode 2048 16.32 / 16.26 15.96 / 15.95 -2.06% 4096 16.24 / 16.22 15.91 / 15.90 -2.00% 8192 16.02 / 16.03 15.68 / 15.70 -2.09% 16384 15.64 / 15.62 15.32 / 15.33 -1.95% The call site now passes !g->glm53: the unchecked variant those models always ran, bit for bit (tests/test_glm53_kda asserts the two variants agree on all-valid selections), and the checked one only for a GLM 5.3 Flash graph, should it ever reach that call. Re-measured the same way: frontier main decode branch decode 2048 16.42 / 16.33 16.32 / 16.30 -0.40% 4096 16.32 / 16.21 16.20 / 16.19 -0.43% 8192 16.01 / 16.01 16.01 / 15.93 -0.25% 16384 15.66 / 15.66 15.64 / 15.62 -0.19% which is inside main's own run-to-run spread; prefill is within 0.08% either way. Greedy generation stays byte-identical to main on 1,471- and 3,841-token prompts before and after. The split availability guard also caches its environment lookup instead of calling getenv per layer. The same protocol on DeepSeek V4 Flash (MXFP4): byte-identical to main, prefill and decode within 0.25% at every frontier, every suite OK. On the full GLM 5.3 the five suites that fail (DeepSeek official-vector fixtures and a 30k-token recall test) fail identically on main, with the same 55 assertions and the same golden-vector statistics. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
… aggregate Every decode optimisation on this branch already had its own DS4_METAL_DISABLE_GLM53_* switch, read inline with getenv at each call site. antirez#954 lays its pre-M5 work out the same way and adds an aggregate that turns the whole set off, so one variable is an A/B against the pre-branch paths. This does the same: the eight switches are a table read once each through glm53_flash_feature_enabled(), and DS4_METAL_DISABLE_GLM53_FLASH_TUNING disables all of them. The table is the list of what this branch changes on the decode path. With the aggregate set, greedy generation on a 1,471-token prompt is byte-identical to main and runs at 22.28 tok/s against main's 22.21, with 146,046 encoder acquisitions over the run against 86,610 on the default path: the unfused dispatch structure is back, not just the speed. The default path is unchanged (28.30 tok/s, byte-identical to main). Two of antirez#954's pieces were checked for GLM 5.3 Flash and neither applies: the greedy chain's ceiling here is the 0.1 ms per token the GPU idles at the token boundary (DS4_METAL_GPU_BUSY_PROFILE: 35.2 ms busy per 35.3 ms token), and GLM's indexed prefill already skips the indexer query projection while a chunk fits the dense window. The findings document records both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
The branch was rebased onto b0a147a, 24 upstream commits of Metal tensor-parallel and DSpark work. Only the Makefile's test list and the mHC producer kernel conflicted; upstream had refactored that kernel into a shared body, which the branch now templates on the mix-weight type so the f16, the bf16 and upstream's fused expand4 kernels are three instantiations of it. Greedy generation is byte-identical to the synced main on the 1,471-, 3,841- and 10,352-token prompts, and the synced main's output matches 110afdd's on all three. DeepSeek V4 Flash (MXFP4) and the full GLM 5.3 (IQ2_XXS) are byte-identical to it too. ds4-bench main / branch / branch / main against b0a147a: decode +32.10% at ctx 2048 and +30.45..30.70% at 4096 to 16384, prefill within 0.17%. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
Author
|
@ivanfioravanti can you take a look at this, and maybe test on spark to check I haven't broken cuda? |
On a one-line chat prompt ("Write a short story about a lighthouse keeper.",
36 tokens) the phased kernels decode at 28.93 tok/s against 29.04 with the
generic kernel: the generic kernel's row traffic is a few megabytes per layer
there, and the three extra dispatches per DSA layer cost more than they
save. The crossover is between 36 and 134 selected rows -- measured from the
CLI at ctx 4096, exact against generic on the same prompt: -0.4% at 36 rows,
+0.3% at 134, +1.1% at 207, +2.2% at 308, +4.3% at 603, +7.5% at 992, +10.7%
at 1,500 -- so the exact path now engages from 128 rows, as the split kernel
engaged from 512. Both kernels are exact, so a generation crossing the
threshold changes nothing but speed: the lighthouse transcript keeps its md5
(b27ccba0d468d445c882694d5428c6e4, identical to main's) at 28.97 tok/s, and
the 1,471-token prompt stays byte-identical to main at 28.24 tok/s with the
exact path engaged throughout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdqebBLGJ81wT7ybao3Tav
Author
|
checked the GLM-5.3-Flash-Q2.gguf results
Method — tok/s, each column the mean of its two interleaved runs:
|
Author
Quantized the
|
| frontier | main prefill | branch prefill | prefill | main decode | branch decode | decode |
|---|---|---|---|---|---|---|
| 2048 | 429.1 | 429.4 | +0.1% | 24.41 | 33.44 | +37.0% |
| 4096 | 389.6 | 390.0 | +0.1% | 23.97 | 32.48 | +35.5% |
| 8192 | 391.8 | 391.6 | -0.0% | 23.91 | 32.31 | +35.1% |
| 16384 | 388.8 | 388.9 | +0.0% | 23.74 | 32.06 | +35.0% |
Author
|
@GiorgioOppo can you test? |
Author
ds4 interleaved benchmark:
|
| prompt | promessi_sposi.txt (1,329,139 bytes) |
| frontiers | 2048 to 262144, step x2 |
| decode | 128 greedy tokens per frontier |
| SSD streaming | off |
| warmup | none |
Arm A command:
./ds4-bench -m /Users/jw/ds4/gguf/GLM-5.3-Flash-Q4_K.gguf \
--prompt-file /Users/jw/ds4/speed-bench/promessi_sposi.txt --ctx-start 2048 --ctx-max 262144 --step-mul 2 --gen-tokens 128
Arm B is the same command, run in the branch worktree.
Each arm is built and run in its own detached worktree, because the Metal
shaders are loaded from the metal/ directory of the tree the binary runs
in, so a binary built at one commit but run in another tree would measure
the wrong shaders. Nothing is checked out while a run is in flight.
Machine
| chip | Apple M3 Ultra |
| model identifier | Mac15,14 |
| CPU | 32 cores (24P + 8E) |
| GPU | 80 cores |
| memory | 512 GiB |
| macOS | 26.5.2 (25F84) |
| compiler | Apple clang version 21.0.0 (clang-2100.1.1.101) |
Interleave design
Order ABBABAAB — 8 runs (4 x main, 4 x branch).
| drift cancelled | mean position | |
|---|---|---|
| linear (steady warm-up or slow-down) | yes | 4.500 vs 4.500 |
| quadratic (thermal saturation) | yes | 25.50 vs 25.50 |
Both linear and quadratic drift cancel exactly.
Throughput
| frontier | main prefill | branch prefill | prefill | main decode | branch decode | decode |
|---|---|---|---|---|---|---|
| 2048 | 427.66 / 426.76 / 427.24 / 426.86 | 425.83 / 426.51 / 429.24 / 429.96 | +0.18% | 21.12 / 21.13 / 21.08 / 21.05 | 27.78 / 27.76 / 27.75 / 27.76 | +31.61% |
| 4096 | 390.36 / 389.86 / 390.19 / 390.13 | 390.35 / 390.17 / 390.25 / 390.47 | +0.04% | 20.74 / 20.81 / 20.73 / 20.70 | 27.05 / 27.06 / 27.04 / 27.06 | +30.40% |
| 8192 | 392.27 / 391.94 / 391.98 / 392.02 | 392.20 / 392.01 / 392.07 / 392.41 | +0.03% | 20.69 / 20.70 / 20.66 / 20.64 | 26.94 / 26.96 / 26.95 / 26.95 | +30.37% |
| 16384 | 389.55 / 389.16 / 389.36 / 389.21 | 389.12 / 388.90 / 389.34 / 389.58 | -0.02% | 20.55 / 20.61 / 20.57 / 20.58 | 26.79 / 26.76 / 26.80 / 26.79 | +30.17% |
| 32768 | 384.17 / 383.64 / 383.77 / 383.57 | 383.59 / 383.13 / 383.92 / 383.94 | -0.04% | 20.45 / 20.43 / 20.45 / 20.43 | 26.57 / 26.57 / 26.62 / 26.59 | +30.08% |
| 65536 | 373.33 / 373.48 / 372.95 / 372.97 | 372.88 / 372.81 / 373.15 / 372.68 | -0.08% | 20.21 / 20.20 / 20.21 / 20.20 | 26.13 / 26.16 / 26.16 / 26.17 | +29.45% |
| 131072 | 353.02 / 352.88 / 353.05 / 352.84 | 353.13 / 352.69 / 352.70 / 353.06 | -0.01% | 19.74 / 19.75 / 19.73 / 19.73 | 25.39 / 25.38 / 25.39 / 25.38 | +28.61% |
| 262144 | 317.18 / 317.01 / 316.86 / 317.13 | 316.86 / 316.92 / 316.89 / 317.10 | -0.03% | 18.87 / 18.87 / 18.87 / 18.84 | 23.98 / 24.06 / 24.00 / 23.98 | +27.26% |
Each cell lists that arm's runs in the order they ran; the delta compares the arms' means. Decode is gen_tps over the full generation at each frontier.
Time to first token (gen_first_ms, lower is better)
| frontier | main | branch | delta |
|---|---|---|---|
| 2048 | 47.95 / 48.23 / 48.10 / 48.21 | 38.38 / 38.38 / 38.49 / 38.06 | -20.35% |
| 4096 | 48.47 / 48.50 / 48.71 / 48.52 | 38.48 / 38.50 / 38.52 / 38.50 | -20.70% |
| 8192 | 48.72 / 48.87 / 48.66 / 48.90 | 38.40 / 38.59 / 38.59 / 38.53 | -21.03% |
| 16384 | 48.92 / 48.96 / 49.34 / 49.14 | 38.81 / 39.52 / 38.79 / 38.66 | -20.67% |
| 32768 | 49.44 / 50.20 / 50.66 / 50.50 | 39.69 / 38.85 / 39.04 / 39.18 | -21.93% |
| 65536 | 49.50 / 49.44 / 52.87 / 49.62 | 39.74 / 39.58 / 40.21 / 39.46 | -21.07% |
| 131072 | 50.50 / 50.36 / 51.13 / 50.46 | 40.42 / 40.52 / 40.56 / 40.40 | -20.03% |
| 262144 | 52.26 / 52.18 / 52.24 / 52.23 | 42.25 / 42.18 / 42.35 / 41.96 | -19.23% |
Repeatability
Worst spread within a single arm's own runs: 0.97% (branch prefill_tps at ctx 2048). A delta is only meaningful well above this.
| frontier | main prefill | branch prefill | main decode | branch decode |
|---|---|---|---|---|
| 2048 | 0.21% | 0.97% | 0.38% | 0.11% |
| 4096 | 0.13% | 0.08% | 0.53% | 0.07% |
| 8192 | 0.08% | 0.10% | 0.29% | 0.07% |
| 16384 | 0.10% | 0.17% | 0.29% | 0.15% |
| 32768 | 0.16% | 0.21% | 0.10% | 0.19% |
| 65536 | 0.14% | 0.13% | 0.05% | 0.15% |
| 131072 | 0.06% | 0.12% | 0.10% | 0.04% |
| 262144 | 0.10% | 0.08% | 0.16% | 0.33% |
Raw data — run timeline, engine load, per-run CSV
Run order
| # | run | arm | started | ended | exit |
|---|---|---|---|---|---|
| 1 | a_1 |
main | 18:25:25 | 18:47:58 | 0 |
| 2 | b_1 |
branch | 18:47:58 | 19:10:21 | 0 |
| 3 | b_2 |
branch | 19:10:21 | 19:32:44 | 0 |
| 4 | a_2 |
main | 19:32:44 | 19:55:18 | 0 |
| 5 | b_3 |
branch | 19:55:18 | 20:17:41 | 0 |
| 6 | a_3 |
main | 20:17:41 | 20:40:15 | 0 |
| 7 | a_4 |
main | 20:40:15 | 21:02:49 | 0 |
| 8 | b_4 |
branch | 21:02:49 | 21:25:11 | 0 |
The first run of a session pays the model load from disk, so its wall time is not comparable to the others. Throughput is measured inside the run and is unaffected.
Engine load (from a_1)
ds4: Metal device Apple M3 Ultra, 512.00 GiB RAM
ds4: memory: KV 2.92 GiB (raw 0.00 + compressed 2.92) + buffers 3.16 GiB + resident model 177.76 GiB = 183.85 GiB planned
ds4: GLM session ctx=262273 (model max=1048576); full-attention prefill/work cap=4096; compact indexed decode is used beyond the cap
ds4: GLM compact indexed prefill chunk=2048 score_rows=1023 score_scratch=255.88 MiB
Per-run CSV
a_1
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,427.66,128,21.12,47.953,127,21.13,202327452
4096,2048,390.36,128,20.74,48.465,127,20.76,251356572
8192,4096,392.27,128,20.69,48.716,127,20.70,349414812
16384,8192,389.55,128,20.55,48.921,127,20.57,545531292
32768,16384,384.17,128,20.45,49.441,127,20.47,937764252
65536,32768,373.33,128,20.21,49.500,127,20.22,0
131072,65536,353.02,128,19.74,50.496,127,19.75,0
262144,131072,317.18,128,18.87,52.262,127,18.88,0
a_2
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,426.76,128,21.13,48.226,127,21.15,202327452
4096,2048,389.86,128,20.81,48.502,127,20.83,251356572
8192,4096,391.94,128,20.70,48.866,127,20.72,349414812
16384,8192,389.16,128,20.61,48.963,127,20.62,545531292
32768,16384,383.64,128,20.43,50.198,127,20.45,937764252
65536,32768,373.48,128,20.20,49.442,127,20.21,0
131072,65536,352.88,128,19.75,50.364,127,19.76,0
262144,131072,317.01,128,18.87,52.177,127,18.88,0
a_3
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,427.24,128,21.08,48.105,127,21.10,202327452
4096,2048,390.19,128,20.73,48.714,127,20.75,251356572
8192,4096,391.98,128,20.66,48.662,127,20.68,349414812
16384,8192,389.36,128,20.57,49.341,127,20.59,545531292
32768,16384,383.77,128,20.45,50.664,127,20.47,937764252
65536,32768,372.95,128,20.21,52.872,127,20.23,0
131072,65536,353.05,128,19.73,51.128,127,19.75,0
262144,131072,316.86,128,18.87,52.244,127,18.88,0
a_4
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,426.86,128,21.05,48.214,127,21.06,202327452
4096,2048,390.13,128,20.70,48.522,127,20.71,251356572
8192,4096,392.02,128,20.64,48.902,127,20.65,349414812
16384,8192,389.21,128,20.58,49.144,127,20.60,545531292
32768,16384,383.57,128,20.43,50.502,127,20.45,937764252
65536,32768,372.97,128,20.20,49.616,127,20.21,0
131072,65536,352.84,128,19.73,50.460,127,19.74,0
262144,131072,317.13,128,18.84,52.230,127,18.85,0
b_1
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,425.83,128,27.78,38.385,127,27.82,202327452
4096,2048,390.35,128,27.05,38.480,127,27.08,251356572
8192,4096,392.20,128,26.94,38.404,127,26.97,349414812
16384,8192,389.12,128,26.79,38.810,127,26.82,545531292
32768,16384,383.59,128,26.57,39.692,127,26.60,937764252
65536,32768,372.88,128,26.13,39.736,127,26.15,0
131072,65536,353.13,128,25.39,40.423,127,25.41,0
262144,131072,316.86,128,23.98,42.251,127,24.00,0
b_2
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,426.51,128,27.76,38.377,127,27.80,202327452
4096,2048,390.17,128,27.06,38.496,127,27.09,251356572
8192,4096,392.01,128,26.96,38.586,127,26.99,349414812
16384,8192,388.90,128,26.76,39.522,127,26.80,545531292
32768,16384,383.13,128,26.57,38.849,127,26.60,937764252
65536,32768,372.81,128,26.16,39.583,127,26.19,0
131072,65536,352.69,128,25.38,40.518,127,25.41,0
262144,131072,316.92,128,24.06,42.179,127,24.08,0
b_3
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,429.24,128,27.75,38.494,127,27.79,202327452
4096,2048,390.25,128,27.04,38.524,127,27.07,251356572
8192,4096,392.07,128,26.95,38.588,127,26.99,349414812
16384,8192,389.34,128,26.80,38.788,127,26.83,545531292
32768,16384,383.92,128,26.62,39.042,127,26.65,937764252
65536,32768,373.15,128,26.16,40.211,127,26.19,0
131072,65536,352.70,128,25.39,40.558,127,25.41,0
262144,131072,316.89,128,24.00,42.352,127,24.02,0
b_4
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,429.96,128,27.76,38.065,127,27.80,202327452
4096,2048,390.47,128,27.06,38.496,127,27.09,251356572
8192,4096,392.41,128,26.95,38.534,127,26.98,349414812
16384,8192,389.58,128,26.79,38.659,127,26.82,545531292
32768,16384,383.94,128,26.59,39.183,127,26.62,937764252
65536,32768,372.68,128,26.17,39.458,127,26.19,0
131072,65536,353.06,128,25.38,40.401,127,25.40,0
262144,131072,317.10,128,23.98,41.959,127,24.00,0
Artifacts on disk: /Users/jw/.cache/ds4-bench/runs/ds4-bench-glm53flash-metal-exact-GLM-5.3-Flash-Q4_K-ABBABAAB-20260903-182525
If you want I can try SSD streaming with a short prompt, my Mac wouldn't be able to sustain a long benchmark |
|
Reproduced on an M5 Max, 40 GPU cores, 128 GB, Metal — the Apple numbers on this PR so far are all M3 Ultra, so this is a different point on the hardware curve. Measured PR head Four independent runs:
One difference worth flagging: prefill is not flat here. It is a small but consistent cost — slower at 0 of 32 frontier-pairs faster, median −1.9%. Your Q2 numbers show prefill within 0.1% across four frontiers. Different chip, and possibly different enough not to matter, but it reproduced in all four of our runs so it is not noise on our side. On the spread: three of the four runs fall inside 1.2 pp; run 2 sits 3.5 pp above the rest with no cause we could identify. We ran a fourth deliberately from the coldest measurable state (fans stopped, die 48.4 °C) to test whether a cold start explained it — it did not, returning +17.7%. So we are quoting the median with the spread shown rather than a point estimate. An earlier draft of this comment said +16.5%, which was one run's answer to a question we had asked once. Full method, per-frontier tables for all four runs, per-rep breakdowns and the raw CSVs: Thanks for the Q2 numbers upthread — they are what made this worth measuring here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
+32.1% engine-only decode at ctx 2048 for GLM 5.3 Flash on Metal, prefill
flat, every change bit-exact against the path it replaces. Forty-two commits on
top of b0a147a.
Measured main / branch / branch / main with
ds4-benchonpromessi_sposi.txt, 128 greedy tokens per frontier, each side built in itsown worktree so each reads its own
metal/*.metal, same unchanged GGUF.Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2,
GLM-5.3-Flash-Q4_Kfully resident. Both runs of each arm are shown; deltas compare the means.
Bit-exact. Greedy generation from this tip is byte-identical to main's over
128 tokens on 1,471-, 3,841- and 10,352-token prompts (ctx 4096 / 8192 /
16384), default mode on both sides, no switches — verified against 110afdd and
again against b0a147a after the rebase. Two non-exact optimisations tried along
the way (widened BF16 loads, the grouped/split DSA kernel) are not on the
default path.
Other models are untouched. Under the same protocol, DeepSeek V4 Flash
(MXFP4) and the full GLM 5.3 (
glm-dsa, IQ2_XXS, which takes the GLM 5.2path) are byte-identical to main and within 0.5% of it on prefill and decode;
their suites match main's (the full GLM 5.3 fails the same five DeepSeek-fixture
and long-recall suites on both trees).
What changed
DSA attention runs the generic kernel's arithmetic, staged and shared —
the largest gain, +27% over main on the short prompt, bit-identical to the
generic kernel. That kernel is slow structurally: one threadgroup per head, 64
heads each re-reading every selected cache row twice.
kernel_glm_attention_indexed_decode_exact_*keeps every floating-pointoperation, operand and ordering and reorganises the work into four phased
dispatches: scores with 16 rows staged per threadgroup for all 64 heads, the
generic 256-thread partition and reduction tree for the softmax denominator, a
pipelined weighted sum over 8 heads × 64 columns per threadgroup with rows
prefetched three stages ahead, and the generic quantised value dot over 256
threadgroups.
tests/test_glm53_kdarequiresmemcmpequality with thegeneric kernel at 8 to 4096 selected rows with out-of-range and
UINT32_MAXrows in the selection. The non-exact split kernel measured +16.86% first; this
recovers all but 1.5% of it (28.25 vs 28.67 tok/s on the short prompt), exactly.
It engages from 128 selected rows; below that the generic kernel's row traffic
is a few megabytes per layer and the extra dispatches cost more than they save
(-0.4% at 36 rows, +2.2% at 308, +10.7% at 1,500).
Fused mHC producer with BF16 mix weights — +5.67% (t=75.4), bit-exact.
glm53_graph_hc_preissued four dispatches per site, twice per layer, 45layers. DeepSeek V4 already fused this; GLM could not use it only because it
stores the mix weights BF16 where DeepSeek stores F16, so the kernel body is
now a template over the weight type (three instantiations after the rebase:
f16, bf16 and upstream's fused expand4).
KDA gate chain, five dispatches to two — +0.74% for pairing
f_a/g_aand
f_b/g_bin a two-slot version of the qkv kernel, +0.30% for foldingbetainto the first pair with a three-slot kernel. Bit-exact.HC expansion folded into its producer at all 90 sites — +0.46% for
kda_output (new BF16 epilogue kernel), +0.48% for the other two combined. The
simdgroup that finishes an output row expands it into the four HC streams in
its epilogue instead of a separate dispatch reading it straight back.
Shared down-projection fused with the HC expand — +0.77%, bit-exact. On the
resident path the routed stage has already run when the shared expert does, so
DeepSeek's existing kernel does the down-projection, the routed add and the
expand in one dispatch.
Widened BF16 matvec loads — tried and dropped. ~+5.4%, but the lanes
accumulate different k, so it is not bit-identical to the scalar path, and an
exact wide variant needs two cross-lane shuffles per element, which costs what
the widening saved. The scalar row helper is the only path.
Requantizer for BF16 tensors (
gguf-tools/glm53-requant-bf16, a modelartifact, not engine work): KDA→Q8_0 measured +13.4% and the head a further
+1.8% at an earlier tip. Refuses an output aliasing the mmapped input, builds
via
mkstemp, checks every header-derived size for overflow.Instrumentation, knobs, tests.
DS4_GLM_DECODE_ABLATEgainedkda(plusfour substages),
hcandhead;DS4_GLM_DECODE_REPEATis a non-destructivecounterpart that dispatches an idempotent stage twice. Five prefill constants
are settable, defaults unchanged.
tests/test_glm53_kdahad never linked andwas not in
make test; fixed, with BF16 matvec cases at the model's widths, anF16-vs-BF16 equivalence case for the templated producer, and the DSA check
above covering the generic, split and exact kernels. The uniform KDA decay
hoist is kept as a cleanup (+0.06%, noise).
Tried and rejected
Left unacted: the SSD-streaming full-attention cap of 8192 is 3.81% slower than
the resident path's 4096 on prefill;
DS4_GLM_FULL_ATTN_STREAMING_CAPexists tosettle whether it pays for itself by reducing re-streaming.
Rollback switches
Every optimisation is individually disableable, following #954's layout, and
DS4_METAL_DISABLE_GLM53_FLASH_TUNINGturns all of them off at once:DS4_METAL_DISABLE_GLM53_HC_PRODUCER_FUSE,..._KDA_GATE_PAIR,..._KDA_GATE_TRIO,..._KDA_OUT_HC_EXPAND,..._ATTN_OUT_HC_EXPAND,..._FFN_HC_EXPAND_ADD,..._SHARED_DOWN_HC_EXPAND,..._DSA_EXACT. Withthe aggregate set, output is byte-identical to main at 22.28 tok/s against
main's 22.21 on the same prompt, with the unfused dispatch structure back.
Diagnostics:
DS4_GLM_DECODE_ABLATE,DS4_GLM_DECODE_REPEAT,DS4_METAL_ENCODER_COUNT.Notes
make,make -C gguf-tools,make test(DeepSeek fixtures),./ds4_test --metal-kernelsand--metal-tensor-equivalenceon GLM 5.3 Flash pass; builds are warning-free;git diff --checkclean.DS4_METAL_ENCODER_COUNT). At the 4.6 µs measured on one fusion that is ~2.9 ms, ~7% of the step — an estimate of launch overhead, not a measured floor.selected_rows_valid = !g->glm53: GLM 5.2 and the full GLM 5.3 keep the unchecked variant they always ran (their selections are always in range; the checked one cost them 2%), and only a GLM 5.3 Flash graph, whose pool selector pads withUINT32_MAXsentinels, would take the checked one — and it no longer reaches that kernel at all.--dump-frontier-logits-dirdumps prefill logits and never touches the decode graph; decode changes are verified by byte-comparing greedy generations.metal/*.metalat run time, so two binaries built around a shader edit compare nothing; use separate worktrees or the per-fileDS4_METAL_*_SOURCEoverrides.Still open
Per-stage budgets, the corrected decode analysis and the measurements behind
every number above:
speed-bench/glm53_decode_findings.md.