cuda: reduce GB10 Q8 attention-output prefill overhead

[JordiPosthumus]

Summary

Optimize the existing Q8 attention-output prefill path on single-GPU GB10:

• Quantize eight independent 32-value activation blocks per thread block,
  preserving the original maximum-reduction tree and rounding.
• Let the exact grouped INT8 MMA kernel read the Q8 layout already prepared
  at startup, avoiding repeated unaligned weight loads.
• Pad the shared scale rows by one element to separate same-column bank
  accesses, using 192 additional bytes of shared memory per thread block.

Why

On the tested IQ2XXS/Q2 model, these two stages accounted for roughly 4% and
23% of prefill GPU time. The activation kernel used one warp per block with
block-wide barriers. The grouped MMA kernel read the interleaved 34-byte Q8
format even when the same codes and scales were already available separately
in an aligned artifact. Its power-of-two shared scale-row stride also maps
different rows to the same banks. The padded path changes that stride, not
the scale values or accumulation order. This bank-conflict explanation follows
the address layout; hardware counters were unavailable (ERR_NVGPUCTRPERM).

The patch changes scheduling and addressing, not model math: Q8 codes, scales,
rounding, INT8 MMA operations and floating-point accumulation order are kept.
It neither allocates nor repacks weights at request time. Missing artifacts
retain the raw path. Decode, multi-GPU, non-GB10 and fewer-than-eight-token
calls retain their previous dispatch. The optional FP16 cuBLAS path is unchanged.

DS4_CUDA_NO_Q8_0_QUANT_WARPS=1, DS4_CUDA_NO_Q8_MMA_ALIGNED=1, and
DS4_CUDA_NO_Q8_MMA_SCALE_PADDING=1 allow independent rollback. The padded
specialization retains the compiled-kernel capability check. No model format,
cache, context or sampling changes.

Validation

Current head 7cd29f0f319b2b42ebdbf75993616bba4d63246b is one commit directly
on f4d03f6c. Runtime and fixtures are unchanged by the rebase; Makefile cleanup
was reconciled with upstream.

On 2026-09-05 a clean standalone GB10 tree passed
make -j2 ds4-server ds4_test tests/test_cuda_q8_prefill CUDA_ARCH=sm_121,
./ds4_test --server, all 28 shapes/eight combinations, memcheck and synccheck
(zero errors). No engine-source adapter was required. The exact head also
passed M3 make -B -j2 all ds4_test, ./ds4_test --server and
make -B -j2 cpu (CPU compile/link only); SDK 27 emits 27 existing Metal
deprecation warnings.

The following unchanged-runtime timing/model evidence predates the rebase:
base b0a147a7, candidate 67d0c9f. GB10, 128 GB unified memory, CUDA 13,
native sm_121a; DeepSeek-V4-Flash-Vision-Exp
IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8, 2048-token chunks, 262144-token allocation.

Separate upstream-baseline/candidate ds4-bench binaries (only the CUDA object
differs), 128 teacher-forced decode tokens per frontier:

Frontier	Before prefill	After prefill	Gain	Before / after decode
8192	691.47 t/s	765.13 t/s	+10.65%	14.49 / 14.41 t/s
32768	675.29 t/s	747.91 t/s	+10.75%	13.72 / 13.66 t/s

The warm decode rows differ by less than 0.6%; this run does not resolve such
a small change, and no decode improvement is claimed. The first 2048-token row was
warmup-dominated and is excluded from the speed claim. All three full
129280-element frontier vectors were byte-identical.

To isolate the incremental padding benefit, balanced eight-arm ABBA/BAAB runs
on the production integration retained the first two optimizations in both
arms, with explicit warmups and two resident sessions:

Measured suffix	Padding off	Padding on	Difference
32768 -> 36864	711.19 t/s	729.02 t/s	+2.51%
131045 -> 135141	416.58 t/s	422.60 t/s	+1.45%

All full-vocabulary logits matched byte-for-byte at the suffix frontier and
16 following teacher-forced decode steps in every arm, including the unaligned
131045 frontier: 30,768,640 non-self float comparisons across both runs.
These incremental timings predate the final host-side capability guard and
comment cleanup; the separate upstream-binary table above uses the final
source. These are measured exact comparisons, not a proof for every future
compiler, GPU or input. Neither table predicts a decode speedup.

The final production integration, including the separate #978 bounds fix,
also reproduced its retained baseline's 8,403,200 float logits byte-for-byte
at 32768 -> 36864 and 64 teacher-forced decode steps, with the vision encoder
loaded and both resident sessions allocated.
Post-restart text/image cold-to-warm checks also passed on two GB10 servers,
which resumed ordinary workloads. This is not a long-context soak.

Model-free tests exercise the public API across 28 shapes and all eight
optimization combinations: partial blocks/warps/tiles, short batches, grouped
strides, maximum 8192 width, unaligned-width fallback, and present/absent
artifacts. Outputs and guards are exact; final API memcheck and synccheck both
report zero errors. Separate numeric kernel tests include zero, subnormal,
nonfinite and rounding boundaries; a 184-shape padding fixture also passed
both sanitizers. Default Mac build,
make cpu, and ./ds4_test --server passed without loading a Mac model.
The full model-backed aggregate suite was not run.

Commands and build prerequisites

make cuda-spark
make tests/test_cuda_q8_prefill CUDA_ARCH=sm_121
./tests/test_cuda_q8_prefill
compute-sanitizer --tool memcheck --error-exitcode 99 --report-api-errors no \
  ./tests/test_cuda_q8_prefill
compute-sanitizer --tool synccheck --error-exitcode 99 --report-api-errors no \
  ./tests/test_cuda_q8_prefill

# Same options for the original and candidate binaries; MODEL is the GGUF above.
DS4_CUDA_Q8_F16_CACHE_RESERVE_MB=4096 \
DS4_CUDA_NO_CUBLAS_ATTENTION_OUTPUT_A=1 DS4_CUDA_DECODE_GRAPHS=0 \
DS4_BENCH_FORCE_SNAPSHOT=1 ./ds4-bench -m "$MODEL" \
  --backend cuda --warm-weights --prefill-chunk 2048 --ctx-alloc 262144 \
  --prompt-file speed-bench/promessi_sposi.txt \
  --ctx-start 2048 --ctx-max 32768 --step-mul 4 --gen-tokens 128 \
  --teacher-forced-decode --csv /tmp/q8-speed.csv \
  --dump-frontier-logits-dir /tmp/q8-logits

The benchmark environment is test-only: it isolates the existing Q8 path,
not the optional FP16 cuBLAS path, and preserves snapshots between frontiers.
--report-api-errors no suppresses handled host-registration fallback notices,
not memory errors. The new CUDA test is model-free and skips non-GB10 devices.

The earlier b0a147a7 CUDA engine needed an unrelated non-Apple TP guard fix;
that evaluation applied the prerequisite identically to both arms. The refreshed
upstream base includes the fix. The existing CUDA long-context smoke target
also omits ds4_image.o; after supplying that link dependency,
make cuda-regression CUDA_ARCH=sm_121 passed. Neither prerequisite is hidden
in this patch. Its new focused test target links the needed image object.

One new test fixture initially requested an artifact below the existing 2 MiB
admission floor; its dimensions were corrected, without relaxing assertions.
The sanitizer was rerun using its absolute toolkit path after a noninteractive
SSH PATH failure. Neither was a kernel failure. Non-GB10 hardware and the full
model-backed aggregate suite were not tested.

Related work

This uses the existing startup artifact format. It complements #766's GB10
decode-side use of those weights, but does not depend on #766. A separately
reproduced raw-loader end-of-allocation overread is fixed independently in #978;
the numerical baseline fixture here explicitly retains raw-load padding.
The padding amendment stays in this PR because it affects the same aligned
prefill kernel; it is not a second competing PR. Closed #621 has padding in
a different WMMA/indexer path, not this Q8 scale layout.