Add sm70 FlashAttention config case for DKQ=256, DV=256, ncols=64

[mistrjirka]

Overview

Added SM70 FlashAttention config improving performance around 20% based on the model. Previously the config for 256x256 head fell back onto the Ampere config which was suboptimal for SM70.

Upstream fallback:
255 registers/thread
552 B stack

Edited FA config:
255 registers/thread
56 B stack

My goal was to optimize the Qwen 3.8 27B performance in agentic workflows and it uses the DKQ=256, DV=256, ncols=64 which configuration was suboptimal.

To simulate agentic turn I used 100k cached KV tokens + 1024 PP and 512 TG.

Simulated Agentic Turn

repeats: 4

-d 100000 -pg 1024,512 

Model	PP upstream	PP edited	PP Δ	TG upstream	TG edited	TG Δ
Qwen3.8 27B	276.1	352.9	+27.84%	16.27	16.37	+0.60%
Ornith 1.5 35B	485.5	580.4	+19.54%	59.26	59.39	+0.21%
Gemma 4 12B	483.4	486.7	+0.67%	39.45	39.46	+0.03%

Prompt Processing Scaling

repeats: 10
repeats for 100k PP: 5

Model	KV depth	Upstream	Edited	Δ
Qwen3.8	100k	275.18	349.18	+26.89%
Qwen3.8	65k	356.23	434.18	+21.88%
Ornith + MMQ	65k	760.65	895.97	+17.79%
Qwen3.8	16k	609.02	657.61	+7.98%
Ornith + MMQ	16k	1171.40	1238.42	+5.72%
Qwen3.8	4k	736.72	754.63	+2.43%

MMQ flag means adding GGML_CUDA_FORCE_MMQ=ON during compilation

Model	pp512	pp2048	pp4096
Ornith 1.5	865 → 891 (+2.94%)	840 → 872 (+3.77%)	821 → 837 (+2.00%)
Gemma 4	1896 → 1918 (+1.15%)	1831 → 1872 (+2.22%)	1767 → 1810 (+2.43%)
Qwen3.8	897 → 899 (+0.26%)	880 → 884 (+0.46%)	856 → 867 (+1.29%)

Additional information

It would be good to test it on other SM70 gpus then just purely v100 tested here.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: yes, GPT-5.6 Sol. I was trying to optimize performance on Qwen 3.8 27B. GPT 5.6 Sol was used during the performance exploration.

[ggml-gh-bot]

Hi @mistrjirka, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[mistrjirka]

Hi @mistrjirka, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

* **Multiple open PRs from a new contributor**: We limit new contributors (those without a previously merged PR) to **1 open PR at a time**. You currently have 2 open PRs.

* **AI-generated content**: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

The AI generated content is disclosed and I had written this PR description by hand, so what exactly is the issue?

[amgomez]

Second V100 data point: 2x Tesla V100-SXM2, Qwen3.8-27B Q8, prefill +11.6% at 70k and +17.2% at 120k

Your note asks for tests on other sm_70 GPUs. This is still V100, but a different machine
and a different measurement path (llama-server instead of llama-bench), so it may still be
useful. The patch is the only difference between the two arms.

Setup

• GPUs: 2x Tesla V100-SXM2 32 GB (sm_70), driver 535.247, layer split across both cards
• Host: DGX-1, Ubuntu 20.04. The server runs in a container under Slurm
• Build: CUDA 12.6.3, gcc-10, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DBUILD_SHARED_LIBS=OFF
• Arms: master a32af33de (b10715) and the same commit with this PR cherry-picked
  (b10717). Same image recipe, same model file, same server flags
• Model: Qwen3.8-27B-UD-Q8_K_XL (arch qwen35, 24 heads / 4 KV heads,
  attention.key_length = attention.value_length = 256)
• Server flags: --ctx-size 262144 -sm tensor -ngl 99 -ub 2048 -cms 2048 -np 1 --spec-type draft-mtp --spec-draft-n-max 3 --temp 1.0 --top-p 0.95 --top-k 20

Method

Requests go to /completion with cache_prompt: false and a unique prefix per request, so
no prompt cache can hit. The size of each point is calibrated against the server's own
/tokenize, so the depths below are measured token counts and not nominal ones. The first
request of each point is discarded, then three are measured and the median is reported. One
decode probe per point reuses the cache left behind by the prefill.

Prefill

depth (tokens)	master b10715	+ this PR	delta
71,713	955.9 t/s	1067.1 t/s	+11.6%
122,869	761.7 t/s	892.7 t/s	+17.2%

The three repetitions of each cell agree within 0.2% (957 / 956 / 956 against
1065 / 1067 / 1067), so the difference is well outside the run to run spread, and it grows
with depth. In wall clock time, a cold 122,869 token prompt goes from 161 s to 138 s.

Decode is unchanged

50.8 -> 49.5 t/s at the shallow point and 40.7 -> 40.3 t/s at the deep one. Both arms
generated the same number of tokens (141 and 160) with the same MTP draft acceptance
(91/147 and 99/180), so the outputs match and only the timing differs. With one sample per
cell that difference is noise, and it agrees with the +0.6% you measured on TG.

This is expected from the dispatch rather than from the table. On Volta fattn.cu selects
the tile kernel while Q->ne[1] * gqa_ratio_eff <= 16, and only above that does it use the
MMA kernel that reads these configs. This model has gqa_ratio 6, so gqa_ratio_eff is 2
and a single decoded token gives 2.

Caveats

The gain here is smaller than the +22% at 65k and +27% at 100k in the PR description, and I
did not isolate why. The quantization, the number of cards, -sm tensor, -ub 2048 and
measuring through the server rather than llama-bench all differ. Sign and shape agree, the
magnitude does not.

The model that motivated this on our side is Qwen3.8-Flash-Next (qwen4exp), which declares
the same 256/256 head sizes, so it should benefit as well. We have not measured that one
yet.

Happy to re-run on this machine if another configuration would help.