fix(attention): fallback for GPUs without flash MMA kernels (sm70)

[Th-Underscore]

Summary

Volta/Turing GPUs (700 <= cc < 800) crash on TTS prefills that select the CUDA MMA flash-attention kernel, which has no usable device code there ("flash_attn_ext_f16 has no device code compatible with CUDA arch 700", followed by MUL_MAT failures). This PR auto-resolves flash vs eager attention from CUDA compute capability and falls back to the exact repeat-KV + matmul/softmax graph:

• New engine::core::attention_fallback unit: preference parsing (per-model <family>.attention session option + AUDIOCPP_ATTENTION env), CC gating via the CUDA driver. ggml_backend_supports_op() cannot be used for detection: on Volta it returns true for shapes that later crash at launch.
• Auto fallback + session options wired into higgs_audio_tts and breeze_tts (backbone, depth, encoder, decoder); process-wide AUDIOCPP_ATTENTION=eager backstop in the shared SDPA/GQA/QwenDecoder modules; --log trace lines for the resolved path.
• Fix for a latent QwenDecoder prefix-concat dtype assert exposed by the eager path (cached prefix KV now cast on every path, not just flash).
• Unit test (attention_fallback_test), breeze_tts model-spec entry, docs.

The change is additive: unknown backends fail open to flash, constructor defaults keep allow_flash_attention=true, and the flash graph is untouched, so families that do not opt in keep byte-identical behavior.

Validation

Build:

scripts/build_linux.sh --backend cuda --with-tests --cuda-arch "70"

Backend: CUDA, Tesla V100-SXM2-16GB (sm70, underclocked to 765 MHz — wall times understate full-clock performance). Models: higgs-audio-v3-tts-4b-q8_0.gguf, breeze-tts-2-q8_0.gguf, fish-audio-s2-pro-q8_0.gguf. Voice ref: VCTK p225 15 s concat preset.

Case	Result
Higgs --task tts, auto, short text	allow_flash=0, 3.0 s audio in 2.8 s wall, no CUDA errors
Higgs forced attention=flash, short text	works (small shapes take the TILE path)
Higgs auto, 1100-char tagged text, chunk=160	max_tokens before EOC — reproduced on flash too; bracket tags (no Higgs parser) derail AR, unrelated to this PR
Higgs auto, same text tags stripped, chunk=160	53.9 s audio in 33.0 s wall
BreezeTTS2 --task clon, auto	all four gates 0, 3.4 s audio in 7.9 s wall
Fish S2 Pro regression (no session option, flash preserved)	2.0 s audio in 3.4 s wall
Higgs --backend cpu	fail-open (allow_flash=1), generates
Server smoke test (/v1/audio/speech, Higgs)	HTTP 200, 176 KB wav, zero flash_attn/MUL_MAT errors
ctest	39/39 pass, including new attention_fallback_test

Checked families/routes: higgs_audio_tts (CLI + server), breeze_tts (CLI streaming + offline), fish_audio (CLI regression), CPU backend. All other families keep default behavior (fail-open).

Eager vs flash is the same op with ulp-level logit differences (AR trajectories can diverge, e.g. 2.32 s vs 2.56 s output for one short sentence); eager is ~1.5x slower on short samples.

Known limitations

• Auto CC-gating is wired only into Higgs and BreezeTTS2; Fish/qwen3-tts still hardcode flash (Fish works on V100 via TILE-shaped graphs; qwen3 untested there). The header documents the one-line adoption recipe for other families.
• On sm70, auto selects eager even for small decode shapes where TILE would work — conservative but safe.
• Validated on V100 CUDA + CPU only.

[0xShug0]

@Th-Underscore Thanks! I was just about to rebase it myself.

[0xShug0]

@Th-Underscore The Qwen eager path is essentially dead in the current code and is not reached by any model’s default path, so that change is safe. Interestingly, I could not reproduce the issue locally on RTX 5090, SM 120, and CUDA runtime 13.2. Higgs TTS works correctly with eager. Would you like to share the exact log?

The concern is PR is broad. It exposes a global user-facing knob for what is basically a backend compatibility workaround for old CUDA devices. On Ampere/Ada/Blackwell, users may never need it. AUDIOCPP_ATTENTION is also too generic and misleading. And AUDIOCPP_ATTENTION=eager is a process-wide shared-attention override so every model loaded in that server process that uses the shared SDPA/GQA/QwenDecoder flash paths can be affected.

Another option is to remove the auto fallback too and make this fully explicit (with clear doc). For example, users on affected old CUDA GPUs can set higgs_audio_tts.cuda_flash_attention=off or breeze_tts.cuda_flash_attention=off. That avoids changing defaults for any backend/GPU and keeps the workaround clearly scoped to the model session that needs it.

Another issue: The documented prefixed option breeze_tts.attention=eager is rejected with the released GGUF: unknown BreezeTTS session option: breeze_tts.attention. The runtime should read runtime::find_option(options.options, {"breeze_tts.attention"}) .