Breeze-TTS 2 performance: weight packing + fused bf16 rounding - #393
Conversation
|
My bad. During testing, I set cfg=4, which was too large and caused quality degradation. Both BF16 and Q4 exhibited the issue, so it's not entirely a Q4 problem. This means the performance benefit from Q4 is still worth considering. This is an autoregressive model and is bandwidth‑sensitive. Text: CFG = 4, HIP, seed = 42: CFG = 1, HIP, seed = 42: CFG = 4, official, seed = 42 With CFG = 4, the last word “reasoning” in this audio is missing; with CFG = 1, it’s normal. All parameters, including the seed, are the same — only the CFG value differs. |
|
Strange — I'm currently running into a weird issue. The content generated with the Vulkan backend and the HIP backend is coming out wrong. I'll keep investigating. |
@0xShug0 After more testing, I found that the official framework also suffers from quality degradation under specific seeds… Specifically, with long text, the volume of the last sentence becomes quieter. I kept thinking there was a bug somewhere, but it turns out the official implementation reproduces it too. So far I haven't found any other bugs — if you have time, feel free to take a look. |
|
So far, my testing approach has been roughly as follows:
|
This is a summary of the 'quality degradation' issue that I have observed. Breeze TTS tail-volume drop: not an engine bugReported symptom: with certain prompt/style combos, the last sentence of Verdict: model-intrinsic trajectory degeneration, not a backend bug. Why:
Practical advice: re-roll with a different seed. Systematic mitigation |
|
@IIIIIllllIIIIIlllll Thanks for the investigation! I’ll be AFK most of the day today and will look at the issues later tonight. The model has two sampling stages, which MAY make it prone to collapse due to small token dirft. Or it could be due to training data. For most of the models replacing the sampling impl is acutally fine and won't cause collapse. |
|
@IIIIIllllIIIIIlllll Could you isolate the transpose1d fast path changes into a separate PR? It would make it easier to track commits that introduce behavioral changes. Also if all models are safe for this change (no performance regression, and no crash due to layout) the the function really doesn't need to mix Please run regression tests on at least the popular models and keep the status updated in the PR. I can fill in the remaining models afterward. |
OK |
nsys on the 2080 Ti shows the f32 -> bf16 -> f32 cast pairs behind every activation rounding point cost ~19% of GPU time on the bf16 path and ~29% on the q4_k path (144k tiny cpy kernels per 20-token run), because ggml has no fused round-to-bf16 op and the CUDA backend runs each ggml_cast as a separate kernel. Add GGML_UNARY_OP_ROUND_BF16 (CPU + CUDA implementations; HIP shares the ggml-cuda sources) that rounds f32 values to bf16 precision in one pass, bit-identical to the cast round trip (same __float2bfloat16 / __bfloat162float sequence as cpy). The qwen decoder activation cast policy gains a fused_round flag, enabled for CUDA/HIP only; Vulkan keeps the round trip. Non-contiguous views also keep the round trip, as the unary op requires contiguous input. Verified on RTX 2080 Ti with Breeze-TTS 2: generated codes are bit-identical to the round trip build in all four test cases (bf16/q4_k, fixed 100-token case and both Chinese regression prompts). RTF on the fixed 100-token case: bf16 0.760 -> 0.695, q4_k 0.484 -> 0.419; Chinese regression q4_k 0.861 -> 0.736 (short) and 0.556 -> 0.464 (long).
Rounding points fed by non-contiguous views (rope/cache paths) still used the cast round trip: a strided f32 -> bf16 cpy plus a contiguous bf16 -> f32 cpy, ~10% of GPU time on the q4_k path. Add a row-strided variant of the round_bf16 kernel (dst is contiguous by construction) and relax the backend/framework gates from ggml_is_contiguous to ggml_is_contiguous_rows, so those points fuse too. Codes remain bit-identical in all four test cases. RTF on RTX 2080 Ti, q4_k: 100-token 0.419 -> 0.399, Chinese long 0.464 -> 0.441; bf16 100-token 0.695 -> 0.677.
…PRE_AMPERE Upstream disables CUDA graphs below sm_80. Keep that default, but add an env-var escape hatch so pre-Ampere behavior can be tested without recompiling. On the RTX 2080 Ti (sm_75) Breeze-TTS 2 decode the graphs do capture and replay correctly (bit-identical codes), but RTF is neutral to slightly worse (0.399 without vs 0.408 with on the q4_k 100-token case), so the upstream default stands for this workload.
ggml_round_bf16 now always produces a contiguous f32 result regardless of input type (f32/f16/bf16), matching the cast round trip bit for bit: bf16 input is already rounded so the op degenerates to an exact widening, f16 input rounds through bf16 and widens, both landing on the same real values as cast -> bf16 -> cast -> f32. This fixes a HIP crash where rounding points fed by the bf16 KV cache hit an f32/f16-only assert in the unary kernel, and recovers the fusion for f16 inputs (CUDA f16 KV cache paths) that the previous f32-only gate skipped. The activation cast no longer needs per-type special cases. Verified bit-identical codes in all 8 cases (CUDA + HIP x q4_k/bf16 x 100-token + 2 Chinese regression prompts). RTF, q4_k 100-token: CUDA 0.417 -> 0.405, HIP 0.73 (unchanged); HIP q4_k vs pre-fusion baseline: 0.84 -> 0.73, long 0.92 -> 0.80, bf16 1.50 -> 1.37.
The col2im path (mul_mat + ggml_col2im_1d) only ran on CUDA/HIP/Metal; Vulkan fell back to ggml_conv_transpose_1d, whose Vulkan shader is a naive per-element kernel. All ops the col2im path needs are already supported by the Vulkan backend, including col2im_1d (f32/f16 pipelines). Breeze-TTS 2 speech decoder on Radeon 8060S: 190 ms -> 98 ms; greedy output codes identical to the generic path (wav correlation 0.99998).
CFG combines logits as uncond + scale * (cond - uncond), which is exactly cond at the default guidance_scale of 1. Running the unconditional backbone there is pure waste: skipping it removes half the backbone prefill and decode work. The depth projector's logits_cfg also gets a scale == 1 shortcut that copies the conditional half directly, avoiding an inexact uncond + 1 * (cond - uncond) round trip. guidance_scale = 0 (pure unconditional) is now accepted as well. On an RTX 2080 Ti, Breeze-TTS 2 fixed 100-token case, native weights: RTF 0.705 -> 0.605; greedy output is bit-identical with and without the skip. guidance_scale = 1.5 still runs the full CFG path unchanged.
The official Breeze-TTS 2 inference runs the backbone and depth decoder with bf16 activations and a bf16 KV cache. A pure fp32 AR loop drifts into degenerate trajectories on some prompts (mispronounced tokens, repetition collapse, missing EOS), so round activations to bf16 at every op boundary via the qwen decoder activation_cast policy, mirroring the reference torch bf16 semantics. CUDA/HIP use the fused round-to-bf16 op; Vulkan uses the cast round trip. KV cache stays F16 on CUDA and Vulkan: bf16 flash attention is only accelerated with native bf16 MMA (sm_80+) and is ~3x slower on older GPUs. HIP uses a bf16 KV cache like the reference. (Ported onto the perf branch; fused_round requires the ROUND_BF16 op from the preceding commits.)
1d3b5a9 to
330e347
Compare
|
@IIIIIllllIIIIIlllll Merged! It's on dev so have some time to test before release. Found the old doc for lifting the Metal fast path. Back then we only had ~20 models. Now it’s 60+, so full regression is probably too much to ask. https://github.com/0xShug0/audio.cpp/blob/main/docs/reports/metal_convtranspose_performance.md. |
I will try to test a portion of them. |
|
Regression results for the conv_transpose1d col2im fast path on Vulkan. Tested on Radeon 8060S (RADV), comparing
No crashes, no layout errors, no perf regressions; vibevoice is ~2x faster with the fast path. VibeVoice's waveform divergence comes from the model, not the change: its diffusion sampler amplifies any tiny numeric difference. Cross-check — HIP (which has always used the col2im path) vs Vulkan without this change diverges even more (waveform corr 0.48), so the delta introduced by enabling col2im on Vulkan is smaller than an ordinary backend swap. Output is deterministic per build (same-build rerun corr = 1.0). |
|
Extended regression pass on the col2im gate (pre = 6f64b48, post = 3935ee0, Radeon 8060S / gfx1151, Vulkan, fixed seed, identical requests):
Note on cosyvoice3: both builds emit the same near-empty 0.06 s output on this GPU, bit-identical pre/post. That is a pre-existing issue on the 8060S and is not related to this change — cosyvoice3 never routes through ConvTranspose1dModule, and the identical pre/post output confirms this PR doesn't affect it. RTF deltas are attributable to this PR alone: pre and mid differ by exactly this one commit, and repeated runs are stable within each build. Magnitude scales with how conv-transpose-heavy the codec/vocoder is. Together with the earlier 5 models, coverage is now 12 models. Full procedure + per-model run scripts: |
|
@IIIIIllllIIIIIlllll Looks good. Didn't realize cosyvoice3 was that slow. I’ve been fighting with an interesting model for a few days. I’ll try to pick up cosyvoice3 before merging dev into main or just remove it from dev before the merge. |
|
Correction on the RTF numbers in my previous two comments: those runs executed all
Takeaways unchanged: no correctness regression anywhere (all waveform corr ≥ 0.9998 except chaotic vibevoice, which was qualified separately), and the speedup is real where conv-transpose is a meaningful share of runtime — vibevoice ~1.85×, pocket_tts's mimi decoder stage 60.7 → 19.8 ms (3×) on the 2080 Ti. Apologies for the noisy first numbers. |
|
@0xShug0 That should be all I can do. |
|
Oh! There's also room for optimization in terms of VRAM usage, but that's a separate piece of work — and I'm already giving it a try. The direction is: split peech_encoder.cpp into two graphs; the transformer graph; and the O(T²) time complexity of the encoder attention — that's a major overhaul. |
Thank you for keeping pushing on this! Sorry I dropped the ball on the new models. I reached out to the Breeze team and see if they could help. |
What's going on? |
Haha, just on a short vacation this week and also got distracted by LiveAvatar audio-to-video model :) |
Haha, I thought I had run into some difficult technical issue. I'll do my best to optimize this model thoroughly, so you'll have less work to do later. |
2 participants
Tested on RTX 2080 Ti (CUDA), gfx1151 (HIP), and Radeon 8060S (Vulkan). Fixed 100-token case, RTF:
Changes
breeze_tts/generator.cpp): load q/k/v as one packed tensor and gate/up as one packed tensor per layer (same approach as the higgs_audio_tts loader). Fewer, larger GEMMs per layer; bit-identical output on the native path, and it makes quantized weight storage (_type=q4_ketc., already supported by the framework) measurably faster. Also documentsweight_typein the Breeze model page.GGML_UNARY_OP_ROUND_BF16 ggml CPU + CUDA/HIP): rounds f32→bf16→f32 in a single kernel, bit-identical to the cast round trip. On the 2080 Ti the cast pairs behind every bf16 activation rounding point cost ~19–29% of GPU time in small kernel launches. The qwen decoder's activation-cast policy gains afused_round` flag (default off; CUDA/HIP only). Supports f32/f16/bf16 inputs with contiguous rows; anything else falls back to the round trip.GGML_CUDA_GRAPHS_PRE_AMPEREenv var: upstream disables graphs below sm_80; this adds an opt-in escape hatch for testing. Default behavior unchanged — on the 2080 Ti graphs were correct but RTF-neutral for this workload, so the default stands.Notes
fused_roundflag not enabled by any in-tree model yet — Breeze's activation-cast enablement comes with a separate bf16 activation fix (see Feadback: breeze-tts-2 #380: this model is trained/tuned in bf16 and the f32 path mispronounces). Models already usingactivation_cast(redae, fireredtts3, personaplex) can opt in later.