Feadback: breeze-tts-2

[IIIIIllllIIIIIlllll]

Long story short – I know nobody wants to read AI-generated walls of text that look detailed but are actually hollow.

I reached out to a few netizens who are interested in breeze-tts-2, and they all ran side‑by‑side comparisons using audio.cpp and the official framework. Everyone confirmed the same thing: audio.cpp can't match the official runtime with acceleration enabled in terms of raw peak performance, but it uses much less VRAM – under 8 GB, whereas the official one needs at least 16 GB (with acceleration). Without acceleration, the official default is eager mode, which gets completely crushed by audio.cpp.

CUDA (2080 Ti) → RTF 0.82
AMD HIP (8060S) → RTF 1.60

Even an oldie like the 2080 Ti can do real‑time generation! Hooray!

We've checked against the official runtime – everything matches, nothing to report.

The one thing worth mentioning is precision. Intuitively, fp32 activations feel "more correct" (higher precision), but the actual audio output is worse – mispronunciations and occasional runaway repetition. The model was trained and tuned in bf16, and only behaves well when you stick to bf16. For human ears, bf16 is the better choice. Those netizens and I all confirmed this – it's unlikely that all of us are hallucinating at the same time (joking). (Fixed on the breeze-bf16-activations branch.)

On AMD (HIP) there are still some performance and consistency quirks – it's slower than CUDA (which is expected for AMD), and previously the outputs weren't identical; both issues have been fixed in my own fork, and if needed, that could come in handy in the future.

Supplementary note: the RTF figures are based on tests run on the BF16-fixed branch; on the unfixed branch, performance can be up to 30% higher. This is a tough trade-off, and for now I haven't found a way to optimize it.

[0xShug0]

@IIIIIllllIIIIIlllll Thank you for the feedback! There’s still room for performance optimization, which is why it’s still in dev. The trade-off is really about time: spending more time on deeper optimizations vs making models available sooner to a broader group of users who may struggle with Python setup, such as compiling FlashAttention. I hope the community can help drive further optimizations for this model.

For optimization, you can check Higgs Audio TTS as a reference. It may require a combination of multiple techniques. If we confirm there’s a trade-off, the optimizations can be made opt-in.

[0xShug0]

@IIIIIllllIIIIIlllll Quickly checked the Breeze acceleration path. It broadly matches the early audio.cpp release 0.1 design pattern: use aggressive session-owned caching and static execution paths to reduce per-request and per-token overhead 😄 . It captures the full RVQ depth loop as a single graph and uses a streaming codec graph. Those actually are the parts that cpp could mirror.

[0xShug0]

I opened a discussion on Hugging Face. The Breeze TTS 2 license includes this clause: “Ensure that each recipient independently accepts this Agreement before accessing or using the Model Materials or Derivative Model.”

This is the first model license I’ve seen with this kind of requirement. The obligation is placed on the distributor/audio.cpp. That means we may only be able to provide the code and ask users to convert the weights themselves or provide a gated repo. Anyway this does look good for audio.cpp integrations. If the license term isn’t updated, we’ll likely need to move the model to community_models.

[IIIIIllllIIIIIlllll]

@0xShug0 Indeed, moving to community_models requires users to accept an agreement, which is a bit of a hassle—or one could simply use the raw safetensors weights.

[0xShug0]

@0xShug0 Indeed, moving to community_models requires users to accept an agreement, which is a bit of a hassle—or one could simply use the raw safetensors weights.

Haha I forgot safetensor and yes that's could be the workaround (maybe with on-the-fly conversion).

I don't think they read their license. "Commercial Purpose includes, without limitation: ....(b) hosting or making the Model Materials, a Derivative Model, or their functionality available through an API, SaaS offering, application, plug-in, website, or other service;....". To me, HF is a website and service.

[0xShug0]

All good now. They updated the license.

[juntaosun]

When will it be merged into the main branch?

[0xShug0]

@juntaosun Maybe around this weekend, so we have time to incorporate more optimization PRs.

[IIIIIllllIIIIIlllll]

2080TI:
Load-time weight quantization (weight_type=q4_k/q8_0 session option): the decode loop is memory-bound, so reading q4_k weights instead of bf was the single biggest win (RTF 0.77→0.49 on a 2080 Ti). In most cases, there is no noticeable loss of quality; only F32 ever produced mispronunciations, everything ≤bf16 was fine. Occasional static noise bugs occur; overall, it is not suitable for practical application.
Fused the bf16 rounding: the bf16-parity fix's f32→bf16→f32 cast pairs were 19-29% of GPU time (144k tiny cpy kernels per 20 tokens). Added GGML_UNARY_OP_ROUND_BF16 (CPU/CUDA/HIP) doing it in one kernel, plus packed QKV and gate/up weights. Codes stay bit-identical; another -10-20%.
Numbers (100-token fixed case, bit-identical codes): CUDA q4_k 0.484→0.405.

[0xShug0]

@IIIIIllllIIIIIlllll Would you like to submit a PR for this? Otherwise, I’ll pick it up later this week and just want to avoid any duplicated effort.

For the “occasional static noise” issue, is this a model-quality problem or cpp implementation bug? If it’s the latter, I’d like to look into fixing it later. I’d appreciate it if you could share your test cases and the exact commands you used. Some test cases that reproduce the mispronunciations would also be super useful.

[IIIIIllllIIIIIlllll]

@IIIIIllllIIIIIlllll Would you like to submit a PR for this? Otherwise, I’ll pick it up later this week and just want to avoid any duplicated effort.

For the “occasional static noise” issue, is this a model-quality problem or cpp implementation bug? If it’s the latter, I’d like to look into fixing it later. I’d appreciate it if you could share your test cases and the exact commands you used. Some test cases that reproduce the mispronunciations would also be super useful.

No problem. The occasional noise is due to the aggressive quantization in Q4 — I used fairly aggressive settings to test performance, and just when I thought "this should be fine," I happened to notice the quality degradation. I'll organize the test results and submit a PR tomorrow.