feat: add Chatterbox Turbo TTS model family - #394
Conversation
Adds chatterbox_turbo as a new model family: Resemble AI's distilled 350M-parameter Chatterbox variant with a GPT2-style T3 backbone and a 2-step meanflow-distilled S3Gen decoder, loaded from cstr's third-party GGUF conversion (built-in voice only; custom voice cloning is not yet implemented). Extends the shared chatterbox S3Gen flow decoder to support meanflow distillation (a second "r" end-time embedding and a non-CFG Euler solve via compute_s3_flow_cfm_meanflow), gated behind the presence of a time_embed_mixer tensor so the base Chatterbox 10-step CFG decoder is unaffected. Also pins the S3 flow encoder's token embedding table to F16 regardless of requested storage type, since ggml_get_rows on CUDA doesn't support K-quant lookups for that table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
@pannagaps Thanks for the PR! I’ll be AFK most of the day today and will test the PR tonight. |
|
@pannagaps Could you move the models to the community_models/? The core models are expected to have runtime tests on at least CUDA, Vulkan, and Metal and will go throught more tests. Since the status is "testing", it's better to keep it under community models so users have the right expectations around support and compatibility. Also please repack the GGUF to make it audio.cpp-native (self-contained, spec v1) and provide conversion scripts. Otherwise, we end up adding unnecessary complexity and small compatibility layers just to support third-party GGUFs, all of which could be avoided with a proper GGUF offline conversion. You can check this PR and commit to get the idea -- cbd7930 |
|
Got it, makes sense. I’ll move the models to community_models/ and repack the GGUF to be audio.cpp-native, along with the conversion scripts. |
- Removed obsolete assets and dot prefix source files from chatterbox_turbo model. - Deleted S3Gen name bridge and tokenizer implementation, simplifying the model structure. - Introduced a new Python script to repack the third-party chatterbox-turbo GGUF files into a format compatible with audio.cpp. - The repacking script includes tensor name translation to match the base Chatterbox's expectations, ensuring seamless integration. - Updated tokenizer handling to create separate vocab, merges, and special tokens files instead of relying on runtime GGUF metadata.
|
hi @0xShug0 , thanks |
|
@pannagaps Thanks! Mergd. There are some contract issues and I will fix them after merge. |
Summary
chatterbox_turboas a new model family: Resemble AI's distilled 350M-parameter Chatterbox variant (GPT2-style T3 backbone, GPT2 BPE tokenizer with 19 built-in emotion/style tags, 2-step meanflow-distilled S3Gen decoder), loaded fromcstr/chatterbox-turbo-GGUF(a third-party GGUF conversion, MIT-relicensed bycstrfor theirCrispASRproject — not published by ResembleAI or audio.cpp).chatterboxS3Gen flow decoder to support meanflow distillation: a second "r" (end-time) sinusoidal embedding mixed into the time embedding via a newtime_embed_mixerlinear layer, and a non-CFG, non-batch-doubled Euler solve (compute_s3_flow_cfm_meanflow). This is gated behind detecting aflow.decoder.estimator.time_embed_mixer.weighttensor, so the base Chatterbox 10-step CFG decoder path is unaffected.ggml_get_rowson CUDA doesn't support K-quant lookups for that table (needed for Turbo's K-quant S3Gen package).docs/tts.mdandREADME.md, and addsmodel_specs/chatterbox_turbo.json.Current limitation: only the built-in default voice baked into the T3 GGUF is supported; custom voice cloning via
--voice-refis rejected with an explicit error, since it depends on the turbo GGUF's speaker-encoder/S3-tokenizer sections, whose exact tensor layout hasn't been validated yet. Markedstatus: testingin the model spec until that lands.Framework changes
src/models/chatterbox/{s3gen_flow.cpp,s3gen_inference.cpp}andinclude/engine/models/chatterbox/s3gen_flow.h(shared with the basechatterboxfamily) were touched to add the meanflow decoder path. The base Chatterbox 10-step CFG decoder is unaffected: the new code paths only activate whenS3FlowDecoderWeights::meanflowis set (detected from the presence of thetime_embed_mixertensor), which the base GGUF package does not have.Test plan
python3 tools/check_loader_catalog_sync.py --self-test && python3 tools/check_loader_catalog_sync.py— passes, loader/spec/catalog in syncaudiocpp_cli --task clon --family chatterbox_turbo --model models/Chatterbox-Turbo-GGUF/chatterbox-turbo-t3-q8_0.gguf --backend cuda --text "Hello from Chatterbox Turbo." --out out.wavchatterboxfamily output is unchanged (regression check on the shared S3Gen flow decoder)