Skip to content

feat: add Chatterbox Turbo TTS model family - #394

Merged
0xShug0 merged 4 commits into
0xShug0:mainfrom
pannagaps:feat/chatterbox-turbo
Sep 4, 2026
Merged

feat: add Chatterbox Turbo TTS model family#394
0xShug0 merged 4 commits into
0xShug0:mainfrom
pannagaps:feat/chatterbox-turbo

Conversation

@pannagaps

@pannagaps pannagaps commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds chatterbox_turbo as a new model family: Resemble AI's distilled 350M-parameter Chatterbox variant (GPT2-style T3 backbone, GPT2 BPE tokenizer with 19 built-in emotion/style tags, 2-step meanflow-distilled S3Gen decoder), loaded from cstr/chatterbox-turbo-GGUF (a third-party GGUF conversion, MIT-relicensed by cstr for their CrispASR project — not published by ResembleAI or audio.cpp).
  • Extends the shared chatterbox S3Gen flow decoder to support meanflow distillation: a second "r" (end-time) sinusoidal embedding mixed into the time embedding via a new time_embed_mixer linear layer, and a non-CFG, non-batch-doubled Euler solve (compute_s3_flow_cfm_meanflow). This is gated behind detecting a flow.decoder.estimator.time_embed_mixer.weight tensor, so the base Chatterbox 10-step CFG decoder path is unaffected.
  • Pins the S3 flow encoder's token embedding lookup table to F16 regardless of requested weight storage type, since ggml_get_rows on CUDA doesn't support K-quant lookups for that table (needed for Turbo's K-quant S3Gen package).
  • Documents the new family in docs/tts.md and README.md, and adds model_specs/chatterbox_turbo.json.

Current limitation: only the built-in default voice baked into the T3 GGUF is supported; custom voice cloning via --voice-ref is rejected with an explicit error, since it depends on the turbo GGUF's speaker-encoder/S3-tokenizer sections, whose exact tensor layout hasn't been validated yet. Marked status: testing in the model spec until that lands.

Framework changes

src/models/chatterbox/{s3gen_flow.cpp,s3gen_inference.cpp} and include/engine/models/chatterbox/s3gen_flow.h (shared with the base chatterbox family) were touched to add the meanflow decoder path. The base Chatterbox 10-step CFG decoder is unaffected: the new code paths only activate when S3FlowDecoderWeights::meanflow is set (detected from the presence of the time_embed_mixer tensor), which the base GGUF package does not have.

Test plan

  • python3 tools/check_loader_catalog_sync.py --self-test && python3 tools/check_loader_catalog_sync.py — passes, loader/spec/catalog in sync
  • Build with audiocpp_cli --task clon --family chatterbox_turbo --model models/Chatterbox-Turbo-GGUF/chatterbox-turbo-t3-q8_0.gguf --backend cuda --text "Hello from Chatterbox Turbo." --out out.wav
  • Confirm base chatterbox family output is unchanged (regression check on the shared S3Gen flow decoder)

pannagaps and others added 2 commits September 2, 2026 09:58
Adds chatterbox_turbo as a new model family: Resemble AI's distilled
350M-parameter Chatterbox variant with a GPT2-style T3 backbone and a
2-step meanflow-distilled S3Gen decoder, loaded from cstr's third-party
GGUF conversion (built-in voice only; custom voice cloning is not yet
implemented).

Extends the shared chatterbox S3Gen flow decoder to support meanflow
distillation (a second "r" end-time embedding and a non-CFG Euler
solve via compute_s3_flow_cfm_meanflow), gated behind the presence of
a time_embed_mixer tensor so the base Chatterbox 10-step CFG decoder
is unaffected. Also pins the S3 flow encoder's token embedding table
to F16 regardless of requested storage type, since ggml_get_rows on
CUDA doesn't support K-quant lookups for that table.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@0xShug0

0xShug0 commented Sep 2, 2026

Copy link
Copy Markdown
Owner

@pannagaps Thanks for the PR! I’ll be AFK most of the day today and will test the PR tonight.

@0xShug0 0xShug0 added the new model Request for new model support label Sep 2, 2026
@0xShug0

0xShug0 commented Sep 2, 2026

Copy link
Copy Markdown
Owner

@pannagaps Could you move the models to the community_models/? The core models are expected to have runtime tests on at least CUDA, Vulkan, and Metal and will go throught more tests. Since the status is "testing", it's better to keep it under community models so users have the right expectations around support and compatibility.

Also please repack the GGUF to make it audio.cpp-native (self-contained, spec v1) and provide conversion scripts. Otherwise, we end up adding unnecessary complexity and small compatibility layers just to support third-party GGUFs, all of which could be avoided with a proper GGUF offline conversion. You can check this PR and commit to get the idea -- cbd7930

@pannagaps

Copy link
Copy Markdown
Contributor Author

Got it, makes sense. I’ll move the models to community_models/ and repack the GGUF to be audio.cpp-native, along with the conversion scripts.

- Removed obsolete assets and dot prefix source files from chatterbox_turbo model.
- Deleted S3Gen name bridge and tokenizer implementation, simplifying the model structure.
- Introduced a new Python script to repack the third-party chatterbox-turbo GGUF files into a format compatible with audio.cpp.
- The repacking script includes tensor name translation to match the base Chatterbox's expectations, ensuring seamless integration.
- Updated tokenizer handling to create separate vocab, merges, and special tokens files instead of relying on runtime GGUF metadata.
@pannagaps

Copy link
Copy Markdown
Contributor Author

hi @0xShug0 ,
Im done with the changes requested please verify and let me know.

thanks

@0xShug0

0xShug0 commented Sep 4, 2026

Copy link
Copy Markdown
Owner

@pannagaps Thanks! Mergd.

There are some contract issues and I will fix them after merge.

@0xShug0
0xShug0 merged commit d8ecb8b into 0xShug0:main Sep 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new model Request for new model support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants