[haaziq-sl]

Overview

Run KittenTTS2's speech model and waveform decoder on NVIDIA GPUs from the CLI or Python. The runtime now downloads the published default and student_w4 CUDA decoders from Hugging Face automatically. Users do not need to export a decoder.

The PR contains two commits on top of KittenML:py-interface:

1. Losslessly repack TQ2_1 weights into Q2_0 when loading GPU buffers, with repack and backend tests.
2. Add CUDA inference options, placement checks, decoder downloads, Linux/Windows CUDA wheels, and the corresponding documentation and validation fixes.

CPU inference remains the default. The GPU packages are kitten-tts-cpp-cu12 on Linux and kitten-tts-cpp-cu13 on Windows; both use the kitten_tts_cpp import.

After installing the GPU wheel and matching PyTorch build:

from kitten_tts_cpp import KittenTTS

with KittenTTS(decoder="student_w4", gpu_layers=99, decoder_device="cuda") as model:
    audio = model.generate("Hello from the GPU.", voice="Bruno")

Validation

Hardware testing on October 24 used an RTX 3090 on Linux and an L4 on Windows Server 2022, with model revision c93cb7c271baa51d9ae8204d5277988906f235a2.

• All 10 CUDA wheels passed on Python 3.10-3.14: installation, CPU and GPU synthesis, both published CUDA decoders, online/offline reuse, voice selection, file output, streaming, cancellation, and error handling.
• All five Linux wheels also generated audio without an NVIDIA driver. Linux and Windows rejected incompatible PyTorch installations with installation guidance.
• Documented Linux CPU/CUDA builds, the Windows CLI, source-distribution installation, and asset export/publication checks passed.
• All 310 GGUF tensors reconstructed the reference exactly. Six CUDA decoder parity cases had a maximum error of 2.98e-7.
• Six seeded CPU WAV files were byte-identical to the base branch.

Full results and limitations are in validation.json. The new wheel CI run is separate from these hardware tests. The inherited CANN workflow has no active jobs and already failed validation on the previous PR head; the inherited Python type-check requires a self-hosted runner.

Limitations

• student_w8 has no published CUDA decoder. GPU output is not promised to be bit-identical to CPU output.
• The first CUDA call on Windows can take over a minute; reuse the loaded model.
• H100 performance has not been benchmarked. The wheels use PTX fallback there.
• The portable scalar CPU LM exceeded an existing parity threshold; the documented native AVX2 build passed. The older published CPU W4 decoder also differs from current eager Python, while matching the same TorchScript model exactly. These observations are recorded in the validation file.

Requirements

• I have read and agree with the contributing guidelines.
• AI usage disclosure: YES. Codex assisted with implementation fixes, documentation, testing, and this PR description.