From 9c8a9f27723c31aa738880c3f6ca6005f5fe3fef Mon Sep 17 00:00:00 2001 From: Warren B Date: Sun, 30 Aug 2026 22:44:57 +0100 Subject: [PATCH] gguf: stop quantizing the tensors that are audible The packer's entire quantize/keep policy was one shape predicate at tensor_source.cpp:1988-1997: quantize any tensor that is rank-2, ends in ".weight", and does not contain "embed" or "codebook". Vocoders, decoders and output heads are protected by nothing. Conv-stack vocoders survive at F32 purely by being rank-3 or higher, and normalisation weights survive purely by being rank-1. Verified by parsing the headers of 22 published GGUFs: - redae/decoder.istft_head.out.weight is Q8_0 [896,1922] in FireRedTTS3-Instruct and FireRedAudio. That layer's output IS the magnitude/phase spectrum, exponentiated immediately after, with nothing downstream to absorb the error. - VibeVoice-7B ships a half-quantized decoder: convs BF16, FFN matmuls Q8_0, 158 tensors. - IndexTTS2.5's s2mel flow decoder is linear rather than conv, so 139 tensors land at Q8_0 where OmniVoice's conv vocoder is F32 for free. - 27 rank-2 norm projections in IndexTTS2.5 are Q8_0 because the norm protection is only a rank test. Adds a declarative rule table with a reason string per entry, covering iSTFT and spectrogram heads, vocoders, codec and latent-to-audio decoders, normalisation projections, and output heads and final projections. A name match demotes the tensor to F16, reusing the existing lookup-table branch rather than adding a mechanism. The table was validated against 13,115 real tensor names from the installed packages, with no false positives. Deliberately not covered: bare "decoder", which would move redae/decoder.qwen2.layers.* -- a 24-layer transformer -- for nothing, and every encoder or analysis-side name. GgufConversionOptions::quantize_audible_tensors restores the old policy. Measured cost of re-cutting the installed packages: omnivoice-q8_0 +8 MiB (+0.6%) 1 tensor fireredtts3-instruct-q8_0 +98 MiB (+2.5%) 15 tensors index-tts2_5-q8_0 +138 MiB (+4.2%) 106 tensors vibevoice-7b-q8_0 +918 MiB (+9.1%) 59 tensors The last row is almost entirely one lm_head.weight [152064,3584]. Separately, two package defaults move off 4-bit. MiniMax Music 3 defaulted to legacy q4_0 for both its language model and its flow transformer -- including the transformer's proj_out, which feeds the vocoder -- while q8_0 and bf16 packages sat unused. q4_0 is a legacy format with one scale per 32-element block and no zero point; q4_k is already supported here and is strictly better at the same bitrate. PersonaPlex moves q4_k -> q8_0. The Music3 default change needs the session fallback too: session.cpp hard-coded the q4_0 filenames as its value_or fallbacks, and spec option defaults are metadata that never reaches the runtime options map, so changing the spec alone would make a fresh install of the new default package throw "file does not exist". The fallback now probes for the highest-quality tier present, so a q8_0 install and an existing q4_0 install both resolve without the caller naming a file. This does not change GGUFs that already exist. Installed models keep the precision they were built with; a package has to be re-converted from its original weights for the policy to reach audio. docs/gguf.md says so explicitly, documents the rule table and the override, and is now honest that the lower-bit tiers are untested. Tests: tests/unittests/test_quantization_policy.cpp writes a 17-tensor safetensors fixture, packs it four ways with the real converter, and asserts the resulting dtype and exact byte count of every tensor. The regression guard matters most: FFN, attention, the decoder-side transformer stack and the tokenizer encoder must still come out q8_0, and embed/codebook must still be f16, so a widened pattern fails loudly. Build: cmake -S . -B build -DENGINE_BUILD_TESTS=ON && cmake --build build Test: ctest -R quantization_policy_test (no model weights required) Backend tested: CPU (pure conversion logic); full suite 39/39. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ATa5YkLUPMDPRL7w1gCo9p --- CMakeLists.txt | 7 + docs/gguf.md | 123 +++++++ .../engine/framework/assets/tensor_source.h | 21 ++ model_specs/minimax_music3.json | 10 +- model_specs/personaplex.json | 6 +- .../minimax_music3/session.cpp | 22 +- src/framework/assets/tensor_source.cpp | 135 +++++++- tests/unittests/test_quantization_policy.cpp | 301 ++++++++++++++++++ 8 files changed, 611 insertions(+), 14 deletions(-) create mode 100644 tests/unittests/test_quantization_policy.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 63195ff21..e0030a02d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2032,6 +2032,13 @@ if (ENGINE_BUILD_TESTS) COMMAND audio_chunking_test ) + add_engine_unittest(quantization_policy_test tests/unittests/test_quantization_policy.cpp) + + add_test( + NAME quantization_policy_test + COMMAND quantization_policy_test + ) + add_engine_unittest(chinese_normalization_test tests/unittests/test_chinese_normalization.cpp) add_test( diff --git a/docs/gguf.md b/docs/gguf.md index fd7ec4908..b78d9160f 100644 --- a/docs/gguf.md +++ b/docs/gguf.md @@ -116,6 +116,41 @@ Additional lower-bit checks: | `personaplex` | `q4_k` | Pass | | `voxtral_realtime` | `q4_k` | Pass (quick CUDA check; transcripts match Q8 except one capitalization-only difference) | +**What the matrix does not cover.** The three tables above test `orig`, 16-bit and +`q8_0`, plus the three `q4_k` spot checks listed here. That is the whole of the recorded +validation. In particular: + +- `minimax_music3`, `minimax_h3` and `audiosr` appear in **no row of any table**. Nothing + about those families has been recorded through this process. +- There is no 4-bit column in the main table, so no family's 4-bit package is validated + against its own 16-bit or `orig` output. +- Every "Pass" label is a load-and-run judgement plus, for ASR, a transcript comparison. + There is no perceptual metric anywhere in the repo — no MOS, PESQ, SI-SDR or + mel-cepstral distortion — so "Pass (drift)" is not quantified and cannot be compared + between packages. Timbre, prosody and speaker-identity drift, which is what + quantization actually costs a TTS or music model, is not measured at all. + +`minimax_music3` and `personaplex` shipped a 4-bit package as their default while a +`q8_0` package sat published and unused. Both now default to `q8_0`; the 4-bit packages +remain installable by id. `minimax_h3` still defaults to `q4_k` because no +higher-precision package is published for it. + +| Family | Old default | New default | Package size | +|---|---|---|---| +| `minimax_music3` | `minimax_music3_q4_0` | `minimax_music3_q8_0` | 8045 MiB -> 12902 MiB (`minimax_music3_bf16` would be 22547 MiB) | +| `personaplex` | `personaplex_7b_v1_q4_k` | `personaplex_7b_v1_q8_0` | roughly 4 GB -> 7 GB for a 7B model | +| `minimax_h3` | `minimax_h3_q4_k` | unchanged | only 4-bit packages are published | + +The MiniMax Music 3 figures are the sum of the five component GGUFs, computed from the +installed `q4_0` package's tensor shapes; the `q4_0` total reproduces the installed files +to within 1 MiB. Re-cutting the `q8_0` package with the tensor policy below would take it +to 13634 MiB. + +`q4_0`, which `minimax_music3` used, is a legacy format: one scale per 32-element block +and no zero point. `q4_k` carries per-super-block scales and mins at essentially the same +bitrate and is what every other 4-bit user in the repo is already on. If a 4-bit MiniMax +Music 3 package is re-cut, cut it as `q4_k`. + Q8 packaging notes: - `chatterbox` Q8 is intentionally mixed type. Graph-sensitive scalar, norm, @@ -145,6 +180,94 @@ Q8 packaging notes: check it was smaller and faster than Q8_0, while transcript output matched Q8_0 except for one capitalization-only difference. +## Which Tensors Get Quantized + +Two rules decide the storage type of every tensor the converter writes. Both live in +`convert_tensor_sources_to_gguf` (`src/framework/assets/tensor_source.cpp`). + +**1. The shape rule.** A tensor is a candidate for the requested quantized type only when +it is a float source, its name ends in `.weight`, it is exactly rank 2, and its last +dimension divides the target's block size (32 for `q8_0`, 256 for the `_k` formats). +Everything else keeps its source dtype. This is why biases and rank-1 norms come out F32, +and why conv kernels do: a Conv1d kernel is rank 3 and its row length is the kernel size +(3, 7, 11, 16), never a multiple of 32. + +**2. The audible-tensor rule.** The shape rule is blind to what a tensor *does*. It +protects conv-stack vocoders by accident of rank and protects nothing at all once a +vocoder, a codec decoder or a norm is written as a matmul. The converter therefore also +holds back tensors by name, and stores them at F16 instead — the same fallback embedding +and codebook tables already use. + +A tensor belongs on that list when quantization error in it reaches the waveform with no +remaining layer to absorb it. That is the whole test. Mid-stack attention and FFN weights +fail it and stay quantized. + +| Group | Patterns | Why | +|---|---|---| +| iSTFT and spectrogram heads | `*istft*`, `*stft_head*`, `*spec_head*`, `*spectrogram_head*`, `*mag_head*`, `*phase_head*` | The layer's output *is* the magnitude/phase spectrum, exponentiated on the way into the iSTFT. `redae/decoder.istft_head.out.weight` shipped as `Q8_0 [896,1922]` in FireRedTTS3-Instruct and FireRedAudio. | +| Waveform vocoders | `*vocoder*`, `*bigvgan*`, `*hifigan*`, `*hifi_gan*`, `hift.*`, `hift/*`, `*.hift.*`, `*/hift.*`, `*waveform_decoder*`, `*wave_decoder*` | Last stage before the WAV. Conv stacks already survived on rank; this makes it deliberate and covers the rank-2 layers inside the same stacks. The HiFT patterns are anchored because `shift` contains `hift`. | +| Codec and latent-to-audio decoders | `*acoustic_decoder*`, `*audio_decoder*`, `*codec_decoder*`, `*codec.decoder.*`, `*codec/decoder.*`, `*_tokenizer.decoder.*`, `*_tokenizer/decoder.*`, `*mel_decoder*`, `*s2mel*` | VibeVoice-7B shipped 52 Q8_0 FFN matmuls inside `model.acoustic_tokenizer.decoder`; IndexTTS2.5 shipped its whole s2mel flow decoder at Q8_0 while OmniVoice's conv vocoder stayed F32 for free. Encoder-side names are deliberately absent: analysis error is absorbed by the decoder that follows it. | +| Normalisation projections | `*norm*`, `*adaln*` | Norms survive today only because they are rank 1. Written as a learned rank-2 projection — `*_norm.project_layer.weight`, an adaLN modulation — a norm is quantized like any other matmul, and its error is multiplied across every channel it scales. | +| Output heads and final projections | `*head.weight`, `*heads.weight`, `*proj_out.weight`, `*final_proj.weight`, `*final_layer.linear.weight` | The head emits the distribution the sampler draws audio tokens from, with no later layer to absorb the error. In every package inspected the head has exactly the shape of the embedding table, which the converter already stores at F16 — so keeping the head at F16 is what makes the two ends of the model agree. | + +Matching is case-insensitive over the whole logical tensor name, `*` stands for any run of +characters, and the first matching rule wins. The list is +`gguf_audible_tensor_rules()` in `src/framework/assets/tensor_source.cpp`; each entry +carries the reason it exists. Add to it there, and add a case to +`tests/unittests/test_quantization_policy.cpp` in the same change. + +Note what is deliberately **not** on the list. `decoder` on its own is not a pattern: a +name like `redae/decoder.qwen2.layers.*` is a 24-layer transformer that still has the +whole rest of the decoder downstream, and excluding it would move gigabytes to F16 for +nothing. Only the synthesis-side decoders named above are protected. + +### Overriding the policy + +`--keep-type *=` still wins over everything, including this list, so +a packager who knows better can quantize a protected tensor deliberately: + +```bash +# keep the policy, but quantize this one head anyway +audiocpp_gguf ... --type q8_0 --keep-type 'lm_head.weight=q8_0' +``` + +The whole policy can also be turned off from a programmatic caller by setting +`GgufConversionOptions::quantize_audible_tensors`, which reproduces the pre-policy output +byte for byte. Prefer the per-tensor override: it records which tensor was quantized and +at what type, instead of turning the protection off wholesale. + +### What this costs + +The exclusion list makes packages bigger. Measured against the shipped packages installed +on the development machine, by re-deciding every tensor with the new policy at the same +requested type: + +| Package | Size | Growth | Tensors moved to F16 | +|---|---|---|---| +| `dramabox-q8_0.gguf` | 18027 MiB | +34 MiB (+0.2%) | 2 | +| `parakeet-tdt-0.6b-v3-q8_0.gguf` | 872 MiB | +5 MiB (+0.5%) | 1 | +| `omnivoice-q8_0.gguf` | 1277 MiB | +8 MiB (+0.6%) | 1 | +| `fireredtts3-instruct-q8_0.gguf` | 3943 MiB | +98 MiB (+2.5%) | 15 | +| `index-tts2_5-q8_0.gguf` | 3324 MiB | +138 MiB (+4.2%) | 106 | +| `firered-audio-q8_0.gguf` | 13308 MiB | +1006 MiB (+7.6%) | 15 | +| `vibevoice-7b-q8_0.gguf` | 10080 MiB | +918 MiB (+9.1%) | 59 | +| `MiniMax-Music3 language_model_q4_0.gguf` | 5729 MiB | +1123 MiB (+19.6%) | 1 | + +Almost all of the growth in the last three rows is a single `lm_head.weight`, which is a +`[248320, 4096]`, `[152064, 3584]` and `[200000, 4096]` matrix respectively. If a package +needs that back, `--keep-type '=q8_0'` restores the old size exactly. + +`orig`, `f16` and `bf16` conversions are unaffected: no quantization happens on those +paths, so the policy is inert. + +### This does not change GGUF files that already exist + +The policy runs at conversion time. It has no effect on any GGUF already on disk or +already published — an installed `index-tts2_5-q8_0.gguf` keeps the Q8_0 s2mel decoder it +was built with. For a package to benefit, it has to be converted again from the original +safetensors weights with a converter built from this change, and re-uploaded. Until then +the only thing that changes for a user is which *package* a spec's default points at. + ## Build The Converter ```bash diff --git a/include/engine/framework/assets/tensor_source.h b/include/engine/framework/assets/tensor_source.h index 6735f2081..de7a4d5a8 100644 --- a/include/engine/framework/assets/tensor_source.h +++ b/include/engine/framework/assets/tensor_source.h @@ -178,11 +178,32 @@ struct GgufTensorTypeOverride { std::string pattern; TensorStorageType storage_type = TensorStorageType::Native; }; +// One entry of the audible-tensor exclusion list the GGUF packer applies before it +// quantizes anything. `pattern` is matched against the whole logical tensor name, +// lowercased, with `*` standing for any run of characters. `reason` is documentation: +// it says why this tensor may not be quantized, and it is what the next contributor +// reads before adding a rule of their own. See docs/gguf.md. +struct GgufAudibleTensorRule { + std::string_view pattern; + std::string_view reason; +}; +// The exclusion list itself, in match order. +[[nodiscard]] const std::vector & gguf_audible_tensor_rules(); +// The reason `tensor_name` is held back from quantization, or an empty view when the +// packer may quantize it. Matching is case-insensitive over the full logical name. +// This answers only the name question; the packer still applies its shape and dtype +// rules on top, and a `--keep-type` override still wins over both. +[[nodiscard]] std::string_view gguf_audible_tensor_reason(std::string_view tensor_name); struct GgufConversionOptions { std::optional bnb_nf4_type; std::vector excluded_tensor_prefixes; std::vector type_overrides; std::vector folded_weight_norm_patterns; + // Quantize audible tensors anyway, restoring the pre-policy shape-only behaviour. + // Prefer a per-tensor `--keep-type =` override to this switch: it + // says which tensor the packager decided to quantize and at what type, instead of + // turning the whole policy off. + bool quantize_audible_tensors = false; }; void convert_tensor_sources_to_gguf(const std::vector & inputs, const std::filesystem::path & output_path, TensorStorageType weight_type, diff --git a/model_specs/minimax_music3.json b/model_specs/minimax_music3.json index 97c165638..50bf91e17 100644 --- a/model_specs/minimax_music3.json +++ b/model_specs/minimax_music3.json @@ -103,21 +103,21 @@ "type": "string", "description": "Language model component GGUF file relative to the model root.", "required": false, - "default": "language_model_q4_0.gguf" + "default": "language_model_q8_0.gguf" }, { "name": "rvq_depth_decoder_gguf", "type": "string", "description": "RVQ depth decoder component GGUF file relative to the model root.", "required": false, - "default": "rvq_depth_decoder_bf16.gguf" + "default": "rvq_depth_decoder_q8_0.gguf" }, { "name": "flow_transformer_gguf", "type": "string", "description": "Flow transformer component GGUF file relative to the model root.", "required": false, - "default": "transformer_q4_0.gguf" + "default": "transformer_q8_0.gguf" }, { "name": "graph_context_mb", @@ -157,7 +157,6 @@ { "id": "minimax_music3_q4_0", "display_name": "MiniMax Music 3 Q4_0 GGUF", - "default": true, "format": "gguf", "precision": "q4_0", "target_directory": "MiniMax-Music3-GGUF", @@ -180,6 +179,7 @@ { "id": "minimax_music3_q8_0", "display_name": "MiniMax Music 3 Q8_0 GGUF", + "default": true, "format": "gguf", "precision": "q8_0", "target_directory": "MiniMax-Music3-GGUF", @@ -224,7 +224,7 @@ ], "dependencies": [], "ui": { - "recommended_package": "minimax_music3_q4_0", + "recommended_package": "minimax_music3_q8_0", "tags": [ "Music", "GGUF" diff --git a/model_specs/personaplex.json b/model_specs/personaplex.json index c5fb8b92a..2587773e3 100644 --- a/model_specs/personaplex.json +++ b/model_specs/personaplex.json @@ -163,7 +163,7 @@ { "id": "personaplex_7b_v1_q4_k", "display_name": "PersonaPlex 7B v1 Q4_K GGUF", - "default": true, + "default": false, "format": "gguf", "precision": "q4_k", "target_directory": "PersonaPlex-GGUF", @@ -175,7 +175,7 @@ { "id": "personaplex_7b_v1_q8_0", "display_name": "PersonaPlex 7B v1 Q8_0 GGUF", - "default": false, + "default": true, "format": "gguf", "precision": "q8_0", "target_directory": "PersonaPlex-GGUF", @@ -187,7 +187,7 @@ ], "dependencies": [], "ui": { - "recommended_package": "personaplex_7b_v1_q4_k", + "recommended_package": "personaplex_7b_v1_q8_0", "tags": [ "TTS", "Stream", diff --git a/src/community_models/minimax_music3/session.cpp b/src/community_models/minimax_music3/session.cpp index 6a2676c8b..d45dd2ba6 100644 --- a/src/community_models/minimax_music3/session.cpp +++ b/src/community_models/minimax_music3/session.cpp @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -52,26 +53,41 @@ std::filesystem::path resolve_component_gguf_path( return path; } +// The published packages ship the same component under different quantization +// tiers, and the model root only ever contains the tier that was installed. +// Prefer the highest-quality name present so a q8_0 install and a legacy q4_0 +// install both resolve without the caller naming a file explicitly. +std::string default_component_gguf( + const MiniMaxMusic3Assets & assets, + std::initializer_list candidates) { + for (const char * candidate : candidates) { + if (engine::io::is_existing_file(assets.model_root / candidate)) { + return candidate; + } + } + return *candidates.begin(); +} + std::shared_ptr select_component_assets( std::shared_ptr base, const std::unordered_map & options) { auto selected = std::make_shared(*base); const std::string language_model_gguf = - runtime::find_option(options, {"minimax_music3.language_model_gguf"}).value_or("language_model_q4_0.gguf"); + runtime::find_option(options, {"minimax_music3.language_model_gguf"}).value_or(default_component_gguf(*base, {"language_model_q8_0.gguf", "language_model_q4_0.gguf"})); selected->language_model_weights = assets::open_tensor_source(resolve_component_gguf_path( *base, "minimax_music3.language_model_gguf", language_model_gguf)); const std::string depth_decoder_gguf = - runtime::find_option(options, {"minimax_music3.rvq_depth_decoder_gguf"}).value_or("rvq_depth_decoder_q8_0.gguf"); + runtime::find_option(options, {"minimax_music3.rvq_depth_decoder_gguf"}).value_or(default_component_gguf(*base, {"rvq_depth_decoder_q8_0.gguf", "rvq_depth_decoder_bf16.gguf"})); selected->depth_decoder_weights = assets::open_tensor_source(resolve_component_gguf_path( *base, "minimax_music3.rvq_depth_decoder_gguf", depth_decoder_gguf)); const std::string flow_transformer_gguf = - runtime::find_option(options, {"minimax_music3.flow_transformer_gguf"}).value_or("transformer_q4_0.gguf"); + runtime::find_option(options, {"minimax_music3.flow_transformer_gguf"}).value_or(default_component_gguf(*base, {"transformer_q8_0.gguf", "transformer_q4_0.gguf"})); selected->transformer_weights = assets::open_tensor_source(resolve_component_gguf_path( *base, "minimax_music3.flow_transformer_gguf", diff --git a/src/framework/assets/tensor_source.cpp b/src/framework/assets/tensor_source.cpp index 189ea2515..f1c0c688e 100644 --- a/src/framework/assets/tensor_source.cpp +++ b/src/framework/assets/tensor_source.cpp @@ -50,6 +50,37 @@ std::optional find_tensor_type_override( return std::nullopt; } +// Wildcard match where `*` stands for any run of characters, anywhere in the pattern. +// `tensor_type_override_matches` above only understands a trailing `*` because +// `--keep-type` takes a prefix; the audible-tensor rules need `*` in the middle to +// name things like `*_tokenizer.decoder.*`. +bool audible_tensor_pattern_matches(std::string_view name, std::string_view pattern) { + size_t name_index = 0; + size_t pattern_index = 0; + size_t star_index = std::string_view::npos; + size_t name_resume = 0; + while (name_index < name.size()) { + if (pattern_index < pattern.size() && pattern[pattern_index] == '*') { + star_index = pattern_index; + ++pattern_index; + name_resume = name_index; + } else if (pattern_index < pattern.size() && pattern[pattern_index] == name[name_index]) { + ++pattern_index; + ++name_index; + } else if (star_index != std::string_view::npos) { + pattern_index = star_index + 1; + ++name_resume; + name_index = name_resume; + } else { + return false; + } + } + while (pattern_index < pattern.size() && pattern[pattern_index] == '*') { + ++pattern_index; + } + return pattern_index == pattern.size(); +} + core::TensorShape shape_from_dims(const std::vector & dims) { if (dims.empty() || dims.size() > core::kMaxTensorRank) { throw std::runtime_error("tensor rank must be between 1 and 4"); @@ -293,6 +324,98 @@ void set_backend_tensor_from_f32( } // namespace +// The GGUF packer's audible-tensor policy. +// +// Everything else in the packer decides by *shape*: a tensor is quantized when it is a +// rank-2 `.weight` whose rows divide the block size. That rule protects norms because +// they happen to be rank 1 and conv-stack vocoders because their kernels happen to be +// rank 3, and it protects nothing at all once a vocoder, a codec decoder or a norm is +// written as a matmul. This list is the part that decides by *meaning*. +// +// A tensor belongs here when quantization error in it reaches the waveform with no +// remaining layer to absorb it. That is the whole test. Mid-stack attention and FFN +// weights fail it — they have the rest of the network downstream — and must stay +// quantized, or every package the project ships grows for nothing. +// +// Matching is over the lowercased full logical name, first rule wins, `*` is any run of +// characters. A matched tensor is stored at F16 instead of the requested quantized type, +// exactly as embedding and codebook tables already are. A `--keep-type =` +// override still wins over this list, so a packager who knows better can quantize any +// one of these deliberately. +const std::vector & gguf_audible_tensor_rules() { + static const std::vector rules = { + // Inverse-STFT and spectrogram heads. This layer's output *is* the magnitude + // and phase spectrum, exponentiated on the way into the iSTFT, so its error + // lands in the samples. `redae/decoder.istft_head.out.weight` shipped as + // Q8_0 [896,1922] in FireRedTTS3-Instruct and FireRedAudio. + {"*istft*", "inverse-STFT head: its output is the spectrum fed to the iSTFT"}, + {"*stft_head*", "STFT head: its output is a spectrum"}, + {"*spec_head*", "spectrogram head: its output is a spectrum"}, + {"*spectrogram_head*", "spectrogram head: its output is a spectrum"}, + {"*mag_head*", "magnitude head: its output is a spectrum magnitude"}, + {"*phase_head*", "phase head: its output is a spectrum phase"}, + + // Waveform vocoders — the last stage before the WAV. Conv-stack vocoders + // already survive on rank; these rules make that deliberate and extend it to + // the rank-2 layers sitting inside the same stacks. + {"*vocoder*", "vocoder: last stage before the waveform"}, + {"*bigvgan*", "BigVGAN vocoder"}, + {"*hifigan*", "HiFi-GAN vocoder"}, + {"*hifi_gan*", "HiFi-GAN vocoder"}, + // Anchored rather than `*hift*`: that substring also occurs inside `shift`. + {"hift.*", "HiFT vocoder"}, + {"hift/*", "HiFT vocoder"}, + {"*.hift.*", "HiFT vocoder"}, + {"*/hift.*", "HiFT vocoder"}, + {"*waveform_decoder*", "waveform decoder"}, + {"*wave_decoder*", "waveform decoder"}, + + // Codec and latent-to-audio decoder stacks. Encoder-side tensors are + // deliberately absent: analysis error is absorbed by the decoder that follows, + // synthesis error is not. VibeVoice-7B shipped 52 Q8_0 FFN matmuls inside + // `model.acoustic_tokenizer.decoder`, and IndexTTS2.5 shipped its whole s2mel + // flow decoder at Q8_0 while OmniVoice's conv vocoder stayed F32 for free. + {"*acoustic_decoder*", "acoustic decoder: synthesis side"}, + {"*audio_decoder*", "audio decoder: synthesis side"}, + {"*codec_decoder*", "codec decoder: synthesis side"}, + {"*codec.decoder.*", "codec decoder: synthesis side"}, + {"*codec/decoder.*", "codec decoder: synthesis side"}, + {"*_tokenizer.decoder.*", "audio tokenizer decoder: synthesis side"}, + {"*_tokenizer/decoder.*", "audio tokenizer decoder: synthesis side"}, + {"*mel_decoder*", "mel decoder: emits the spectrogram the vocoder renders"}, + {"*s2mel*", "s2mel flow decoder: emits the mel spectrogram"}, + + // Normalisation projections. Norms survive today only because they are rank 1. + // Written as a learned rank-2 projection — `*_norm.project_layer.weight`, an + // adaLN modulation — a norm is quantized like any other matmul, and its error + // is multiplied across every channel it scales. + {"*norm*", "normalisation projection: scales every channel it gates"}, + {"*adaln*", "adaLN modulation: scales and shifts every channel it gates"}, + + // Output heads and final projections. The head emits the distribution the + // sampler draws audio tokens from, with no later layer to absorb the error. In + // every package inspected the head has exactly the shape of the embedding + // table, which this packer already stores at F16 — so keeping the head at F16 + // is what makes the two ends of the model agree. + {"*head.weight", "output head: emits the distribution the sampler draws from"}, + {"*heads.weight", "output heads: emit the distribution the sampler draws from"}, + {"*proj_out.weight", "final projection: feeds the vocoder or codec decoder"}, + {"*final_proj.weight", "final projection: feeds the vocoder or codec decoder"}, + {"*final_layer.linear.weight", "DiT final layer: emits the flow-matching output"}, + }; + return rules; +} + +std::string_view gguf_audible_tensor_reason(std::string_view tensor_name) { + const std::string normalized_name = lower_ascii(tensor_name); + for (const auto & rule : gguf_audible_tensor_rules()) { + if (audible_tensor_pattern_matches(normalized_name, rule.pattern)) { + return rule.reason; + } + } + return {}; +} + void set_backend_tensor_from_f32_parallel( ggml_tensor * tensor, std::string_view name, @@ -1994,12 +2117,18 @@ void convert_tensor_sources_to_gguf(const std::vector & input item.name.compare(item.name.size() - 7, 7, ".weight") == 0; const bool is_lookup_table = normalized_name.find("embed") != std::string::npos || normalized_name.find("codebook") != std::string::npos; - const bool can_quantize = !preserve_source_dtype && source_is_float && name_is_weight && item.shape.size() == 2 && - !is_lookup_table && item.shape.back() % ggml_blck_size(requested_type) == 0; + // Shape and dtype say the tensor *could* be quantized; the audible-tensor + // policy says whether it should be. Both fall back to F16, the same way + // lookup tables already do. + const bool quantizable_shape = !preserve_source_dtype && source_is_float && name_is_weight && + item.shape.size() == 2 && item.shape.back() % ggml_blck_size(requested_type) == 0; + const bool is_audible_tensor = quantizable_shape && !options.quantize_audible_tensors && + !gguf_audible_tensor_reason(item.name).empty(); + const bool can_quantize = quantizable_shape && !is_lookup_table && !is_audible_tensor; const bool use_requested = !preserve_source_dtype && (!ggml_is_quantized(requested_type) ? source_is_float && !item.shape.empty() : can_quantize); const bool use_f16_lookup = !preserve_source_dtype && ggml_is_quantized(requested_type) && - source_is_float && is_lookup_table; + source_is_float && (is_lookup_table || is_audible_tensor); const auto type_override = find_tensor_type_override(item.name, options.type_overrides); const bool use_override = type_override.has_value() && *type_override != TensorStorageType::Native; const ggml_type override_type = use_override ? ggml_type_for_tensor_storage(*type_override) : source_type; diff --git a/tests/unittests/test_quantization_policy.cpp b/tests/unittests/test_quantization_policy.cpp new file mode 100644 index 000000000..5c479c1e3 --- /dev/null +++ b/tests/unittests/test_quantization_policy.cpp @@ -0,0 +1,301 @@ +#include "engine/framework/assets/tensor_source.h" + +#include "engine/framework/io/safetensors.h" + +#include "test_assert.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +using engine::assets::GgufConversionOptions; +using engine::assets::TensorStorageType; +using engine::test::require; +using engine::test::require_eq; + +// Every test tensor is [4, 64]: rank 2, last dimension divisible by the Q8_0 block +// size of 32, so the packer's shape rule alone would quantize all of them. What each +// one is stored as is therefore decided by the name policy and nothing else. +constexpr int64_t kRows = 4; +constexpr int64_t kCols = 64; + +// ggml_row_size(Q8_0, 64) = 2 blocks x sizeof(block_q8_0) = 2 x 34 = 68 bytes per row. +constexpr size_t kQ8Bytes = 4 * 68; +constexpr size_t kF16Bytes = 4 * 64 * 2; +constexpr size_t kF32Bytes = 4 * 64 * 4; + +std::vector f32_payload(int64_t elements) { + std::vector bytes(static_cast(elements) * sizeof(float)); + for (int64_t index = 0; index < elements; ++index) { + const float value = static_cast(index % 17) * 0.25F - 2.0F; + std::memcpy(bytes.data() + static_cast(index) * sizeof(float), &value, sizeof(float)); + } + return bytes; +} + +engine::io::SafeTensorWriteEntry matrix(const std::string & name) { + return {name, "F32", {kRows, kCols}, f32_payload(kRows * kCols)}; +} + +engine::io::SafeTensorWriteEntry vector_1d(const std::string & name) { + return {name, "F32", {kCols}, f32_payload(kCols)}; +} + +engine::io::SafeTensorWriteEntry conv_kernel(const std::string & name) { + return {name, "F32", {4, 4, 7}, f32_payload(4 * 4 * 7)}; +} + +// --- names ------------------------------------------------------------------ +// Ordinary mid-stack weights. These MUST stay quantized: they have the rest of the +// network downstream to absorb the error, and protecting them would grow every +// package the project ships. +constexpr const char * kFfnWeight = "model.layers.0.mlp.down_proj.weight"; +constexpr const char * kAttentionWeight = "model.layers.0.self_attn.q_proj.weight"; +// A transformer stack that merely has "decoder" in its path. Deliberately NOT +// protected -- the RedAE decoder's own iSTFT head is what needs protecting, and a +// blanket "decoder" exclusion would move a whole 24-layer Qwen stack to F16. +constexpr const char * kCodecDecoderBackbone = "redae.decoder.qwen2.layers.0.mlp.down_proj.weight"; +// Analysis side of an audio tokenizer. Its error is absorbed by the decoder that +// follows it, so it stays quantized while the matching decoder tensor does not. +constexpr const char * kTokenizerEncoder = "model.acoustic_tokenizer.encoder.stages.0.ffn.linear1.weight"; + +// Lookup tables. Pre-existing behaviour, demoted to F16 rather than quantized. +constexpr const char * kEmbedding = "model.embed_tokens.weight"; +constexpr const char * kCodebook = "quantizer.quantizers.0.codebook.weight"; + +// Audible tensors, newly protected. +constexpr const char * kIstftHead = "redae.decoder.istft_head.out.weight"; +constexpr const char * kVocoderLinear = "vocoder.resblocks.0.linear.weight"; +constexpr const char * kNormProjection = "flow.transformer.layers.0.attention_norm.project_layer.weight"; +constexpr const char * kAdaLnModulation = "flow.final_layer.adaLN_modulation.1.weight"; +constexpr const char * kLmHead = "lm_head.weight"; +constexpr const char * kMelHead = "gpt.mel_head.weight"; +constexpr const char * kProjOut = "proj_out.weight"; +constexpr const char * kTokenizerDecoder = "model.acoustic_tokenizer.decoder.stages.0.ffn.linear1.weight"; + +// Shapes the policy must not touch. +constexpr const char * kLayerNorm = "model.layers.0.input_layernorm.weight"; +constexpr const char * kBias = "model.layers.0.mlp.down_proj.bias"; +constexpr const char * kConvKernel = "vocoder.conv_pre.weight"; + +std::filesystem::path scratch_root() { + return std::filesystem::temp_directory_path() / "audiocpp_quantization_policy_test"; +} + +std::filesystem::path write_source(const std::filesystem::path & root) { + std::filesystem::create_directories(root); + const std::vector entries{ + matrix(kFfnWeight), + matrix(kAttentionWeight), + matrix(kCodecDecoderBackbone), + matrix(kTokenizerEncoder), + matrix(kEmbedding), + matrix(kCodebook), + matrix(kIstftHead), + matrix(kVocoderLinear), + matrix(kNormProjection), + matrix(kAdaLnModulation), + matrix(kLmHead), + matrix(kMelHead), + matrix(kProjOut), + matrix(kTokenizerDecoder), + vector_1d(kLayerNorm), + vector_1d(kBias), + conv_kernel(kConvKernel), + }; + const auto path = root / "model.safetensors"; + engine::io::write_safetensors_file(path, entries); + return path; +} + +// Packs the fixture at Q8_0 with `options` and returns the result, opened for reading. +std::shared_ptr pack( + const std::filesystem::path & root, + const std::filesystem::path & source, + const std::string & output_name, + GgufConversionOptions options) { + const auto output = root / output_name; + engine::assets::convert_tensor_sources_to_gguf( + {{source, ""}}, + output, + TensorStorageType::Q8_0, + /*overwrite=*/true, + /*embed_sidecars=*/false, + root, + {}, + std::nullopt, + std::move(options)); + return engine::assets::open_tensor_source(output); +} + +void require_stored_as( + const engine::assets::TensorSource & packed, + const char * name, + const std::string & dtype, + size_t bytes) { + require_eq(packed.require_metadata(name).dtype, dtype, std::string("dtype of ") + name); + require_eq(packed.require_tensor_data(name).bytes.size(), bytes, std::string("byte size of ") + name); +} + +// The name predicate on its own, with no conversion involved. This is the surface a +// contributor reads and adds to, so it is asserted directly as well as through the +// packer. +void test_audible_tensor_reasons() { + require(!engine::assets::gguf_audible_tensor_rules().empty(), "the exclusion list is not empty"); + + require(!engine::assets::gguf_audible_tensor_reason(kIstftHead).empty(), "istft head is audible"); + require(!engine::assets::gguf_audible_tensor_reason(kVocoderLinear).empty(), "vocoder linear is audible"); + require(!engine::assets::gguf_audible_tensor_reason(kNormProjection).empty(), "norm projection is audible"); + require(!engine::assets::gguf_audible_tensor_reason(kAdaLnModulation).empty(), "adaLN modulation is audible"); + require(!engine::assets::gguf_audible_tensor_reason(kLmHead).empty(), "lm head is audible"); + require(!engine::assets::gguf_audible_tensor_reason(kMelHead).empty(), "mel head is audible"); + require(!engine::assets::gguf_audible_tensor_reason(kProjOut).empty(), "proj_out is audible"); + require(!engine::assets::gguf_audible_tensor_reason(kTokenizerDecoder).empty(), "tokenizer decoder is audible"); + + require(engine::assets::gguf_audible_tensor_reason(kFfnWeight).empty(), "an FFN weight is not audible"); + require(engine::assets::gguf_audible_tensor_reason(kAttentionWeight).empty(), "an attention weight is not audible"); + require( + engine::assets::gguf_audible_tensor_reason(kCodecDecoderBackbone).empty(), + "a decoder-side transformer stack is not audible"); + require( + engine::assets::gguf_audible_tensor_reason(kTokenizerEncoder).empty(), + "the analysis side of a tokenizer is not audible"); + + // Matching is case-insensitive and covers the whole name, not just a prefix. + require( + !engine::assets::gguf_audible_tensor_reason("MODEL.DECODER.ISTFT_HEAD.OUT.WEIGHT").empty(), + "matching is case-insensitive"); + // "shift" contains "hift"; the HiFT rules are anchored so it must not match. + require( + engine::assets::gguf_audible_tensor_reason("dit.audio_scale_shift_table.weight").empty(), + "a scale-shift table is not mistaken for a HiFT vocoder"); +} + +// The behaviour that already shipped. If any of this changes, every published GGUF +// stops matching what the converter would now produce. +void test_existing_behaviour_is_unchanged(const engine::assets::TensorSource & packed) { + require_stored_as(packed, kFfnWeight, "q8_0", kQ8Bytes); + require_stored_as(packed, kAttentionWeight, "q8_0", kQ8Bytes); + require_stored_as(packed, kCodecDecoderBackbone, "q8_0", kQ8Bytes); + require_stored_as(packed, kTokenizerEncoder, "q8_0", kQ8Bytes); + + // Lookup tables were, and remain, demoted to F16 rather than quantized. + require_stored_as(packed, kEmbedding, "f16", kF16Bytes); + require_stored_as(packed, kCodebook, "f16", kF16Bytes); + + // The shape rule keeps rank-1 and rank-3 tensors at their source dtype. The new + // name policy must not pull them down to F16 -- `vocoder.conv_pre.weight` matches + // a vocoder rule by name and must still come out F32. + require_stored_as(packed, kLayerNorm, "f32", kCols * 4); + require_stored_as(packed, kBias, "f32", kCols * 4); + require_stored_as(packed, kConvKernel, "f32", 4 * 4 * 7 * 4); +} + +void test_audible_tensors_are_not_quantized(const engine::assets::TensorSource & packed) { + require_stored_as(packed, kIstftHead, "f16", kF16Bytes); + require_stored_as(packed, kVocoderLinear, "f16", kF16Bytes); + require_stored_as(packed, kNormProjection, "f16", kF16Bytes); + require_stored_as(packed, kAdaLnModulation, "f16", kF16Bytes); + require_stored_as(packed, kLmHead, "f16", kF16Bytes); + require_stored_as(packed, kMelHead, "f16", kF16Bytes); + require_stored_as(packed, kProjOut, "f16", kF16Bytes); + require_stored_as(packed, kTokenizerDecoder, "f16", kF16Bytes); +} + +// A packager who knows better can still quantize one protected tensor by name. +void test_per_tensor_override_wins( + const std::filesystem::path & root, + const std::filesystem::path & source) { + GgufConversionOptions options; + options.type_overrides.push_back({kIstftHead, TensorStorageType::Q8_0}); + const auto packed = pack(root, source, "override.gguf", std::move(options)); + + require_stored_as(*packed, kIstftHead, "q8_0", kQ8Bytes); + // Only the named tensor moves; the rest of the policy still holds. + require_stored_as(*packed, kVocoderLinear, "f16", kF16Bytes); + require_stored_as(*packed, kLmHead, "f16", kF16Bytes); + require_stored_as(*packed, kFfnWeight, "q8_0", kQ8Bytes); +} + +// The whole-policy escape hatch reproduces the pre-policy output exactly. +void test_policy_can_be_disabled( + const std::filesystem::path & root, + const std::filesystem::path & source) { + GgufConversionOptions options; + options.quantize_audible_tensors = true; + const auto packed = pack(root, source, "unprotected.gguf", std::move(options)); + + require_stored_as(*packed, kIstftHead, "q8_0", kQ8Bytes); + require_stored_as(*packed, kVocoderLinear, "q8_0", kQ8Bytes); + require_stored_as(*packed, kNormProjection, "q8_0", kQ8Bytes); + require_stored_as(*packed, kAdaLnModulation, "q8_0", kQ8Bytes); + require_stored_as(*packed, kLmHead, "q8_0", kQ8Bytes); + require_stored_as(*packed, kTokenizerDecoder, "q8_0", kQ8Bytes); + + // Disabling the audible-tensor policy must not disable the lookup-table rule. + require_stored_as(*packed, kEmbedding, "f16", kF16Bytes); + require_stored_as(*packed, kCodebook, "f16", kF16Bytes); + require_stored_as(*packed, kConvKernel, "f32", 4 * 4 * 7 * 4); +} + +// A 16-bit target quantizes nothing, so the policy must be inert there: every float +// weight still lands at BF16 and no tensor is pushed to F16 behind the packager's back. +void test_policy_is_inert_for_non_quantized_targets( + const std::filesystem::path & root, + const std::filesystem::path & source) { + const auto output = root / "bf16.gguf"; + engine::assets::convert_tensor_sources_to_gguf( + {{source, ""}}, + output, + TensorStorageType::BF16, + /*overwrite=*/true, + /*embed_sidecars=*/false, + root, + {}, + std::nullopt, + {}); + const auto packed = engine::assets::open_tensor_source(output); + + require_stored_as(*packed, kIstftHead, "bf16", kF16Bytes); + require_stored_as(*packed, kLmHead, "bf16", kF16Bytes); + require_stored_as(*packed, kFfnWeight, "bf16", kF16Bytes); + require_stored_as(*packed, kConvKernel, "bf16", 4 * 4 * 7 * 2); +} + +} // namespace + +int main() { + try { + const auto root = scratch_root(); + std::error_code cleanup_error; + std::filesystem::remove_all(root, cleanup_error); + const auto source = write_source(root); + + test_audible_tensor_reasons(); + + const auto packed = pack(root, source, "policy.gguf", {}); + test_existing_behaviour_is_unchanged(*packed); + test_audible_tensors_are_not_quantized(*packed); + + test_per_tensor_override_wins(root, source); + test_policy_can_be_disabled(root, source); + test_policy_is_inert_for_non_quantized_targets(root, source); + + std::filesystem::remove_all(root, cleanup_error); + std::cout << "quantization_policy_test passed\n"; + } catch (const std::exception & ex) { + std::cerr << "quantization_policy_test failed: " << ex.what() << "\n"; + return 1; + } + return 0; +}