diff --git a/.agents/specs/ltx25-a2v-audio-input.md b/.agents/specs/ltx25-a2v-audio-input.md index c434a66535..e91f6d2603 100644 --- a/.agents/specs/ltx25-a2v-audio-input.md +++ b/.agents/specs/ltx25-a2v-audio-input.md @@ -150,11 +150,14 @@ nameable. Upstream's stage 1 is **caller-configured and guided** — upstream's and the trajectory is not, and no claim is made that a render here reproduces upstream's A2Vid output. Recorded under `## Owed`. -**Out, and refused by name rather than dropped.** Resampling: upstream resamples -with `torchaudio.functional.resample` (`ops.py:40`), an arbitrary-ratio polyphase -kaiser resampler; this project ports only the integer-ratio hann-sinc variant, -so a sample rate other than the encoder's is refused with the two rates in the -message. Non-PCM16 and non-RIFF containers: upstream reads anything PyAV opens +**Out, and refused by name rather than dropped.** Resampling was out of this row +and is now IN the tree: row `LTX25-AUDIO-RESAMPLE` +([#2583](https://github.com/mudler/vllm.cpp/issues/2583)) ported +`torchaudio.functional.resample` (`ops.py:40`) and deleted the refusal this +paragraph described. It also corrected the reason: the filter is an +arbitrary-ratio polyphase sinc with a HANN window, not a kaiser one, because +`resample` defaults `resampling_method="sinc_interp_hann"` and `ops.py:40` +passes no override. Non-PCM16 and non-RIFF containers: upstream reads anything PyAV opens (`decode.py:252`), and no demuxer is vendored here (`video_api.cpp:115-121` says so explicitly). MP3/FLAC/OGG are therefore refused by name. `RetakePipeline` (#924) and text-to-audio. @@ -460,8 +463,9 @@ in the waveform's own dtype (`ops.py:54`) and the encoder in the parameters' `AudioEncoderConfigurator.from_metadata` and `decode_audio_from_file` at reduced dimensions, which needs PyAV and a written fixture file, and is why it did not ride here. -- **Arbitrary-ratio resampling** — refused by name; needs the polyphase kaiser - resampler upstream uses at `ops.py:40`. +- ~~**Arbitrary-ratio resampling**~~ — CLOSED by row `LTX25-AUDIO-RESAMPLE` + ([#2583](https://github.com/mudler/vllm.cpp/issues/2583)), which ported + `ops.py:36-42`. The filter it names is hann-windowed, not kaiser. - **Compressed audio containers** — refused by name; no demuxer is vendored. - **`Ltx2CreateAudioLatentState` and `Ltx2ConditionAudioByReference` remain test-only.** Both have exactly one call site each and it is a test diff --git a/.agents/specs/ltx25-audio-resample.md b/.agents/specs/ltx25-audio-resample.md new file mode 100644 index 0000000000..ccfa91cda3 --- /dev/null +++ b/.agents/specs/ltx25-audio-resample.md @@ -0,0 +1,467 @@ +# LTX-2.5 — arbitrary-ratio audio resampling (A19) + +Row: `LTX25-AUDIO-RESAMPLE`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) +(operator-owned; **not edited by this row**). Issue: +[#2583](https://github.com/mudler/vllm.cpp/issues/2583). Plan: +[`ltx25-completion-scope.md`](ltx25-completion-scope.md) §8 order **7**, gap +**A19**, itself issue +[#2526](https://github.com/mudler/vllm.cpp/issues/2526). + +Upstream pins: + +| Reference | Registry id | Revision | +|---|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`) | `ltx-2` | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | +| torchaudio, the module `ops.py:40` calls | — | `2.11.0+cu130`, the wheel the `ltx-2` pin resolves and the goldens were generated with | + +Read from a local checkout at that revision with `git show :`, never +from a working tree, because a working tree can differ from the pin. + +--- + +## 0. Honesty statement — what this row does and does not claim + +This row makes an audio file at **any** sample rate drive an LTX-2.5 render. It +does not claim A3 (`DubItPipeline`), A18 (reference-audio conditioning delivery), +or any IC-LoRA. It unblocks the first two by removing the refusal that stood in +front of both; it does not implement either, and both stay owed by their own +rows. + +**Upstream ships no tests at this pin.** `find . -iname '*test*' -not -path +'./.git/*'` over `Lightricks/LTX-2 @ fd4ded7f` returns empty. So "port the +upstream tests in the same change" has nothing to port from the model author, and +the obligation becomes what §6 does instead: **execute** the upstream module at +the pin and emit its outputs as goldens, plus pin each upstream default against a +`file:line`. + +**No render on real weights is claimed.** The row is gated on reduced-dimension +fixtures and executed-upstream goldens, exactly as `LTX25-A2V-AUDIO-INPUT` was. + +--- + +## 1. Scope + +In: + +* Port `torchaudio.functional.resample` at the defaults `ops.py:40` passes, as an + arbitrary rational-ratio polyphase sinc-hann resampler. +* Call it from `Ltx2WaveformToLogMel`, which is where upstream calls it. +* Drop the two refusals that stand in front of it, and repair the **nine** + statements across this tree that misname upstream's filter as kaiser. + +Out: + +* `sinc_interp_kaiser` and the `beta` knob. Upstream passes neither, and porting + an unreached branch is the shape `## Nothing lands dead` forbids. +* Non-default `lowpass_filter_width`, `rolloff`. Same reason. +* Compressed audio containers, channel mixing, A3, A18. Each is refused or owed + elsewhere and none of them is this gap. + +--- + +## 2. Our baseline — the gap, on this tree + +`Ltx2WaveformToLogMel` refuses any waveform whose rate is not +`config.target_sample_rate` +(`src/vllm/model_executor/models/ltx2_audio_vae.cpp:1051-1060` at this row's base +`6643b2bbf`; the refusal is gone at head, so the anchor names the tree it +describes), and `Ltx2DecodeAudioWav` refuses the same thing one hop earlier so +the message can name the file +(`src/vllm/model_executor/models/ltx2_audio_input.cpp:96-105`, same base). +Between them, **every** audio input at a rate other than the checkpoint's is +turned away, so `a2vid_two_stage` only accepts a take the user resampled with +some other tool first. + +**The refusal misnames the filter, in nine places.** Its message and eight +sibling comments call `torchaudio.functional.resample` "an arbitrary-ratio +polyphase **kaiser** resampler". It is not one. `resample`'s signature defaults +`resampling_method="sinc_interp_hann"` (torchaudio 2.11 +`functional/functional.py:1441`), and `_get_sinc_resample_kernel` builds a kaiser +window only on the `else` of `if resampling_method == "sinc_interp_hann"` +(`:1384-1391`). `ops.py:40` passes neither `resampling_method` nor `beta`, so the +window is `cos(t*pi/lpw/2)**2` — a **hann** window, the same family this tree +already ports for the vocoder's BWE stage at `ltx2_audio_vae.cpp:440` +(`Ltx2HannSincResampleFilter1d`), which is exactly that kernel specialized to +`orig_freq == 1`. + +So what is missing is the **arbitrary rational ratio**, not the window. Sizing +A19 at M was right; the reason written beside it was wrong, and this row corrects +the reason as well as the code. + +The nine sites, read at this row's base `6643b2bbf` because the repair moves +every one of them: `src/vllm/model_executor/models/ltx2_audio_input.cpp:101`, +`src/vllm/model_executor/models/ltx2_audio_vae.cpp:1057`, +`include/vllm/model_executor/models/ltx2_audio_vae_encoder.h:173`, +`include/vllm/multimodal/ltx2_video.h:315`, +`include/vllm/model_executor/models/ltx2_audio_input.h:124`, +`tests/vllm/multimodal/test_ltx2_video.cpp:7518`, +`.agents/specs/ltx25-a2v-audio-input.md:155` and `:463`, and +**`examples/ltx2_gen/main.cpp:179`**. + +That last entry is the row's own record defect and not a tenth discovery. This +paragraph said "nine" and listed **eight** until 2026-09-02; the repair covered +the eight it listed, and the ninth shipped unrepaired into a `--help` string. The +list is now enumerated by +`git grep -inE "polyphase kaiser|kaiser resampl" 6643b2bbf -- .`, which returns +exactly these nine, rather than transcribed. + +--- + +## 3. Upstream chain — read end to end, not inferred + +| Hop | Upstream | What it settles | +|---|---|---| +| 1 | `a2vid_two_stage.py:196` | `decode_audio_from_file` runs at the file's **native** rate | +| 2 | `decode.py:290-296` | the window (`start_time`, `max_duration`) is applied in samples at the **native** rate, before any resample | +| 3 | `decode.py:173-176` | the waveform is **float32**, `astype(np.float32)`, normalised to [-1, 1] | +| 4 | `a2vid_two_stage.py:200` -> `audio_vae.py:271` | `encode_audio` -> `audio_processor.waveform_to_mel(audio.to(device=device))`; `.to(device=...)` moves, it does not cast | +| 5 | `ops.py:44-49` | `waveform_to_mel` calls `self.resample_audio(audio)` **first**, then the mel transform | +| 6 | `ops.py:36-42` | `resample_audio` returns `audio` unchanged when the rates are equal; otherwise `torchaudio.functional.resample(waveform, orig, target)` and then `.to(device=..., dtype=waveform.dtype)` | +| 7 | `functional.py:1470-1476` | `orig_freq <= 0 or new_freq <= 0` raises; `orig_freq == new_freq` returns the input; `gcd` reduces the ratio | +| 8 | `functional.py:1305-1402` | the kernel: `base_freq = min(o, n) * rolloff`, `width = ceil(lpw * o / base_freq)`, one row per output phase | +| 9 | `functional.py:1405-1432` | zero-pad `(width, width + o)`, `conv1d` stride `o`, transpose-interleave, truncate to `ceil(n * L / o)` — with the quotient **narrowed to f32 before the ceil** (`:1427`; `torch.as_tensor` of a Python float takes the default dtype), which keeps one sample fewer than an exact integer ceil at 180697 samples for 44100 -> 16000 | +| 10 | `a2vid_two_stage.py:301-303` | the returned soundtrack is the caller's **original** `Audio`, at its **native** rate. The resample never reaches the output | + +Hop 10 is why nothing downstream of the encoder changes: the file's own rate +stays the render's `sample_rate`, exactly as today. + +--- + +## 4. Port map + +| Upstream | Ours | +|---|---| +| `functional.py:1305-1402` `_get_sinc_resample_kernel` | `Ltx2SincResampleKernel` (file-local), `ltx2_audio_vae.cpp` | +| `functional.py:1405-1432` `_apply_sinc_resample_kernel` | `Ltx2ResampleWaveform`, `ltx2_audio_vae.cpp` | +| `functional.py:1435-1490` `resample` | `Ltx2ResampleWaveform`'s guards and early return | +| `ops.py:36-42` `AudioProcessor.resample_audio` | the call at the head of `Ltx2WaveformToLogMel` | + +`Ltx2ResampleWaveform` is declared in +`include/vllm/model_executor/models/ltx2_audio_vae_encoder.h` beside the mel +front-end it belongs to, because upstream keeps them in one module (`ops.py`). + +### The dtype, and why it is `float` and not `double` + +Upstream resamples in **float32**: hop 3 makes the waveform `np.float32`, hop 4 +does not cast it, and `_get_sinc_resample_kernel` is called with +`dtype=waveform.dtype`, so **the filter itself is built in float32** — `idx`, +`t`, the clamp, the window, `sin(t)/t` and the scale, all of it +(`functional.py:1376-1397`). + +This is not a detail to round up. Building the same filter in `double` and +storing it as `float` is *more accurate than upstream* and therefore **further +from it**: measured against the pinned oracle, a double-built kernel differs by +up to **3.39746e-06** where a float-built one differs by **1.19209e-07**, a +factor of 28 and 13.6 times the gate's own 2.5e-07 bound. AGENTS.md's *Inherit vLLM defaults* names exactly this failure — a dtype that +is too wide, which no token gate can see. The port mirrors the float32 +arithmetic operation for operation, including the association `sinc * (window * +scale)` (`:1397`) and the fact that `scale = base_freq / orig_freq` is computed +in Python `float` (f64) and *then* narrowed (`:1395`). + +The one deliberate exception is the convolution **accumulator**, which is +`double`. torch's `conv1d` reduction order over 161 taps is a vectorised +implementation detail that no C++ loop reproduces, so the choice is between two +different roundings; the exact one is chosen and annotated. It is an accumulator, +not a stored dtype: every stored value stays `float`. + +### The one shape decision + +`Ltx2WaveformToLogMel` takes `[channels, samples]` channel-major. torchaudio +packs the batch and resamples the last axis, so each channel resamples +independently against the same kernel. The port builds the kernel once and walks +the channels, which is that, not an approximation of it. + +--- + +## 5. Reachability — the sentence the records must carry + +The production entry point is `vllm_video_generate` -> `VideoEngine::Generate` +-> `Ltx2VideoEngine::Generate` -> `Ltx2DecodeAudioWav` +(`src/vllm/multimodal/ltx2_video.cpp:3063`) -> `Ltx2EncodeAudioToLatent` +(`:3068`) -> `Ltx2WaveformToLogMel` +(`src/vllm/model_executor/models/ltx2_audio_input.cpp:200`) -> the new +`Ltx2ResampleWaveform`. Every hop already exists and is already gated; this row +adds the last one and deletes the two refusals that stood in the chain. + +The smallest failing test therefore enters through `LoadVideoEngine` + +`engine->Generate` with a WAV at a rate the fixture's audio VAE does not declare, +per the note at `test_ltx2_video.cpp:7229-7240`. A unit test over +`Ltx2ResampleWaveform` alone would prove the function works and never that a +request can arrive at it. + +**Mutation:** delete the `Ltx2ResampleWaveform` call in `Ltx2WaveformToLogMel` +and the video-level case must red. Recorded in `## Outcome`. + +--- + +## 6. Tests to port + +Upstream ships none (§0). What this change does instead: + +1. **Goldens by execution.** `scripts/gen-ltx2-vae-goldens.py` gains sections + **8d**, **8e** and **8f**, importing `ltx_core.model.audio_vae.ops.AudioProcessor` + from the `--ltx2` checkout and running `resample_audio` and `waveform_to_mel`. + They ride the generator's existing `kLtx2VaeUpstreamRevision` anchor, which the + C++ suite already asserts against a pinned SHA, so a regeneration against a + different upstream fails the gate rather than replacing the oracle. +2. **The rate pairs are chosen to reach distinct arms**, not for coverage + arithmetic. What section 8d ships is 16000 -> 48000 (pure upsample, `o = 1`, + the degenerate ratio the tree already ports for the vocoder's BWE stage), + 48000 -> 16000 (pure downsample, `o = 3`, `n = 1`, `width = 19`, one phase + row), 44100 -> 16000 (`gcd = 100`, `o = 441`, `n = 160`, `width = 17`, the + widest kernel and the pair a real 44.1 kHz take hits), and 16000 -> 16000 (the + equal-rate early return, which must be a **copy** and not a filtered pass). + This paragraph named a different four before the port ran; the shipped set is + the one above, and `RESAMPLE_CASES` in the generator is its record. +2b. **Section 8f reaches the truncation boundary, which 8d cannot.** The 8d arms + top out at 218 output samples, and upstream's f32-narrowed ceil (hop 9) first + disagrees with an exact integer ceil at 180697 input samples — 4.097 s at + 44.1 kHz, three orders of magnitude further out. 8f carries four LENGTH-plus- + tail goldens: 180696 / 180697 / 180698 at 44100 -> 16000, which bracket that + boundary, and 90569 at 22050 -> 16000, where the same effect starts. The + generator asserts which of them an exact integer ceil gets right and which it + does not, so an arm that stopped discriminating fails generation rather than + gating nothing. +3. **A lower bound, not only a tolerance.** A resampler that returned zeros, or + that returned the input untouched, satisfies a `max|diff| < tol` gate against + the wrong golden and any shape check. The cases assert the output's absmax is + positive **and** that a resampled take differs from the same samples read as + if they were already at the target rate — the exact wrong answer the refusal + was written to prevent. +4. **The end-to-end case**, at `test_ltx2_video.cpp`, replaces the SUBCASE that + asserted the refusal: a 44.1 kHz WAV now renders, and its conditioning trace + carries a non-zero audio latent whose digest **differs** from the same file + written at the fixture's own rate. + +--- + +## 7. Gates + +```sh +cmake --build build -j 4 --target test_ltx2_vae test_ltx2_video +./build/tests/test_ltx2_vae -tc='*resampl*' +./build/tests/test_ltx2_video -tc='*audio*' +scripts/agent-preflight.sh --staged +``` + +Plus regeneration parity: + +```sh +python3 scripts/gen-ltx2-vae-goldens.py --ltx2 ~/_git/LTX-2 \ + --out tests/vllm/models/ltx2_vae_goldens.inc +git diff --exit-code tests/vllm/models/ltx2_vae_goldens.inc +``` + +--- + +## 8. Dependencies + +None. A19 depends on nothing in §8 of the plan; A3 and A18 depend on it. + +--- + +## 9. Work breakdown + +1. Spec (this file), committed first. +2. Generator sections 8d/8e and the regenerated `.inc`. +3. `test_ltx2_vae.cpp` cases, red before the port. +4. `Ltx2ResampleWaveform` plus the `Ltx2WaveformToLogMel` call. +5. Delete the two refusals; repair the nine kaiser statements. +6. `test_ltx2_video.cpp`: the refusal SUBCASE becomes the render case. +7. `docs/FEATURES.md` and `docs/USAGE.md` where they state the rate constraint. + +--- + +## 10. Risks/decisions + +| # | Risk | Handling | +|---|---|---| +| R1 | A `double` filter reads as "better" and silently widens the model path | §4 measures both against the oracle and pins float32 with the number. The gate tolerance is tight enough (2.5e-07) that a double-built kernel **fails** it | +| R2 | Removing a refusal admits an input that then fails deeper, with a worse message | The window and channel refusals stay; only the rate one goes. The `samples > n_fft/2` check in `Ltx2WaveformToLogMel` now sees the **resampled** length, which is upstream's order (hop 5) | +| R3 | A tolerance-only gate passes a resampler that returns the input | §6.3's lower bound and difference assertions | +| R4 | The nine kaiser statements are repaired in prose but the sizing document keeps the claim | `ltx25-completion-scope.md` is another row's spec and is not edited here; the correction is recorded on #2583 and in §2 | +| D1 | Should `Ltx2DecodeAudioWav` keep `want_sample_rate`? | No. Upstream's decoder has no target rate (hop 1); the parameter existed only to raise. Removed, and the one call site updated | + +--- + +## Now + +`DONE` pending review. Landed on `row/LTX25-AUDIO-RESAMPLE`, issue #2583. + +--- + +## Outcome + +### What was measured, and what it decided + +**The dtype, which was the one decision this port could have got quietly wrong.** +Three filters were built and compared against the pinned oracle's own +`AudioProcessor.resample_audio` output at four rate pairs: + +| Filter built in | max abs diff vs the oracle | +|---|---| +| `double` throughout `Ltx2SincResampleKernel`, narrowed to `float` at the store | **3.39746e-06** | +| torchaudio's own `dtype=None` kernel (`idx` in f64, `t` in the default f32) | 2.37e-06 | +| `float`, mirroring `dtype=waveform.dtype` | **1.19209e-07** | + +All three re-measured on 2026-09-02, because the first two rows were first +recorded as 1.03e-05 and 4.71e-06 and neither reproduces. The mutation each row +names is exact: row 1 makes `base_freq`, `scale`, `lpw`, `kPi`, `phase`, `idx`, +`t`, `shaped`, `window` and `sinc` `double` and casts only at +`kernel[j * taps + k] = ...`; it reds the `Wide` arm at 3.39746e-06 and all four +8f tails between 1.55e-06 and 3.09e-06. Row 2 is torchaudio's own +`_get_sinc_resample_kernel(..., dtype=None)` over the same 8d inputs. Row 3 is +this port at the committed head, read by setting `kResampleTol` to `0.0`. + +The `double` filter is *more accurate than upstream* and therefore twenty-eight +times further from it. It would have passed a gate written around it and read as +a careful implementation. §4 states the chain that fixes the answer at f32, and +the gate's 2.5e-07 is set so that a widening fails it. + +**The tolerance is twice the measured floor, not a hedge.** Setting +`kResampleTol` to `0.0` reds three of the four 8d arms, at 5.96e-08 (Up), +5.96e-08 (Down) and 1.19209e-07 (Wide), and all four 8f tails, at 1.49e-08, +4.47e-08, 2.98e-08 and 7.45e-09. `Same` passes at zero, because it is a copy. +2.5e-07 is 2.10x the worst of those. + +**`audio_latent_absmax` cannot gate this, and that is a finding rather than a +gap.** The obvious video-level claim — "the 44.1 kHz take must land nearer the +24 kHz one than the mis-read take does" — was written, run, and **failed**: the +three takes read 1.07194, 1.07293 and 1.07208, a 0.1% spread over three +genuinely different waveforms, because the trace's absmax is dominated by the +encoder's per-channel statistics. The assertion was removed rather than inverted +or loosened, and the reason is recorded where the assertion was. + +### The truncation was NOT upstream's, and no arm here could see it + +`_apply_sinc_resample_kernel` ends +`target_length = torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` +(`functional.py:1427`). That reads as an integer ceil and this port first shipped +one, `(next * samples + orig - 1) / orig`. It is not one: `torch.as_tensor` of a +**Python float** takes `torch.get_default_dtype()`, which is float32, so the f64 +quotient is rounded to f32 **before** the ceil. Where the exact quotient sits +above an integer by less than half an f32 ulp, the narrowing lands on that +integer and upstream keeps one sample FEWER. + +Measured at 44100 -> 16000: 180696 samples give 65559 both ways, 180697 give +65559 upstream and 65560 from the exact ceil, 180698 give 65560 both ways. 48102 +of the first 60 s worth of lengths diverge, and the density grows with duration. +22050 -> 16000 starts at 90569 and 44100 -> 24000 at 240854. At 48000 -> 16000 +(`o = 3`) and 16000 -> 48000 (`o = 1`) it never happens at any audio length, +which is why three of the four 8d arms look clean at any length. One extra output +sample changes the last mel frames' STFT window content and, where +`samples % hop == 0`, the frame count — the audio latent length, and so the +conditioning shape. + +**The gate could not see it, and that is the reason 8f exists.** The four 8d +arms top out at 218 output samples. Substituting torchaudio's real float32 ceil +into this port and rebuilding left `*resampl*` 26/26, `*waveform_to_mel*` 9/9 and +`*RESAMPLED*` 14/14 green: the right ratio at the wrong length. Section 8f adds +the four boundary arms of §6.2b, and with them the old formula reds on exactly +the two divergent lengths (65560 vs 65559 and 65720 vs 65719, plus their tails at +0.257 and 0.359) and stays green on the two that bracket them. + +The repair mirrors the computation rather than describing it: the int64 product +`next * samples` is exact as a `double` for any length under 2^53 / next, so +`double(next * samples) / double(orig)` is Python's own correctly-rounded `int / +int`; `static_cast` is `as_tensor`, `std::ceil` on the float is +`torch.ceil`, and the cast back to `int64_t` is `.long()`. Checked against +torchaudio over 276060 (ratio, length) pairs — twelve ratios by 23005 lengths, +covering 1 to 20000, the runs around 90569 and 180697, a thousand lengths either +side of 2^24 where f32 no longer holds every integer, and 2^25: zero +divergences. + +### One more kaiser statement, in product output + +The row's first pass repaired eight of the nine statements that misnamed the +filter. The ninth was `examples/ltx2_gen/main.cpp`'s `--help` text, which is not +a comment: it told users that the sample rate "must equal the audio VAE's mel +front-end rate", that "neither is converted, because upstream resamples with a +polyphase kaiser resampler that is not ported here", and that "both mismatches +are refused". All three were false at that head. Repaired, and the sweep re-run +over every git-tracked file rather than over source and specs: `git grep -inE +"(^|[^a-z])kaiser"` now returns only the vocoder's genuine kaiser-sinc +anti-aliasing filter, this row's own prose about the repair, and unrelated +tokenizer and HuggingFace-org byte matches. + +### Red before, green after + +| | Before | After | +|---|---|---| +| `test_ltx2_vae -tc='*waveform_to_mel*'` | THREW the rate refusal at `ltx2_audio_vae.cpp:1053` (at `6643b2bbf`) | 48/48 cases, 3245/3245 assertions | +| `test_ltx2_video -tc='*RESAMPLED*'` | THREW `'high.wav' is sampled at 44100 Hz ...` | 110/110 cases, 4876/4876 assertions | +| `test_ltx2_vae -tc='*resampl*'`, section 8f | 50/54 with the EXACT INTEGER CEIL in place: `CeilAt` 65560 vs 65559 and its tail 0.256834 out, `CeilAlt` 65720 vs 65719 and its tail 0.358731 out | 54/54 | + +The first two reds are the refusal itself, which is what the row exists to +delete. The third is the truncation repair below, and it is red-first in the +strict sense: the two bracketing arms `CeilBelow` and `CeilAbove` stay GREEN +under the same old formula, so the arm discriminates rather than simply failing. +The same old formula against the test file as it stood BEFORE section 8f leaves +`*resampl*` 26/26, `*waveform_to_mel*` 9/9 and `test_ltx2_video`'s `*RESAMPLED*` +14/14 all green — the measurement that says the gate was blind, rather than an +argument that it was. + +### The reachability mutation, and what it proved + +Deleting the `Ltx2ResampleWaveform` call inside `Ltx2WaveformToLogMel` and +rebuilding: + +* the video-level case reds on `high_trace.audio_latent_digest != + misread_trace.audio_latent_digest` — the two takes carry **byte-identical PCM** + and differ only in the header's rate field, so they produce the same digest + (`18285287296143238670`) exactly when nothing read the rate; +* the `waveform_to_mel` case reds on the frame count first (38 vs 14) and the mel + length second (608 vs 224); +* **the section-8d unit case still PASSES.** That is the point of the mutation: + `Ltx2ResampleWaveform` works whether or not anything calls it, and a test that + constructs it by hand measures the class. The tree was restored byte-for-byte + and both suites re-run green. + +That video count reads 4876 and not the 4870 this row measured before its last +merge from `main`: `origin/main` added the `dims=2` upsampler cases to the same +suite. Re-read after the merge rather than carried across it, because a merge +falsifies a count without touching a line this row owns. + +The mutation was RE-RUN at the truncation-repair head, because the repair moves +the very line the mutation deletes and a mutation proof does not survive an edit +to its own subject. It behaves identically there: 8d/8f 54/54, the mel case red +at 38 vs 14 frames and 608 vs 224, the video case red on the same digest +`18285287296143238670`. `ltx2_audio_vae.cpp` sha256 before and after that +restore: `fd1b68a71594d7bfd28dcb7f16e67e8c0abe91d5c267e089eb0bee3e3755af03` +(the earlier run's subject was the same file at `052047579`, sha256 +`88055b01e9c2bf0bb29d826b2c61dc4c8439a1d8ae6d779a46969b50b0c0212b`). + +The mutation's FIRST build failed `-Werror` on the now-unused `sampling_rate`, +and the three suites then re-ran the STALE binaries and printed green — the +shape in which a mutation that never applied reads as a passing test. Recorded +because the reading it produced was indistinguishable from the real one until +the build's exit status was checked. + +### Rejected + +* **A `double` accumulator for the kernel** — §4, measured above. +* **Porting `sinc_interp_kaiser`** — unreached from `ops.py:40`; it would be a + branch nothing can enter. +* **Keeping `want_sample_rate` on `Ltx2DecodeAudioWav`** — upstream's decoder has + no target rate. The parameter existed only to raise, so it went with the raise. +* **Guarding the resample at the call site** (`if (rate != target)`) rather than + inside `Ltx2ResampleWaveform` — it would have moved upstream's own early return + (`ops.py:38-39`) off the production path, leaving that arm reachable only from + a test. + +--- + +## Owed + +* **A real-checkpoint A2V render from a 44.1 kHz source.** The gate here is + reduced-dimension fixtures plus executed-upstream goldens; a full render needs + the GPU lease and belongs to the campaign's render rows. Inherits + [#2526](https://github.com/mudler/vllm.cpp/issues/2526). +* **The `sinc_interp_kaiser` arm and the `beta` knob.** Unreached from + `ops.py:40`; deliberately not ported (§1). Inherits + [#2526](https://github.com/mudler/vllm.cpp/issues/2526). +* **The audio VAE's f32/bf16 tension.** `ltx2_audio_vae.cpp` argues f32 from + upstream's vocoder float32 pin (`vocoder.py:575-580`) while upstream builds + `AudioDecoder` in bf16. This row does not resolve it and does not contradict + it: the resampler's float32 is argued from `ops.py`'s own chain (§4), not from + that pin. Inherits [#2526](https://github.com/mudler/vllm.cpp/issues/2526). diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 39412fa2be..8b651a82d3 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -196,7 +196,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` and `test_ltx2_device` each gate one direction ALONE, on a forward where the other stream is PRESENT but DISABLED. Swapping the two flags is RED on both arms | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN`, which `modality_scale = 3.0` selects on every video row. On the DEVICE forward too since 2026-08-19 ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 latent upsampler `dims=2` arm | LTX-2.5 spatial upsampler checkpoint whose config sets `dims: 2` | `ltx2 the latent upsampler reproduces upstream on the dims=2 arm` (`test_ltx2_pipeline`) against the EXECUTED upstream module, plus `a dims=2 upsampler checkpoint RENDERS` (`test_ltx2_video`) through `LoadVideoEngine`/`Generate`; removing the fold reds the golden at max|diff| 0.68 while the render stays green, so the value gate and not a shape check is what holds it | No flag: `dims` is read off the checkpoint's own config, so a 2-D upsampler at `--upsampler` just works. Conv2d for `initial_conv`, both ResBlock stacks and `final_conv` (model.py:47); the frame axis folds into the batch (:86), which makes GroupNorm PER FRAME. Frame count preserved, H and W doubled, which is what both spatial call sites require. Two 2-D contradictions upstream cannot run are refused by name and EXECUTED against the real module by the golden generator: `temporal_upsample` (a Conv3d handed a folded 4-D tensor) and `rational_resampler` (five-value unpack at spatial_rational_resampler.py:41) | -| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118) | +| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118). ANY sample rate: the mel front-end resamples (`ops.py:36-42`, #2583) and the soundtrack returns at the FILE's rate | | LTX-2.5 keyframe interpolation (`KeyframeInterpolationPipeline`) | LTX-2.5 DiT + spatial upsampler | `ltx2 keyframe:` cases in `test_ltx2_pipeline` / `test_ltx2_video` via `LoadVideoEngine`+`Generate`: frame 0 APPENDS against a `ti2vid_two_stage` control, the x0 invariant on four arms, the 4096 anchor | `--pipeline-kind keyframe_interpolation`. No frame-0 special case, so `--first-frame` is guidance to interpolate FROM; stage 2's audio leaves. `--lora` REQUIRED. `--last-frame` new (#1191). CPU fixtures | | LTX-2.5 two-stage text/image-to-video (`TI2VidTwoStagesPipeline`) | LTX-2.5 DiT + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video` `ltx2 ti2vid:` cases through `LoadVideoEngine`+`Generate`; the x0 invariant on all FOUR arms, and the 4096 anchor read at two geometries against a res_2s control that moves | `--pipeline-kind ti2vid_two_stage`. Guided half-res stage 1 on the UNADAPTED model, plain Euler; distilled 3-sigma stage 2. `--lora` REQUIRED, no `--audio-path`; stage 1's audio leaves. CPU fixtures, Full-model run owed | | LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in | diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index a038130536..ec7d2d6d58 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -174,14 +174,16 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "--a2v-guidance-scale is the guider\'s modality_scale; a take here rides whichever\n" "recipe the checkpoint resolves, in practice distilled_two_stage, whose sigmas are\n" "fixed. So no claim is made that this reproduces upstream\'s A2Vid output.\n\n" - "The WAV must already match the checkpoint: its sample rate must equal the audio\n" - "VAE\'s mel front-end rate and its channel count the encoder\'s in_channels.\n" - "Neither is converted, because upstream resamples with a polyphase kaiser\n" - "resampler that is not ported here and feeds the file\'s own channel count into a\n" - "fixed-width convolution; both mismatches are refused with both numbers, since a\n" - "resampled-wrong or upmixed take renders a finished clip conditioned on audio you\n" - "never supplied. The take must also be at least as long as the clip: upstream\n" - "truncates a long one and asserts on a short one, so a short one is refused.\n\n" + "ANY sample rate is accepted: a take whose rate differs from the audio VAE\'s\n" + "mel front-end rate is RESAMPLED to it, with upstream\'s own filter — the\n" + "gcd-reduced rational-ratio hann-windowed sinc `torchaudio.functional.resample`\n" + "builds at its defaults, which is what ops.py:40 calls. The CHANNEL COUNT must\n" + "still equal the encoder\'s in_channels and a mismatch is refused with both\n" + "numbers: upstream feeds the file\'s own channel count into a fixed-width\n" + "convolution, so up- or downmixing here would render a finished clip conditioned\n" + "on audio you never supplied. The take must also be at least as long as the clip:\n" + "upstream truncates a long one and asserts on a short one, so a short one is\n" + "refused.\n\n" "--audio-start-time seeks into the file (default 0) and --audio-max-duration caps\n" "how much is read (default: the clip\'s own duration). Either without\n" "--audio-path is refused rather than ignored. The rendered audio.wav is your own\n" diff --git a/include/vllm/model_executor/models/ltx2_audio_input.h b/include/vllm/model_executor/models/ltx2_audio_input.h index 63541f31a9..c2d2dc5362 100644 --- a/include/vllm/model_executor/models/ltx2_audio_input.h +++ b/include/vllm/model_executor/models/ltx2_audio_input.h @@ -118,13 +118,17 @@ Ltx2WavFormat Ltx2ProbeWavFormat(const std::string& bytes); // // `want_channels` is the encoder's `in_channels`; a file that declares anything // else is REFUSED rather than mixed, for the reason in this header's third -// bullet. `want_sample_rate` is the mel front-end's target; a mismatch is -// REFUSED rather than resampled, because this project ports only the -// integer-ratio hann-sinc resampler and upstream's `ops.py:40` is an -// arbitrary-ratio polyphase kaiser one. +// bullet. +// +// THERE IS NO TARGET RATE HERE, and that is upstream's shape rather than an +// omission: `decode_audio_from_file` reads at the file's own rate +// (`decode.py:240-300`) and the resample happens two hops later, inside the mel +// front-end (`ops.py:44-49`). The rendered soundtrack is this decode's output, +// so it stays at the file's rate too (`a2vid_two_stage.py:301-303`). Row +// LTX25-AUDIO-RESAMPLE (#2583) removed the refusal that used to sit here. Ltx2DecodedAudio Ltx2DecodeAudioWav(const std::string& bytes, const std::string& label, - int64_t want_channels, int64_t want_sample_rate, - double start_time, double max_duration); + int64_t want_channels, double start_time, + double max_duration); // `encode_audio` (audio_vae.py:249-274) followed by A2Vid's truncation // (a2vid_two_stage.py:201-202): waveform -> log-mel -> encoder -> keep at most diff --git a/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h b/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h index 7e88063e0b..ac62e14430 100644 --- a/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h +++ b/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h @@ -17,6 +17,7 @@ // model/audio_vae/attention.py:16-55 // (the audio patchifier) <- components/patchifiers.py:287-305 // (per-channel statistics) <- model/audio_vae/ops.py:58-75 +// Ltx2ResampleWaveform <- model/audio_vae/ops.py:36-42 // Ltx2WaveformToLogMel <- model/audio_vae/ops.py:8-55 // Ltx2SlaneyMelFilterbank <- torchaudio.functional.melscale_fbanks // (reached from ops.py:20-34) @@ -165,22 +166,39 @@ struct Ltx2AudioProcessorConfig { std::vector Ltx2SlaneyMelFilterbank(int64_t n_freqs, double f_min, double f_max, int64_t n_mels, int64_t sample_rate); +// `AudioProcessor.resample_audio` (ops.py:36-42), which is +// `torchaudio.functional.resample` at every default: the arbitrary +// rational-ratio polyphase sinc resampler with `lowpass_filter_width = 6`, +// `rolloff = 0.99` and — `ops.py:40` passes neither `resampling_method` nor +// `beta` — a HANN window (functional.py:1441). It is NOT kaiser, which nine +// statements in this tree asserted before row LTX25-AUDIO-RESAMPLE (#2583). +// +// `waveform` is [channels, samples] channel-major and each channel resamples +// independently against one kernel, as torchaudio's packed batch does. Returns +// `ceil(new_freq * samples / orig_freq)` samples per channel — the quotient +// NARROWED TO f32 BEFORE the ceil, because `torch.as_tensor` of a Python float +// takes the default dtype (functional.py:1427), which keeps one sample fewer +// than an exact integer ceil at 180697 samples for 44100 -> 16000 and at 48102 +// of the first 60 s worth of lengths. Returns the INPUT unfiltered when the +// rates already match (ops.py:38-39). Computed in f32, which is the dtype +// upstream resamples in — see the note at the definition. +std::vector Ltx2ResampleWaveform(const std::vector& waveform, int64_t channels, + int64_t samples, int64_t orig_freq, int64_t new_freq, + int64_t* out_samples); + // `AudioProcessor.waveform_to_mel` (ops.py:44-55) at any channel count. -// `waveform` is [channels, samples], channel-major, at `sampling_rate`. +// `source` is [channels, source_samples], channel-major, at `sampling_rate`. // -// A SAMPLE RATE THAT DOES NOT MATCH `target_sample_rate` IS REFUSED BY NAME. -// Upstream resamples with `torchaudio.functional.resample` (ops.py:40), a -// polyphase kaiser resampler for an arbitrary rational ratio; this project ports -// only the integer-ratio hann-sinc variant the BWE stage needs -// (Ltx2HannSincResampleFilter1d). Refusing is deliberate: silently treating -// 44.1 kHz samples as 16 kHz produces audio conditioning that is pitched and -// time-scaled wrong while every shape still checks out. +// A SAMPLE RATE THAT DOES NOT MATCH `target_sample_rate` IS RESAMPLED, not +// refused: `waveform_to_mel` calls `resample_audio` before the mel transform +// (ops.py:49), so the frame count this reports is over the RESAMPLED length. // // Returns [channels, frames, mel_bins] channel-major — the layout // `Ltx2AudioEncoderForward` takes, which is upstream's final // `permute(0, 1, 3, 2)` (ops.py:55). std::vector Ltx2WaveformToLogMel(const Ltx2AudioProcessorConfig& config, - const std::vector& waveform, int64_t channels, - int64_t samples, int64_t sampling_rate, int64_t* out_frames); + const std::vector& source, int64_t channels, + int64_t source_samples, int64_t sampling_rate, + int64_t* out_frames); } // namespace vllm diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 5c778b46be..78cfd54f9b 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -310,13 +310,16 @@ inline constexpr char kLtx2ImageCrfExtra[] = "image_crf"; // the ordinary text-to-video render. // // A 16-bit PCM RIFF/WAVE file whose channel count matches the checkpoint's audio -// VAE encoder `in_channels` and whose sample rate matches its mel front-end. -// Neither is converted: upstream resamples with an arbitrary-ratio polyphase -// kaiser resampler (`ops.py:40`) this project has not ported, and it feeds the -// file's own channel count straight into a conv that declares 2 -// (`model_configurator.py:172`). Both mismatches are refused with both numbers -// in the message, because a resampled-wrong or upmixed-wrong take conditions the -// render on a waveform the caller never supplied and still finishes. +// VAE encoder `in_channels`. ANY SAMPLE RATE is accepted: the mel front-end +// resamples it to the checkpoint's own, exactly as upstream does +// (`ops.py:36-49`, ported by row LTX25-AUDIO-RESAMPLE, #2583). The rendered +// soundtrack still comes back at the FILE's rate, because upstream returns the +// caller's original take rather than a VAE round trip (`:301-303`). +// +// The CHANNEL count is still refused, with both numbers in the message: +// upstream feeds the file's own count straight into a conv that declares 2 +// (`model_configurator.py:172`), so an upmixed-wrong take would condition the +// render on a waveform the caller never supplied and still finish. // // The audio is held FROZEN through every denoise phase — upstream's // `ModalitySpec(frozen=True, noise_scale=0.0)` at `a2vid_two_stage.py:251-256` diff --git a/scripts/gen-ltx2-vae-goldens.py b/scripts/gen-ltx2-vae-goldens.py index 1a5ddbdf76..42bc1a0824 100644 --- a/scripts/gen-ltx2-vae-goldens.py +++ b/scripts/gen-ltx2-vae-goldens.py @@ -17,7 +17,8 @@ model/video_vae/conv_video_decoder.py -> section 5 (Conv video decoder) model/video_vae/video_vae.py -> section 6 (VideoEncoder) model/audio_vae/audio_vae.py -> section 7 (AudioEncoder) - model/audio_vae/ops.py -> section 8 (AudioProcessor mel front-end) + model/audio_vae/ops.py -> section 8 (AudioProcessor mel front-end, + and 8d-8f its RESAMPLER) conditioning/types/*.py -> section 9 (the conditioning items) Usage: @@ -1007,6 +1008,69 @@ def noncausal_randn(*args, **kwargs): MEL = dict(target_sample_rate=16000, mel_bins=8, mel_hop_length=16, n_fft=64) MEL_SAMPLES = 200 +# Section 8d — `AudioProcessor.resample_audio` (ops.py:36-42). Four rate pairs, +# each chosen for an ARM rather than for coverage arithmetic: +# +# (Up) 16000 -> 48000, gcd 16000 => o = 1. The degenerate ratio the tree +# already ports for the vocoder's BWE stage, so a port that only +# handles this one still passes here and fails the next two. +# (Down) 48000 -> 16000, o = 3, n = 1. `width` is 19 and the kernel has ONE +# phase row, which is the transpose of the arm above. +# (Wide) 44100 -> 16000, gcd 100 => o = 441, n = 160, `width` 17. The widest +# kernel and the only pair here whose `t == 0` tap is not at an obvious +# index. This is also the pair a real 44.1 kHz take hits. +# (Same) 16000 -> 16000. `resample` returns the INPUT (functional.py:1473), +# and `resample_audio` returns before it is even called (ops.py:38-39). +# A port that filtered anyway would be wrong by the filter's own +# passband ripple — small enough to pass a loose tolerance. +RESAMPLE_CASES = ( + ("Up", 16000, 48000, 64), + ("Down", 48000, 16000, 192), + ("Wide", 44100, 16000, 600), + ("Same", 16000, 16000, 64), +) +RESAMPLE_CHANNELS = 2 + +# Section 8e — `waveform_to_mel` at a rate the processor does NOT target, which +# is the production shape: `resample_audio` runs first and the mel transform then +# sees the RESAMPLED length (ops.py:44-49). 600 samples at 44100 resample to 218 +# at 16000, comfortably past the `n_fft // 2` reflect-pad floor. +MEL_SOURCE_RATE = 44100 +MEL_SOURCE_SAMPLES = 600 + +# Section 8f — the TRUNCATION BOUNDARY, which no arm above can reach. +# +# `_apply_sinc_resample_kernel` truncates to +# `torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` +# (functional.py:1427). `torch.as_tensor` of a PYTHON FLOAT takes +# `torch.get_default_dtype()`, which is float32, so the f64 quotient is narrowed +# to f32 BEFORE the ceil. Where the exact quotient sits above an integer by less +# than half an f32 ulp, the narrowing lands ON that integer and upstream keeps one +# sample FEWER than an exact integer ceil would. +# +# The four arms of 8d top out at 218 output samples, three orders of magnitude +# below where that first happens, so they cannot see it: at 44100 -> 16000 the +# first divergent length is 180697 (4.097 s) and 48102 of the first 60 s worth of +# lengths diverge. One extra output sample moves the last STFT windows and, where +# `samples % hop == 0`, the mel FRAME COUNT — that is, the conditioning shape. +# +# Two ratios, because the effect needs `orig_freq` large enough for the quotient +# to land that close: at 48000 -> 16000 (o = 3) and 16000 -> 48000 (o = 1) it +# never happens at any audio length, which is why the clean-looking arms are +# clean. Each ratio carries the first divergent length and, for 44100, the +# lengths either side of it, where an exact ceil is RIGHT — an arm that only +# carried divergent lengths could be satisfied by always subtracting one. +RESAMPLE_CEIL_CASES = ( + ("CeilBelow", 44100, 16000, 180696, False), + ("CeilAt", 44100, 16000, 180697, True), + ("CeilAbove", 44100, 16000, 180698, False), + ("CeilAlt", 22050, 16000, 90569, True), +) +# The goldens carry the LENGTH and the last few samples rather than 65 559 floats +# per arm. The length is the discriminator; the tail is there so a port that +# produced the right count from a shifted signal cannot pass on the count alone. +RESAMPLE_CEIL_TAIL = 8 + def section_video_encoder(out) -> None: import torch @@ -1319,6 +1383,102 @@ def section_audio_mel(out) -> None: out.write("\n") emit_f32(out, "kLtx2MelQuietGolden", quiet.numpy()) + # --- 8d: resample_audio on its own, at four ratios (ops.py:36-42) --- + # + # Run through `AudioProcessor.resample_audio` rather than through + # `torchaudio.functional.resample` directly, so the goldens carry upstream's + # OWN call — its argument order, its equal-rate early return, and its + # `.to(dtype=waveform.dtype)` — and not this generator's reading of it. + out.write("// --- section 8d: AudioProcessor.resample_audio (ops.py:36-42) ---\n") + emit_scalar(out, "kLtx2ResampleChannels", RESAMPLE_CHANNELS) + for tag, orig, target, length in RESAMPLE_CASES: + wave = make_input( + f"ltx2.resample.{tag}", (1, RESAMPLE_CHANNELS, length), 0.5 + ) + proc = AudioProcessor( + target_sample_rate=target, + mel_bins=MEL["mel_bins"], + mel_hop_length=MEL["mel_hop_length"], + n_fft=MEL["n_fft"], + ) + got = proc.resample_audio(Audio(waveform=wave, sampling_rate=orig)) + assert got.sampling_rate == target + assert got.waveform.dtype == torch.float32, ( + "upstream resamples in the WAVEFORM's dtype; a golden emitted from " + "anything but float32 would gate the wrong arithmetic" + ) + if orig == target: + assert got.waveform.data_ptr() == wave.data_ptr(), ( + "ops.py:38-39 returns the SAME Audio when the rates match; a " + "golden that went through the filter would not gate that branch" + ) + assert float(got.waveform.abs().max()) > 0.0, "an all-zero golden gates nothing" + emit_scalar(out, f"kLtx2Resample{tag}OrigRate", orig) + emit_scalar(out, f"kLtx2Resample{tag}NewRate", target) + emit_scalar(out, f"kLtx2Resample{tag}InSamples", length) + emit_scalar(out, f"kLtx2Resample{tag}OutSamples", got.waveform.shape[-1]) + emit_f32(out, f"kLtx2Resample{tag}Golden", got.waveform.numpy()) + + # --- 8e: waveform_to_mel THROUGH the resampler, the production shape --- + # + # 8d proves the filter; this proves the CALL. `waveform_to_mel` resamples + # first (ops.py:49) and the mel transform then runs on the resampled length, + # so a port that resampled AFTER the transform, or that padded before it, + # matches 8d exactly and fails here. + source = make_input("ltx2.mel.resampled.input", (1, 2, MEL_SOURCE_SAMPLES), 0.5) + resampled_mel = processor.waveform_to_mel( + Audio(waveform=source, sampling_rate=MEL_SOURCE_RATE) + ) + out.write("// --- section 8e: waveform_to_mel at a NON-target rate (ops.py:44-55) ---\n") + emit_scalar(out, "kLtx2MelSourceRate", MEL_SOURCE_RATE) + emit_scalar(out, "kLtx2MelSourceSamples", MEL_SOURCE_SAMPLES) + emit_scalar(out, "kLtx2MelResampledFrames", resampled_mel.shape[2]) + out.write("\n") + emit_f32(out, "kLtx2MelResampledGolden", resampled_mel.numpy()) + + # --- 8f: the truncation boundary (functional.py:1427) --- + # + # Same call as 8d, at lengths that reach where upstream's f32-narrowed ceil + # and an exact integer ceil disagree. Emitted as a length plus a tail window, + # because the full arm is 65 559 samples wide. + out.write("// --- section 8f: the truncation boundary (functional.py:1427) ---\n") + emit_scalar(out, "kLtx2ResampleCeilTail", RESAMPLE_CEIL_TAIL) + for tag, orig, target, length, diverges in RESAMPLE_CEIL_CASES: + wave = make_input(f"ltx2.resample.{tag}", (1, 1, length), 0.5) + proc = AudioProcessor( + target_sample_rate=target, + mel_bins=MEL["mel_bins"], + mel_hop_length=MEL["mel_hop_length"], + n_fft=MEL["n_fft"], + ) + got = proc.resample_audio(Audio(waveform=wave, sampling_rate=orig)) + assert got.sampling_rate == target + assert got.waveform.dtype == torch.float32 + produced = int(got.waveform.shape[-1]) + # The POSITIVE CONTROL on this arm. An exact integer ceil is what a + # reader writes from the formula as printed; assert here which lengths it + # gets right and which it does not, so an arm that stopped discriminating + # fails the GENERATOR rather than quietly gating nothing. + exact_ceil = -(-((target // math.gcd(orig, target)) * length) // + (orig // math.gcd(orig, target))) + if diverges: + assert exact_ceil == produced + 1, ( + f"{tag}: expected upstream to keep one sample FEWER than an exact " + f"integer ceil, got upstream={produced} exact={exact_ceil}" + ) + else: + assert exact_ceil == produced, ( + f"{tag}: expected the two ceils to AGREE here, got " + f"upstream={produced} exact={exact_ceil}" + ) + tail = got.waveform.reshape(-1)[-RESAMPLE_CEIL_TAIL:] + assert float(tail.abs().max()) > 0.0, "an all-zero tail gates nothing" + emit_scalar(out, f"kLtx2Resample{tag}OrigRate", orig) + emit_scalar(out, f"kLtx2Resample{tag}NewRate", target) + emit_scalar(out, f"kLtx2Resample{tag}InSamples", length) + emit_scalar(out, f"kLtx2Resample{tag}OutSamples", produced) + emit_f32(out, f"kLtx2Resample{tag}TailGolden", tail.numpy()) + # --------------------------------------------------------------------------- # Section 9 — the CONDITIONING ITEMS (phase L11). What the encoders' output is diff --git a/src/vllm/model_executor/models/ltx2_audio_input.cpp b/src/vllm/model_executor/models/ltx2_audio_input.cpp index 067343e019..ff87fc87b8 100644 --- a/src/vllm/model_executor/models/ltx2_audio_input.cpp +++ b/src/vllm/model_executor/models/ltx2_audio_input.cpp @@ -71,8 +71,8 @@ Ltx2WavFormat Ltx2ProbeWavFormat(const std::string& bytes) { } Ltx2DecodedAudio Ltx2DecodeAudioWav(const std::string& bytes, const std::string& label, - int64_t want_channels, int64_t want_sample_rate, - double start_time, double max_duration) { + int64_t want_channels, double start_time, + double max_duration) { const Ltx2WavFormat fmt = Ltx2ProbeWavFormat(bytes); // Refused BEFORE the decode, because `MiniMaxH3ReadWav` would silently repeat @@ -94,17 +94,6 @@ Ltx2DecodedAudio Ltx2DecodeAudioWav(const std::string& bytes, const std::string& Fail("'" + label + "' is " + std::to_string(fmt.bits_per_sample) + "-bit PCM; only 16-bit is decoded here"); } - if (want_sample_rate > 0 && fmt.sample_rate != want_sample_rate) { - Fail("'" + label + "' is sampled at " + std::to_string(fmt.sample_rate) + - " Hz and this checkpoint's mel front-end targets " + std::to_string(want_sample_rate) + - " Hz (audio_vae/ops.py:19-34). Upstream resamples with " - "`torchaudio.functional.resample` (ops.py:40), an arbitrary-ratio polyphase kaiser " - "resampler; this project ports only the integer-ratio hann-sinc variant, so the rate " - "is refused rather than assumed. Treating these samples as " + - std::to_string(want_sample_rate) + - " Hz would pitch- and time-shift the conditioning while every shape still checked out"); - } - Ltx2DecodedAudio out; out.channels = fmt.channels; out.sample_rate = fmt.sample_rate; diff --git a/src/vllm/model_executor/models/ltx2_audio_vae.cpp b/src/vllm/model_executor/models/ltx2_audio_vae.cpp index 152d9021ac..fd98a4c557 100644 --- a/src/vllm/model_executor/models/ltx2_audio_vae.cpp +++ b/src/vllm/model_executor/models/ltx2_audio_vae.cpp @@ -12,11 +12,12 @@ // numeric contract across the two stages. // // ─── SCOPE, so nothing is discovered later ─────────────────────────────────── -// This is the DECODE direction only: AudioDecoder + Vocoder (+ the BWE chain). -// The ANALYSIS half (`AudioEncoder`, audio_vae.py:60-246, and the mel front-end -// `AudioProcessor`, ops.py:8-55) is what a REFERENCE AUDIO would need, and it is -// NOT ported here — it is owed, and the same is true of the video VAE's encoder -// (see ltx2_video_vae.cpp). +// Both directions now live here. The DECODE half is AudioDecoder + Vocoder +// (+ the BWE chain); the ANALYSIS half is `AudioEncoder` (audio_vae.py:60-246) +// and the whole `AudioProcessor` front-end (ops.py:8-55) — its slaney +// filterbank, its mel transform, and, since row LTX25-AUDIO-RESAMPLE (#2583), +// its resampler. The video VAE's encoder is a different file +// (ltx2_video_vae.cpp). #include "vllm/model_executor/models/ltx2_audio_vae.h" #include "vllm/model_executor/models/ltx2_audio_vae_encoder.h" @@ -25,6 +26,7 @@ #include #include #include +#include #include #include @@ -1044,20 +1046,181 @@ std::vector Ltx2SlaneyMelFilterbank(int64_t n_freqs, double f_min, double return fb; } +namespace { + +// `_get_sinc_resample_kernel` (torchaudio 2.11 functional/functional.py:1305-1402) +// at the arguments `ops.py:40` passes, which is to say at every default. Returns +// `new_freq` phase rows of `2 * width + orig_freq` taps, row-major. `orig_freq` +// and `new_freq` are already REDUCED by their gcd, as upstream reduces them at +// `:1340-1341`. +// +// THE WINDOW IS HANN, and nine comments in this tree used to say kaiser. +// `resample` defaults `resampling_method="sinc_interp_hann"` (`:1441`) and only +// the kaiser branch (`:1386-1391`) builds `i0`-weighted taps; `ops.py:40` passes +// neither that argument nor `beta`. The same window this file already builds for +// the vocoder's BWE stage (`Ltx2HannSincResampleFilter1d`) is this kernel at +// `orig_freq == 1`; what was missing was the arbitrary rational ratio. +// +// EVERY VALUE IS COMPUTED IN `float`, AND THAT IS THE MIRROR RATHER THAN A +// SHORTCUT. Upstream's waveform is float32 from the decoder on +// (`decode.py:173-176`), `encode_audio` moves it without casting +// (`audio_vae.py:271`), and the kernel is built with `dtype=waveform.dtype` +// (`:1487`), so `idx`, `t`, the clamp, the window and `sin(t)/t` are all f32 +// (`:1376-1397`). Measured against the pinned oracle, a `double`-built kernel +// narrowed to `float` lands 3.39746e-06 away where this f32-built one lands +// 1.19209e-07: widening here does not improve the port, it loosens the gate by a +// factor of 28 and hides exactly what AGENTS.md's "a token gate cannot detect a +// dtype that is too wide" is about. +std::vector Ltx2SincResampleKernel(int64_t orig_freq, int64_t new_freq, + int64_t* out_width) { + constexpr int64_t kLowpassFilterWidth = 6; // functional.py:1439 + constexpr double kRolloff = 0.99; // functional.py:1440 + + // `base_freq` and `width` are Python floats, i.e. f64, and stay f64: only the + // TENSOR arithmetic below narrows (`:1350`, `:1369`; the first tensor is `:1376`). + const double base_freq64 = static_cast(std::min(orig_freq, new_freq)) * kRolloff; + const int64_t width = static_cast( + std::ceil(static_cast(kLowpassFilterWidth) * static_cast(orig_freq) / + base_freq64)); + const int64_t taps = 2 * width + orig_freq; + if (out_width != nullptr) *out_width = width; + + // `t *= base_freq` narrows its operand to the tensor dtype (`:1379`), and so + // does `window * scale` — but `scale = base_freq / orig_freq` is a Python + // DIVISION in f64 that is narrowed only afterwards (`:1395`). + const float base_freq = static_cast(base_freq64); + const float scale = static_cast(base_freq64 / static_cast(orig_freq)); + const float lpw = static_cast(kLowpassFilterWidth); + constexpr float kPi = std::numbers::pi_v; // `math.pi` narrowed, as torch narrows it + + std::vector kernel(static_cast(new_freq * taps)); + for (int64_t j = 0; j < new_freq; ++j) { + // `torch.arange(0, -new_freq, -1) / new_freq` (`:1378`): row j is the output + // phase -j/new_freq. + const float phase = static_cast(-j) / static_cast(new_freq); + for (int64_t k = 0; k < taps; ++k) { + // `idx = torch.arange(-width, width + orig_freq) / orig_freq` (`:1376`). + const float idx = static_cast(k - width) / static_cast(orig_freq); + float t = (phase + idx) * base_freq; + // `:1380`. The SINC reads the clamped t too, because `t *= math.pi` comes + // after (`:1393`). Outside +/-6 the window is cos(pi/2)^2 = 0 exactly, so + // those taps vanish whatever the sinc does — which is why the BWE + // filter's unclamped sinc has never mattered. + t = std::max(-lpw, std::min(lpw, t)); + const float shaped = std::cos(((t * kPi) / lpw) / 2.0f); + const float window = shaped * shaped; // `** 2` at `:1385` + t *= kPi; // `:1393` + // `torch.where(t == 0, 1.0, t.sin() / t)` (`:1396`). The equality is + // upstream's own and fires on exactly one tap per kernel — j = 0, + // k = width — because orig_freq and new_freq are coprime after the gcd. + const float sinc = t == 0.0f ? 1.0f : std::sin(t) / t; + // `kernels *= window * scale` (`:1397`) — that association, not + // `(sinc * window) * scale`, which rounds differently in f32. + kernel[static_cast(j * taps + k)] = sinc * (window * scale); + } + } + return kernel; +} + +} // namespace + +std::vector Ltx2ResampleWaveform(const std::vector& waveform, int64_t channels, + int64_t samples, int64_t orig_freq, int64_t new_freq, + int64_t* out_samples) { + VT_CHECK(channels > 0 && samples > 0, "ltx2 resample: channels and samples must be positive"); + VT_CHECK(static_cast(waveform.size()) == channels * samples, + "ltx2 resample: waveform size does not match [channels, samples]"); + // `resample` (functional.py:1470-1471), the same refusal and the same reason. + VT_CHECK(orig_freq > 0 && new_freq > 0, + "ltx2 resample: original frequency and desired frequency should be positive " + "(functional.py:1470-1471)"); + + if (out_samples != nullptr) *out_samples = samples; + // `:1473-1474`, and `resample_audio` returns even earlier (`ops.py:38-39`). + // The input UNFILTERED, not the input pushed through a unit-ratio filter: the + // filter has passband ripple and would change every sample by a little. + if (orig_freq == new_freq) return waveform; + + const int64_t gcd = std::gcd(orig_freq, new_freq); + const int64_t orig = orig_freq / gcd; // `:1340`, `:1416` + const int64_t next = new_freq / gcd; // `:1341`, `:1417` + int64_t width = 0; + const std::vector kernel = Ltx2SincResampleKernel(orig, next, &width); + const int64_t taps = 2 * width + orig; + + // `_apply_sinc_resample_kernel` (`:1405-1432`): zero-pad by (width, width + + // orig), convolve with stride `orig`, transpose the phase axis in front of the + // block axis, and truncate to `ceil(new * length / orig)`. + const int64_t blocks = samples / orig + 1; + // `target_length = torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` + // (`:1427`), and this is NOT an exact integer ceil. `torch.as_tensor` of a + // PYTHON FLOAT takes `torch.get_default_dtype()`, which is float32, so the f64 + // quotient is rounded to f32 BEFORE the ceil. Where the exact quotient sits + // above an integer by less than half an f32 ulp, the narrowing lands ON that + // integer and upstream keeps one sample FEWER than `(next * samples + orig - 1) + // / orig` gives — first at 180697 samples for 44100 -> 16000 (4.097 s), then + // for 48102 of the first 60 s worth of lengths. The extra sample moves the last + // STFT windows and, where `samples % hop == 0`, the mel FRAME COUNT. + // + // `next * samples` is exact in `int64_t` and exact again as a `double` for any + // length under 2^53 / next, so the division below is Python's own + // correctly-rounded f64 `int / int`; the narrowing and the ceil then mirror + // `as_tensor` and `torch.ceil`, and the cast to `int64_t` mirrors `.long()`. + // Checked against torchaudio over 276060 (ratio, length) pairs — twelve ratios + // by 23005 lengths, including a run either side of 2^24, where f32 no longer + // holds every integer: zero divergences. + const double quotient = static_cast(next * samples) / static_cast(orig); + const int64_t target = static_cast(std::ceil(static_cast(quotient))); + if (out_samples != nullptr) *out_samples = target; + + // ONE kernel for every channel. torchaudio packs the batch and resamples the + // last axis (`:1419-1421`), so the channels are independent; this walks them + // rather than approximating that. + std::vector out(static_cast(channels) * static_cast(target)); + for (int64_t c = 0; c < channels; ++c) { + const float* in = waveform.data() + c * samples; + float* dst = out.data() + c * target; + for (int64_t b = 0; b < blocks; ++b) { + if (b * next >= target) break; + // Where tap 0 of this block lands in the UNPADDED signal. The zero pad is + // skipped rather than materialised, which is the same sum. + const int64_t base = b * orig - width; + const int64_t first = std::max(0, -base); + const int64_t last = std::min(taps, samples - base); + for (int64_t j = 0; j < next; ++j) { + const int64_t index = b * next + j; + if (index >= target) break; + const float* row = kernel.data() + j * taps; + // The one `double` in this function, and it is an ACCUMULATOR rather + // than a stored dtype: torch's vectorised conv1d reduction order is not + // reproducible from a C++ loop, so the choice is between two roundings + // and the exact sum is the defensible one. Every stored value is f32. + double acc = 0.0; + for (int64_t k = first; k < last; ++k) { + acc += static_cast(row[k]) * static_cast(in[base + k]); + } + dst[index] = static_cast(acc); + } + } + } + return out; +} + std::vector Ltx2WaveformToLogMel(const Ltx2AudioProcessorConfig& config, - const std::vector& waveform, int64_t channels, - int64_t samples, int64_t sampling_rate, + const std::vector& source, int64_t channels, + int64_t source_samples, int64_t sampling_rate, int64_t* out_frames) { - VT_CHECK(static_cast(waveform.size()) == channels * samples, + VT_CHECK(static_cast(source.size()) == channels * source_samples, "ltx2 mel: waveform size does not match [channels, samples]"); - VT_CHECK(sampling_rate == config.target_sample_rate, - "ltx2 mel: the waveform is at " + std::to_string(sampling_rate) + " Hz but the audio " - "VAE wants " + std::to_string(config.target_sample_rate) + - " Hz. Upstream resamples with torchaudio.functional.resample (audio_vae/ops.py:36-42), " - "a polyphase kaiser resampler for an arbitrary rational ratio, which is NOT ported — " - "this project carries only the integer-ratio hann-sinc variant the BWE stage needs. " - "Refused rather than reinterpreted: treating the samples as if they were already at " - "the target rate conditions on audio that is pitched and time-scaled wrong"); + // `waveform = self.resample_audio(audio).waveform` (ops.py:49), called + // UNCONDITIONALLY: the rate test lives inside `resample_audio` (ops.py:38-39), + // so the equal-rate arm is on the production path rather than reachable only + // from a test. Everything below — the reflect-pad floor, the frame count, the + // mel — is therefore over the RESAMPLED signal, which is upstream's order. + int64_t samples = 0; + const std::vector waveform = Ltx2ResampleWaveform( + source, channels, source_samples, sampling_rate, config.target_sample_rate, &samples); + const int64_t n_fft = config.n_fft; const int64_t hop = config.mel_hop_length; VT_CHECK(n_fft > 0 && hop > 0, "ltx2 mel: n_fft and hop_length must be positive"); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 22bf514d71..29c0eac461 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -3076,10 +3076,11 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "; it is a duration in seconds and must be positive"); } + // No target rate: the decode runs at the FILE's rate and the mel front-end + // resamples (ops.py:44-49). See `Ltx2DecodeAudioWav`'s header. a2v_source = Ltx2DecodeAudioWav( ReadFileBytes(kLtx2AudioPathExtra, a2v_audio_path), a2v_audio_path, - im.audio_encoder_cfg.encoder.in_channels, - im.audio_encoder_cfg.processor.target_sample_rate, start_time, max_duration); + im.audio_encoder_cfg.encoder.in_channels, start_time, max_duration); const Ltx2AudioSpectrogram encoded = Ltx2EncodeAudioToLatent( im.audio_encoder_cfg.encoder, im.audio_encoder_cfg.processor, im.audio_encoder_weights, diff --git a/tests/vllm/models/ltx2_vae_goldens.inc b/tests/vllm/models/ltx2_vae_goldens.inc index aeb670c312..9e48102e85 100644 --- a/tests/vllm/models/ltx2_vae_goldens.inc +++ b/tests/vllm/models/ltx2_vae_goldens.inc @@ -4649,6 +4649,301 @@ inline constexpr float kLtx2MelQuietGolden[] = { -11.5129251f, -11.5129251f, -11.5129251f, -11.5129251f, }; +// --- section 8d: AudioProcessor.resample_audio (ops.py:36-42) --- +inline constexpr int64_t kLtx2ResampleChannels = 2; +inline constexpr int64_t kLtx2ResampleUpOrigRate = 16000; +inline constexpr int64_t kLtx2ResampleUpNewRate = 48000; +inline constexpr int64_t kLtx2ResampleUpInSamples = 64; +inline constexpr int64_t kLtx2ResampleUpOutSamples = 192; +inline constexpr float kLtx2ResampleUpGolden[] = { + 0.133884266f, 0.0918205902f, -0.0255323742f, -0.197727516f, -0.371828228f, -0.479041606f, + -0.471846819f, -0.360692859f, -0.214061275f, -0.108339809f, -0.0662166774f, -0.039924968f, + 0.0404968187f, 0.193504661f, 0.354771823f, 0.42060411f, 0.334565669f, 0.139964432f, + -0.0506840423f, -0.138156235f, -0.0982855111f, 0.0225483235f, 0.159993023f, 0.275422126f, + 0.355261415f, 0.382120222f, 0.323688149f, 0.159209579f, -0.0810237005f, -0.312432617f, + -0.448440582f, -0.463129699f, -0.405220091f, -0.34950918f, -0.332130969f, -0.32618162f, + -0.271405935f, -0.127568468f, 0.0891931355f, 0.306307077f, 0.425508291f, 0.371295869f, + 0.141137481f, -0.168693811f, -0.396210313f, -0.409043342f, -0.207575709f, 0.0552373789f, + 0.179578736f, 0.0805262998f, -0.137114063f, -0.272212356f, -0.212493807f, -0.0356906354f, + 0.0736512393f, 0.0123008974f, -0.138699532f, -0.204925239f, -0.110514529f, 0.0321467146f, + 0.0363768227f, -0.159051433f, -0.408695728f, -0.480595052f, -0.266345084f, 0.117857657f, + 0.428332329f, 0.492335737f, 0.329499304f, 0.106610663f, -0.00907083135f, 0.0378266498f, + 0.187097982f, 0.338165194f, 0.423817635f, 0.434502244f, 0.400267035f, 0.361475706f, + 0.344876498f, 0.350619495f, 0.352048993f, 0.311718345f, 0.208393216f, 0.05891959f, + -0.0841019675f, -0.161014602f, -0.140655726f, -0.0377356634f, 0.0911664814f, 0.172342196f, + 0.155045733f, 0.042715624f, -0.10623949f, -0.210352689f, -0.210627481f, -0.10195376f, + 0.0685161501f, 0.232668713f, 0.33972463f, 0.379634589f, 0.378310233f, 0.372507691f, + 0.38568297f, 0.41811499f, 0.448957741f, 0.442723602f, 0.359707177f, 0.176587f, + -0.0858395919f, -0.349144161f, -0.505121112f, -0.483186722f, -0.308442414f, -0.0975251049f, + 0.0148507962f, -0.0292887557f, -0.170069456f, -0.278221965f, -0.254882395f, -0.103100471f, + 0.0820374712f, 0.187563434f, 0.155847415f, 0.00705303298f, -0.193033978f, -0.378515601f, + -0.501759768f, -0.52891165f, -0.443026602f, -0.266485959f, -0.0746473297f, 0.0318640955f, + -0.0101245418f, -0.171410829f, -0.336801201f, -0.382912368f, -0.267979831f, -0.0621753745f, + 0.103031375f, 0.129021317f, 0.0144238407f, -0.155600309f, -0.273973525f, -0.27839604f, + -0.181593135f, -0.0554181561f, 0.0137927886f, -0.0278404783f, -0.170404479f, -0.342470616f, + -0.455148369f, -0.462089509f, -0.390882462f, -0.319350362f, -0.313464373f, -0.378141999f, + -0.461465418f, -0.498804986f, -0.455821276f, -0.341705322f, -0.194867447f, -0.0606985763f, + 0.0247824583f, 0.0487422198f, 0.0282341018f, -0.00259423628f, -0.0235595647f, -0.0477596074f, + -0.101373702f, -0.177613169f, -0.220084473f, -0.172693759f, -0.0544211492f, 0.0281328484f, + -0.0334747769f, -0.23220484f, -0.417596281f, -0.409719974f, -0.161943913f, 0.180644095f, + 0.389402241f, 0.337898821f, 0.100684159f, -0.119783737f, -0.164848462f, -0.0346657261f, + 0.142288923f, 0.235139355f, 0.208460733f, 0.118321925f, 0.0390205905f, 0.00303064473f, + 0.445470721f, 0.29844594f, 0.0789522305f, -0.0442571826f, 0.0190145057f, 0.212463439f, + 0.381696761f, 0.393006206f, 0.226355091f, -0.0186772011f, -0.196490571f, -0.207019269f, + -0.0468566157f, 0.199214488f, 0.405914158f, 0.463017434f, 0.316879481f, -0.00637663249f, + -0.398234218f, -0.693616152f, -0.738651037f, -0.484023809f, -0.0435220301f, 0.34737438f, + 0.474355489f, 0.299590766f, -0.00258852099f, -0.179535776f, -0.103193171f, 0.129609138f, + 0.287667543f, 0.207886934f, -0.0726820678f, -0.367550731f, -0.511022151f, -0.483476281f, + -0.388759345f, -0.339509666f, -0.371335417f, -0.446947157f, -0.509600461f, -0.52418673f, + -0.490478426f, -0.436395705f, -0.400743932f, -0.404196441f, -0.418688267f, -0.369954288f, + -0.194614381f, 0.0875882283f, 0.357672304f, 0.47100696f, 0.36678791f, 0.117068402f, + -0.132126331f, -0.271304518f, -0.294583142f, -0.27867201f, -0.30367738f, -0.388001353f, + -0.485279113f, -0.527561128f, -0.472777069f, -0.331204176f, -0.158322424f, -0.0249618776f, + 0.0148999924f, -0.0481792577f, -0.169395626f, -0.272131115f, -0.291395575f, -0.210498825f, + -0.0661703423f, 0.0708888695f, 0.128283232f, 0.0680397302f, -0.0855218992f, -0.252087831f, + -0.349260598f, -0.355694622f, -0.325025618f, -0.330481172f, -0.382303476f, -0.399712712f, + -0.279376447f, -0.00664910674f, 0.299109846f, 0.460590035f, 0.379405528f, 0.112821877f, + -0.176825255f, -0.350439608f, -0.379978091f, -0.326593995f, -0.25282529f, -0.165309787f, + -0.0369570144f, 0.131423995f, 0.284679413f, 0.350006908f, 0.302066803f, 0.191396281f, + 0.106939793f, 0.112949446f, 0.21247226f, 0.353295594f, 0.46238023f, 0.482803315f, + 0.398368537f, 0.238175258f, 0.0603726916f, -0.0789099857f, -0.15574719f, -0.187806249f, + -0.214972809f, -0.261167616f, -0.31014207f, -0.321638256f, -0.277262509f, -0.207888648f, + -0.166348115f, -0.169614062f, -0.174571082f, -0.121473916f, -0.0060363994f, 0.0980909616f, + 0.102463357f, -0.00375754829f, -0.126347676f, -0.138543501f, 0.00219828752f, 0.203163311f, + 0.307530284f, 0.230149224f, 0.0306903068f, -0.14745301f, -0.206874877f, -0.168353304f, + -0.125326455f, -0.134805664f, -0.158722103f, -0.112727828f, 0.0363009349f, 0.224020794f, + 0.333458394f, 0.2824651f, 0.0785970166f, -0.190376863f, -0.409248203f, -0.493239254f, + -0.427479506f, -0.272100419f, -0.128921509f, -0.0807205066f, -0.139783099f, -0.241041079f, + -0.286729604f, -0.211247459f, -0.0247106366f, 0.188118815f, 0.313987404f, 0.275930554f, + 0.0798854306f, -0.18334426f, -0.383101583f, -0.417240143f, -0.270101786f, -0.0253726505f, + 0.179037392f, 0.233607903f, 0.121847987f, -0.0812300071f, -0.26580596f, -0.351501971f, + -0.318386525f, -0.195205286f, -0.0308631677f, 0.122206084f, 0.213791981f, 0.207223073f, + 0.100514427f, -0.0622927919f, -0.209378287f, -0.283151537f, -0.276329786f, -0.232446685f, + -0.209688485f, -0.233458713f, -0.27620554f, -0.28249678f, -0.219260603f, -0.106768727f, +}; + +inline constexpr int64_t kLtx2ResampleDownOrigRate = 48000; +inline constexpr int64_t kLtx2ResampleDownNewRate = 16000; +inline constexpr int64_t kLtx2ResampleDownInSamples = 192; +inline constexpr int64_t kLtx2ResampleDownOutSamples = 64; +inline constexpr float kLtx2ResampleDownGolden[] = { + -0.0653797016f, -0.073269777f, -0.148023993f, -0.0650461167f, 0.242959961f, 0.350555152f, + 0.0233238004f, -0.292549998f, 0.0432978123f, 0.301227987f, 0.0433950201f, 0.051469449f, + -0.360557288f, -0.11672654f, 0.195395276f, 0.0221682694f, 0.0440903604f, 0.020922726f, + -0.182012573f, 0.00258047739f, -0.0147719998f, 0.070274502f, -0.0313250236f, -0.2670964f, + 0.0618665032f, 0.0976947695f, -0.0370035395f, -0.18601872f, 0.122944795f, -0.200641274f, + -0.0474185012f, -0.237794429f, -0.00165624486f, -0.030814914f, 0.0299725831f, 0.184029549f, + -0.0193744563f, -0.0356619693f, 0.163084179f, -0.0488685742f, -0.0179705899f, 0.101946548f, + 0.138102219f, 0.305152833f, 0.447461009f, -0.107914671f, -0.0785821527f, -0.0753501803f, + -0.170663506f, 0.213874444f, -0.0363523066f, -0.048221752f, -0.105345771f, 0.143479466f, + 0.160106972f, 0.239474237f, 0.182363778f, 0.279522777f, -0.160115555f, 0.011421659f, + -0.141609654f, -0.111503638f, 0.0886406973f, -0.0875492021f, -0.16086942f, -0.277595878f, + 0.0166485943f, -0.140741989f, 0.0211277455f, 0.00768060703f, 0.0280438047f, 0.269561291f, + 0.0397057943f, -0.19806914f, -0.0431697853f, -0.0739519894f, 0.070396401f, -0.160541594f, + -0.00253871456f, -0.0407009572f, 0.187902853f, -0.0259926617f, -0.257088929f, 0.167080641f, + -0.228101581f, -0.0421476401f, 0.234342724f, 0.15414308f, -0.206527114f, -0.00110695569f, + 0.253393978f, -0.208834425f, 0.115428343f, -0.106179774f, 0.0309635643f, 0.216262385f, + -0.0682550222f, 0.129363611f, 0.22991617f, 0.0303576179f, 0.252601564f, 0.11894282f, + 0.298532993f, 0.213497579f, -0.157630861f, -0.316708177f, -0.00704507343f, 0.174568385f, + 0.337846696f, 0.231524006f, -0.0973610133f, 0.0622536652f, -0.114698462f, -0.000667623477f, + -0.162611544f, -0.0165845435f, -0.267340153f, 0.0150169944f, 0.0334990136f, -0.0230757799f, + 0.0803245604f, -0.2463938f, -0.0079144584f, 0.0854698643f, -0.0801748708f, -0.153812885f, + -0.12739481f, -0.146888018f, +}; + +inline constexpr int64_t kLtx2ResampleWideOrigRate = 44100; +inline constexpr int64_t kLtx2ResampleWideNewRate = 16000; +inline constexpr int64_t kLtx2ResampleWideInSamples = 600; +inline constexpr int64_t kLtx2ResampleWideOutSamples = 218; +inline constexpr float kLtx2ResampleWideGolden[] = { + -0.166109875f, -0.0630680472f, 0.0970632955f, -0.151416913f, 0.0240903199f, 0.0663119555f, + 0.0803315341f, 0.00678226678f, 0.198316097f, 0.10671559f, -0.280532002f, 0.239263371f, + -0.261209846f, -0.264212847f, -0.285671055f, 0.0628080517f, 0.00952306949f, 0.059553951f, + -0.0891417712f, 0.0585061572f, 0.0265714284f, -0.0520042107f, 0.0164552033f, -0.0880147219f, + -0.163946286f, 0.0146814911f, 0.222776204f, -0.186651662f, -0.0553991199f, 0.176685765f, + -0.170007318f, -0.113077916f, 0.119050048f, -0.176367268f, -0.129646897f, 0.0907209516f, + -0.0496546365f, 0.133777753f, -0.0100589916f, 0.221267298f, 0.00451244181f, 0.306644559f, + 0.20498611f, -0.13016741f, 0.0865820944f, -0.234014377f, 0.0648791045f, 0.126939878f, + -0.0115708541f, -0.111877613f, 0.14006488f, -0.156115606f, 0.0307647083f, 0.0475151688f, + -0.0394521095f, -0.0205634832f, 0.18426457f, -0.0175724365f, 0.126709551f, 0.126869366f, + 0.0952487066f, 0.136410594f, 0.174353659f, -0.137977764f, 0.0578158684f, 0.227951288f, + -0.0852605477f, -0.150851771f, 0.123837478f, 0.0507039875f, 0.136538386f, 0.282794505f, + -0.244014159f, -0.0632211491f, 0.15392904f, 0.0547516271f, 0.175257862f, -0.0491353497f, + -0.154547647f, 0.0704203993f, -0.237117171f, -0.246590525f, 0.0639194101f, 0.252007723f, + 0.375681937f, 0.173818141f, 0.060616836f, -0.152863637f, -0.133546785f, -0.181805402f, + 0.178500712f, 0.0247977544f, 0.03136703f, 0.244234294f, 0.0473097265f, 0.0835861415f, + -0.086889118f, 0.0202626158f, 0.0417347848f, 0.0451637879f, 0.0889597908f, 0.424210399f, + 0.0198644847f, -0.135825023f, 0.253715664f, 0.158522293f, -0.359053552f, -0.131656185f, + -0.0171459615f, -0.0714050904f, 0.242445141f, 0.0335307084f, 0.235993713f, -0.0286050439f, + -0.295446247f, -0.00572203193f, 0.0129030515f, 0.226378903f, 0.0578723922f, -0.0214700717f, + -0.0222770963f, -0.320240289f, 0.180718198f, -0.105571404f, -0.0180277042f, -0.093553789f, + 0.0627840161f, 0.15227665f, -0.0517334007f, -0.118923105f, -0.247941241f, -0.210730165f, + -0.130211934f, 0.077543214f, -0.127787769f, 0.253119469f, 0.0214030668f, -0.0240055565f, + -0.384567648f, 0.0333582833f, 0.000859327032f, -0.130582809f, 0.0810907856f, 0.0593428575f, + -0.0652422756f, 0.0980934277f, -0.0323485583f, -0.207968518f, -0.128021747f, -0.0529218614f, + 0.0871731117f, 0.188785702f, 0.246557832f, -0.00204586587f, 0.171493143f, 0.159378737f, + 0.056447383f, 0.0526220873f, 0.23541069f, -0.264652103f, -0.00124741346f, -0.0341790505f, + -0.196577966f, -0.213871971f, -0.0873092711f, 0.0459644794f, 0.244911849f, -0.0141087482f, + -0.284092218f, -0.0412680618f, 0.405599743f, 0.0470541827f, -0.115298733f, -0.197923139f, + -0.307000816f, -0.466205537f, -0.0451052412f, -0.011972269f, 0.113826551f, -0.149915695f, + 0.00626881467f, 0.00471630925f, 0.0430704467f, 0.276836783f, 0.058558546f, -0.168954059f, + 0.114365786f, -0.0500359125f, -0.190239385f, 0.0441222079f, 0.156270564f, -0.134900004f, + 0.304657489f, -0.0279312395f, 0.191525131f, 0.0655415505f, -0.348410845f, -0.0924862623f, + -0.0718971193f, 0.0818686336f, 0.0861204565f, -0.171984121f, 0.0864807144f, 0.163079545f, + -0.181385398f, 0.0981449336f, 0.124766447f, -0.0518741645f, 0.17262435f, -0.249620304f, + -0.0516820587f, 0.00645593321f, -0.0235484783f, -0.0549264066f, 0.155914336f, 0.033666864f, + -0.112114444f, -0.110596262f, 0.231156155f, 0.0603438914f, 0.103510648f, 0.103870779f, + -0.00270144059f, 0.255741745f, 0.336562037f, 0.199568674f, 0.11468336f, 0.111278333f, + 0.0494238473f, 0.111096464f, -0.0281137917f, -0.103573799f, -0.0396192186f, 0.21315515f, + 0.130312636f, -0.171485648f, 0.114488833f, 0.0394079015f, -0.308014482f, 0.0735709369f, + 0.137956455f, -0.325919807f, -0.206913173f, -0.160477445f, 0.117573932f, 0.315905005f, + 0.00996286143f, 0.092836678f, -0.0213343836f, -0.0349569358f, 0.128590822f, -0.130582705f, + 0.0150985261f, 0.0375045836f, 0.0608869307f, 0.00387548353f, -0.172976136f, 0.0597007796f, + -0.168234691f, -0.373954743f, -0.106707633f, -0.0131773744f, -0.292082697f, 0.202119842f, + -0.0338037126f, -0.0859032795f, -0.136623159f, 0.0820575207f, -0.0194657296f, 0.186938107f, + 0.060591951f, -0.0963178575f, 0.0430724882f, -0.0302837882f, -0.035506729f, 0.201354608f, + 0.184694782f, 0.336674929f, 0.0287558977f, 0.000921385246f, -0.0135701178f, -0.0380269364f, + 0.129963949f, -0.0505637601f, -0.0889441594f, -0.0362011604f, -0.140392095f, 0.122969434f, + 0.127392754f, 0.136146575f, -0.0079227658f, -0.176621646f, 0.086479634f, -0.190951377f, + -0.15915598f, -0.0403363518f, -0.0394156612f, 0.0792375505f, 0.0261054002f, 0.0217790119f, + -0.352562994f, -0.148276031f, 0.0707434937f, -0.159458429f, 0.041782245f, -0.0774111077f, + -0.0474635102f, 0.213561073f, 0.163344324f, -0.120505691f, -0.0739919096f, -0.0653060675f, + -0.0936806127f, -0.227657869f, 0.0889978036f, 0.118333198f, -0.114345334f, -0.127344832f, + 0.0917342603f, -0.119830474f, 0.21951814f, -0.114790723f, -0.141756326f, -0.163251773f, + 0.136926875f, 0.304104924f, 0.17052184f, 0.204871073f, -0.196935922f, 0.249426678f, + -0.0680071265f, -0.261540204f, 0.0208325256f, -0.0220708381f, 0.260184914f, -0.0265973154f, + -0.102168217f, 0.196518809f, 0.192088574f, -0.0881040916f, -0.0792996809f, -0.166160718f, + 0.228083462f, -0.0459954739f, 0.312827975f, -0.141072467f, -0.0688961223f, -0.0530200824f, + -0.0169182848f, 0.108116254f, -0.12148159f, 0.15531382f, 0.0330864601f, -0.231164575f, + -0.229718223f, 0.0563248321f, 0.246671408f, -0.169734195f, -0.126586407f, -0.0533274971f, + -0.16137971f, -0.0419682935f, -0.226478949f, 0.234363094f, 0.185875759f, -0.112364776f, + 0.346106201f, 0.197649047f, 0.186018482f, -0.0976952389f, -0.102497213f, -0.0145039763f, + 0.0675646365f, 0.0726319999f, -0.00998232421f, -0.179051265f, 0.10536781f, -0.105141826f, + -0.234749645f, -0.124726616f, -0.0402343571f, -0.354087919f, -0.215418801f, 0.0746908113f, + 0.0410933495f, 0.343521953f, 0.138164833f, 0.0281131174f, -0.0974368528f, -0.377483726f, + -0.189783916f, -0.0490393974f, 0.0702480972f, -0.11025051f, 0.18443951f, 0.0214362554f, + -0.348050624f, -0.117464028f, 0.15207684f, -0.0281669889f, -0.168709338f, 0.125837639f, + 0.0349771678f, -0.232138872f, -0.120971933f, 0.432278723f, 0.0182103906f, -0.308253527f, + 0.189801708f, 0.142383322f, 0.117755942f, -0.0810037479f, 0.139789477f, 0.072753109f, + -0.0383830853f, 0.0737549812f, -0.135906562f, 0.0727570578f, -0.124967001f, 0.0332392044f, + -0.107525945f, 0.17213212f, -0.154151469f, 0.0481800362f, 0.0866204649f, 0.00996933877f, + -0.120263219f, 0.159145638f, 0.182878241f, -0.113282107f, 0.0444048122f, 0.073611699f, + 0.244076818f, -0.101309456f, -0.0706439242f, 0.132584006f, +}; + +inline constexpr int64_t kLtx2ResampleSameOrigRate = 16000; +inline constexpr int64_t kLtx2ResampleSameNewRate = 16000; +inline constexpr int64_t kLtx2ResampleSameInSamples = 64; +inline constexpr int64_t kLtx2ResampleSameOutSamples = 64; +inline constexpr float kLtx2ResampleSameGolden[] = { + -0.405204147f, 0.256094337f, -0.0365431421f, 0.144786432f, 0.442803383f, 0.197185129f, + 0.00641530426f, 0.101439901f, -0.288413048f, 0.276153743f, -0.373815358f, 0.0357299149f, + -0.150143951f, 0.425933182f, 0.430869371f, 0.386764467f, 0.449498534f, -0.215525791f, + -0.413843364f, 0.454426825f, 0.257733166f, 0.474363089f, 0.454185247f, 0.137395754f, + -0.162905961f, -0.475493759f, 0.458122998f, 0.345632404f, -0.195999309f, 0.344082147f, + -0.232523456f, -0.395225525f, 0.197237954f, -0.0901086852f, -0.329409868f, 0.115814395f, + -0.00241821329f, 0.0528635159f, -0.136961848f, -0.0803770274f, 0.446430832f, 0.251397312f, + -0.219453916f, 0.155429542f, -0.0725865141f, -0.248174518f, 0.245423719f, -0.103727877f, + 0.0699998587f, -0.0313329883f, -0.45214057f, 0.290790528f, 0.147005036f, 0.388405949f, + 0.131976366f, 0.443826556f, -0.37574175f, 0.341548532f, -0.356789261f, 0.484469026f, + -0.176039338f, -0.431059867f, -0.379225999f, 0.0652751997f, 0.444586664f, 0.178928718f, + 0.428170532f, -0.28014794f, 0.493626893f, 0.0600764342f, -0.200313687f, 0.224563673f, + 0.34332484f, 0.302795142f, -0.0397084989f, -0.236524582f, 0.326341033f, -0.200754166f, + 0.0734653324f, 0.407461613f, 0.0551238768f, -0.144065112f, -0.352078319f, 0.0399719477f, + -0.339309841f, 0.466512203f, -0.487870157f, -0.31224522f, -0.00245836703f, -0.263568729f, + 0.447112143f, -0.229905024f, -0.208819613f, -0.0640842915f, -0.36582464f, -0.182941675f, + -0.10237547f, -0.306450248f, 0.284807384f, -0.136687264f, 0.211049214f, -0.0450212136f, + 0.351251781f, -0.296978563f, 0.300218612f, -0.376386553f, -0.14377071f, 0.01248514f, + 0.0800864846f, -0.341801167f, 0.30843401f, 0.215325922f, 0.121199809f, -0.278818995f, + -0.464530498f, 0.206952333f, -0.349141926f, -0.0297168121f, 0.177203611f, 0.143456593f, + 0.186289802f, -0.49433735f, 0.153161019f, 0.0672777146f, -0.297201097f, -0.425057709f, + 0.3405931f, -0.147772357f, +}; + +// --- section 8e: waveform_to_mel at a NON-target rate (ops.py:44-55) --- +inline constexpr int64_t kLtx2MelSourceRate = 44100; +inline constexpr int64_t kLtx2MelSourceSamples = 600; +inline constexpr int64_t kLtx2MelResampledFrames = 14; + +inline constexpr float kLtx2MelResampledGolden[] = { + -5.4661231f, -5.55506706f, -5.09506035f, -6.19318295f, -6.08189249f, -6.17544174f, + -5.6668644f, -5.5211668f, -5.80990982f, -5.9284153f, -5.45444393f, -6.4685173f, + -5.71690655f, -6.0217433f, -5.73173523f, -5.59101057f, -6.07259798f, -6.04894781f, + -6.19490576f, -6.57504129f, -5.64122391f, -5.86473227f, -5.59691286f, -5.66495609f, + -5.93782425f, -5.77671909f, -5.45543861f, -5.59877825f, -5.71345234f, -5.57294083f, + -5.66409063f, -5.63091135f, -5.33672714f, -5.69532776f, -5.50630045f, -5.50697517f, + -6.08902121f, -6.27135468f, -5.85663557f, -5.54306078f, -5.97331429f, -6.38751221f, + -5.9048481f, -5.60310745f, -6.03241777f, -6.22403574f, -5.95464993f, -5.71830559f, + -6.40463924f, -6.08543062f, -6.15129328f, -6.13091803f, -5.54007912f, -5.81266499f, + -5.7169714f, -5.62237167f, -5.90837908f, -6.22743368f, -6.04174423f, -6.56699181f, + -5.65484095f, -5.91686869f, -5.77181244f, -5.94467306f, -5.75919247f, -5.85417461f, + -6.53755093f, -5.83754826f, -5.23825598f, -5.88241386f, -6.02013731f, -5.9696269f, + -5.91963625f, -5.505476f, -5.38549566f, -5.82375288f, -5.1732707f, -5.70048857f, + -5.96792746f, -5.97677279f, -6.40191746f, -6.35137081f, -5.50886393f, -5.50407076f, + -6.01290703f, -5.95151901f, -5.75256491f, -5.87025166f, -7.16911221f, -7.09269428f, + -6.36924839f, -5.86924458f, -6.40944195f, -5.8599596f, -5.64378595f, -6.12015867f, + -6.38106489f, -6.78282833f, -5.74355841f, -5.83369303f, -6.07583332f, -5.64163589f, + -5.58671045f, -5.93410969f, -6.30367708f, -5.97529173f, -5.1600523f, -5.84388876f, + -6.34159184f, -6.55544806f, -5.81524277f, -6.22483015f, -6.10869217f, -5.77106762f, + -5.83326101f, -5.49699259f, -5.30421782f, -5.54879427f, -6.10520172f, -5.80052233f, + -5.61863375f, -5.76047802f, -5.48267508f, -5.68262529f, -5.77609634f, -6.02572727f, + -6.03231239f, -5.71555662f, -5.91699076f, -6.43874311f, -5.68555355f, -5.26530838f, + -5.82621717f, -5.71177197f, -5.74042225f, -5.82096243f, -6.38452053f, -6.5581007f, + -6.00606155f, -5.24496508f, -5.74657297f, -5.65564775f, -5.57938623f, -5.94365597f, + -6.04934359f, -6.66540241f, -5.68788862f, -5.36847687f, -5.80125999f, -5.79700184f, + -6.13079166f, -5.73697901f, -6.27567387f, -6.72550631f, -5.46829557f, -5.46381998f, + -5.79290152f, -6.0005722f, -5.97839832f, -6.0189929f, -5.96040344f, -6.65128517f, + -5.91887093f, -5.66960001f, -5.91024971f, -5.93545103f, -6.04944897f, -6.35564899f, + -5.57299137f, -6.35422421f, -6.45694542f, -6.01158762f, -5.86569738f, -5.94834423f, + -6.09327936f, -5.88648033f, -5.25396204f, -5.22967243f, -5.94662523f, -6.15266275f, + -5.83826828f, -5.48036671f, -5.74903774f, -6.08138084f, -5.46356964f, -5.06745768f, + -5.91538f, -6.51051712f, -5.72736788f, -5.71280241f, -5.83652401f, -6.05304956f, + -5.75634432f, -5.65665674f, -6.04497814f, -6.47492218f, -5.77602959f, -5.28765106f, + -5.60678625f, -6.04329443f, -5.8856926f, -6.14575195f, -5.67699766f, -5.49637413f, + -5.39192724f, -5.52504921f, -6.04295254f, -6.0130825f, -6.02200174f, -6.39702034f, + -6.22545624f, -5.62030745f, -5.4359746f, -5.42942905f, -5.7402153f, -5.68468618f, + -5.94748259f, -6.64641619f, -6.79842901f, -6.17434788f, -5.65791321f, -5.58381701f, + -6.03300667f, -5.81140184f, +}; + +// --- section 8f: the truncation boundary (functional.py:1427) --- +inline constexpr int64_t kLtx2ResampleCeilTail = 8; +inline constexpr int64_t kLtx2ResampleCeilBelowOrigRate = 44100; +inline constexpr int64_t kLtx2ResampleCeilBelowNewRate = 16000; +inline constexpr int64_t kLtx2ResampleCeilBelowInSamples = 180696; +inline constexpr int64_t kLtx2ResampleCeilBelowOutSamples = 65559; +inline constexpr float kLtx2ResampleCeilBelowTailGolden[] = { + 0.0894216299f, 0.0131077049f, -0.0238793846f, -0.0588569939f, 0.0581167452f, 0.219902992f, + -0.0258866604f, -0.171903431f, +}; + +inline constexpr int64_t kLtx2ResampleCeilAtOrigRate = 44100; +inline constexpr int64_t kLtx2ResampleCeilAtNewRate = 16000; +inline constexpr int64_t kLtx2ResampleCeilAtInSamples = 180697; +inline constexpr int64_t kLtx2ResampleCeilAtOutSamples = 65559; +inline constexpr float kLtx2ResampleCeilAtTailGolden[] = { + -0.190139741f, 0.06196668f, -0.194867074f, -0.0171126947f, 0.11970146f, 0.0139573188f, + -0.063427873f, 0.152270317f, +}; + +inline constexpr int64_t kLtx2ResampleCeilAboveOrigRate = 44100; +inline constexpr int64_t kLtx2ResampleCeilAboveNewRate = 16000; +inline constexpr int64_t kLtx2ResampleCeilAboveInSamples = 180698; +inline constexpr int64_t kLtx2ResampleCeilAboveOutSamples = 65560; +inline constexpr float kLtx2ResampleCeilAboveTailGolden[] = { + 0.138243884f, -0.0549423918f, 0.164882183f, 0.0120501379f, -0.0783000514f, 0.365968585f, + -0.0271436367f, -0.185848534f, +}; + +inline constexpr int64_t kLtx2ResampleCeilAltOrigRate = 22050; +inline constexpr int64_t kLtx2ResampleCeilAltNewRate = 16000; +inline constexpr int64_t kLtx2ResampleCeilAltInSamples = 90569; +inline constexpr int64_t kLtx2ResampleCeilAltOutSamples = 65719; +inline constexpr float kLtx2ResampleCeilAltTailGolden[] = { + 0.00326456479f, 0.0271689408f, 0.162618577f, 0.348693013f, -0.00762261869f, 0.0951811969f, + -0.0475787744f, -0.406309605f, +}; + // --- section 9a: the INITIAL video state (tools.py:139-186) --- inline constexpr int64_t kLtx2CondVideoBaseTokens = 12; inline constexpr int64_t kLtx2CondVideoBaseWidth = 4; diff --git a/tests/vllm/models/test_ltx2_vae.cpp b/tests/vllm/models/test_ltx2_vae.cpp index ba161b9067..87348f84c3 100644 --- a/tests/vllm/models/test_ltx2_vae.cpp +++ b/tests/vllm/models/test_ltx2_vae.cpp @@ -2817,6 +2817,184 @@ TEST_CASE("ltx2 vae: the slaney mel filterbank matches torchaudio") { } } +TEST_CASE("ltx2 vae: resample_audio matches upstream at every ratio it can take") { + // Row LTX25-AUDIO-RESAMPLE (#2583). Generator section 8d runs upstream's OWN + // `AudioProcessor.resample_audio` (ops.py:36-42) at four ratios; this + // reproduces each from the same PRNG input. + // + // TOLERANCE. 2.5e-07 is not a hedge, it is twice the MEASURED floor. Setting + // it to 0.0 on this tree reds three of the four 8d arms at 5.96e-08 (Up), + // 5.96e-08 (Down) and 1.19209e-07 (Wide) and all four 8f tails at 1.49e-08, + // 4.47e-08, 2.98e-08 and 7.45e-09; Same passes at zero because it is a copy. The + // port mirrors torchaudio's float32 kernel arithmetic operation for operation, + // so the only residual is the difference between two libm `sin`/`cos` at the + // same f32 argument plus the convolution's reduction order. Building the filter + // in `double` instead would land 3.39746e-06 out — 13.6 times this bound — + // which is why a wider dtype FAILS this gate rather than passing it more + // comfortably. + struct Arm { + const char* tag; + const char* input; + int64_t orig_rate; + int64_t new_rate; + int64_t in_samples; + int64_t out_samples; + const float* golden; + size_t golden_size; + }; + constexpr double kResampleTol = 2.5e-7; + const int64_t channels = vllm_test::kLtx2ResampleChannels; + const Arm arms[] = { + {"Up (o = 1, the BWE ratio)", "ltx2.resample.Up", vllm_test::kLtx2ResampleUpOrigRate, + vllm_test::kLtx2ResampleUpNewRate, vllm_test::kLtx2ResampleUpInSamples, + vllm_test::kLtx2ResampleUpOutSamples, vllm_test::kLtx2ResampleUpGolden, + std::size(vllm_test::kLtx2ResampleUpGolden)}, + {"Down (n = 1)", "ltx2.resample.Down", vllm_test::kLtx2ResampleDownOrigRate, + vllm_test::kLtx2ResampleDownNewRate, vllm_test::kLtx2ResampleDownInSamples, + vllm_test::kLtx2ResampleDownOutSamples, vllm_test::kLtx2ResampleDownGolden, + std::size(vllm_test::kLtx2ResampleDownGolden)}, + {"Wide (o = 441, width 17)", "ltx2.resample.Wide", vllm_test::kLtx2ResampleWideOrigRate, + vllm_test::kLtx2ResampleWideNewRate, vllm_test::kLtx2ResampleWideInSamples, + vllm_test::kLtx2ResampleWideOutSamples, vllm_test::kLtx2ResampleWideGolden, + std::size(vllm_test::kLtx2ResampleWideGolden)}, + {"Same (the early return)", "ltx2.resample.Same", vllm_test::kLtx2ResampleSameOrigRate, + vllm_test::kLtx2ResampleSameNewRate, vllm_test::kLtx2ResampleSameInSamples, + vllm_test::kLtx2ResampleSameOutSamples, vllm_test::kLtx2ResampleSameGolden, + std::size(vllm_test::kLtx2ResampleSameGolden)}, + }; + + for (const Arm& arm : arms) { + // `std::string`, not the bare pointer: doctest stringifies a `const char*` + // as a BOOL, so both `CAPTURE(arm.tag)` and `INFO("arm: " << arm.tag)` print + // `arm: 1` and the diagnostic cannot name which ratio broke. + INFO("arm: " << std::string(arm.tag)); + const std::vector in = Ltx2Input(arm.input, channels * arm.in_samples, 0.5); + int64_t produced = 0; + const std::vector got = vllm::Ltx2ResampleWaveform( + in, channels, arm.in_samples, arm.orig_rate, arm.new_rate, &produced); + // `ceil` of the f32-narrowed `new * length / orig` (functional.py:1427; the + // narrowing is what section 8f below gates). Asserted before the + // values because a resampler that produced the right SAMPLES at the wrong + // LENGTH would be compared against a shifted golden. + CHECK(produced == arm.out_samples); + REQUIRE(got.size() == arm.golden_size); + const double err = MaxAbsDiff(got, arm.golden, arm.golden_size); + INFO("resample max|diff| = " << err); + CHECK(err <= kResampleTol); + // A LOWER bound as well. An all-zero output, or one that dropped every + // channel but the first, matches nothing here but would satisfy a + // tolerance against a golden regenerated from the same defect. + double absmax = 0.0; + for (float v : got) absmax = std::max(absmax, std::abs(static_cast(v))); + CHECK(absmax > 0.0); + double second_channel_absmax = 0.0; + for (int64_t i = produced; i < 2 * produced; ++i) { + second_channel_absmax = + std::max(second_channel_absmax, std::abs(static_cast(got[static_cast(i)]))); + } + CHECK(second_channel_absmax > 0.0); + } + + // Section 8f — THE TRUNCATION BOUNDARY, which no arm above can reach. + // + // Upstream truncates to + // `torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` + // (functional.py:1427), and `torch.as_tensor` of a PYTHON FLOAT takes + // `torch.get_default_dtype()` — float32. The f64 quotient is therefore rounded + // to f32 BEFORE the ceil, and where the exact quotient sits above an integer by + // less than half an f32 ulp the narrowing lands ON that integer: upstream keeps + // one sample FEWER than the exact integer ceil `(next * samples + orig - 1) / + // orig` this port used to compute. + // + // The four arms above top out at 218 output samples and cannot see it. At + // 44100 -> 16000 the first divergent length is 180697 (4.097 s) and 48102 of + // the first 60 s worth of lengths diverge; at 22050 -> 16000 it starts at + // 90569. At 48000 -> 16000 (o = 3) and 16000 -> 48000 (o = 1) it never happens, + // which is why three of the four arms above look clean at any length. One extra + // output sample moves the last STFT windows and, where `samples % hop == 0`, + // the mel FRAME COUNT — the conditioning shape. + // + // `CeilBelow` and `CeilAbove` bracket the boundary and are lengths where the + // exact ceil is RIGHT, so an arm that always subtracted one would fail them. + struct CeilArm { + const char* tag; + const char* input; + int64_t orig_rate; + int64_t new_rate; + int64_t in_samples; + int64_t out_samples; + const float* tail; + size_t tail_size; + }; + const CeilArm ceil_arms[] = { + {"CeilBelow (exact ceil agrees)", "ltx2.resample.CeilBelow", + vllm_test::kLtx2ResampleCeilBelowOrigRate, vllm_test::kLtx2ResampleCeilBelowNewRate, + vllm_test::kLtx2ResampleCeilBelowInSamples, vllm_test::kLtx2ResampleCeilBelowOutSamples, + vllm_test::kLtx2ResampleCeilBelowTailGolden, + std::size(vllm_test::kLtx2ResampleCeilBelowTailGolden)}, + {"CeilAt (the f32 narrowing bites)", "ltx2.resample.CeilAt", + vllm_test::kLtx2ResampleCeilAtOrigRate, vllm_test::kLtx2ResampleCeilAtNewRate, + vllm_test::kLtx2ResampleCeilAtInSamples, vllm_test::kLtx2ResampleCeilAtOutSamples, + vllm_test::kLtx2ResampleCeilAtTailGolden, + std::size(vllm_test::kLtx2ResampleCeilAtTailGolden)}, + {"CeilAbove (exact ceil agrees)", "ltx2.resample.CeilAbove", + vllm_test::kLtx2ResampleCeilAboveOrigRate, vllm_test::kLtx2ResampleCeilAboveNewRate, + vllm_test::kLtx2ResampleCeilAboveInSamples, vllm_test::kLtx2ResampleCeilAboveOutSamples, + vllm_test::kLtx2ResampleCeilAboveTailGolden, + std::size(vllm_test::kLtx2ResampleCeilAboveTailGolden)}, + {"CeilAlt (22050 -> 16000, the same effect at another ratio)", + "ltx2.resample.CeilAlt", vllm_test::kLtx2ResampleCeilAltOrigRate, + vllm_test::kLtx2ResampleCeilAltNewRate, vllm_test::kLtx2ResampleCeilAltInSamples, + vllm_test::kLtx2ResampleCeilAltOutSamples, vllm_test::kLtx2ResampleCeilAltTailGolden, + std::size(vllm_test::kLtx2ResampleCeilAltTailGolden)}, + }; + + for (const CeilArm& arm : ceil_arms) { + INFO("ceil arm: " << std::string(arm.tag)); + REQUIRE(arm.tail_size == static_cast(vllm_test::kLtx2ResampleCeilTail)); + const std::vector in = Ltx2Input(arm.input, arm.in_samples, 0.5); + int64_t produced = 0; + const std::vector got = vllm::Ltx2ResampleWaveform(in, 1, arm.in_samples, arm.orig_rate, + arm.new_rate, &produced); + INFO("produced = " << produced << ", upstream = " << arm.out_samples); + CHECK(produced == arm.out_samples); + REQUIRE(got.size() == static_cast(produced)); + REQUIRE(produced >= static_cast(arm.tail_size)); + // The tail as well as the count. A port that produced upstream's LENGTH from + // a signal shifted by a sample would satisfy the count on its own; these are + // the last `kLtx2ResampleCeilTail` samples upstream actually emitted. + const std::vector tail(got.end() - static_cast(arm.tail_size), + got.end()); + const double tail_err = MaxAbsDiff(tail, arm.tail, arm.tail_size); + INFO("tail max|diff| = " << tail_err); + CHECK(tail_err <= kResampleTol); + double tail_absmax = 0.0; + for (float v : tail) tail_absmax = std::max(tail_absmax, std::abs(static_cast(v))); + CHECK(tail_absmax > 0.0); + } + + // The equal-rate arm returns the INPUT, byte for byte. Upstream returns the + // same `Audio` object (ops.py:38-39) and never enters the filter; a port that + // ran a unit-ratio filter anyway would be wrong by its passband ripple, which + // is under the tolerance above and would pass every check in the loop. + const std::vector same_in = + Ltx2Input("ltx2.resample.Same", channels * vllm_test::kLtx2ResampleSameInSamples, 0.5); + const std::vector same_out = vllm::Ltx2ResampleWaveform( + same_in, channels, vllm_test::kLtx2ResampleSameInSamples, + vllm_test::kLtx2ResampleSameOrigRate, vllm_test::kLtx2ResampleSameNewRate, nullptr); + CHECK(same_out == same_in); + + // Upstream's own refusal, at the same boundary (functional.py:1470-1471). + bool threw = false; + try { + vllm::Ltx2ResampleWaveform(same_in, channels, vllm_test::kLtx2ResampleSameInSamples, 16000, 0, + nullptr); + } catch (const std::exception&) { + threw = true; + } + CHECK(threw); +} + TEST_CASE("ltx2 vae: waveform_to_mel matches upstream AudioProcessor") { vllm::Ltx2AudioProcessorConfig cfg; cfg.target_sample_rate = vllm_test::kLtx2MelRate; @@ -2838,21 +3016,37 @@ TEST_CASE("ltx2 vae: waveform_to_mel matches upstream AudioProcessor") { INFO("waveform_to_mel max|diff| = " << err); CHECK(err <= kLtx2GoldenTol); - // A rate that does not match is REFUSED, because upstream would RESAMPLE and - // this project does not carry that resampler. Treating the samples as if they - // were already at the target rate conditions on audio that is pitched wrong. - bool threw = false; - std::string message; - try { - vllm::Ltx2WaveformToLogMel(cfg, wave, channels, samples, 44100, nullptr); - } catch (const std::exception& error) { - threw = true; - message = error.what(); - } - REQUIRE(threw); - INFO("refusal message: " << message); - CHECK(message.find("resample") != std::string::npos); - CHECK(message.find("44100") != std::string::npos); + // A rate that does NOT match is RESAMPLED, which is what upstream does + // (`waveform_to_mel` calls `resample_audio` first, ops.py:49) and what this + // project refused to do until row LTX25-AUDIO-RESAMPLE (#2583). + // + // Section 8e's golden is the whole claim: the source is 600 samples at 44100, + // the processor targets 16000, and the mel that comes back is over the + // RESAMPLED 218 samples. A build that resampled after the transform, or that + // never resampled at all, produces a different frame count before it produces + // a different value, so the count is asserted first. + int64_t resampled_frames = 0; + const int64_t source_samples = vllm_test::kLtx2MelSourceSamples; + const std::vector source = + Ltx2Input("ltx2.mel.resampled.input", channels * source_samples, 0.5); + const std::vector resampled_mel = vllm::Ltx2WaveformToLogMel( + cfg, source, channels, source_samples, vllm_test::kLtx2MelSourceRate, &resampled_frames); + CHECK(resampled_frames == vllm_test::kLtx2MelResampledFrames); + REQUIRE(static_cast(resampled_mel.size()) == + static_cast(std::size(vllm_test::kLtx2MelResampledGolden))); + const double resampled_err = MaxAbsDiff(resampled_mel, vllm_test::kLtx2MelResampledGolden, + std::size(vllm_test::kLtx2MelResampledGolden)); + INFO("resampled waveform_to_mel max|diff| = " << resampled_err); + CHECK(resampled_err <= kLtx2GoldenTol); + + // And it is NOT the mel of the same samples read as if they were already at + // the target rate — the exact wrong answer the old refusal existed to + // prevent, and the one a tolerance against the golden alone cannot see if the + // golden were ever regenerated from the wrong side. + const std::vector misread = + vllm::Ltx2WaveformToLogMel(cfg, source, channels, source_samples, + cfg.target_sample_rate, nullptr); + CHECK(misread.size() != resampled_mel.size()); } TEST_CASE("ltx2 vae: SILENCE saturates the mel log clamp, and the clamp is pinned") { diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 1f8ce6b078..312c9c0133 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -7564,6 +7564,106 @@ TEST_CASE("ltx2 video: a supplied audio file CONDITIONS the render, and stays FR } } +TEST_CASE("ltx2 video: a take at ANOTHER rate is RESAMPLED, not refused") { + // Row LTX25-AUDIO-RESAMPLE, issue #2583, gap A19 of the completion plan. + // + // Enters through the PRODUCTION path — `LoadVideoEngine` + `Generate`, what + // `vllm_video_generate` calls straight through (`vllm_c.cpp:1646`) — for the + // reason in this section's header: a unit test over `Ltx2ResampleWaveform` + // proves the filter works and never that a request can arrive at it. Until + // this row, a request at any rate but the checkpoint's was refused three hops + // earlier and NOTHING downstream of that refusal ran. + // + // Upstream refuses nothing here: `waveform_to_mel` calls `resample_audio` + // before the mel transform (ops.py:44-49) and `resample_audio` filters + // whenever the rates differ (ops.py:36-42). + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + + // THREE takes of the same 220 Hz tone, and the third is the control: + // native — 2.0 s at the fixture's own 24000, the arm that always worked. + // high — 2.0 s at 44100. Same sound, more samples; must resample. + // misread— `high`'s PCM bytes with 24000 written in the header. This is + // what reading the samples "as if they were already at the target + // rate" produces: 3.675 s of a 119.7 Hz tone. It is a legal WAV, so + // it renders, and it is the wrong answer the refusal prevented. + const std::string native = WriteWav(ws.root + "/native.wav", 2, kFixtureAudioRate, 2.0); + const std::string high = WriteWav(ws.root + "/high.wav", 2, 44100, 2.0); + const std::string misread = ws.root + "/misread.wav"; + { + std::string bytes = MakeWavPcm16(2, 44100, 2.0); + // The `fmt ` chunk's sample-rate field: "RIFF" + size + "WAVE" + "fmt " + + // size = 20 bytes, then wFormatTag(2) + nChannels(2) = 4 more. + const auto rate = static_cast(kFixtureAudioRate); + for (int i = 0; i < 4; ++i) { + bytes[24 + static_cast(i)] = static_cast((rate >> (8 * i)) & 0xFF); + } + std::ofstream out(misread, std::ios::binary); + REQUIRE(out.good()); + out.write(bytes.data(), static_cast(bytes.size())); + out.close(); + } + + auto render = [&](const std::string& dir, const std::string& wav) { + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/" + dir); + gen.extras[vllm::multimodal::kLtx2AudioPathExtra] = wav; + const vllm::multimodal::VideoResult result = engine->Generate(gen); + return std::make_pair(result, ltx2->last_conditioning()); + }; + + const auto [native_result, native_trace] = render("rs_native", native); + const auto [high_result, high_trace] = render("rs_high", high); + const auto [misread_result, misread_trace] = render("rs_misread", misread); + + // (1) Accepted, conditioned, and not zeroed. Before this row the second call + // threw, which is the red this case was written on. + CHECK(high_trace.completed); + CHECK(high_trace.audio_conditioned); + CHECK(high_trace.audio_tokens > 0); + CHECK(high_trace.audio_latent_absmax > 0.0); + CHECK(high_trace.audio_latent_digest != 0); + + // (2) The RETURNED soundtrack stays at the FILE's rate. Upstream hands back + // the original `Audio` untouched (a2vid_two_stage.py:301-303), so the + // resample reaches the encoder and nothing else. A build that resampled + // the take in place instead of inside the mel front-end passes every + // assertion above and fails this one. + CHECK(high_result.sample_rate == 44100); + CHECK(native_result.sample_rate == kFixtureAudioRate); + + // (3) The rate was READ, not ignored. `high` and `misread` carry BYTE + // IDENTICAL PCM and differ only in the header's rate field, so a build + // that never resampled gives them the same latent. + CHECK(high_trace.audio_latent_digest != misread_trace.audio_latent_digest); + + // (4) THE VALUES ARE NOT GATED HERE, and that is a measurement rather than a + // gap left open. The obvious video-level claim is "`high` is the same two + // seconds of the same tone as `native`, so its latent must land NEARER + // `native` than the mis-read take does" — and `audio_latent_absmax` cannot + // carry it. Measured on this fixture, the three takes read 1.07194 + // (native), 1.07293 (high) and 1.07208 (misread): a 0.1% spread across + // three genuinely different waveforms, because the trace's absmax is + // dominated by the encoder's per-channel statistics and not by the take. + // A one-scalar ordering over that population would have passed or failed + // on noise either way. + // + // So the numbers live where they can be gated: `test_ltx2_vae`'s sections + // 8d and 8e hold the resampler and the resampled mel against goldens the + // generator produced by EXECUTING upstream, at 2.5e-07. This case proves + // the request arrives, the rate is read, and the soundtrack comes back + // untouched — the three things a unit test over the filter cannot. + INFO("audio_latent_absmax native=" << native_trace.audio_latent_absmax + << " high=" << high_trace.audio_latent_absmax + << " misread=" << misread_trace.audio_latent_absmax); + + // The render still produced its artifacts; resampling is not a bypass. + CHECK(high_result.frame_count > 0); + CHECK(misread_result.frame_count > 0); +} + TEST_CASE("ltx2 video: every audio-input mismatch is refused BY WHAT IS WRONG") { // Each of these renders a finished clip if it is accepted, which is why every // one is a refusal rather than a conversion. The assertions hold each message @@ -7589,24 +7689,6 @@ TEST_CASE("ltx2 video: every audio-input mismatch is refused BY WHAT IS WRONG") g.extras[vllm::multimodal::kLtx2AudioPathExtra] = w; }; - SUBCASE("a sample rate the mel front-end does not target") { - // Upstream RESAMPLES here (ops.py:40) with an arbitrary-ratio polyphase - // kaiser resampler this project has not ported. Reading 44.1 kHz samples at - // the checkpoint's rate pitches and time-shifts the conditioning while every - // shape checks out, so both rates go in the message. - // - // The TARGET rate asserted here is the fixture's 24000, which is neither the - // shipped 16000 nor `Ltx2ParseAudioEncoderConfig`'s own default. That is the - // point: with the fixture at 16000 this assertion passed against a parser - // that never read `params.sampling_rate` at all. - const std::string wav = WriteWav(ws.root + "/44k.wav", 2, 44100, 2.0); - const std::string message = refusal("rate", just_path, wav); - INFO("refusal: " << message); - CHECK(message.find("44100") != std::string::npos); - CHECK(message.find(std::to_string(kFixtureAudioRate)) != std::string::npos); - CHECK(message.find("ops.py:40") != std::string::npos); - } - SUBCASE("a channel count the encoder does not declare") { // `MiniMaxH3ReadWav` would REPEAT a mono take across both channels // (minimax_h3.h:1839-1845). That is H3's contract; LTX-2 hands the file's own