From 6643b2bbf7f514a5950386a5ec157d3d6757dc2d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 15:08:51 +0000 Subject: [PATCH 1/5] spec(LTX25-AUDIO-RESAMPLE): A19 is the arbitrary ratio, and upstream's window is hann (#2583) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ltx25-completion-scope.md` places A19 at order 7 because it gates A3 and A18: `Ltx2WaveformToLogMel` refuses every rate but the audio VAE's own, so any real take has to arrive pre-resampled. Upstream refuses nothing here; `ops.py:44-49` calls `resample_audio` before the mel transform, unconditionally. The reason written beside that refusal, in nine places across this tree, is wrong. It calls `torchaudio.functional.resample` "an arbitrary-ratio polyphase kaiser resampler". `resample` defaults `resampling_method="sinc_interp_hann"` (torchaudio 2.11 functional.py:1441) and only the `sinc_interp_kaiser` branch builds a kaiser window (:1384-1391); `ops.py:40` passes neither that argument nor `beta`. The window is hann -- the same family `Ltx2HannSincResampleFilter1d` already ports for the vocoder's BWE stage, which is this kernel at `orig_freq == 1`. What is missing is the rational ratio, not the filter. The spec's §4 records the measurement that decides the dtype, because it is the one place this port could quietly go wrong in a way no gate would see. Upstream resamples in float32 all the way down, kernel included, and building the same filter in double lands 1.03e-05 from the oracle where float lands 1.79e-07 -- a wider dtype reading as a looser gate rather than as an error. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-audio-resample.md | 278 ++++++++++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 .agents/specs/ltx25-audio-resample.md diff --git a/.agents/specs/ltx25-audio-resample.md b/.agents/specs/ltx25-audio-resample.md new file mode 100644 index 000000000..41079641b --- /dev/null +++ b/.agents/specs/ltx25-audio-resample.md @@ -0,0 +1,278 @@ +# LTX-2.5 — arbitrary-ratio audio resampling (A19) + +Row: `LTX25-AUDIO-RESAMPLE`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) +(operator-owned; **not edited by this row**). Issue: +[#2583](https://github.com/mudler/vllm.cpp/issues/2583). Plan: +[`ltx25-completion-scope.md`](ltx25-completion-scope.md) §8 order **7**, gap +**A19**, itself issue +[#2526](https://github.com/mudler/vllm.cpp/issues/2526). + +Upstream pins: + +| Reference | Registry id | Revision | +|---|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`) | `ltx-2` | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | +| torchaudio, the module `ops.py:40` calls | — | `2.11.0+cu130`, the wheel the `ltx-2` pin resolves and the goldens were generated with | + +Read from a local checkout at that revision with `git show :`, never +from a working tree, because a working tree can differ from the pin. + +--- + +## 0. Honesty statement — what this row does and does not claim + +This row makes an audio file at **any** sample rate drive an LTX-2.5 render. It +does not claim A3 (`DubItPipeline`), A18 (reference-audio conditioning delivery), +or any IC-LoRA. It unblocks the first two by removing the refusal that stood in +front of both; it does not implement either, and both stay owed by their own +rows. + +**Upstream ships no tests at this pin.** `find . -iname '*test*' -not -path +'./.git/*'` over `Lightricks/LTX-2 @ fd4ded7f` returns empty. So "port the +upstream tests in the same change" has nothing to port from the model author, and +the obligation becomes what §6 does instead: **execute** the upstream module at +the pin and emit its outputs as goldens, plus pin each upstream default against a +`file:line`. + +**No render on real weights is claimed.** The row is gated on reduced-dimension +fixtures and executed-upstream goldens, exactly as `LTX25-A2V-AUDIO-INPUT` was. + +--- + +## 1. Scope + +In: + +* Port `torchaudio.functional.resample` at the defaults `ops.py:40` passes, as an + arbitrary rational-ratio polyphase sinc-hann resampler. +* Call it from `Ltx2WaveformToLogMel`, which is where upstream calls it. +* Drop the two refusals that stand in front of it, and repair the **nine** + statements across this tree that misname upstream's filter as kaiser. + +Out: + +* `sinc_interp_kaiser` and the `beta` knob. Upstream passes neither, and porting + an unreached branch is the shape `## Nothing lands dead` forbids. +* Non-default `lowpass_filter_width`, `rolloff`. Same reason. +* Compressed audio containers, channel mixing, A3, A18. Each is refused or owed + elsewhere and none of them is this gap. + +--- + +## 2. Our baseline — the gap, on this tree + +`Ltx2WaveformToLogMel` refuses any waveform whose rate is not +`config.target_sample_rate` +(`src/vllm/model_executor/models/ltx2_audio_vae.cpp:1051-1060`), and +`Ltx2DecodeAudioWav` refuses the same thing one hop earlier so the message can +name the file (`src/vllm/model_executor/models/ltx2_audio_input.cpp:96-105`). +Between them, **every** audio input at a rate other than the checkpoint's is +turned away, so `a2vid_two_stage` only accepts a take the user resampled with +some other tool first. + +**The refusal misnames the filter, in nine places.** Its message and eight +sibling comments call `torchaudio.functional.resample` "an arbitrary-ratio +polyphase **kaiser** resampler". It is not one. `resample`'s signature defaults +`resampling_method="sinc_interp_hann"` (torchaudio 2.11 +`functional/functional.py:1441`), and `_get_sinc_resample_kernel` builds a kaiser +window only on the `else` of `if resampling_method == "sinc_interp_hann"` +(`:1384-1391`). `ops.py:40` passes neither `resampling_method` nor `beta`, so the +window is `cos(t*pi/lpw/2)**2` — a **hann** window, the same family this tree +already ports for the vocoder's BWE stage at `ltx2_audio_vae.cpp:440` +(`Ltx2HannSincResampleFilter1d`), which is exactly that kernel specialized to +`orig_freq == 1`. + +So what is missing is the **arbitrary rational ratio**, not the window. Sizing +A19 at M was right; the reason written beside it was wrong, and this row corrects +the reason as well as the code. + +The nine sites: `ltx2_audio_input.cpp:101`, `ltx2_audio_vae.cpp:1057`, +`ltx2_audio_vae_encoder.h:173`, `ltx2_video.h:315`, `ltx2_audio_input.h:124`, +`test_ltx2_video.cpp:7518`, and `ltx25-a2v-audio-input.md:155` and `:463`. + +--- + +## 3. Upstream chain — read end to end, not inferred + +| Hop | Upstream | What it settles | +|---|---|---| +| 1 | `a2vid_two_stage.py:196` | `decode_audio_from_file` runs at the file's **native** rate | +| 2 | `decode.py:290-296` | the window (`start_time`, `max_duration`) is applied in samples at the **native** rate, before any resample | +| 3 | `decode.py:173-176` | the waveform is **float32**, `astype(np.float32)`, normalised to [-1, 1] | +| 4 | `a2vid_two_stage.py:200` -> `audio_vae.py:271` | `encode_audio` -> `audio_processor.waveform_to_mel(audio.to(device=device))`; `.to(device=...)` moves, it does not cast | +| 5 | `ops.py:44-49` | `waveform_to_mel` calls `self.resample_audio(audio)` **first**, then the mel transform | +| 6 | `ops.py:36-42` | `resample_audio` returns `audio` unchanged when the rates are equal; otherwise `torchaudio.functional.resample(waveform, orig, target)` and then `.to(device=..., dtype=waveform.dtype)` | +| 7 | `functional.py:1470-1476` | `orig_freq <= 0 or new_freq <= 0` raises; `orig_freq == new_freq` returns the input; `gcd` reduces the ratio | +| 8 | `functional.py:1305-1402` | the kernel: `base_freq = min(o, n) * rolloff`, `width = ceil(lpw * o / base_freq)`, one row per output phase | +| 9 | `functional.py:1405-1432` | zero-pad `(width, width + o)`, `conv1d` stride `o`, transpose-interleave, truncate to `ceil(n * L / o)` | +| 10 | `a2vid_two_stage.py:301-303` | the returned soundtrack is the caller's **original** `Audio`, at its **native** rate. The resample never reaches the output | + +Hop 10 is why nothing downstream of the encoder changes: the file's own rate +stays the render's `sample_rate`, exactly as today. + +--- + +## 4. Port map + +| Upstream | Ours | +|---|---| +| `functional.py:1305-1402` `_get_sinc_resample_kernel` | `Ltx2SincResampleKernel` (file-local), `ltx2_audio_vae.cpp` | +| `functional.py:1405-1432` `_apply_sinc_resample_kernel` | `Ltx2ResampleWaveform`, `ltx2_audio_vae.cpp` | +| `functional.py:1435-1490` `resample` | `Ltx2ResampleWaveform`'s guards and early return | +| `ops.py:36-42` `AudioProcessor.resample_audio` | the call at the head of `Ltx2WaveformToLogMel` | + +`Ltx2ResampleWaveform` is declared in +`include/vllm/model_executor/models/ltx2_audio_vae_encoder.h` beside the mel +front-end it belongs to, because upstream keeps them in one module (`ops.py`). + +### The dtype, and why it is `float` and not `double` + +Upstream resamples in **float32**: hop 3 makes the waveform `np.float32`, hop 4 +does not cast it, and `_get_sinc_resample_kernel` is called with +`dtype=waveform.dtype`, so **the filter itself is built in float32** — `idx`, +`t`, the clamp, the window, `sin(t)/t` and the scale, all of it +(`functional.py:1376-1397`). + +This is not a detail to round up. Building the same filter in `double` and +storing it as `float` is *more accurate than upstream* and therefore **further +from it**: measured against the pinned oracle, a double-built kernel differs by +up to **1.03e-05** where a float-built one differs by **1.79e-07**, a factor of +57. AGENTS.md's *Inherit vLLM defaults* names exactly this failure — a dtype that +is too wide, which no token gate can see. The port mirrors the float32 +arithmetic operation for operation, including the association `sinc * (window * +scale)` (`:1397`) and the fact that `scale = base_freq / orig_freq` is computed +in Python `float` (f64) and *then* narrowed (`:1395`). + +The one deliberate exception is the convolution **accumulator**, which is +`double`. torch's `conv1d` reduction order over 161 taps is a vectorised +implementation detail that no C++ loop reproduces, so the choice is between two +different roundings; the exact one is chosen and annotated. It is an accumulator, +not a stored dtype: every stored value stays `float`. + +### The one shape decision + +`Ltx2WaveformToLogMel` takes `[channels, samples]` channel-major. torchaudio +packs the batch and resamples the last axis, so each channel resamples +independently against the same kernel. The port builds the kernel once and walks +the channels, which is that, not an approximation of it. + +--- + +## 5. Reachability — the sentence the records must carry + +The production entry point is `vllm_video_generate` -> `VideoEngine::Generate` +-> `Ltx2VideoEngine::Generate` -> `Ltx2DecodeAudioWav` +(`src/vllm/multimodal/ltx2_video.cpp:3063`) -> `Ltx2EncodeAudioToLatent` +(`:3068`) -> `Ltx2WaveformToLogMel` +(`src/vllm/model_executor/models/ltx2_audio_input.cpp:200`) -> the new +`Ltx2ResampleWaveform`. Every hop already exists and is already gated; this row +adds the last one and deletes the two refusals that stood in the chain. + +The smallest failing test therefore enters through `LoadVideoEngine` + +`engine->Generate` with a WAV at a rate the fixture's audio VAE does not declare, +per the note at `test_ltx2_video.cpp:7229-7240`. A unit test over +`Ltx2ResampleWaveform` alone would prove the function works and never that a +request can arrive at it. + +**Mutation:** delete the `Ltx2ResampleWaveform` call in `Ltx2WaveformToLogMel` +and the video-level case must red. Recorded in `## Outcome`. + +--- + +## 6. Tests to port + +Upstream ships none (§0). What this change does instead: + +1. **Goldens by execution.** `scripts/gen-ltx2-vae-goldens.py` gains sections + **8d** and **8e**, importing `ltx_core.model.audio_vae.ops.AudioProcessor` + from the `--ltx2` checkout and running `resample_audio` and `waveform_to_mel`. + They ride the generator's existing `kLtx2VaeUpstreamRevision` anchor, which the + C++ suite already asserts against a pinned SHA, so a regeneration against a + different upstream fails the gate rather than replacing the oracle. +2. **The rate pairs are chosen to reach distinct arms**, not for coverage + arithmetic: 16000 -> 24000 (pure upsample, `o = 2`), 44100 -> 24000 + (`gcd = 300`, `o = 147`, `width = 12`, the widest kernel), 48000 -> 24000 + (pure downsample, `o = 2`), and 24000 -> 24000 (the equal-rate early return, + which must be a **copy** and not a filtered pass). +3. **A lower bound, not only a tolerance.** A resampler that returned zeros, or + that returned the input untouched, satisfies a `max|diff| < tol` gate against + the wrong golden and any shape check. The cases assert the output's absmax is + positive **and** that a resampled take differs from the same samples read as + if they were already at the target rate — the exact wrong answer the refusal + was written to prevent. +4. **The end-to-end case**, at `test_ltx2_video.cpp`, replaces the SUBCASE that + asserted the refusal: a 44.1 kHz WAV now renders, and its conditioning trace + carries a non-zero audio latent whose digest **differs** from the same file + written at the fixture's own rate. + +--- + +## 7. Gates + +```sh +cmake --build build -j 4 --target test_ltx2_vae test_ltx2_video +./build/tests/test_ltx2_vae -tc='*resampl*' +./build/tests/test_ltx2_video -tc='*audio*' +scripts/agent-preflight.sh --staged +``` + +Plus regeneration parity: + +```sh +python3 scripts/gen-ltx2-vae-goldens.py --ltx2 ~/_git/LTX-2 \ + --out tests/vllm/models/ltx2_vae_goldens.inc +git diff --exit-code tests/vllm/models/ltx2_vae_goldens.inc +``` + +--- + +## 8. Dependencies + +None. A19 depends on nothing in §8 of the plan; A3 and A18 depend on it. + +--- + +## 9. Work breakdown + +1. Spec (this file), committed first. +2. Generator sections 8d/8e and the regenerated `.inc`. +3. `test_ltx2_vae.cpp` cases, red before the port. +4. `Ltx2ResampleWaveform` plus the `Ltx2WaveformToLogMel` call. +5. Delete the two refusals; repair the nine kaiser statements. +6. `test_ltx2_video.cpp`: the refusal SUBCASE becomes the render case. +7. `docs/FEATURES.md` and `docs/USAGE.md` where they state the rate constraint. + +--- + +## 10. Risks/decisions + +| # | Risk | Handling | +|---|---|---| +| R1 | A `double` filter reads as "better" and silently widens the model path | §4 measures both against the oracle and pins float32 with the number. The gate tolerance is tight enough (2.5e-07) that a double-built kernel **fails** it | +| R2 | Removing a refusal admits an input that then fails deeper, with a worse message | The window and channel refusals stay; only the rate one goes. The `samples > n_fft/2` check in `Ltx2WaveformToLogMel` now sees the **resampled** length, which is upstream's order (hop 5) | +| R3 | A tolerance-only gate passes a resampler that returns the input | §6.3's lower bound and difference assertions | +| R4 | The nine kaiser statements are repaired in prose but the sizing document keeps the claim | `ltx25-completion-scope.md` is another row's spec and is not edited here; the correction is recorded on #2583 and in §2 | +| D1 | Should `Ltx2DecodeAudioWav` keep `want_sample_rate`? | No. Upstream's decoder has no target rate (hop 1); the parameter existed only to raise. Removed, and the one call site updated | + +--- + +## Now + +`ACTIVE`. Implementation in flight on `row/LTX25-AUDIO-RESAMPLE`, issue #2583. + +--- + +## Owed + +* **A real-checkpoint A2V render from a 44.1 kHz source.** The gate here is + reduced-dimension fixtures plus executed-upstream goldens; a full render needs + the GPU lease and belongs to the campaign's render rows. Inherits + [#2526](https://github.com/mudler/vllm.cpp/issues/2526). +* **The `sinc_interp_kaiser` arm and the `beta` knob.** Unreached from + `ops.py:40`; deliberately not ported (§1). Inherits + [#2526](https://github.com/mudler/vllm.cpp/issues/2526). +* **The audio VAE's f32/bf16 tension.** `ltx2_audio_vae.cpp` argues f32 from + upstream's vocoder float32 pin (`vocoder.py:575-580`) while upstream builds + `AudioDecoder` in bf16. This row does not resolve it and does not contradict + it: the resampler's float32 is argued from `ops.py`'s own chain (§4), not from + that pin. Inherits [#2526](https://github.com/mudler/vllm.cpp/issues/2526). From 05204757966f2b5215ce17357a47a5b8e4107883 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 15:37:50 +0000 Subject: [PATCH 2/5] feat(LTX25-AUDIO-RESAMPLE): resample the take instead of refusing it, and the window is hann (#2583) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #2583. Order 7 of `ltx25-completion-scope.md` §8, gap A19. `Ltx2WaveformToLogMel` refused every sample rate but the audio VAE's own, and `Ltx2DecodeAudioWav` refused it one hop earlier so the message could name the file. Between them, an audio-to-video render only accepted a take the caller had already resampled with some other tool. Upstream refuses nothing here: `waveform_to_mel` calls `resample_audio` before the mel transform (ops.py:44-49) and `resample_audio` filters whenever the rates differ (ops.py:36-42). `Ltx2ResampleWaveform` is `torchaudio.functional.resample` at the arguments ops.py:40 passes, which is to say at every default: gcd-reduced rational ratio, `lowpass_filter_width = 6`, `rolloff = 0.99`, one kernel row per output phase, conv with stride `orig`, and truncation to `ceil(new * length / orig)`. It is called from `Ltx2WaveformToLogMel` unconditionally, because upstream's rate test lives inside `resample_audio` and guarding at the call site would move that early return off the production path. The filter is HANN, and nine statements in this tree said kaiser. `resample` defaults `resampling_method="sinc_interp_hann"` (torchaudio 2.11 functional.py:1441) and only the kaiser branch (:1386-1391) builds i0-weighted taps; ops.py:40 passes neither that argument nor `beta`. It is the same window `Ltx2HannSincResampleFilter1d` already builds for the vocoder's BWE stage, which is this kernel at `orig_freq == 1`. What was missing was the rational ratio. All nine are repaired here, including the two in `ltx25-a2v-audio-input.md`, which is the row that wrote them. THE DTYPE IS f32 AND THAT IS THE MIRROR, NOT A SHORTCUT. Upstream's waveform is float32 from the decoder on (decode.py:173-176), `encode_audio` moves it without casting (audio_vae.py:271), and the kernel is built with `dtype=waveform.dtype`, so the filter itself is f32 (:1376-1397). Measured against the pinned oracle at four rate pairs, a `double`-built kernel narrowed to `float` lands 1.03e-05 away; the f32 one lands 1.79e-07. Widening would not have improved the port, it would have loosened the gate by fifty-seven and read as care. The gate is 2.5e-07, twice the measured floor: at 0.0 three of the four arms red at 5.96e-08, 5.96e-08 and 1.19e-07, and the equal-rate arm passes because it is a copy. Sections 8d and 8e of the golden generator run upstream's own `AudioProcessor` at those four ratios and through `waveform_to_mel` at a non-target rate, riding the existing revision anchor. Both reds were the refusal itself. `test_ltx2_video`'s case enters through `LoadVideoEngine` + `Generate` and pairs a 44.1 kHz take with the same PCM bytes relabelled 24 kHz, which is the wrong answer the refusal existed to prevent: deleting the production call site makes those two carry the SAME latent digest while the unit case over `Ltx2ResampleWaveform` stays green. One assertion was written, run, and removed rather than loosened. `audio_latent_absmax` reads 1.07194 / 1.07293 / 1.07208 across three genuinely different takes, so it cannot order them; the reason is recorded where the assertion was and the values are gated at the VAE level instead. This file's own scope note claimed the analysis half was not ported. The mel front-end has lived here since the A2V input row, so that sentence was already false; repaired in flow. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-a2v-audio-input.md | 18 +- .agents/specs/ltx25-audio-resample.md | 73 ++++- docs/FEATURES.md | 2 +- .../model_executor/models/ltx2_audio_input.h | 16 +- .../models/ltx2_audio_vae_encoder.h | 34 ++- include/vllm/multimodal/ltx2_video.h | 17 +- scripts/gen-ltx2-vae-goldens.py | 86 +++++- .../models/ltx2_audio_input.cpp | 15 +- .../model_executor/models/ltx2_audio_vae.cpp | 177 ++++++++++-- src/vllm/multimodal/ltx2_video.cpp | 5 +- tests/vllm/models/ltx2_vae_goldens.inc | 257 ++++++++++++++++++ tests/vllm/models/test_ltx2_vae.cpp | 140 +++++++++- tests/vllm/multimodal/test_ltx2_video.cpp | 118 ++++++-- 13 files changed, 861 insertions(+), 97 deletions(-) diff --git a/.agents/specs/ltx25-a2v-audio-input.md b/.agents/specs/ltx25-a2v-audio-input.md index c434a6653..e91f6d260 100644 --- a/.agents/specs/ltx25-a2v-audio-input.md +++ b/.agents/specs/ltx25-a2v-audio-input.md @@ -150,11 +150,14 @@ nameable. Upstream's stage 1 is **caller-configured and guided** — upstream's and the trajectory is not, and no claim is made that a render here reproduces upstream's A2Vid output. Recorded under `## Owed`. -**Out, and refused by name rather than dropped.** Resampling: upstream resamples -with `torchaudio.functional.resample` (`ops.py:40`), an arbitrary-ratio polyphase -kaiser resampler; this project ports only the integer-ratio hann-sinc variant, -so a sample rate other than the encoder's is refused with the two rates in the -message. Non-PCM16 and non-RIFF containers: upstream reads anything PyAV opens +**Out, and refused by name rather than dropped.** Resampling was out of this row +and is now IN the tree: row `LTX25-AUDIO-RESAMPLE` +([#2583](https://github.com/mudler/vllm.cpp/issues/2583)) ported +`torchaudio.functional.resample` (`ops.py:40`) and deleted the refusal this +paragraph described. It also corrected the reason: the filter is an +arbitrary-ratio polyphase sinc with a HANN window, not a kaiser one, because +`resample` defaults `resampling_method="sinc_interp_hann"` and `ops.py:40` +passes no override. Non-PCM16 and non-RIFF containers: upstream reads anything PyAV opens (`decode.py:252`), and no demuxer is vendored here (`video_api.cpp:115-121` says so explicitly). MP3/FLAC/OGG are therefore refused by name. `RetakePipeline` (#924) and text-to-audio. @@ -460,8 +463,9 @@ in the waveform's own dtype (`ops.py:54`) and the encoder in the parameters' `AudioEncoderConfigurator.from_metadata` and `decode_audio_from_file` at reduced dimensions, which needs PyAV and a written fixture file, and is why it did not ride here. -- **Arbitrary-ratio resampling** — refused by name; needs the polyphase kaiser - resampler upstream uses at `ops.py:40`. +- ~~**Arbitrary-ratio resampling**~~ — CLOSED by row `LTX25-AUDIO-RESAMPLE` + ([#2583](https://github.com/mudler/vllm.cpp/issues/2583)), which ported + `ops.py:36-42`. The filter it names is hann-windowed, not kaiser. - **Compressed audio containers** — refused by name; no demuxer is vendored. - **`Ltx2CreateAudioLatentState` and `Ltx2ConditionAudioByReference` remain test-only.** Both have exactly one call site each and it is a test diff --git a/.agents/specs/ltx25-audio-resample.md b/.agents/specs/ltx25-audio-resample.md index 41079641b..492c7ae44 100644 --- a/.agents/specs/ltx25-audio-resample.md +++ b/.agents/specs/ltx25-audio-resample.md @@ -258,7 +258,78 @@ None. A19 depends on nothing in §8 of the plan; A3 and A18 depend on it. ## Now -`ACTIVE`. Implementation in flight on `row/LTX25-AUDIO-RESAMPLE`, issue #2583. +`DONE` pending review. Landed on `row/LTX25-AUDIO-RESAMPLE`, issue #2583. + +--- + +## Outcome + +### What was measured, and what it decided + +**The dtype, which was the one decision this port could have got quietly wrong.** +Three filters were built and compared against the pinned oracle's own +`AudioProcessor.resample_audio` output at four rate pairs: + +| Filter built in | max abs diff vs the oracle | +|---|---| +| `double`, narrowed to `float` | **1.03e-05** | +| torchaudio's own f64-internal path (`dtype=None`) | 4.71e-06 | +| `float`, mirroring `dtype=waveform.dtype` | **1.79e-07** | + +The `double` filter is *more accurate than upstream* and therefore fifty-seven +times further from it. It would have passed a gate written around it and read as +a careful implementation. §4 states the chain that fixes the answer at f32, and +the gate's 2.5e-07 is set so that a widening fails it. + +**The tolerance is twice the measured floor, not a hedge.** Setting +`kResampleTol` to `0.0` reds three of the four arms, at 5.96e-08 (Up), 5.96e-08 +(Down) and 1.19e-07 (Wide). `Same` passes at zero, because it is a copy. + +**`audio_latent_absmax` cannot gate this, and that is a finding rather than a +gap.** The obvious video-level claim — "the 44.1 kHz take must land nearer the +24 kHz one than the mis-read take does" — was written, run, and **failed**: the +three takes read 1.07194, 1.07293 and 1.07208, a 0.1% spread over three +genuinely different waveforms, because the trace's absmax is dominated by the +encoder's per-channel statistics. The assertion was removed rather than inverted +or loosened, and the reason is recorded where the assertion was. + +### Red before, green after + +| | Before | After | +|---|---|---| +| `test_ltx2_vae -tc='*waveform_to_mel*'` | THREW the rate refusal at `ltx2_audio_vae.cpp:1053` | 48/48 cases, 3217/3217 assertions | +| `test_ltx2_video -tc='*RESAMPLED*'` | THREW `'high.wav' is sampled at 44100 Hz ...` | 110/110 cases, 4870/4870 assertions | + +Both reds are the refusal itself, which is what the row exists to delete. + +### The reachability mutation, and what it proved + +Deleting the `Ltx2ResampleWaveform` call inside `Ltx2WaveformToLogMel` and +rebuilding: + +* the video-level case reds on `high_trace.audio_latent_digest != + misread_trace.audio_latent_digest` — the two takes carry **byte-identical PCM** + and differ only in the header's rate field, so they produce the same digest + (`18285287296143238670`) exactly when nothing read the rate; +* the `waveform_to_mel` case reds on the frame count first (38 vs 14) and the mel + length second (608 vs 224); +* **the section-8d unit case still PASSES.** That is the point of the mutation: + `Ltx2ResampleWaveform` works whether or not anything calls it, and a test that + constructs it by hand measures the class. The tree was restored byte-for-byte + (sha256 `88055b01e9c2bf0bb29d826b2c61dc4c8439a1d8ae6d779a46969b50b0c0212b`) and + both suites re-run green. + +### Rejected + +* **A `double` accumulator for the kernel** — §4, measured above. +* **Porting `sinc_interp_kaiser`** — unreached from `ops.py:40`; it would be a + branch nothing can enter. +* **Keeping `want_sample_rate` on `Ltx2DecodeAudioWav`** — upstream's decoder has + no target rate. The parameter existed only to raise, so it went with the raise. +* **Guarding the resample at the call site** (`if (rate != target)`) rather than + inside `Ltx2ResampleWaveform` — it would have moved upstream's own early return + (`ops.py:38-39`) off the production path, leaving that arm reachable only from + a test. --- diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3ce536fa6..950db32c7 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -195,7 +195,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` and `test_ltx2_device` each gate one direction ALONE, on a forward where the other stream is PRESENT but DISABLED. Swapping the two flags is RED on both arms | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN`, which `modality_scale = 3.0` selects on every video row. On the DEVICE forward too since 2026-08-19 ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | -| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118) | +| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118). ANY sample rate: the mel front-end resamples (`ops.py:36-42`, #2583) and the soundtrack returns at the FILE's rate | | LTX-2.5 keyframe interpolation (`KeyframeInterpolationPipeline`) | LTX-2.5 DiT + spatial upsampler | `ltx2 keyframe:` cases in `test_ltx2_pipeline` / `test_ltx2_video` via `LoadVideoEngine`+`Generate`: frame 0 APPENDS against a `ti2vid_two_stage` control, the x0 invariant on four arms, the 4096 anchor | `--pipeline-kind keyframe_interpolation`. No frame-0 special case, so `--first-frame` is guidance to interpolate FROM; stage 2's audio leaves. `--lora` REQUIRED. `--last-frame` new (#1191). CPU fixtures | | LTX-2.5 two-stage text/image-to-video (`TI2VidTwoStagesPipeline`) | LTX-2.5 DiT + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video` `ltx2 ti2vid:` cases through `LoadVideoEngine`+`Generate`; the x0 invariant on all FOUR arms, and the 4096 anchor read at two geometries against a res_2s control that moves | `--pipeline-kind ti2vid_two_stage`. Guided half-res stage 1 on the UNADAPTED model, plain Euler; distilled 3-sigma stage 2. `--lora` REQUIRED, no `--audio-path`; stage 1's audio leaves. CPU fixtures, Full-model run owed | | LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in | diff --git a/include/vllm/model_executor/models/ltx2_audio_input.h b/include/vllm/model_executor/models/ltx2_audio_input.h index 63541f31a..c2d2dc536 100644 --- a/include/vllm/model_executor/models/ltx2_audio_input.h +++ b/include/vllm/model_executor/models/ltx2_audio_input.h @@ -118,13 +118,17 @@ Ltx2WavFormat Ltx2ProbeWavFormat(const std::string& bytes); // // `want_channels` is the encoder's `in_channels`; a file that declares anything // else is REFUSED rather than mixed, for the reason in this header's third -// bullet. `want_sample_rate` is the mel front-end's target; a mismatch is -// REFUSED rather than resampled, because this project ports only the -// integer-ratio hann-sinc resampler and upstream's `ops.py:40` is an -// arbitrary-ratio polyphase kaiser one. +// bullet. +// +// THERE IS NO TARGET RATE HERE, and that is upstream's shape rather than an +// omission: `decode_audio_from_file` reads at the file's own rate +// (`decode.py:240-300`) and the resample happens two hops later, inside the mel +// front-end (`ops.py:44-49`). The rendered soundtrack is this decode's output, +// so it stays at the file's rate too (`a2vid_two_stage.py:301-303`). Row +// LTX25-AUDIO-RESAMPLE (#2583) removed the refusal that used to sit here. Ltx2DecodedAudio Ltx2DecodeAudioWav(const std::string& bytes, const std::string& label, - int64_t want_channels, int64_t want_sample_rate, - double start_time, double max_duration); + int64_t want_channels, double start_time, + double max_duration); // `encode_audio` (audio_vae.py:249-274) followed by A2Vid's truncation // (a2vid_two_stage.py:201-202): waveform -> log-mel -> encoder -> keep at most diff --git a/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h b/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h index 7e88063e0..73c0b5806 100644 --- a/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h +++ b/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h @@ -17,6 +17,7 @@ // model/audio_vae/attention.py:16-55 // (the audio patchifier) <- components/patchifiers.py:287-305 // (per-channel statistics) <- model/audio_vae/ops.py:58-75 +// Ltx2ResampleWaveform <- model/audio_vae/ops.py:36-42 // Ltx2WaveformToLogMel <- model/audio_vae/ops.py:8-55 // Ltx2SlaneyMelFilterbank <- torchaudio.functional.melscale_fbanks // (reached from ops.py:20-34) @@ -165,22 +166,35 @@ struct Ltx2AudioProcessorConfig { std::vector Ltx2SlaneyMelFilterbank(int64_t n_freqs, double f_min, double f_max, int64_t n_mels, int64_t sample_rate); +// `AudioProcessor.resample_audio` (ops.py:36-42), which is +// `torchaudio.functional.resample` at every default: the arbitrary +// rational-ratio polyphase sinc resampler with `lowpass_filter_width = 6`, +// `rolloff = 0.99` and — `ops.py:40` passes neither `resampling_method` nor +// `beta` — a HANN window (functional.py:1441). It is NOT kaiser, which nine +// statements in this tree asserted before row LTX25-AUDIO-RESAMPLE (#2583). +// +// `waveform` is [channels, samples] channel-major and each channel resamples +// independently against one kernel, as torchaudio's packed batch does. Returns +// `ceil(new_freq * samples / orig_freq)` samples per channel, and returns the +// INPUT unfiltered when the rates already match (ops.py:38-39). Computed in f32, +// which is the dtype upstream resamples in — see the note at the definition. +std::vector Ltx2ResampleWaveform(const std::vector& waveform, int64_t channels, + int64_t samples, int64_t orig_freq, int64_t new_freq, + int64_t* out_samples); + // `AudioProcessor.waveform_to_mel` (ops.py:44-55) at any channel count. -// `waveform` is [channels, samples], channel-major, at `sampling_rate`. +// `source` is [channels, source_samples], channel-major, at `sampling_rate`. // -// A SAMPLE RATE THAT DOES NOT MATCH `target_sample_rate` IS REFUSED BY NAME. -// Upstream resamples with `torchaudio.functional.resample` (ops.py:40), a -// polyphase kaiser resampler for an arbitrary rational ratio; this project ports -// only the integer-ratio hann-sinc variant the BWE stage needs -// (Ltx2HannSincResampleFilter1d). Refusing is deliberate: silently treating -// 44.1 kHz samples as 16 kHz produces audio conditioning that is pitched and -// time-scaled wrong while every shape still checks out. +// A SAMPLE RATE THAT DOES NOT MATCH `target_sample_rate` IS RESAMPLED, not +// refused: `waveform_to_mel` calls `resample_audio` before the mel transform +// (ops.py:49), so the frame count this reports is over the RESAMPLED length. // // Returns [channels, frames, mel_bins] channel-major — the layout // `Ltx2AudioEncoderForward` takes, which is upstream's final // `permute(0, 1, 3, 2)` (ops.py:55). std::vector Ltx2WaveformToLogMel(const Ltx2AudioProcessorConfig& config, - const std::vector& waveform, int64_t channels, - int64_t samples, int64_t sampling_rate, int64_t* out_frames); + const std::vector& source, int64_t channels, + int64_t source_samples, int64_t sampling_rate, + int64_t* out_frames); } // namespace vllm diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 5c778b46b..78cfd54f9 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -310,13 +310,16 @@ inline constexpr char kLtx2ImageCrfExtra[] = "image_crf"; // the ordinary text-to-video render. // // A 16-bit PCM RIFF/WAVE file whose channel count matches the checkpoint's audio -// VAE encoder `in_channels` and whose sample rate matches its mel front-end. -// Neither is converted: upstream resamples with an arbitrary-ratio polyphase -// kaiser resampler (`ops.py:40`) this project has not ported, and it feeds the -// file's own channel count straight into a conv that declares 2 -// (`model_configurator.py:172`). Both mismatches are refused with both numbers -// in the message, because a resampled-wrong or upmixed-wrong take conditions the -// render on a waveform the caller never supplied and still finishes. +// VAE encoder `in_channels`. ANY SAMPLE RATE is accepted: the mel front-end +// resamples it to the checkpoint's own, exactly as upstream does +// (`ops.py:36-49`, ported by row LTX25-AUDIO-RESAMPLE, #2583). The rendered +// soundtrack still comes back at the FILE's rate, because upstream returns the +// caller's original take rather than a VAE round trip (`:301-303`). +// +// The CHANNEL count is still refused, with both numbers in the message: +// upstream feeds the file's own count straight into a conv that declares 2 +// (`model_configurator.py:172`), so an upmixed-wrong take would condition the +// render on a waveform the caller never supplied and still finish. // // The audio is held FROZEN through every denoise phase — upstream's // `ModalitySpec(frozen=True, noise_scale=0.0)` at `a2vid_two_stage.py:251-256` diff --git a/scripts/gen-ltx2-vae-goldens.py b/scripts/gen-ltx2-vae-goldens.py index 1a5ddbdf7..ad0dc84a7 100644 --- a/scripts/gen-ltx2-vae-goldens.py +++ b/scripts/gen-ltx2-vae-goldens.py @@ -17,7 +17,8 @@ model/video_vae/conv_video_decoder.py -> section 5 (Conv video decoder) model/video_vae/video_vae.py -> section 6 (VideoEncoder) model/audio_vae/audio_vae.py -> section 7 (AudioEncoder) - model/audio_vae/ops.py -> section 8 (AudioProcessor mel front-end) + model/audio_vae/ops.py -> section 8 (AudioProcessor mel front-end, + and 8d/8e its RESAMPLER) conditioning/types/*.py -> section 9 (the conditioning items) Usage: @@ -1007,6 +1008,36 @@ def noncausal_randn(*args, **kwargs): MEL = dict(target_sample_rate=16000, mel_bins=8, mel_hop_length=16, n_fft=64) MEL_SAMPLES = 200 +# Section 8d — `AudioProcessor.resample_audio` (ops.py:36-42). Four rate pairs, +# each chosen for an ARM rather than for coverage arithmetic: +# +# (Up) 16000 -> 48000, gcd 16000 => o = 1. The degenerate ratio the tree +# already ports for the vocoder's BWE stage, so a port that only +# handles this one still passes here and fails the next two. +# (Down) 48000 -> 16000, o = 3, n = 1. `width` is 19 and the kernel has ONE +# phase row, which is the transpose of the arm above. +# (Wide) 44100 -> 16000, gcd 100 => o = 441, n = 160, `width` 17. The widest +# kernel and the only pair here whose `t == 0` tap is not at an obvious +# index. This is also the pair a real 44.1 kHz take hits. +# (Same) 16000 -> 16000. `resample` returns the INPUT (functional.py:1473), +# and `resample_audio` returns before it is even called (ops.py:38-39). +# A port that filtered anyway would be wrong by the filter's own +# passband ripple — small enough to pass a loose tolerance. +RESAMPLE_CASES = ( + ("Up", 16000, 48000, 64), + ("Down", 48000, 16000, 192), + ("Wide", 44100, 16000, 600), + ("Same", 16000, 16000, 64), +) +RESAMPLE_CHANNELS = 2 + +# Section 8e — `waveform_to_mel` at a rate the processor does NOT target, which +# is the production shape: `resample_audio` runs first and the mel transform then +# sees the RESAMPLED length (ops.py:44-49). 600 samples at 44100 resample to 218 +# at 16000, comfortably past the `n_fft // 2` reflect-pad floor. +MEL_SOURCE_RATE = 44100 +MEL_SOURCE_SAMPLES = 600 + def section_video_encoder(out) -> None: import torch @@ -1319,6 +1350,59 @@ def section_audio_mel(out) -> None: out.write("\n") emit_f32(out, "kLtx2MelQuietGolden", quiet.numpy()) + # --- 8d: resample_audio on its own, at four ratios (ops.py:36-42) --- + # + # Run through `AudioProcessor.resample_audio` rather than through + # `torchaudio.functional.resample` directly, so the goldens carry upstream's + # OWN call — its argument order, its equal-rate early return, and its + # `.to(dtype=waveform.dtype)` — and not this generator's reading of it. + out.write("// --- section 8d: AudioProcessor.resample_audio (ops.py:36-42) ---\n") + emit_scalar(out, "kLtx2ResampleChannels", RESAMPLE_CHANNELS) + for tag, orig, target, length in RESAMPLE_CASES: + wave = make_input( + f"ltx2.resample.{tag}", (1, RESAMPLE_CHANNELS, length), 0.5 + ) + proc = AudioProcessor( + target_sample_rate=target, + mel_bins=MEL["mel_bins"], + mel_hop_length=MEL["mel_hop_length"], + n_fft=MEL["n_fft"], + ) + got = proc.resample_audio(Audio(waveform=wave, sampling_rate=orig)) + assert got.sampling_rate == target + assert got.waveform.dtype == torch.float32, ( + "upstream resamples in the WAVEFORM's dtype; a golden emitted from " + "anything but float32 would gate the wrong arithmetic" + ) + if orig == target: + assert got.waveform.data_ptr() == wave.data_ptr(), ( + "ops.py:38-39 returns the SAME Audio when the rates match; a " + "golden that went through the filter would not gate that branch" + ) + assert float(got.waveform.abs().max()) > 0.0, "an all-zero golden gates nothing" + emit_scalar(out, f"kLtx2Resample{tag}OrigRate", orig) + emit_scalar(out, f"kLtx2Resample{tag}NewRate", target) + emit_scalar(out, f"kLtx2Resample{tag}InSamples", length) + emit_scalar(out, f"kLtx2Resample{tag}OutSamples", got.waveform.shape[-1]) + emit_f32(out, f"kLtx2Resample{tag}Golden", got.waveform.numpy()) + + # --- 8e: waveform_to_mel THROUGH the resampler, the production shape --- + # + # 8d proves the filter; this proves the CALL. `waveform_to_mel` resamples + # first (ops.py:49) and the mel transform then runs on the resampled length, + # so a port that resampled AFTER the transform, or that padded before it, + # matches 8d exactly and fails here. + source = make_input("ltx2.mel.resampled.input", (1, 2, MEL_SOURCE_SAMPLES), 0.5) + resampled_mel = processor.waveform_to_mel( + Audio(waveform=source, sampling_rate=MEL_SOURCE_RATE) + ) + out.write("// --- section 8e: waveform_to_mel at a NON-target rate (ops.py:44-55) ---\n") + emit_scalar(out, "kLtx2MelSourceRate", MEL_SOURCE_RATE) + emit_scalar(out, "kLtx2MelSourceSamples", MEL_SOURCE_SAMPLES) + emit_scalar(out, "kLtx2MelResampledFrames", resampled_mel.shape[2]) + out.write("\n") + emit_f32(out, "kLtx2MelResampledGolden", resampled_mel.numpy()) + # --------------------------------------------------------------------------- # Section 9 — the CONDITIONING ITEMS (phase L11). What the encoders' output is diff --git a/src/vllm/model_executor/models/ltx2_audio_input.cpp b/src/vllm/model_executor/models/ltx2_audio_input.cpp index 067343e01..ff87fc87b 100644 --- a/src/vllm/model_executor/models/ltx2_audio_input.cpp +++ b/src/vllm/model_executor/models/ltx2_audio_input.cpp @@ -71,8 +71,8 @@ Ltx2WavFormat Ltx2ProbeWavFormat(const std::string& bytes) { } Ltx2DecodedAudio Ltx2DecodeAudioWav(const std::string& bytes, const std::string& label, - int64_t want_channels, int64_t want_sample_rate, - double start_time, double max_duration) { + int64_t want_channels, double start_time, + double max_duration) { const Ltx2WavFormat fmt = Ltx2ProbeWavFormat(bytes); // Refused BEFORE the decode, because `MiniMaxH3ReadWav` would silently repeat @@ -94,17 +94,6 @@ Ltx2DecodedAudio Ltx2DecodeAudioWav(const std::string& bytes, const std::string& Fail("'" + label + "' is " + std::to_string(fmt.bits_per_sample) + "-bit PCM; only 16-bit is decoded here"); } - if (want_sample_rate > 0 && fmt.sample_rate != want_sample_rate) { - Fail("'" + label + "' is sampled at " + std::to_string(fmt.sample_rate) + - " Hz and this checkpoint's mel front-end targets " + std::to_string(want_sample_rate) + - " Hz (audio_vae/ops.py:19-34). Upstream resamples with " - "`torchaudio.functional.resample` (ops.py:40), an arbitrary-ratio polyphase kaiser " - "resampler; this project ports only the integer-ratio hann-sinc variant, so the rate " - "is refused rather than assumed. Treating these samples as " + - std::to_string(want_sample_rate) + - " Hz would pitch- and time-shift the conditioning while every shape still checked out"); - } - Ltx2DecodedAudio out; out.channels = fmt.channels; out.sample_rate = fmt.sample_rate; diff --git a/src/vllm/model_executor/models/ltx2_audio_vae.cpp b/src/vllm/model_executor/models/ltx2_audio_vae.cpp index 152d9021a..c8b8ea80f 100644 --- a/src/vllm/model_executor/models/ltx2_audio_vae.cpp +++ b/src/vllm/model_executor/models/ltx2_audio_vae.cpp @@ -12,11 +12,12 @@ // numeric contract across the two stages. // // ─── SCOPE, so nothing is discovered later ─────────────────────────────────── -// This is the DECODE direction only: AudioDecoder + Vocoder (+ the BWE chain). -// The ANALYSIS half (`AudioEncoder`, audio_vae.py:60-246, and the mel front-end -// `AudioProcessor`, ops.py:8-55) is what a REFERENCE AUDIO would need, and it is -// NOT ported here — it is owed, and the same is true of the video VAE's encoder -// (see ltx2_video_vae.cpp). +// Both directions now live here. The DECODE half is AudioDecoder + Vocoder +// (+ the BWE chain); the ANALYSIS half is `AudioEncoder` (audio_vae.py:60-246) +// and the whole `AudioProcessor` front-end (ops.py:8-55) — its slaney +// filterbank, its mel transform, and, since row LTX25-AUDIO-RESAMPLE (#2583), +// its resampler. The video VAE's encoder is a different file +// (ltx2_video_vae.cpp). #include "vllm/model_executor/models/ltx2_audio_vae.h" #include "vllm/model_executor/models/ltx2_audio_vae_encoder.h" @@ -25,6 +26,7 @@ #include #include #include +#include #include #include @@ -1044,20 +1046,163 @@ std::vector Ltx2SlaneyMelFilterbank(int64_t n_freqs, double f_min, double return fb; } +namespace { + +// `_get_sinc_resample_kernel` (torchaudio 2.11 functional/functional.py:1305-1402) +// at the arguments `ops.py:40` passes, which is to say at every default. Returns +// `new_freq` phase rows of `2 * width + orig_freq` taps, row-major. `orig_freq` +// and `new_freq` are already REDUCED by their gcd, as upstream reduces them at +// `:1341-1342`. +// +// THE WINDOW IS HANN, and nine comments in this tree used to say kaiser. +// `resample` defaults `resampling_method="sinc_interp_hann"` (`:1441`) and only +// the kaiser branch (`:1386-1391`) builds `i0`-weighted taps; `ops.py:40` passes +// neither that argument nor `beta`. The same window this file already builds for +// the vocoder's BWE stage (`Ltx2HannSincResampleFilter1d`) is this kernel at +// `orig_freq == 1`; what was missing was the arbitrary rational ratio. +// +// EVERY VALUE IS COMPUTED IN `float`, AND THAT IS THE MIRROR RATHER THAN A +// SHORTCUT. Upstream's waveform is float32 from the decoder on +// (`decode.py:173-176`), `encode_audio` moves it without casting +// (`audio_vae.py:271`), and the kernel is built with `dtype=waveform.dtype` +// (`:1483`), so `idx`, `t`, the clamp, the window and `sin(t)/t` are all f32 +// (`:1376-1397`). Measured against the pinned oracle, a `double`-built kernel +// narrowed to `float` lands 1.03e-05 away where an f32-built one lands 1.79e-07: +// widening here does not improve the port, it loosens the gate by a factor of 57 +// and hides exactly what AGENTS.md's "a token gate cannot detect a dtype that is +// too wide" is about. +std::vector Ltx2SincResampleKernel(int64_t orig_freq, int64_t new_freq, + int64_t* out_width) { + constexpr int64_t kLowpassFilterWidth = 6; // functional.py:1439 + constexpr double kRolloff = 0.99; // functional.py:1440 + + // `base_freq` and `width` are Python floats, i.e. f64, and stay f64: only the + // TENSOR arithmetic below narrows (`:1357-1370`). + const double base_freq64 = static_cast(std::min(orig_freq, new_freq)) * kRolloff; + const int64_t width = static_cast( + std::ceil(static_cast(kLowpassFilterWidth) * static_cast(orig_freq) / + base_freq64)); + const int64_t taps = 2 * width + orig_freq; + if (out_width != nullptr) *out_width = width; + + // `t *= base_freq` narrows its operand to the tensor dtype (`:1379`), and so + // does `window * scale` — but `scale = base_freq / orig_freq` is a Python + // DIVISION in f64 that is narrowed only afterwards (`:1395`). + const float base_freq = static_cast(base_freq64); + const float scale = static_cast(base_freq64 / static_cast(orig_freq)); + const float lpw = static_cast(kLowpassFilterWidth); + constexpr float kPi = std::numbers::pi_v; // `math.pi` narrowed, as torch narrows it + + std::vector kernel(static_cast(new_freq * taps)); + for (int64_t j = 0; j < new_freq; ++j) { + // `torch.arange(0, -new_freq, -1) / new_freq` (`:1378`): row j is the output + // phase -j/new_freq. + const float phase = static_cast(-j) / static_cast(new_freq); + for (int64_t k = 0; k < taps; ++k) { + // `idx = torch.arange(-width, width + orig_freq) / orig_freq` (`:1376`). + const float idx = static_cast(k - width) / static_cast(orig_freq); + float t = (phase + idx) * base_freq; + // `:1380`. The SINC reads the clamped t too, because `t *= math.pi` comes + // after (`:1393`). Outside +/-6 the window is cos(pi/2)^2 = 0 exactly, so + // those taps vanish whatever the sinc does — which is why the BWE + // filter's unclamped sinc has never mattered. + t = std::max(-lpw, std::min(lpw, t)); + const float shaped = std::cos(((t * kPi) / lpw) / 2.0f); + const float window = shaped * shaped; // `** 2` at `:1385` + t *= kPi; // `:1393` + // `torch.where(t == 0, 1.0, t.sin() / t)` (`:1396`). The equality is + // upstream's own and fires on exactly one tap per kernel — j = 0, + // k = width — because orig_freq and new_freq are coprime after the gcd. + const float sinc = t == 0.0f ? 1.0f : std::sin(t) / t; + // `kernels *= window * scale` (`:1397`) — that association, not + // `(sinc * window) * scale`, which rounds differently in f32. + kernel[static_cast(j * taps + k)] = sinc * (window * scale); + } + } + return kernel; +} + +} // namespace + +std::vector Ltx2ResampleWaveform(const std::vector& waveform, int64_t channels, + int64_t samples, int64_t orig_freq, int64_t new_freq, + int64_t* out_samples) { + VT_CHECK(channels > 0 && samples > 0, "ltx2 resample: channels and samples must be positive"); + VT_CHECK(static_cast(waveform.size()) == channels * samples, + "ltx2 resample: waveform size does not match [channels, samples]"); + // `resample` (functional.py:1470-1471), the same refusal and the same reason. + VT_CHECK(orig_freq > 0 && new_freq > 0, + "ltx2 resample: original frequency and desired frequency should be positive " + "(functional.py:1470-1471)"); + + if (out_samples != nullptr) *out_samples = samples; + // `:1473-1474`, and `resample_audio` returns even earlier (`ops.py:38-39`). + // The input UNFILTERED, not the input pushed through a unit-ratio filter: the + // filter has passband ripple and would change every sample by a little. + if (orig_freq == new_freq) return waveform; + + const int64_t gcd = std::gcd(orig_freq, new_freq); + const int64_t orig = orig_freq / gcd; // `:1341`, `:1414` + const int64_t next = new_freq / gcd; // `:1342`, `:1415` + int64_t width = 0; + const std::vector kernel = Ltx2SincResampleKernel(orig, next, &width); + const int64_t taps = 2 * width + orig; + + // `_apply_sinc_resample_kernel` (`:1405-1432`): zero-pad by (width, width + + // orig), convolve with stride `orig`, transpose the phase axis in front of the + // block axis, and truncate to `ceil(new * length / orig)`. + const int64_t blocks = samples / orig + 1; + const int64_t target = (next * samples + orig - 1) / orig; // `:1427` + if (out_samples != nullptr) *out_samples = target; + + // ONE kernel for every channel. torchaudio packs the batch and resamples the + // last axis (`:1416-1418`), so the channels are independent; this walks them + // rather than approximating that. + std::vector out(static_cast(channels) * static_cast(target)); + for (int64_t c = 0; c < channels; ++c) { + const float* in = waveform.data() + c * samples; + float* dst = out.data() + c * target; + for (int64_t b = 0; b < blocks; ++b) { + if (b * next >= target) break; + // Where tap 0 of this block lands in the UNPADDED signal. The zero pad is + // skipped rather than materialised, which is the same sum. + const int64_t base = b * orig - width; + const int64_t first = std::max(0, -base); + const int64_t last = std::min(taps, samples - base); + for (int64_t j = 0; j < next; ++j) { + const int64_t index = b * next + j; + if (index >= target) break; + const float* row = kernel.data() + j * taps; + // The one `double` in this function, and it is an ACCUMULATOR rather + // than a stored dtype: torch's vectorised conv1d reduction order is not + // reproducible from a C++ loop, so the choice is between two roundings + // and the exact sum is the defensible one. Every stored value is f32. + double acc = 0.0; + for (int64_t k = first; k < last; ++k) { + acc += static_cast(row[k]) * static_cast(in[base + k]); + } + dst[index] = static_cast(acc); + } + } + } + return out; +} + std::vector Ltx2WaveformToLogMel(const Ltx2AudioProcessorConfig& config, - const std::vector& waveform, int64_t channels, - int64_t samples, int64_t sampling_rate, + const std::vector& source, int64_t channels, + int64_t source_samples, int64_t sampling_rate, int64_t* out_frames) { - VT_CHECK(static_cast(waveform.size()) == channels * samples, + VT_CHECK(static_cast(source.size()) == channels * source_samples, "ltx2 mel: waveform size does not match [channels, samples]"); - VT_CHECK(sampling_rate == config.target_sample_rate, - "ltx2 mel: the waveform is at " + std::to_string(sampling_rate) + " Hz but the audio " - "VAE wants " + std::to_string(config.target_sample_rate) + - " Hz. Upstream resamples with torchaudio.functional.resample (audio_vae/ops.py:36-42), " - "a polyphase kaiser resampler for an arbitrary rational ratio, which is NOT ported — " - "this project carries only the integer-ratio hann-sinc variant the BWE stage needs. " - "Refused rather than reinterpreted: treating the samples as if they were already at " - "the target rate conditions on audio that is pitched and time-scaled wrong"); + // `waveform = self.resample_audio(audio).waveform` (ops.py:49), called + // UNCONDITIONALLY: the rate test lives inside `resample_audio` (ops.py:38-39), + // so the equal-rate arm is on the production path rather than reachable only + // from a test. Everything below — the reflect-pad floor, the frame count, the + // mel — is therefore over the RESAMPLED signal, which is upstream's order. + int64_t samples = 0; + const std::vector waveform = Ltx2ResampleWaveform( + source, channels, source_samples, sampling_rate, config.target_sample_rate, &samples); + const int64_t n_fft = config.n_fft; const int64_t hop = config.mel_hop_length; VT_CHECK(n_fft > 0 && hop > 0, "ltx2 mel: n_fft and hop_length must be positive"); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 3aa77ae2d..e110ac538 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -3060,10 +3060,11 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "; it is a duration in seconds and must be positive"); } + // No target rate: the decode runs at the FILE's rate and the mel front-end + // resamples (ops.py:44-49). See `Ltx2DecodeAudioWav`'s header. a2v_source = Ltx2DecodeAudioWav( ReadFileBytes(kLtx2AudioPathExtra, a2v_audio_path), a2v_audio_path, - im.audio_encoder_cfg.encoder.in_channels, - im.audio_encoder_cfg.processor.target_sample_rate, start_time, max_duration); + im.audio_encoder_cfg.encoder.in_channels, start_time, max_duration); const Ltx2AudioSpectrogram encoded = Ltx2EncodeAudioToLatent( im.audio_encoder_cfg.encoder, im.audio_encoder_cfg.processor, im.audio_encoder_weights, diff --git a/tests/vllm/models/ltx2_vae_goldens.inc b/tests/vllm/models/ltx2_vae_goldens.inc index aeb670c31..ad118dc88 100644 --- a/tests/vllm/models/ltx2_vae_goldens.inc +++ b/tests/vllm/models/ltx2_vae_goldens.inc @@ -4649,6 +4649,263 @@ inline constexpr float kLtx2MelQuietGolden[] = { -11.5129251f, -11.5129251f, -11.5129251f, -11.5129251f, }; +// --- section 8d: AudioProcessor.resample_audio (ops.py:36-42) --- +inline constexpr int64_t kLtx2ResampleChannels = 2; +inline constexpr int64_t kLtx2ResampleUpOrigRate = 16000; +inline constexpr int64_t kLtx2ResampleUpNewRate = 48000; +inline constexpr int64_t kLtx2ResampleUpInSamples = 64; +inline constexpr int64_t kLtx2ResampleUpOutSamples = 192; +inline constexpr float kLtx2ResampleUpGolden[] = { + 0.133884266f, 0.0918205902f, -0.0255323742f, -0.197727516f, -0.371828228f, -0.479041606f, + -0.471846819f, -0.360692859f, -0.214061275f, -0.108339809f, -0.0662166774f, -0.039924968f, + 0.0404968187f, 0.193504661f, 0.354771823f, 0.42060411f, 0.334565669f, 0.139964432f, + -0.0506840423f, -0.138156235f, -0.0982855111f, 0.0225483235f, 0.159993023f, 0.275422126f, + 0.355261415f, 0.382120222f, 0.323688149f, 0.159209579f, -0.0810237005f, -0.312432617f, + -0.448440582f, -0.463129699f, -0.405220091f, -0.34950918f, -0.332130969f, -0.32618162f, + -0.271405935f, -0.127568468f, 0.0891931355f, 0.306307077f, 0.425508291f, 0.371295869f, + 0.141137481f, -0.168693811f, -0.396210313f, -0.409043342f, -0.207575709f, 0.0552373789f, + 0.179578736f, 0.0805262998f, -0.137114063f, -0.272212356f, -0.212493807f, -0.0356906354f, + 0.0736512393f, 0.0123008974f, -0.138699532f, -0.204925239f, -0.110514529f, 0.0321467146f, + 0.0363768227f, -0.159051433f, -0.408695728f, -0.480595052f, -0.266345084f, 0.117857657f, + 0.428332329f, 0.492335737f, 0.329499304f, 0.106610663f, -0.00907083135f, 0.0378266498f, + 0.187097982f, 0.338165194f, 0.423817635f, 0.434502244f, 0.400267035f, 0.361475706f, + 0.344876498f, 0.350619495f, 0.352048993f, 0.311718345f, 0.208393216f, 0.05891959f, + -0.0841019675f, -0.161014602f, -0.140655726f, -0.0377356634f, 0.0911664814f, 0.172342196f, + 0.155045733f, 0.042715624f, -0.10623949f, -0.210352689f, -0.210627481f, -0.10195376f, + 0.0685161501f, 0.232668713f, 0.33972463f, 0.379634589f, 0.378310233f, 0.372507691f, + 0.38568297f, 0.41811499f, 0.448957741f, 0.442723602f, 0.359707177f, 0.176587f, + -0.0858395919f, -0.349144161f, -0.505121112f, -0.483186722f, -0.308442414f, -0.0975251049f, + 0.0148507962f, -0.0292887557f, -0.170069456f, -0.278221965f, -0.254882395f, -0.103100471f, + 0.0820374712f, 0.187563434f, 0.155847415f, 0.00705303298f, -0.193033978f, -0.378515601f, + -0.501759768f, -0.52891165f, -0.443026602f, -0.266485959f, -0.0746473297f, 0.0318640955f, + -0.0101245418f, -0.171410829f, -0.336801201f, -0.382912368f, -0.267979831f, -0.0621753745f, + 0.103031375f, 0.129021317f, 0.0144238407f, -0.155600309f, -0.273973525f, -0.27839604f, + -0.181593135f, -0.0554181561f, 0.0137927886f, -0.0278404783f, -0.170404479f, -0.342470616f, + -0.455148369f, -0.462089509f, -0.390882462f, -0.319350362f, -0.313464373f, -0.378141999f, + -0.461465418f, -0.498804986f, -0.455821276f, -0.341705322f, -0.194867447f, -0.0606985763f, + 0.0247824583f, 0.0487422198f, 0.0282341018f, -0.00259423628f, -0.0235595647f, -0.0477596074f, + -0.101373702f, -0.177613169f, -0.220084473f, -0.172693759f, -0.0544211492f, 0.0281328484f, + -0.0334747769f, -0.23220484f, -0.417596281f, -0.409719974f, -0.161943913f, 0.180644095f, + 0.389402241f, 0.337898821f, 0.100684159f, -0.119783737f, -0.164848462f, -0.0346657261f, + 0.142288923f, 0.235139355f, 0.208460733f, 0.118321925f, 0.0390205905f, 0.00303064473f, + 0.445470721f, 0.29844594f, 0.0789522305f, -0.0442571826f, 0.0190145057f, 0.212463439f, + 0.381696761f, 0.393006206f, 0.226355091f, -0.0186772011f, -0.196490571f, -0.207019269f, + -0.0468566157f, 0.199214488f, 0.405914158f, 0.463017434f, 0.316879481f, -0.00637663249f, + -0.398234218f, -0.693616152f, -0.738651037f, -0.484023809f, -0.0435220301f, 0.34737438f, + 0.474355489f, 0.299590766f, -0.00258852099f, -0.179535776f, -0.103193171f, 0.129609138f, + 0.287667543f, 0.207886934f, -0.0726820678f, -0.367550731f, -0.511022151f, -0.483476281f, + -0.388759345f, -0.339509666f, -0.371335417f, -0.446947157f, -0.509600461f, -0.52418673f, + -0.490478426f, -0.436395705f, -0.400743932f, -0.404196441f, -0.418688267f, -0.369954288f, + -0.194614381f, 0.0875882283f, 0.357672304f, 0.47100696f, 0.36678791f, 0.117068402f, + -0.132126331f, -0.271304518f, -0.294583142f, -0.27867201f, -0.30367738f, -0.388001353f, + -0.485279113f, -0.527561128f, -0.472777069f, -0.331204176f, -0.158322424f, -0.0249618776f, + 0.0148999924f, -0.0481792577f, -0.169395626f, -0.272131115f, -0.291395575f, -0.210498825f, + -0.0661703423f, 0.0708888695f, 0.128283232f, 0.0680397302f, -0.0855218992f, -0.252087831f, + -0.349260598f, -0.355694622f, -0.325025618f, -0.330481172f, -0.382303476f, -0.399712712f, + -0.279376447f, -0.00664910674f, 0.299109846f, 0.460590035f, 0.379405528f, 0.112821877f, + -0.176825255f, -0.350439608f, -0.379978091f, -0.326593995f, -0.25282529f, -0.165309787f, + -0.0369570144f, 0.131423995f, 0.284679413f, 0.350006908f, 0.302066803f, 0.191396281f, + 0.106939793f, 0.112949446f, 0.21247226f, 0.353295594f, 0.46238023f, 0.482803315f, + 0.398368537f, 0.238175258f, 0.0603726916f, -0.0789099857f, -0.15574719f, -0.187806249f, + -0.214972809f, -0.261167616f, -0.31014207f, -0.321638256f, -0.277262509f, -0.207888648f, + -0.166348115f, -0.169614062f, -0.174571082f, -0.121473916f, -0.0060363994f, 0.0980909616f, + 0.102463357f, -0.00375754829f, -0.126347676f, -0.138543501f, 0.00219828752f, 0.203163311f, + 0.307530284f, 0.230149224f, 0.0306903068f, -0.14745301f, -0.206874877f, -0.168353304f, + -0.125326455f, -0.134805664f, -0.158722103f, -0.112727828f, 0.0363009349f, 0.224020794f, + 0.333458394f, 0.2824651f, 0.0785970166f, -0.190376863f, -0.409248203f, -0.493239254f, + -0.427479506f, -0.272100419f, -0.128921509f, -0.0807205066f, -0.139783099f, -0.241041079f, + -0.286729604f, -0.211247459f, -0.0247106366f, 0.188118815f, 0.313987404f, 0.275930554f, + 0.0798854306f, -0.18334426f, -0.383101583f, -0.417240143f, -0.270101786f, -0.0253726505f, + 0.179037392f, 0.233607903f, 0.121847987f, -0.0812300071f, -0.26580596f, -0.351501971f, + -0.318386525f, -0.195205286f, -0.0308631677f, 0.122206084f, 0.213791981f, 0.207223073f, + 0.100514427f, -0.0622927919f, -0.209378287f, -0.283151537f, -0.276329786f, -0.232446685f, + -0.209688485f, -0.233458713f, -0.27620554f, -0.28249678f, -0.219260603f, -0.106768727f, +}; + +inline constexpr int64_t kLtx2ResampleDownOrigRate = 48000; +inline constexpr int64_t kLtx2ResampleDownNewRate = 16000; +inline constexpr int64_t kLtx2ResampleDownInSamples = 192; +inline constexpr int64_t kLtx2ResampleDownOutSamples = 64; +inline constexpr float kLtx2ResampleDownGolden[] = { + -0.0653797016f, -0.073269777f, -0.148023993f, -0.0650461167f, 0.242959961f, 0.350555152f, + 0.0233238004f, -0.292549998f, 0.0432978123f, 0.301227987f, 0.0433950201f, 0.051469449f, + -0.360557288f, -0.11672654f, 0.195395276f, 0.0221682694f, 0.0440903604f, 0.020922726f, + -0.182012573f, 0.00258047739f, -0.0147719998f, 0.070274502f, -0.0313250236f, -0.2670964f, + 0.0618665032f, 0.0976947695f, -0.0370035395f, -0.18601872f, 0.122944795f, -0.200641274f, + -0.0474185012f, -0.237794429f, -0.00165624486f, -0.030814914f, 0.0299725831f, 0.184029549f, + -0.0193744563f, -0.0356619693f, 0.163084179f, -0.0488685742f, -0.0179705899f, 0.101946548f, + 0.138102219f, 0.305152833f, 0.447461009f, -0.107914671f, -0.0785821527f, -0.0753501803f, + -0.170663506f, 0.213874444f, -0.0363523066f, -0.048221752f, -0.105345771f, 0.143479466f, + 0.160106972f, 0.239474237f, 0.182363778f, 0.279522777f, -0.160115555f, 0.011421659f, + -0.141609654f, -0.111503638f, 0.0886406973f, -0.0875492021f, -0.16086942f, -0.277595878f, + 0.0166485943f, -0.140741989f, 0.0211277455f, 0.00768060703f, 0.0280438047f, 0.269561291f, + 0.0397057943f, -0.19806914f, -0.0431697853f, -0.0739519894f, 0.070396401f, -0.160541594f, + -0.00253871456f, -0.0407009572f, 0.187902853f, -0.0259926617f, -0.257088929f, 0.167080641f, + -0.228101581f, -0.0421476401f, 0.234342724f, 0.15414308f, -0.206527114f, -0.00110695569f, + 0.253393978f, -0.208834425f, 0.115428343f, -0.106179774f, 0.0309635643f, 0.216262385f, + -0.0682550222f, 0.129363611f, 0.22991617f, 0.0303576179f, 0.252601564f, 0.11894282f, + 0.298532993f, 0.213497579f, -0.157630861f, -0.316708177f, -0.00704507343f, 0.174568385f, + 0.337846696f, 0.231524006f, -0.0973610133f, 0.0622536652f, -0.114698462f, -0.000667623477f, + -0.162611544f, -0.0165845435f, -0.267340153f, 0.0150169944f, 0.0334990136f, -0.0230757799f, + 0.0803245604f, -0.2463938f, -0.0079144584f, 0.0854698643f, -0.0801748708f, -0.153812885f, + -0.12739481f, -0.146888018f, +}; + +inline constexpr int64_t kLtx2ResampleWideOrigRate = 44100; +inline constexpr int64_t kLtx2ResampleWideNewRate = 16000; +inline constexpr int64_t kLtx2ResampleWideInSamples = 600; +inline constexpr int64_t kLtx2ResampleWideOutSamples = 218; +inline constexpr float kLtx2ResampleWideGolden[] = { + -0.166109875f, -0.0630680472f, 0.0970632955f, -0.151416913f, 0.0240903199f, 0.0663119555f, + 0.0803315341f, 0.00678226678f, 0.198316097f, 0.10671559f, -0.280532002f, 0.239263371f, + -0.261209846f, -0.264212847f, -0.285671055f, 0.0628080517f, 0.00952306949f, 0.059553951f, + -0.0891417712f, 0.0585061572f, 0.0265714284f, -0.0520042107f, 0.0164552033f, -0.0880147219f, + -0.163946286f, 0.0146814911f, 0.222776204f, -0.186651662f, -0.0553991199f, 0.176685765f, + -0.170007318f, -0.113077916f, 0.119050048f, -0.176367268f, -0.129646897f, 0.0907209516f, + -0.0496546365f, 0.133777753f, -0.0100589916f, 0.221267298f, 0.00451244181f, 0.306644559f, + 0.20498611f, -0.13016741f, 0.0865820944f, -0.234014377f, 0.0648791045f, 0.126939878f, + -0.0115708541f, -0.111877613f, 0.14006488f, -0.156115606f, 0.0307647083f, 0.0475151688f, + -0.0394521095f, -0.0205634832f, 0.18426457f, -0.0175724365f, 0.126709551f, 0.126869366f, + 0.0952487066f, 0.136410594f, 0.174353659f, -0.137977764f, 0.0578158684f, 0.227951288f, + -0.0852605477f, -0.150851771f, 0.123837478f, 0.0507039875f, 0.136538386f, 0.282794505f, + -0.244014159f, -0.0632211491f, 0.15392904f, 0.0547516271f, 0.175257862f, -0.0491353497f, + -0.154547647f, 0.0704203993f, -0.237117171f, -0.246590525f, 0.0639194101f, 0.252007723f, + 0.375681937f, 0.173818141f, 0.060616836f, -0.152863637f, -0.133546785f, -0.181805402f, + 0.178500712f, 0.0247977544f, 0.03136703f, 0.244234294f, 0.0473097265f, 0.0835861415f, + -0.086889118f, 0.0202626158f, 0.0417347848f, 0.0451637879f, 0.0889597908f, 0.424210399f, + 0.0198644847f, -0.135825023f, 0.253715664f, 0.158522293f, -0.359053552f, -0.131656185f, + -0.0171459615f, -0.0714050904f, 0.242445141f, 0.0335307084f, 0.235993713f, -0.0286050439f, + -0.295446247f, -0.00572203193f, 0.0129030515f, 0.226378903f, 0.0578723922f, -0.0214700717f, + -0.0222770963f, -0.320240289f, 0.180718198f, -0.105571404f, -0.0180277042f, -0.093553789f, + 0.0627840161f, 0.15227665f, -0.0517334007f, -0.118923105f, -0.247941241f, -0.210730165f, + -0.130211934f, 0.077543214f, -0.127787769f, 0.253119469f, 0.0214030668f, -0.0240055565f, + -0.384567648f, 0.0333582833f, 0.000859327032f, -0.130582809f, 0.0810907856f, 0.0593428575f, + -0.0652422756f, 0.0980934277f, -0.0323485583f, -0.207968518f, -0.128021747f, -0.0529218614f, + 0.0871731117f, 0.188785702f, 0.246557832f, -0.00204586587f, 0.171493143f, 0.159378737f, + 0.056447383f, 0.0526220873f, 0.23541069f, -0.264652103f, -0.00124741346f, -0.0341790505f, + -0.196577966f, -0.213871971f, -0.0873092711f, 0.0459644794f, 0.244911849f, -0.0141087482f, + -0.284092218f, -0.0412680618f, 0.405599743f, 0.0470541827f, -0.115298733f, -0.197923139f, + -0.307000816f, -0.466205537f, -0.0451052412f, -0.011972269f, 0.113826551f, -0.149915695f, + 0.00626881467f, 0.00471630925f, 0.0430704467f, 0.276836783f, 0.058558546f, -0.168954059f, + 0.114365786f, -0.0500359125f, -0.190239385f, 0.0441222079f, 0.156270564f, -0.134900004f, + 0.304657489f, -0.0279312395f, 0.191525131f, 0.0655415505f, -0.348410845f, -0.0924862623f, + -0.0718971193f, 0.0818686336f, 0.0861204565f, -0.171984121f, 0.0864807144f, 0.163079545f, + -0.181385398f, 0.0981449336f, 0.124766447f, -0.0518741645f, 0.17262435f, -0.249620304f, + -0.0516820587f, 0.00645593321f, -0.0235484783f, -0.0549264066f, 0.155914336f, 0.033666864f, + -0.112114444f, -0.110596262f, 0.231156155f, 0.0603438914f, 0.103510648f, 0.103870779f, + -0.00270144059f, 0.255741745f, 0.336562037f, 0.199568674f, 0.11468336f, 0.111278333f, + 0.0494238473f, 0.111096464f, -0.0281137917f, -0.103573799f, -0.0396192186f, 0.21315515f, + 0.130312636f, -0.171485648f, 0.114488833f, 0.0394079015f, -0.308014482f, 0.0735709369f, + 0.137956455f, -0.325919807f, -0.206913173f, -0.160477445f, 0.117573932f, 0.315905005f, + 0.00996286143f, 0.092836678f, -0.0213343836f, -0.0349569358f, 0.128590822f, -0.130582705f, + 0.0150985261f, 0.0375045836f, 0.0608869307f, 0.00387548353f, -0.172976136f, 0.0597007796f, + -0.168234691f, -0.373954743f, -0.106707633f, -0.0131773744f, -0.292082697f, 0.202119842f, + -0.0338037126f, -0.0859032795f, -0.136623159f, 0.0820575207f, -0.0194657296f, 0.186938107f, + 0.060591951f, -0.0963178575f, 0.0430724882f, -0.0302837882f, -0.035506729f, 0.201354608f, + 0.184694782f, 0.336674929f, 0.0287558977f, 0.000921385246f, -0.0135701178f, -0.0380269364f, + 0.129963949f, -0.0505637601f, -0.0889441594f, -0.0362011604f, -0.140392095f, 0.122969434f, + 0.127392754f, 0.136146575f, -0.0079227658f, -0.176621646f, 0.086479634f, -0.190951377f, + -0.15915598f, -0.0403363518f, -0.0394156612f, 0.0792375505f, 0.0261054002f, 0.0217790119f, + -0.352562994f, -0.148276031f, 0.0707434937f, -0.159458429f, 0.041782245f, -0.0774111077f, + -0.0474635102f, 0.213561073f, 0.163344324f, -0.120505691f, -0.0739919096f, -0.0653060675f, + -0.0936806127f, -0.227657869f, 0.0889978036f, 0.118333198f, -0.114345334f, -0.127344832f, + 0.0917342603f, -0.119830474f, 0.21951814f, -0.114790723f, -0.141756326f, -0.163251773f, + 0.136926875f, 0.304104924f, 0.17052184f, 0.204871073f, -0.196935922f, 0.249426678f, + -0.0680071265f, -0.261540204f, 0.0208325256f, -0.0220708381f, 0.260184914f, -0.0265973154f, + -0.102168217f, 0.196518809f, 0.192088574f, -0.0881040916f, -0.0792996809f, -0.166160718f, + 0.228083462f, -0.0459954739f, 0.312827975f, -0.141072467f, -0.0688961223f, -0.0530200824f, + -0.0169182848f, 0.108116254f, -0.12148159f, 0.15531382f, 0.0330864601f, -0.231164575f, + -0.229718223f, 0.0563248321f, 0.246671408f, -0.169734195f, -0.126586407f, -0.0533274971f, + -0.16137971f, -0.0419682935f, -0.226478949f, 0.234363094f, 0.185875759f, -0.112364776f, + 0.346106201f, 0.197649047f, 0.186018482f, -0.0976952389f, -0.102497213f, -0.0145039763f, + 0.0675646365f, 0.0726319999f, -0.00998232421f, -0.179051265f, 0.10536781f, -0.105141826f, + -0.234749645f, -0.124726616f, -0.0402343571f, -0.354087919f, -0.215418801f, 0.0746908113f, + 0.0410933495f, 0.343521953f, 0.138164833f, 0.0281131174f, -0.0974368528f, -0.377483726f, + -0.189783916f, -0.0490393974f, 0.0702480972f, -0.11025051f, 0.18443951f, 0.0214362554f, + -0.348050624f, -0.117464028f, 0.15207684f, -0.0281669889f, -0.168709338f, 0.125837639f, + 0.0349771678f, -0.232138872f, -0.120971933f, 0.432278723f, 0.0182103906f, -0.308253527f, + 0.189801708f, 0.142383322f, 0.117755942f, -0.0810037479f, 0.139789477f, 0.072753109f, + -0.0383830853f, 0.0737549812f, -0.135906562f, 0.0727570578f, -0.124967001f, 0.0332392044f, + -0.107525945f, 0.17213212f, -0.154151469f, 0.0481800362f, 0.0866204649f, 0.00996933877f, + -0.120263219f, 0.159145638f, 0.182878241f, -0.113282107f, 0.0444048122f, 0.073611699f, + 0.244076818f, -0.101309456f, -0.0706439242f, 0.132584006f, +}; + +inline constexpr int64_t kLtx2ResampleSameOrigRate = 16000; +inline constexpr int64_t kLtx2ResampleSameNewRate = 16000; +inline constexpr int64_t kLtx2ResampleSameInSamples = 64; +inline constexpr int64_t kLtx2ResampleSameOutSamples = 64; +inline constexpr float kLtx2ResampleSameGolden[] = { + -0.405204147f, 0.256094337f, -0.0365431421f, 0.144786432f, 0.442803383f, 0.197185129f, + 0.00641530426f, 0.101439901f, -0.288413048f, 0.276153743f, -0.373815358f, 0.0357299149f, + -0.150143951f, 0.425933182f, 0.430869371f, 0.386764467f, 0.449498534f, -0.215525791f, + -0.413843364f, 0.454426825f, 0.257733166f, 0.474363089f, 0.454185247f, 0.137395754f, + -0.162905961f, -0.475493759f, 0.458122998f, 0.345632404f, -0.195999309f, 0.344082147f, + -0.232523456f, -0.395225525f, 0.197237954f, -0.0901086852f, -0.329409868f, 0.115814395f, + -0.00241821329f, 0.0528635159f, -0.136961848f, -0.0803770274f, 0.446430832f, 0.251397312f, + -0.219453916f, 0.155429542f, -0.0725865141f, -0.248174518f, 0.245423719f, -0.103727877f, + 0.0699998587f, -0.0313329883f, -0.45214057f, 0.290790528f, 0.147005036f, 0.388405949f, + 0.131976366f, 0.443826556f, -0.37574175f, 0.341548532f, -0.356789261f, 0.484469026f, + -0.176039338f, -0.431059867f, -0.379225999f, 0.0652751997f, 0.444586664f, 0.178928718f, + 0.428170532f, -0.28014794f, 0.493626893f, 0.0600764342f, -0.200313687f, 0.224563673f, + 0.34332484f, 0.302795142f, -0.0397084989f, -0.236524582f, 0.326341033f, -0.200754166f, + 0.0734653324f, 0.407461613f, 0.0551238768f, -0.144065112f, -0.352078319f, 0.0399719477f, + -0.339309841f, 0.466512203f, -0.487870157f, -0.31224522f, -0.00245836703f, -0.263568729f, + 0.447112143f, -0.229905024f, -0.208819613f, -0.0640842915f, -0.36582464f, -0.182941675f, + -0.10237547f, -0.306450248f, 0.284807384f, -0.136687264f, 0.211049214f, -0.0450212136f, + 0.351251781f, -0.296978563f, 0.300218612f, -0.376386553f, -0.14377071f, 0.01248514f, + 0.0800864846f, -0.341801167f, 0.30843401f, 0.215325922f, 0.121199809f, -0.278818995f, + -0.464530498f, 0.206952333f, -0.349141926f, -0.0297168121f, 0.177203611f, 0.143456593f, + 0.186289802f, -0.49433735f, 0.153161019f, 0.0672777146f, -0.297201097f, -0.425057709f, + 0.3405931f, -0.147772357f, +}; + +// --- section 8e: waveform_to_mel at a NON-target rate (ops.py:44-55) --- +inline constexpr int64_t kLtx2MelSourceRate = 44100; +inline constexpr int64_t kLtx2MelSourceSamples = 600; +inline constexpr int64_t kLtx2MelResampledFrames = 14; + +inline constexpr float kLtx2MelResampledGolden[] = { + -5.4661231f, -5.55506706f, -5.09506035f, -6.19318295f, -6.08189249f, -6.17544174f, + -5.6668644f, -5.5211668f, -5.80990982f, -5.9284153f, -5.45444393f, -6.4685173f, + -5.71690655f, -6.0217433f, -5.73173523f, -5.59101057f, -6.07259798f, -6.04894781f, + -6.19490576f, -6.57504129f, -5.64122391f, -5.86473227f, -5.59691286f, -5.66495609f, + -5.93782425f, -5.77671909f, -5.45543861f, -5.59877825f, -5.71345234f, -5.57294083f, + -5.66409063f, -5.63091135f, -5.33672714f, -5.69532776f, -5.50630045f, -5.50697517f, + -6.08902121f, -6.27135468f, -5.85663557f, -5.54306078f, -5.97331429f, -6.38751221f, + -5.9048481f, -5.60310745f, -6.03241777f, -6.22403574f, -5.95464993f, -5.71830559f, + -6.40463924f, -6.08543062f, -6.15129328f, -6.13091803f, -5.54007912f, -5.81266499f, + -5.7169714f, -5.62237167f, -5.90837908f, -6.22743368f, -6.04174423f, -6.56699181f, + -5.65484095f, -5.91686869f, -5.77181244f, -5.94467306f, -5.75919247f, -5.85417461f, + -6.53755093f, -5.83754826f, -5.23825598f, -5.88241386f, -6.02013731f, -5.9696269f, + -5.91963625f, -5.505476f, -5.38549566f, -5.82375288f, -5.1732707f, -5.70048857f, + -5.96792746f, -5.97677279f, -6.40191746f, -6.35137081f, -5.50886393f, -5.50407076f, + -6.01290703f, -5.95151901f, -5.75256491f, -5.87025166f, -7.16911221f, -7.09269428f, + -6.36924839f, -5.86924458f, -6.40944195f, -5.8599596f, -5.64378595f, -6.12015867f, + -6.38106489f, -6.78282833f, -5.74355841f, -5.83369303f, -6.07583332f, -5.64163589f, + -5.58671045f, -5.93410969f, -6.30367708f, -5.97529173f, -5.1600523f, -5.84388876f, + -6.34159184f, -6.55544806f, -5.81524277f, -6.22483015f, -6.10869217f, -5.77106762f, + -5.83326101f, -5.49699259f, -5.30421782f, -5.54879427f, -6.10520172f, -5.80052233f, + -5.61863375f, -5.76047802f, -5.48267508f, -5.68262529f, -5.77609634f, -6.02572727f, + -6.03231239f, -5.71555662f, -5.91699076f, -6.43874311f, -5.68555355f, -5.26530838f, + -5.82621717f, -5.71177197f, -5.74042225f, -5.82096243f, -6.38452053f, -6.5581007f, + -6.00606155f, -5.24496508f, -5.74657297f, -5.65564775f, -5.57938623f, -5.94365597f, + -6.04934359f, -6.66540241f, -5.68788862f, -5.36847687f, -5.80125999f, -5.79700184f, + -6.13079166f, -5.73697901f, -6.27567387f, -6.72550631f, -5.46829557f, -5.46381998f, + -5.79290152f, -6.0005722f, -5.97839832f, -6.0189929f, -5.96040344f, -6.65128517f, + -5.91887093f, -5.66960001f, -5.91024971f, -5.93545103f, -6.04944897f, -6.35564899f, + -5.57299137f, -6.35422421f, -6.45694542f, -6.01158762f, -5.86569738f, -5.94834423f, + -6.09327936f, -5.88648033f, -5.25396204f, -5.22967243f, -5.94662523f, -6.15266275f, + -5.83826828f, -5.48036671f, -5.74903774f, -6.08138084f, -5.46356964f, -5.06745768f, + -5.91538f, -6.51051712f, -5.72736788f, -5.71280241f, -5.83652401f, -6.05304956f, + -5.75634432f, -5.65665674f, -6.04497814f, -6.47492218f, -5.77602959f, -5.28765106f, + -5.60678625f, -6.04329443f, -5.8856926f, -6.14575195f, -5.67699766f, -5.49637413f, + -5.39192724f, -5.52504921f, -6.04295254f, -6.0130825f, -6.02200174f, -6.39702034f, + -6.22545624f, -5.62030745f, -5.4359746f, -5.42942905f, -5.7402153f, -5.68468618f, + -5.94748259f, -6.64641619f, -6.79842901f, -6.17434788f, -5.65791321f, -5.58381701f, + -6.03300667f, -5.81140184f, +}; + // --- section 9a: the INITIAL video state (tools.py:139-186) --- inline constexpr int64_t kLtx2CondVideoBaseTokens = 12; inline constexpr int64_t kLtx2CondVideoBaseWidth = 4; diff --git a/tests/vllm/models/test_ltx2_vae.cpp b/tests/vllm/models/test_ltx2_vae.cpp index ba161b906..ef7d0a775 100644 --- a/tests/vllm/models/test_ltx2_vae.cpp +++ b/tests/vllm/models/test_ltx2_vae.cpp @@ -2817,6 +2817,100 @@ TEST_CASE("ltx2 vae: the slaney mel filterbank matches torchaudio") { } } +TEST_CASE("ltx2 vae: resample_audio matches upstream at every ratio it can take") { + // Row LTX25-AUDIO-RESAMPLE (#2583). Generator section 8d runs upstream's OWN + // `AudioProcessor.resample_audio` (ops.py:36-42) at four ratios; this + // reproduces each from the same PRNG input. + // + // TOLERANCE. 2.5e-07 is not a hedge, it is twice the MEASURED floor. Setting + // it to 0.0 on this tree reds three of the four arms at 5.96e-08 (Up), 5.96e-08 + // (Down) and 1.19e-07 (Wide); Same passes at zero because it is a copy. The + // port mirrors torchaudio's float32 kernel arithmetic operation for operation, + // so the only residual is the difference between two libm `sin`/`cos` at the + // same f32 argument plus the convolution's reduction order. Building the filter + // in `double` instead would land 1.03e-05 out — eighty times this bound — which + // is why a wider dtype FAILS this gate rather than passing it more comfortably. + struct Arm { + const char* tag; + const char* input; + int64_t orig_rate; + int64_t new_rate; + int64_t in_samples; + int64_t out_samples; + const float* golden; + size_t golden_size; + }; + constexpr double kResampleTol = 2.5e-7; + const int64_t channels = vllm_test::kLtx2ResampleChannels; + const Arm arms[] = { + {"Up (o = 1, the BWE ratio)", "ltx2.resample.Up", vllm_test::kLtx2ResampleUpOrigRate, + vllm_test::kLtx2ResampleUpNewRate, vllm_test::kLtx2ResampleUpInSamples, + vllm_test::kLtx2ResampleUpOutSamples, vllm_test::kLtx2ResampleUpGolden, + std::size(vllm_test::kLtx2ResampleUpGolden)}, + {"Down (n = 1)", "ltx2.resample.Down", vllm_test::kLtx2ResampleDownOrigRate, + vllm_test::kLtx2ResampleDownNewRate, vllm_test::kLtx2ResampleDownInSamples, + vllm_test::kLtx2ResampleDownOutSamples, vllm_test::kLtx2ResampleDownGolden, + std::size(vllm_test::kLtx2ResampleDownGolden)}, + {"Wide (o = 441, width 17)", "ltx2.resample.Wide", vllm_test::kLtx2ResampleWideOrigRate, + vllm_test::kLtx2ResampleWideNewRate, vllm_test::kLtx2ResampleWideInSamples, + vllm_test::kLtx2ResampleWideOutSamples, vllm_test::kLtx2ResampleWideGolden, + std::size(vllm_test::kLtx2ResampleWideGolden)}, + {"Same (the early return)", "ltx2.resample.Same", vllm_test::kLtx2ResampleSameOrigRate, + vllm_test::kLtx2ResampleSameNewRate, vllm_test::kLtx2ResampleSameInSamples, + vllm_test::kLtx2ResampleSameOutSamples, vllm_test::kLtx2ResampleSameGolden, + std::size(vllm_test::kLtx2ResampleSameGolden)}, + }; + + for (const Arm& arm : arms) { + INFO("arm: " << arm.tag); // NOT CAPTURE: doctest stringifies a `const char*` as a bool + const std::vector in = Ltx2Input(arm.input, channels * arm.in_samples, 0.5); + int64_t produced = 0; + const std::vector got = vllm::Ltx2ResampleWaveform( + in, channels, arm.in_samples, arm.orig_rate, arm.new_rate, &produced); + // `ceil(new * length / orig)` (functional.py:1427). Asserted before the + // values because a resampler that produced the right SAMPLES at the wrong + // LENGTH would be compared against a shifted golden. + CHECK(produced == arm.out_samples); + REQUIRE(got.size() == arm.golden_size); + const double err = MaxAbsDiff(got, arm.golden, arm.golden_size); + INFO("resample max|diff| = " << err); + CHECK(err <= kResampleTol); + // A LOWER bound as well. An all-zero output, or one that dropped every + // channel but the first, matches nothing here but would satisfy a + // tolerance against a golden regenerated from the same defect. + double absmax = 0.0; + for (float v : got) absmax = std::max(absmax, std::abs(static_cast(v))); + CHECK(absmax > 0.0); + double second_channel_absmax = 0.0; + for (int64_t i = produced; i < 2 * produced; ++i) { + second_channel_absmax = + std::max(second_channel_absmax, std::abs(static_cast(got[static_cast(i)]))); + } + CHECK(second_channel_absmax > 0.0); + } + + // The equal-rate arm returns the INPUT, byte for byte. Upstream returns the + // same `Audio` object (ops.py:38-39) and never enters the filter; a port that + // ran a unit-ratio filter anyway would be wrong by its passband ripple, which + // is under the tolerance above and would pass every check in the loop. + const std::vector same_in = + Ltx2Input("ltx2.resample.Same", channels * vllm_test::kLtx2ResampleSameInSamples, 0.5); + const std::vector same_out = vllm::Ltx2ResampleWaveform( + same_in, channels, vllm_test::kLtx2ResampleSameInSamples, + vllm_test::kLtx2ResampleSameOrigRate, vllm_test::kLtx2ResampleSameNewRate, nullptr); + CHECK(same_out == same_in); + + // Upstream's own refusal, at the same boundary (functional.py:1470-1471). + bool threw = false; + try { + vllm::Ltx2ResampleWaveform(same_in, channels, vllm_test::kLtx2ResampleSameInSamples, 16000, 0, + nullptr); + } catch (const std::exception&) { + threw = true; + } + CHECK(threw); +} + TEST_CASE("ltx2 vae: waveform_to_mel matches upstream AudioProcessor") { vllm::Ltx2AudioProcessorConfig cfg; cfg.target_sample_rate = vllm_test::kLtx2MelRate; @@ -2838,21 +2932,37 @@ TEST_CASE("ltx2 vae: waveform_to_mel matches upstream AudioProcessor") { INFO("waveform_to_mel max|diff| = " << err); CHECK(err <= kLtx2GoldenTol); - // A rate that does not match is REFUSED, because upstream would RESAMPLE and - // this project does not carry that resampler. Treating the samples as if they - // were already at the target rate conditions on audio that is pitched wrong. - bool threw = false; - std::string message; - try { - vllm::Ltx2WaveformToLogMel(cfg, wave, channels, samples, 44100, nullptr); - } catch (const std::exception& error) { - threw = true; - message = error.what(); - } - REQUIRE(threw); - INFO("refusal message: " << message); - CHECK(message.find("resample") != std::string::npos); - CHECK(message.find("44100") != std::string::npos); + // A rate that does NOT match is RESAMPLED, which is what upstream does + // (`waveform_to_mel` calls `resample_audio` first, ops.py:49) and what this + // project refused to do until row LTX25-AUDIO-RESAMPLE (#2583). + // + // Section 8e's golden is the whole claim: the source is 600 samples at 44100, + // the processor targets 16000, and the mel that comes back is over the + // RESAMPLED 218 samples. A build that resampled after the transform, or that + // never resampled at all, produces a different frame count before it produces + // a different value, so the count is asserted first. + int64_t resampled_frames = 0; + const int64_t source_samples = vllm_test::kLtx2MelSourceSamples; + const std::vector source = + Ltx2Input("ltx2.mel.resampled.input", channels * source_samples, 0.5); + const std::vector resampled_mel = vllm::Ltx2WaveformToLogMel( + cfg, source, channels, source_samples, vllm_test::kLtx2MelSourceRate, &resampled_frames); + CHECK(resampled_frames == vllm_test::kLtx2MelResampledFrames); + REQUIRE(static_cast(resampled_mel.size()) == + static_cast(std::size(vllm_test::kLtx2MelResampledGolden))); + const double resampled_err = MaxAbsDiff(resampled_mel, vllm_test::kLtx2MelResampledGolden, + std::size(vllm_test::kLtx2MelResampledGolden)); + INFO("resampled waveform_to_mel max|diff| = " << resampled_err); + CHECK(resampled_err <= kLtx2GoldenTol); + + // And it is NOT the mel of the same samples read as if they were already at + // the target rate — the exact wrong answer the old refusal existed to + // prevent, and the one a tolerance against the golden alone cannot see if the + // golden were ever regenerated from the wrong side. + const std::vector misread = + vllm::Ltx2WaveformToLogMel(cfg, source, channels, source_samples, + cfg.target_sample_rate, nullptr); + CHECK(misread.size() != resampled_mel.size()); } TEST_CASE("ltx2 vae: SILENCE saturates the mel log clamp, and the clamp is pinned") { diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 4736ee63a..db04efcb0 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -7488,6 +7488,106 @@ TEST_CASE("ltx2 video: a supplied audio file CONDITIONS the render, and stays FR } } +TEST_CASE("ltx2 video: a take at ANOTHER rate is RESAMPLED, not refused") { + // Row LTX25-AUDIO-RESAMPLE, issue #2583, gap A19 of the completion plan. + // + // Enters through the PRODUCTION path — `LoadVideoEngine` + `Generate`, what + // `vllm_video_generate` calls straight through (`vllm_c.cpp:1646`) — for the + // reason in this section's header: a unit test over `Ltx2ResampleWaveform` + // proves the filter works and never that a request can arrive at it. Until + // this row, a request at any rate but the checkpoint's was refused three hops + // earlier and NOTHING downstream of that refusal ran. + // + // Upstream refuses nothing here: `waveform_to_mel` calls `resample_audio` + // before the mel transform (ops.py:44-49) and `resample_audio` filters + // whenever the rates differ (ops.py:36-42). + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + + // THREE takes of the same 220 Hz tone, and the third is the control: + // native — 2.0 s at the fixture's own 24000, the arm that always worked. + // high — 2.0 s at 44100. Same sound, more samples; must resample. + // misread— `high`'s PCM bytes with 24000 written in the header. This is + // what reading the samples "as if they were already at the target + // rate" produces: 3.675 s of a 119.7 Hz tone. It is a legal WAV, so + // it renders, and it is the wrong answer the refusal prevented. + const std::string native = WriteWav(ws.root + "/native.wav", 2, kFixtureAudioRate, 2.0); + const std::string high = WriteWav(ws.root + "/high.wav", 2, 44100, 2.0); + const std::string misread = ws.root + "/misread.wav"; + { + std::string bytes = MakeWavPcm16(2, 44100, 2.0); + // The `fmt ` chunk's sample-rate field: "RIFF" + size + "WAVE" + "fmt " + + // size = 20 bytes, then wFormatTag(2) + nChannels(2) = 4 more. + const auto rate = static_cast(kFixtureAudioRate); + for (int i = 0; i < 4; ++i) { + bytes[24 + static_cast(i)] = static_cast((rate >> (8 * i)) & 0xFF); + } + std::ofstream out(misread, std::ios::binary); + REQUIRE(out.good()); + out.write(bytes.data(), static_cast(bytes.size())); + out.close(); + } + + auto render = [&](const std::string& dir, const std::string& wav) { + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/" + dir); + gen.extras[vllm::multimodal::kLtx2AudioPathExtra] = wav; + const vllm::multimodal::VideoResult result = engine->Generate(gen); + return std::make_pair(result, ltx2->last_conditioning()); + }; + + const auto [native_result, native_trace] = render("rs_native", native); + const auto [high_result, high_trace] = render("rs_high", high); + const auto [misread_result, misread_trace] = render("rs_misread", misread); + + // (1) Accepted, conditioned, and not zeroed. Before this row the second call + // threw, which is the red this case was written on. + CHECK(high_trace.completed); + CHECK(high_trace.audio_conditioned); + CHECK(high_trace.audio_tokens > 0); + CHECK(high_trace.audio_latent_absmax > 0.0); + CHECK(high_trace.audio_latent_digest != 0); + + // (2) The RETURNED soundtrack stays at the FILE's rate. Upstream hands back + // the original `Audio` untouched (a2vid_two_stage.py:301-303), so the + // resample reaches the encoder and nothing else. A build that resampled + // the take in place instead of inside the mel front-end passes every + // assertion above and fails this one. + CHECK(high_result.sample_rate == 44100); + CHECK(native_result.sample_rate == kFixtureAudioRate); + + // (3) The rate was READ, not ignored. `high` and `misread` carry BYTE + // IDENTICAL PCM and differ only in the header's rate field, so a build + // that never resampled gives them the same latent. + CHECK(high_trace.audio_latent_digest != misread_trace.audio_latent_digest); + + // (4) THE VALUES ARE NOT GATED HERE, and that is a measurement rather than a + // gap left open. The obvious video-level claim is "`high` is the same two + // seconds of the same tone as `native`, so its latent must land NEARER + // `native` than the mis-read take does" — and `audio_latent_absmax` cannot + // carry it. Measured on this fixture, the three takes read 1.07194 + // (native), 1.07293 (high) and 1.07208 (misread): a 0.1% spread across + // three genuinely different waveforms, because the trace's absmax is + // dominated by the encoder's per-channel statistics and not by the take. + // A one-scalar ordering over that population would have passed or failed + // on noise either way. + // + // So the numbers live where they can be gated: `test_ltx2_vae`'s sections + // 8d and 8e hold the resampler and the resampled mel against goldens the + // generator produced by EXECUTING upstream, at 2.5e-07. This case proves + // the request arrives, the rate is read, and the soundtrack comes back + // untouched — the three things a unit test over the filter cannot. + INFO("audio_latent_absmax native=" << native_trace.audio_latent_absmax + << " high=" << high_trace.audio_latent_absmax + << " misread=" << misread_trace.audio_latent_absmax); + + // The render still produced its artifacts; resampling is not a bypass. + CHECK(high_result.frame_count > 0); + CHECK(misread_result.frame_count > 0); +} + TEST_CASE("ltx2 video: every audio-input mismatch is refused BY WHAT IS WRONG") { // Each of these renders a finished clip if it is accepted, which is why every // one is a refusal rather than a conversion. The assertions hold each message @@ -7513,24 +7613,6 @@ TEST_CASE("ltx2 video: every audio-input mismatch is refused BY WHAT IS WRONG") g.extras[vllm::multimodal::kLtx2AudioPathExtra] = w; }; - SUBCASE("a sample rate the mel front-end does not target") { - // Upstream RESAMPLES here (ops.py:40) with an arbitrary-ratio polyphase - // kaiser resampler this project has not ported. Reading 44.1 kHz samples at - // the checkpoint's rate pitches and time-shifts the conditioning while every - // shape checks out, so both rates go in the message. - // - // The TARGET rate asserted here is the fixture's 24000, which is neither the - // shipped 16000 nor `Ltx2ParseAudioEncoderConfig`'s own default. That is the - // point: with the fixture at 16000 this assertion passed against a parser - // that never read `params.sampling_rate` at all. - const std::string wav = WriteWav(ws.root + "/44k.wav", 2, 44100, 2.0); - const std::string message = refusal("rate", just_path, wav); - INFO("refusal: " << message); - CHECK(message.find("44100") != std::string::npos); - CHECK(message.find(std::to_string(kFixtureAudioRate)) != std::string::npos); - CHECK(message.find("ops.py:40") != std::string::npos); - } - SUBCASE("a channel count the encoder does not declare") { // `MiniMaxH3ReadWav` would REPEAT a mono take across both channels // (minimax_h3.h:1839-1845). That is H3's contract; LTX-2 hands the file's own From 303de55158429c33f08d988a0e976e30e8e1a668 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 20:39:33 +0000 Subject: [PATCH 3/5] fix(LTX25-AUDIO-RESAMPLE): upstream's truncation is an f32 ceil, and no arm here could reach it (#2583) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_apply_sinc_resample_kernel` ends `target_length = torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` (torchaudio 2.11.0 `functional/functional.py:1427`). That reads as an integer ceil and this row first shipped one, `(next * samples + orig - 1) / orig`. It is not one: `torch.as_tensor` of a Python float takes `torch.get_default_dtype()`, which is float32, so the f64 quotient is rounded to f32 BEFORE the ceil. Where the exact quotient sits above an integer by less than half an f32 ulp, the narrowing lands on that integer and upstream keeps one sample FEWER. At 44100 -> 16000 that starts at 180697 input samples (4.097 s) and hits 48102 of the first 60 s worth of lengths; 22050 -> 16000 starts at 90569 and 44100 -> 24000 at 240854. At 48000 -> 16000 (`o = 3`) and 16000 -> 48000 (`o = 1`) it never happens at any audio length, which is why three of the four section-8d arms look clean. One extra output sample moves the last STFT windows and, where `samples % hop == 0`, the mel frame count — the audio latent length, and so the conditioning shape. The gate could not see it. The 8d arms top out at 218 output samples, three orders of magnitude below the boundary: with the wrong formula and the test file as it stood, `*resampl*` reads 26/26, `*waveform_to_mel*` 9/9 and `test_ltx2_video`'s `*RESAMPLED*` 14/14, all green. So this adds generator section 8f, four length-plus-tail goldens that reach the boundary — 180696 / 180697 / 180698 at 44100 -> 16000, which bracket it, and 90569 at 22050 -> 16000, where the same effect starts at another ratio. The generator asserts which of them an exact integer ceil gets right and which it does not, so an arm that stopped discriminating fails generation rather than gating nothing. Against those arms the old formula reds on exactly the two divergent lengths (65560 vs 65559, 65720 vs 65719, tails 0.256834 and 0.358731) and stays green on the two that bracket them. The repair mirrors the computation rather than describing it: `next * samples` is exact in `int64_t` and exact again as a `double` under 2^53 / next, so `double(next * samples) / double(orig)` is Python's own correctly-rounded `int / int`; `static_cast` is `as_tensor`, `std::ceil` is `torch.ceil`, and the cast back is `.long()`. Checked against torchaudio over 276060 (ratio, length) pairs — twelve ratios by 23005 lengths, including either side of 2^24: zero divergences. Also in this change: * `examples/ltx2_gen/main.cpp`'s `--help` was the ninth kaiser statement, and the only one that is product output rather than a comment. It told users the rate "must equal" the mel front-end's, that "neither is converted, because upstream resamples with a polyphase kaiser resampler that is not ported here", and that "both mismatches are refused". All three are false at this head. The sweep is re-run over every git-tracked file rather than over source and specs, which is the file kind the first pass did not consider. * Five stale torchaudio anchors, three of them pointing at blank lines. Every anchor this change touches was re-resolved by grepping the statement at torchaudio 2.11.0+cu130: the kernel's reductions are `:1340-1341`, not `:1341-1342`; `_apply_sinc_resample_kernel`'s are `:1416-1417`; the batch pack is `:1419-1421`; `dtype=waveform.dtype` is `:1487`. * `INFO("arm: " << arm.tag)` printed `arm: 1`, because doctest stringifies a `const char*` as a bool exactly as `CAPTURE` does. Wrapped in `std::string`, so the diagnostic names the ratio that broke. * Three record inaccuracies. Spec §6.2 named four rate pairs the generator does not ship; the double-kernel divergence is 3.39746e-06 and not 1.03e-05, which leaves the conclusion intact at 13.6x the 2.5e-07 bound; and the reachability mutation is re-run at this head, because the repair edits the very line the mutation deletes. Goldens regenerate byte-identically against `ltx-2` `fd4ded7f2`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-audio-resample.md | 136 +++++++++++++++--- examples/ltx2_gen/main.cpp | 18 +-- .../models/ltx2_audio_vae_encoder.h | 10 +- scripts/gen-ltx2-vae-goldens.py | 78 +++++++++- .../model_executor/models/ltx2_audio_vae.cpp | 40 ++++-- tests/vllm/models/ltx2_vae_goldens.inc | 38 +++++ tests/vllm/models/test_ltx2_vae.cpp | 96 ++++++++++++- 7 files changed, 368 insertions(+), 48 deletions(-) diff --git a/.agents/specs/ltx25-audio-resample.md b/.agents/specs/ltx25-audio-resample.md index 492c7ae44..27ff1bb69 100644 --- a/.agents/specs/ltx25-audio-resample.md +++ b/.agents/specs/ltx25-audio-resample.md @@ -104,7 +104,7 @@ The nine sites: `ltx2_audio_input.cpp:101`, `ltx2_audio_vae.cpp:1057`, | 6 | `ops.py:36-42` | `resample_audio` returns `audio` unchanged when the rates are equal; otherwise `torchaudio.functional.resample(waveform, orig, target)` and then `.to(device=..., dtype=waveform.dtype)` | | 7 | `functional.py:1470-1476` | `orig_freq <= 0 or new_freq <= 0` raises; `orig_freq == new_freq` returns the input; `gcd` reduces the ratio | | 8 | `functional.py:1305-1402` | the kernel: `base_freq = min(o, n) * rolloff`, `width = ceil(lpw * o / base_freq)`, one row per output phase | -| 9 | `functional.py:1405-1432` | zero-pad `(width, width + o)`, `conv1d` stride `o`, transpose-interleave, truncate to `ceil(n * L / o)` | +| 9 | `functional.py:1405-1432` | zero-pad `(width, width + o)`, `conv1d` stride `o`, transpose-interleave, truncate to `ceil(n * L / o)` — with the quotient **narrowed to f32 before the ceil** (`:1427`; `torch.as_tensor` of a Python float takes the default dtype), which keeps one sample fewer than an exact integer ceil at 180697 samples for 44100 -> 16000 | | 10 | `a2vid_two_stage.py:301-303` | the returned soundtrack is the caller's **original** `Audio`, at its **native** rate. The resample never reaches the output | Hop 10 is why nothing downstream of the encoder changes: the file's own rate @@ -136,8 +136,8 @@ does not cast it, and `_get_sinc_resample_kernel` is called with This is not a detail to round up. Building the same filter in `double` and storing it as `float` is *more accurate than upstream* and therefore **further from it**: measured against the pinned oracle, a double-built kernel differs by -up to **1.03e-05** where a float-built one differs by **1.79e-07**, a factor of -57. AGENTS.md's *Inherit vLLM defaults* names exactly this failure — a dtype that +up to **3.39746e-06** where a float-built one differs by **1.19209e-07**, a +factor of 28 and 13.6 times the gate's own 2.5e-07 bound. AGENTS.md's *Inherit vLLM defaults* names exactly this failure — a dtype that is too wide, which no token gate can see. The port mirrors the float32 arithmetic operation for operation, including the association `sinc * (window * scale)` (`:1397`) and the fact that `scale = base_freq / orig_freq` is computed @@ -184,16 +184,29 @@ and the video-level case must red. Recorded in `## Outcome`. Upstream ships none (§0). What this change does instead: 1. **Goldens by execution.** `scripts/gen-ltx2-vae-goldens.py` gains sections - **8d** and **8e**, importing `ltx_core.model.audio_vae.ops.AudioProcessor` + **8d**, **8e** and **8f**, importing `ltx_core.model.audio_vae.ops.AudioProcessor` from the `--ltx2` checkout and running `resample_audio` and `waveform_to_mel`. They ride the generator's existing `kLtx2VaeUpstreamRevision` anchor, which the C++ suite already asserts against a pinned SHA, so a regeneration against a different upstream fails the gate rather than replacing the oracle. 2. **The rate pairs are chosen to reach distinct arms**, not for coverage - arithmetic: 16000 -> 24000 (pure upsample, `o = 2`), 44100 -> 24000 - (`gcd = 300`, `o = 147`, `width = 12`, the widest kernel), 48000 -> 24000 - (pure downsample, `o = 2`), and 24000 -> 24000 (the equal-rate early return, - which must be a **copy** and not a filtered pass). + arithmetic. What section 8d ships is 16000 -> 48000 (pure upsample, `o = 1`, + the degenerate ratio the tree already ports for the vocoder's BWE stage), + 48000 -> 16000 (pure downsample, `o = 3`, `n = 1`, `width = 19`, one phase + row), 44100 -> 16000 (`gcd = 100`, `o = 441`, `n = 160`, `width = 17`, the + widest kernel and the pair a real 44.1 kHz take hits), and 16000 -> 16000 (the + equal-rate early return, which must be a **copy** and not a filtered pass). + This paragraph named a different four before the port ran; the shipped set is + the one above, and `RESAMPLE_CASES` in the generator is its record. +2b. **Section 8f reaches the truncation boundary, which 8d cannot.** The 8d arms + top out at 218 output samples, and upstream's f32-narrowed ceil (hop 9) first + disagrees with an exact integer ceil at 180697 input samples — 4.097 s at + 44.1 kHz, three orders of magnitude further out. 8f carries four LENGTH-plus- + tail goldens: 180696 / 180697 / 180698 at 44100 -> 16000, which bracket that + boundary, and 90569 at 22050 -> 16000, where the same effect starts. The + generator asserts which of them an exact integer ceil gets right and which it + does not, so an arm that stopped discriminating fails generation rather than + gating nothing. 3. **A lower bound, not only a tolerance.** A resampler that returned zeros, or that returned the input untouched, satisfies a `max|diff| < tol` gate against the wrong golden and any shape check. The cases assert the output's absmax is @@ -272,18 +285,29 @@ Three filters were built and compared against the pinned oracle's own | Filter built in | max abs diff vs the oracle | |---|---| -| `double`, narrowed to `float` | **1.03e-05** | -| torchaudio's own f64-internal path (`dtype=None`) | 4.71e-06 | -| `float`, mirroring `dtype=waveform.dtype` | **1.79e-07** | - -The `double` filter is *more accurate than upstream* and therefore fifty-seven +| `double` throughout `Ltx2SincResampleKernel`, narrowed to `float` at the store | **3.39746e-06** | +| torchaudio's own `dtype=None` kernel (`idx` in f64, `t` in the default f32) | 2.37e-06 | +| `float`, mirroring `dtype=waveform.dtype` | **1.19209e-07** | + +All three re-measured on 2026-09-02, because the first two rows were first +recorded as 1.03e-05 and 4.71e-06 and neither reproduces. The mutation each row +names is exact: row 1 makes `base_freq`, `scale`, `lpw`, `kPi`, `phase`, `idx`, +`t`, `shaped`, `window` and `sinc` `double` and casts only at +`kernel[j * taps + k] = ...`; it reds the `Wide` arm at 3.39746e-06 and all four +8f tails between 1.55e-06 and 3.09e-06. Row 2 is torchaudio's own +`_get_sinc_resample_kernel(..., dtype=None)` over the same 8d inputs. Row 3 is +this port at the committed head, read by setting `kResampleTol` to `0.0`. + +The `double` filter is *more accurate than upstream* and therefore twenty-eight times further from it. It would have passed a gate written around it and read as a careful implementation. §4 states the chain that fixes the answer at f32, and the gate's 2.5e-07 is set so that a widening fails it. **The tolerance is twice the measured floor, not a hedge.** Setting -`kResampleTol` to `0.0` reds three of the four arms, at 5.96e-08 (Up), 5.96e-08 -(Down) and 1.19e-07 (Wide). `Same` passes at zero, because it is a copy. +`kResampleTol` to `0.0` reds three of the four 8d arms, at 5.96e-08 (Up), +5.96e-08 (Down) and 1.19209e-07 (Wide), and all four 8f tails, at 1.49e-08, +4.47e-08, 2.98e-08 and 7.45e-09. `Same` passes at zero, because it is a copy. +2.5e-07 is 2.10x the worst of those. **`audio_latent_absmax` cannot gate this, and that is a finding rather than a gap.** The obvious video-level claim — "the 44.1 kHz take must land nearer the @@ -293,14 +317,74 @@ genuinely different waveforms, because the trace's absmax is dominated by the encoder's per-channel statistics. The assertion was removed rather than inverted or loosened, and the reason is recorded where the assertion was. +### The truncation was NOT upstream's, and no arm here could see it + +`_apply_sinc_resample_kernel` ends +`target_length = torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` +(`functional.py:1427`). That reads as an integer ceil and this port first shipped +one, `(next * samples + orig - 1) / orig`. It is not one: `torch.as_tensor` of a +**Python float** takes `torch.get_default_dtype()`, which is float32, so the f64 +quotient is rounded to f32 **before** the ceil. Where the exact quotient sits +above an integer by less than half an f32 ulp, the narrowing lands on that +integer and upstream keeps one sample FEWER. + +Measured at 44100 -> 16000: 180696 samples give 65559 both ways, 180697 give +65559 upstream and 65560 from the exact ceil, 180698 give 65560 both ways. 48102 +of the first 60 s worth of lengths diverge, and the density grows with duration. +22050 -> 16000 starts at 90569 and 44100 -> 24000 at 240854. At 48000 -> 16000 +(`o = 3`) and 16000 -> 48000 (`o = 1`) it never happens at any audio length, +which is why three of the four 8d arms look clean at any length. One extra output +sample changes the last mel frames' STFT window content and, where +`samples % hop == 0`, the frame count — the audio latent length, and so the +conditioning shape. + +**The gate could not see it, and that is the reason 8f exists.** The four 8d +arms top out at 218 output samples. Substituting torchaudio's real float32 ceil +into this port and rebuilding left `*resampl*` 26/26, `*waveform_to_mel*` 9/9 and +`*RESAMPLED*` 14/14 green: the right ratio at the wrong length. Section 8f adds +the four boundary arms of §6.2b, and with them the old formula reds on exactly +the two divergent lengths (65560 vs 65559 and 65720 vs 65719, plus their tails at +0.257 and 0.359) and stays green on the two that bracket them. + +The repair mirrors the computation rather than describing it: the int64 product +`next * samples` is exact as a `double` for any length under 2^53 / next, so +`double(next * samples) / double(orig)` is Python's own correctly-rounded `int / +int`; `static_cast` is `as_tensor`, `std::ceil` on the float is +`torch.ceil`, and the cast back to `int64_t` is `.long()`. Checked against +torchaudio over 276060 (ratio, length) pairs — twelve ratios by 23005 lengths, +covering 1 to 20000, the runs around 90569 and 180697, a thousand lengths either +side of 2^24 where f32 no longer holds every integer, and 2^25: zero +divergences. + +### One more kaiser statement, in product output + +The row's first pass repaired eight of the nine statements that misnamed the +filter. The ninth was `examples/ltx2_gen/main.cpp`'s `--help` text, which is not +a comment: it told users that the sample rate "must equal the audio VAE's mel +front-end rate", that "neither is converted, because upstream resamples with a +polyphase kaiser resampler that is not ported here", and that "both mismatches +are refused". All three were false at that head. Repaired, and the sweep re-run +over every git-tracked file rather than over source and specs: `git grep -inE +"(^|[^a-z])kaiser"` now returns only the vocoder's genuine kaiser-sinc +anti-aliasing filter, this row's own prose about the repair, and unrelated +tokenizer and HuggingFace-org byte matches. + ### Red before, green after | | Before | After | |---|---|---| -| `test_ltx2_vae -tc='*waveform_to_mel*'` | THREW the rate refusal at `ltx2_audio_vae.cpp:1053` | 48/48 cases, 3217/3217 assertions | +| `test_ltx2_vae -tc='*waveform_to_mel*'` | THREW the rate refusal at `ltx2_audio_vae.cpp:1053` | 48/48 cases, 3245/3245 assertions | | `test_ltx2_video -tc='*RESAMPLED*'` | THREW `'high.wav' is sampled at 44100 Hz ...` | 110/110 cases, 4870/4870 assertions | +| `test_ltx2_vae -tc='*resampl*'`, section 8f | 50/54 with the EXACT INTEGER CEIL in place: `CeilAt` 65560 vs 65559 and its tail 0.256834 out, `CeilAlt` 65720 vs 65719 and its tail 0.358731 out | 54/54 | -Both reds are the refusal itself, which is what the row exists to delete. +The first two reds are the refusal itself, which is what the row exists to +delete. The third is the truncation repair below, and it is red-first in the +strict sense: the two bracketing arms `CeilBelow` and `CeilAbove` stay GREEN +under the same old formula, so the arm discriminates rather than simply failing. +The same old formula against the test file as it stood BEFORE section 8f leaves +`*resampl*` 26/26, `*waveform_to_mel*` 9/9 and `test_ltx2_video`'s `*RESAMPLED*` +14/14 all green — the measurement that says the gate was blind, rather than an +argument that it was. ### The reachability mutation, and what it proved @@ -316,8 +400,22 @@ rebuilding: * **the section-8d unit case still PASSES.** That is the point of the mutation: `Ltx2ResampleWaveform` works whether or not anything calls it, and a test that constructs it by hand measures the class. The tree was restored byte-for-byte - (sha256 `88055b01e9c2bf0bb29d826b2c61dc4c8439a1d8ae6d779a46969b50b0c0212b`) and - both suites re-run green. + and both suites re-run green. + +The mutation was RE-RUN at the truncation-repair head, because the repair moves +the very line the mutation deletes and a mutation proof does not survive an edit +to its own subject. It behaves identically there: 8d/8f 54/54, the mel case red +at 38 vs 14 frames and 608 vs 224, the video case red on the same digest +`18285287296143238670`. `ltx2_audio_vae.cpp` sha256 before and after that +restore: `fd1b68a71594d7bfd28dcb7f16e67e8c0abe91d5c267e089eb0bee3e3755af03` +(the earlier run's subject was the same file at `052047579`, sha256 +`88055b01e9c2bf0bb29d826b2c61dc4c8439a1d8ae6d779a46969b50b0c0212b`). + +The mutation's FIRST build failed `-Werror` on the now-unused `sampling_rate`, +and the three suites then re-ran the STALE binaries and printed green — the +shape in which a mutation that never applied reads as a passing test. Recorded +because the reading it produced was indistinguishable from the real one until +the build's exit status was checked. ### Rejected diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index a03813053..ec7d2d6d5 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -174,14 +174,16 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "--a2v-guidance-scale is the guider\'s modality_scale; a take here rides whichever\n" "recipe the checkpoint resolves, in practice distilled_two_stage, whose sigmas are\n" "fixed. So no claim is made that this reproduces upstream\'s A2Vid output.\n\n" - "The WAV must already match the checkpoint: its sample rate must equal the audio\n" - "VAE\'s mel front-end rate and its channel count the encoder\'s in_channels.\n" - "Neither is converted, because upstream resamples with a polyphase kaiser\n" - "resampler that is not ported here and feeds the file\'s own channel count into a\n" - "fixed-width convolution; both mismatches are refused with both numbers, since a\n" - "resampled-wrong or upmixed take renders a finished clip conditioned on audio you\n" - "never supplied. The take must also be at least as long as the clip: upstream\n" - "truncates a long one and asserts on a short one, so a short one is refused.\n\n" + "ANY sample rate is accepted: a take whose rate differs from the audio VAE\'s\n" + "mel front-end rate is RESAMPLED to it, with upstream\'s own filter — the\n" + "gcd-reduced rational-ratio hann-windowed sinc `torchaudio.functional.resample`\n" + "builds at its defaults, which is what ops.py:40 calls. The CHANNEL COUNT must\n" + "still equal the encoder\'s in_channels and a mismatch is refused with both\n" + "numbers: upstream feeds the file\'s own channel count into a fixed-width\n" + "convolution, so up- or downmixing here would render a finished clip conditioned\n" + "on audio you never supplied. The take must also be at least as long as the clip:\n" + "upstream truncates a long one and asserts on a short one, so a short one is\n" + "refused.\n\n" "--audio-start-time seeks into the file (default 0) and --audio-max-duration caps\n" "how much is read (default: the clip\'s own duration). Either without\n" "--audio-path is refused rather than ignored. The rendered audio.wav is your own\n" diff --git a/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h b/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h index 73c0b5806..ac62e1443 100644 --- a/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h +++ b/include/vllm/model_executor/models/ltx2_audio_vae_encoder.h @@ -175,9 +175,13 @@ std::vector Ltx2SlaneyMelFilterbank(int64_t n_freqs, double f_min, double // // `waveform` is [channels, samples] channel-major and each channel resamples // independently against one kernel, as torchaudio's packed batch does. Returns -// `ceil(new_freq * samples / orig_freq)` samples per channel, and returns the -// INPUT unfiltered when the rates already match (ops.py:38-39). Computed in f32, -// which is the dtype upstream resamples in — see the note at the definition. +// `ceil(new_freq * samples / orig_freq)` samples per channel — the quotient +// NARROWED TO f32 BEFORE the ceil, because `torch.as_tensor` of a Python float +// takes the default dtype (functional.py:1427), which keeps one sample fewer +// than an exact integer ceil at 180697 samples for 44100 -> 16000 and at 48102 +// of the first 60 s worth of lengths. Returns the INPUT unfiltered when the +// rates already match (ops.py:38-39). Computed in f32, which is the dtype +// upstream resamples in — see the note at the definition. std::vector Ltx2ResampleWaveform(const std::vector& waveform, int64_t channels, int64_t samples, int64_t orig_freq, int64_t new_freq, int64_t* out_samples); diff --git a/scripts/gen-ltx2-vae-goldens.py b/scripts/gen-ltx2-vae-goldens.py index ad0dc84a7..42bc1a082 100644 --- a/scripts/gen-ltx2-vae-goldens.py +++ b/scripts/gen-ltx2-vae-goldens.py @@ -18,7 +18,7 @@ model/video_vae/video_vae.py -> section 6 (VideoEncoder) model/audio_vae/audio_vae.py -> section 7 (AudioEncoder) model/audio_vae/ops.py -> section 8 (AudioProcessor mel front-end, - and 8d/8e its RESAMPLER) + and 8d-8f its RESAMPLER) conditioning/types/*.py -> section 9 (the conditioning items) Usage: @@ -1038,6 +1038,39 @@ def noncausal_randn(*args, **kwargs): MEL_SOURCE_RATE = 44100 MEL_SOURCE_SAMPLES = 600 +# Section 8f — the TRUNCATION BOUNDARY, which no arm above can reach. +# +# `_apply_sinc_resample_kernel` truncates to +# `torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` +# (functional.py:1427). `torch.as_tensor` of a PYTHON FLOAT takes +# `torch.get_default_dtype()`, which is float32, so the f64 quotient is narrowed +# to f32 BEFORE the ceil. Where the exact quotient sits above an integer by less +# than half an f32 ulp, the narrowing lands ON that integer and upstream keeps one +# sample FEWER than an exact integer ceil would. +# +# The four arms of 8d top out at 218 output samples, three orders of magnitude +# below where that first happens, so they cannot see it: at 44100 -> 16000 the +# first divergent length is 180697 (4.097 s) and 48102 of the first 60 s worth of +# lengths diverge. One extra output sample moves the last STFT windows and, where +# `samples % hop == 0`, the mel FRAME COUNT — that is, the conditioning shape. +# +# Two ratios, because the effect needs `orig_freq` large enough for the quotient +# to land that close: at 48000 -> 16000 (o = 3) and 16000 -> 48000 (o = 1) it +# never happens at any audio length, which is why the clean-looking arms are +# clean. Each ratio carries the first divergent length and, for 44100, the +# lengths either side of it, where an exact ceil is RIGHT — an arm that only +# carried divergent lengths could be satisfied by always subtracting one. +RESAMPLE_CEIL_CASES = ( + ("CeilBelow", 44100, 16000, 180696, False), + ("CeilAt", 44100, 16000, 180697, True), + ("CeilAbove", 44100, 16000, 180698, False), + ("CeilAlt", 22050, 16000, 90569, True), +) +# The goldens carry the LENGTH and the last few samples rather than 65 559 floats +# per arm. The length is the discriminator; the tail is there so a port that +# produced the right count from a shifted signal cannot pass on the count alone. +RESAMPLE_CEIL_TAIL = 8 + def section_video_encoder(out) -> None: import torch @@ -1403,6 +1436,49 @@ def section_audio_mel(out) -> None: out.write("\n") emit_f32(out, "kLtx2MelResampledGolden", resampled_mel.numpy()) + # --- 8f: the truncation boundary (functional.py:1427) --- + # + # Same call as 8d, at lengths that reach where upstream's f32-narrowed ceil + # and an exact integer ceil disagree. Emitted as a length plus a tail window, + # because the full arm is 65 559 samples wide. + out.write("// --- section 8f: the truncation boundary (functional.py:1427) ---\n") + emit_scalar(out, "kLtx2ResampleCeilTail", RESAMPLE_CEIL_TAIL) + for tag, orig, target, length, diverges in RESAMPLE_CEIL_CASES: + wave = make_input(f"ltx2.resample.{tag}", (1, 1, length), 0.5) + proc = AudioProcessor( + target_sample_rate=target, + mel_bins=MEL["mel_bins"], + mel_hop_length=MEL["mel_hop_length"], + n_fft=MEL["n_fft"], + ) + got = proc.resample_audio(Audio(waveform=wave, sampling_rate=orig)) + assert got.sampling_rate == target + assert got.waveform.dtype == torch.float32 + produced = int(got.waveform.shape[-1]) + # The POSITIVE CONTROL on this arm. An exact integer ceil is what a + # reader writes from the formula as printed; assert here which lengths it + # gets right and which it does not, so an arm that stopped discriminating + # fails the GENERATOR rather than quietly gating nothing. + exact_ceil = -(-((target // math.gcd(orig, target)) * length) // + (orig // math.gcd(orig, target))) + if diverges: + assert exact_ceil == produced + 1, ( + f"{tag}: expected upstream to keep one sample FEWER than an exact " + f"integer ceil, got upstream={produced} exact={exact_ceil}" + ) + else: + assert exact_ceil == produced, ( + f"{tag}: expected the two ceils to AGREE here, got " + f"upstream={produced} exact={exact_ceil}" + ) + tail = got.waveform.reshape(-1)[-RESAMPLE_CEIL_TAIL:] + assert float(tail.abs().max()) > 0.0, "an all-zero tail gates nothing" + emit_scalar(out, f"kLtx2Resample{tag}OrigRate", orig) + emit_scalar(out, f"kLtx2Resample{tag}NewRate", target) + emit_scalar(out, f"kLtx2Resample{tag}InSamples", length) + emit_scalar(out, f"kLtx2Resample{tag}OutSamples", produced) + emit_f32(out, f"kLtx2Resample{tag}TailGolden", tail.numpy()) + # --------------------------------------------------------------------------- # Section 9 — the CONDITIONING ITEMS (phase L11). What the encoders' output is diff --git a/src/vllm/model_executor/models/ltx2_audio_vae.cpp b/src/vllm/model_executor/models/ltx2_audio_vae.cpp index c8b8ea80f..fd98a4c55 100644 --- a/src/vllm/model_executor/models/ltx2_audio_vae.cpp +++ b/src/vllm/model_executor/models/ltx2_audio_vae.cpp @@ -1052,7 +1052,7 @@ namespace { // at the arguments `ops.py:40` passes, which is to say at every default. Returns // `new_freq` phase rows of `2 * width + orig_freq` taps, row-major. `orig_freq` // and `new_freq` are already REDUCED by their gcd, as upstream reduces them at -// `:1341-1342`. +// `:1340-1341`. // // THE WINDOW IS HANN, and nine comments in this tree used to say kaiser. // `resample` defaults `resampling_method="sinc_interp_hann"` (`:1441`) and only @@ -1065,19 +1065,19 @@ namespace { // SHORTCUT. Upstream's waveform is float32 from the decoder on // (`decode.py:173-176`), `encode_audio` moves it without casting // (`audio_vae.py:271`), and the kernel is built with `dtype=waveform.dtype` -// (`:1483`), so `idx`, `t`, the clamp, the window and `sin(t)/t` are all f32 +// (`:1487`), so `idx`, `t`, the clamp, the window and `sin(t)/t` are all f32 // (`:1376-1397`). Measured against the pinned oracle, a `double`-built kernel -// narrowed to `float` lands 1.03e-05 away where an f32-built one lands 1.79e-07: -// widening here does not improve the port, it loosens the gate by a factor of 57 -// and hides exactly what AGENTS.md's "a token gate cannot detect a dtype that is -// too wide" is about. +// narrowed to `float` lands 3.39746e-06 away where this f32-built one lands +// 1.19209e-07: widening here does not improve the port, it loosens the gate by a +// factor of 28 and hides exactly what AGENTS.md's "a token gate cannot detect a +// dtype that is too wide" is about. std::vector Ltx2SincResampleKernel(int64_t orig_freq, int64_t new_freq, int64_t* out_width) { constexpr int64_t kLowpassFilterWidth = 6; // functional.py:1439 constexpr double kRolloff = 0.99; // functional.py:1440 // `base_freq` and `width` are Python floats, i.e. f64, and stay f64: only the - // TENSOR arithmetic below narrows (`:1357-1370`). + // TENSOR arithmetic below narrows (`:1350`, `:1369`; the first tensor is `:1376`). const double base_freq64 = static_cast(std::min(orig_freq, new_freq)) * kRolloff; const int64_t width = static_cast( std::ceil(static_cast(kLowpassFilterWidth) * static_cast(orig_freq) / @@ -1142,8 +1142,8 @@ std::vector Ltx2ResampleWaveform(const std::vector& waveform, int6 if (orig_freq == new_freq) return waveform; const int64_t gcd = std::gcd(orig_freq, new_freq); - const int64_t orig = orig_freq / gcd; // `:1341`, `:1414` - const int64_t next = new_freq / gcd; // `:1342`, `:1415` + const int64_t orig = orig_freq / gcd; // `:1340`, `:1416` + const int64_t next = new_freq / gcd; // `:1341`, `:1417` int64_t width = 0; const std::vector kernel = Ltx2SincResampleKernel(orig, next, &width); const int64_t taps = 2 * width + orig; @@ -1152,11 +1152,29 @@ std::vector Ltx2ResampleWaveform(const std::vector& waveform, int6 // orig), convolve with stride `orig`, transpose the phase axis in front of the // block axis, and truncate to `ceil(new * length / orig)`. const int64_t blocks = samples / orig + 1; - const int64_t target = (next * samples + orig - 1) / orig; // `:1427` + // `target_length = torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` + // (`:1427`), and this is NOT an exact integer ceil. `torch.as_tensor` of a + // PYTHON FLOAT takes `torch.get_default_dtype()`, which is float32, so the f64 + // quotient is rounded to f32 BEFORE the ceil. Where the exact quotient sits + // above an integer by less than half an f32 ulp, the narrowing lands ON that + // integer and upstream keeps one sample FEWER than `(next * samples + orig - 1) + // / orig` gives — first at 180697 samples for 44100 -> 16000 (4.097 s), then + // for 48102 of the first 60 s worth of lengths. The extra sample moves the last + // STFT windows and, where `samples % hop == 0`, the mel FRAME COUNT. + // + // `next * samples` is exact in `int64_t` and exact again as a `double` for any + // length under 2^53 / next, so the division below is Python's own + // correctly-rounded f64 `int / int`; the narrowing and the ceil then mirror + // `as_tensor` and `torch.ceil`, and the cast to `int64_t` mirrors `.long()`. + // Checked against torchaudio over 276060 (ratio, length) pairs — twelve ratios + // by 23005 lengths, including a run either side of 2^24, where f32 no longer + // holds every integer: zero divergences. + const double quotient = static_cast(next * samples) / static_cast(orig); + const int64_t target = static_cast(std::ceil(static_cast(quotient))); if (out_samples != nullptr) *out_samples = target; // ONE kernel for every channel. torchaudio packs the batch and resamples the - // last axis (`:1416-1418`), so the channels are independent; this walks them + // last axis (`:1419-1421`), so the channels are independent; this walks them // rather than approximating that. std::vector out(static_cast(channels) * static_cast(target)); for (int64_t c = 0; c < channels; ++c) { diff --git a/tests/vllm/models/ltx2_vae_goldens.inc b/tests/vllm/models/ltx2_vae_goldens.inc index ad118dc88..9e48102e8 100644 --- a/tests/vllm/models/ltx2_vae_goldens.inc +++ b/tests/vllm/models/ltx2_vae_goldens.inc @@ -4906,6 +4906,44 @@ inline constexpr float kLtx2MelResampledGolden[] = { -6.03300667f, -5.81140184f, }; +// --- section 8f: the truncation boundary (functional.py:1427) --- +inline constexpr int64_t kLtx2ResampleCeilTail = 8; +inline constexpr int64_t kLtx2ResampleCeilBelowOrigRate = 44100; +inline constexpr int64_t kLtx2ResampleCeilBelowNewRate = 16000; +inline constexpr int64_t kLtx2ResampleCeilBelowInSamples = 180696; +inline constexpr int64_t kLtx2ResampleCeilBelowOutSamples = 65559; +inline constexpr float kLtx2ResampleCeilBelowTailGolden[] = { + 0.0894216299f, 0.0131077049f, -0.0238793846f, -0.0588569939f, 0.0581167452f, 0.219902992f, + -0.0258866604f, -0.171903431f, +}; + +inline constexpr int64_t kLtx2ResampleCeilAtOrigRate = 44100; +inline constexpr int64_t kLtx2ResampleCeilAtNewRate = 16000; +inline constexpr int64_t kLtx2ResampleCeilAtInSamples = 180697; +inline constexpr int64_t kLtx2ResampleCeilAtOutSamples = 65559; +inline constexpr float kLtx2ResampleCeilAtTailGolden[] = { + -0.190139741f, 0.06196668f, -0.194867074f, -0.0171126947f, 0.11970146f, 0.0139573188f, + -0.063427873f, 0.152270317f, +}; + +inline constexpr int64_t kLtx2ResampleCeilAboveOrigRate = 44100; +inline constexpr int64_t kLtx2ResampleCeilAboveNewRate = 16000; +inline constexpr int64_t kLtx2ResampleCeilAboveInSamples = 180698; +inline constexpr int64_t kLtx2ResampleCeilAboveOutSamples = 65560; +inline constexpr float kLtx2ResampleCeilAboveTailGolden[] = { + 0.138243884f, -0.0549423918f, 0.164882183f, 0.0120501379f, -0.0783000514f, 0.365968585f, + -0.0271436367f, -0.185848534f, +}; + +inline constexpr int64_t kLtx2ResampleCeilAltOrigRate = 22050; +inline constexpr int64_t kLtx2ResampleCeilAltNewRate = 16000; +inline constexpr int64_t kLtx2ResampleCeilAltInSamples = 90569; +inline constexpr int64_t kLtx2ResampleCeilAltOutSamples = 65719; +inline constexpr float kLtx2ResampleCeilAltTailGolden[] = { + 0.00326456479f, 0.0271689408f, 0.162618577f, 0.348693013f, -0.00762261869f, 0.0951811969f, + -0.0475787744f, -0.406309605f, +}; + // --- section 9a: the INITIAL video state (tools.py:139-186) --- inline constexpr int64_t kLtx2CondVideoBaseTokens = 12; inline constexpr int64_t kLtx2CondVideoBaseWidth = 4; diff --git a/tests/vllm/models/test_ltx2_vae.cpp b/tests/vllm/models/test_ltx2_vae.cpp index ef7d0a775..87348f84c 100644 --- a/tests/vllm/models/test_ltx2_vae.cpp +++ b/tests/vllm/models/test_ltx2_vae.cpp @@ -2823,13 +2823,15 @@ TEST_CASE("ltx2 vae: resample_audio matches upstream at every ratio it can take" // reproduces each from the same PRNG input. // // TOLERANCE. 2.5e-07 is not a hedge, it is twice the MEASURED floor. Setting - // it to 0.0 on this tree reds three of the four arms at 5.96e-08 (Up), 5.96e-08 - // (Down) and 1.19e-07 (Wide); Same passes at zero because it is a copy. The + // it to 0.0 on this tree reds three of the four 8d arms at 5.96e-08 (Up), + // 5.96e-08 (Down) and 1.19209e-07 (Wide) and all four 8f tails at 1.49e-08, + // 4.47e-08, 2.98e-08 and 7.45e-09; Same passes at zero because it is a copy. The // port mirrors torchaudio's float32 kernel arithmetic operation for operation, // so the only residual is the difference between two libm `sin`/`cos` at the // same f32 argument plus the convolution's reduction order. Building the filter - // in `double` instead would land 1.03e-05 out — eighty times this bound — which - // is why a wider dtype FAILS this gate rather than passing it more comfortably. + // in `double` instead would land 3.39746e-06 out — 13.6 times this bound — + // which is why a wider dtype FAILS this gate rather than passing it more + // comfortably. struct Arm { const char* tag; const char* input; @@ -2862,12 +2864,16 @@ TEST_CASE("ltx2 vae: resample_audio matches upstream at every ratio it can take" }; for (const Arm& arm : arms) { - INFO("arm: " << arm.tag); // NOT CAPTURE: doctest stringifies a `const char*` as a bool + // `std::string`, not the bare pointer: doctest stringifies a `const char*` + // as a BOOL, so both `CAPTURE(arm.tag)` and `INFO("arm: " << arm.tag)` print + // `arm: 1` and the diagnostic cannot name which ratio broke. + INFO("arm: " << std::string(arm.tag)); const std::vector in = Ltx2Input(arm.input, channels * arm.in_samples, 0.5); int64_t produced = 0; const std::vector got = vllm::Ltx2ResampleWaveform( in, channels, arm.in_samples, arm.orig_rate, arm.new_rate, &produced); - // `ceil(new * length / orig)` (functional.py:1427). Asserted before the + // `ceil` of the f32-narrowed `new * length / orig` (functional.py:1427; the + // narrowing is what section 8f below gates). Asserted before the // values because a resampler that produced the right SAMPLES at the wrong // LENGTH would be compared against a shifted golden. CHECK(produced == arm.out_samples); @@ -2889,6 +2895,84 @@ TEST_CASE("ltx2 vae: resample_audio matches upstream at every ratio it can take" CHECK(second_channel_absmax > 0.0); } + // Section 8f — THE TRUNCATION BOUNDARY, which no arm above can reach. + // + // Upstream truncates to + // `torch.ceil(torch.as_tensor(new_freq * length / orig_freq)).long()` + // (functional.py:1427), and `torch.as_tensor` of a PYTHON FLOAT takes + // `torch.get_default_dtype()` — float32. The f64 quotient is therefore rounded + // to f32 BEFORE the ceil, and where the exact quotient sits above an integer by + // less than half an f32 ulp the narrowing lands ON that integer: upstream keeps + // one sample FEWER than the exact integer ceil `(next * samples + orig - 1) / + // orig` this port used to compute. + // + // The four arms above top out at 218 output samples and cannot see it. At + // 44100 -> 16000 the first divergent length is 180697 (4.097 s) and 48102 of + // the first 60 s worth of lengths diverge; at 22050 -> 16000 it starts at + // 90569. At 48000 -> 16000 (o = 3) and 16000 -> 48000 (o = 1) it never happens, + // which is why three of the four arms above look clean at any length. One extra + // output sample moves the last STFT windows and, where `samples % hop == 0`, + // the mel FRAME COUNT — the conditioning shape. + // + // `CeilBelow` and `CeilAbove` bracket the boundary and are lengths where the + // exact ceil is RIGHT, so an arm that always subtracted one would fail them. + struct CeilArm { + const char* tag; + const char* input; + int64_t orig_rate; + int64_t new_rate; + int64_t in_samples; + int64_t out_samples; + const float* tail; + size_t tail_size; + }; + const CeilArm ceil_arms[] = { + {"CeilBelow (exact ceil agrees)", "ltx2.resample.CeilBelow", + vllm_test::kLtx2ResampleCeilBelowOrigRate, vllm_test::kLtx2ResampleCeilBelowNewRate, + vllm_test::kLtx2ResampleCeilBelowInSamples, vllm_test::kLtx2ResampleCeilBelowOutSamples, + vllm_test::kLtx2ResampleCeilBelowTailGolden, + std::size(vllm_test::kLtx2ResampleCeilBelowTailGolden)}, + {"CeilAt (the f32 narrowing bites)", "ltx2.resample.CeilAt", + vllm_test::kLtx2ResampleCeilAtOrigRate, vllm_test::kLtx2ResampleCeilAtNewRate, + vllm_test::kLtx2ResampleCeilAtInSamples, vllm_test::kLtx2ResampleCeilAtOutSamples, + vllm_test::kLtx2ResampleCeilAtTailGolden, + std::size(vllm_test::kLtx2ResampleCeilAtTailGolden)}, + {"CeilAbove (exact ceil agrees)", "ltx2.resample.CeilAbove", + vllm_test::kLtx2ResampleCeilAboveOrigRate, vllm_test::kLtx2ResampleCeilAboveNewRate, + vllm_test::kLtx2ResampleCeilAboveInSamples, vllm_test::kLtx2ResampleCeilAboveOutSamples, + vllm_test::kLtx2ResampleCeilAboveTailGolden, + std::size(vllm_test::kLtx2ResampleCeilAboveTailGolden)}, + {"CeilAlt (22050 -> 16000, the same effect at another ratio)", + "ltx2.resample.CeilAlt", vllm_test::kLtx2ResampleCeilAltOrigRate, + vllm_test::kLtx2ResampleCeilAltNewRate, vllm_test::kLtx2ResampleCeilAltInSamples, + vllm_test::kLtx2ResampleCeilAltOutSamples, vllm_test::kLtx2ResampleCeilAltTailGolden, + std::size(vllm_test::kLtx2ResampleCeilAltTailGolden)}, + }; + + for (const CeilArm& arm : ceil_arms) { + INFO("ceil arm: " << std::string(arm.tag)); + REQUIRE(arm.tail_size == static_cast(vllm_test::kLtx2ResampleCeilTail)); + const std::vector in = Ltx2Input(arm.input, arm.in_samples, 0.5); + int64_t produced = 0; + const std::vector got = vllm::Ltx2ResampleWaveform(in, 1, arm.in_samples, arm.orig_rate, + arm.new_rate, &produced); + INFO("produced = " << produced << ", upstream = " << arm.out_samples); + CHECK(produced == arm.out_samples); + REQUIRE(got.size() == static_cast(produced)); + REQUIRE(produced >= static_cast(arm.tail_size)); + // The tail as well as the count. A port that produced upstream's LENGTH from + // a signal shifted by a sample would satisfy the count on its own; these are + // the last `kLtx2ResampleCeilTail` samples upstream actually emitted. + const std::vector tail(got.end() - static_cast(arm.tail_size), + got.end()); + const double tail_err = MaxAbsDiff(tail, arm.tail, arm.tail_size); + INFO("tail max|diff| = " << tail_err); + CHECK(tail_err <= kResampleTol); + double tail_absmax = 0.0; + for (float v : tail) tail_absmax = std::max(tail_absmax, std::abs(static_cast(v))); + CHECK(tail_absmax > 0.0); + } + // The equal-rate arm returns the INPUT, byte for byte. Upstream returns the // same `Audio` object (ops.py:38-39) and never enters the filter; a port that // ran a unit-ratio filter anyway would be wrong by its passband ripple, which From f6f2b3e3e45607e9601dc3644db4c1b63845753a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 20:55:53 +0000 Subject: [PATCH 4/5] docs(LTX25-AUDIO-RESAMPLE): re-read the video suite's count after the merge that moved it `test_ltx2_video` reads 4876 assertions on the merged head, not the 4870 this row recorded before it: `origin/main` added the `dims=2` upsampler cases to the same suite. The count is re-read after the merge rather than carried across it, because a merge falsifies a number without touching a line this row owns. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-audio-resample.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.agents/specs/ltx25-audio-resample.md b/.agents/specs/ltx25-audio-resample.md index 27ff1bb69..d2098a1df 100644 --- a/.agents/specs/ltx25-audio-resample.md +++ b/.agents/specs/ltx25-audio-resample.md @@ -374,7 +374,7 @@ tokenizer and HuggingFace-org byte matches. | | Before | After | |---|---|---| | `test_ltx2_vae -tc='*waveform_to_mel*'` | THREW the rate refusal at `ltx2_audio_vae.cpp:1053` | 48/48 cases, 3245/3245 assertions | -| `test_ltx2_video -tc='*RESAMPLED*'` | THREW `'high.wav' is sampled at 44100 Hz ...` | 110/110 cases, 4870/4870 assertions | +| `test_ltx2_video -tc='*RESAMPLED*'` | THREW `'high.wav' is sampled at 44100 Hz ...` | 110/110 cases, 4876/4876 assertions | | `test_ltx2_vae -tc='*resampl*'`, section 8f | 50/54 with the EXACT INTEGER CEIL in place: `CeilAt` 65560 vs 65559 and its tail 0.256834 out, `CeilAlt` 65720 vs 65719 and its tail 0.358731 out | 54/54 | The first two reds are the refusal itself, which is what the row exists to @@ -402,6 +402,11 @@ rebuilding: constructs it by hand measures the class. The tree was restored byte-for-byte and both suites re-run green. +That video count reads 4876 and not the 4870 this row measured before its last +merge from `main`: `origin/main` added the `dims=2` upsampler cases to the same +suite. Re-read after the merge rather than carried across it, because a merge +falsifies a count without touching a line this row owns. + The mutation was RE-RUN at the truncation-repair head, because the repair moves the very line the mutation deletes and a mutation proof does not survive an edit to its own subject. It behaves identically there: 8d/8f 54/54, the mel case red From fd0b21bd427f5c8651bca9ed9ffa29adef1bd7a0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 2 Sep 2026 21:15:31 +0000 Subject: [PATCH 5/5] docs(LTX25-AUDIO-RESAMPLE): the list said nine and named eight, which is why the ninth shipped The kaiser sweep's record paragraph claimed nine sites and enumerated eight. The one it left out was `examples/ltx2_gen/main.cpp:179`, and that omission is the whole mechanism by which the first pass repaired the eight it listed and left a false statement in a `--help` string. The list is now the output of `git grep -inE "polyphase kaiser|kaiser resampl" 6643b2bbf -- .`, which returns exactly nine, rather than a transcription of it. Every in-tree anchor in that paragraph and in the gap section names the base `6643b2bbf` it was read at. Those lines describe the tree BEFORE this row, the refusals they cite no longer exist at head, and an anchor that resolves to a different statement than it names is the defect this row has now hit twice. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-audio-resample.md | 29 ++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/.agents/specs/ltx25-audio-resample.md b/.agents/specs/ltx25-audio-resample.md index d2098a1df..ccfa91cda 100644 --- a/.agents/specs/ltx25-audio-resample.md +++ b/.agents/specs/ltx25-audio-resample.md @@ -63,9 +63,11 @@ Out: `Ltx2WaveformToLogMel` refuses any waveform whose rate is not `config.target_sample_rate` -(`src/vllm/model_executor/models/ltx2_audio_vae.cpp:1051-1060`), and -`Ltx2DecodeAudioWav` refuses the same thing one hop earlier so the message can -name the file (`src/vllm/model_executor/models/ltx2_audio_input.cpp:96-105`). +(`src/vllm/model_executor/models/ltx2_audio_vae.cpp:1051-1060` at this row's base +`6643b2bbf`; the refusal is gone at head, so the anchor names the tree it +describes), and `Ltx2DecodeAudioWav` refuses the same thing one hop earlier so +the message can name the file +(`src/vllm/model_executor/models/ltx2_audio_input.cpp:96-105`, same base). Between them, **every** audio input at a rate other than the checkpoint's is turned away, so `a2vid_two_stage` only accepts a take the user resampled with some other tool first. @@ -86,9 +88,22 @@ So what is missing is the **arbitrary rational ratio**, not the window. Sizing A19 at M was right; the reason written beside it was wrong, and this row corrects the reason as well as the code. -The nine sites: `ltx2_audio_input.cpp:101`, `ltx2_audio_vae.cpp:1057`, -`ltx2_audio_vae_encoder.h:173`, `ltx2_video.h:315`, `ltx2_audio_input.h:124`, -`test_ltx2_video.cpp:7518`, and `ltx25-a2v-audio-input.md:155` and `:463`. +The nine sites, read at this row's base `6643b2bbf` because the repair moves +every one of them: `src/vllm/model_executor/models/ltx2_audio_input.cpp:101`, +`src/vllm/model_executor/models/ltx2_audio_vae.cpp:1057`, +`include/vllm/model_executor/models/ltx2_audio_vae_encoder.h:173`, +`include/vllm/multimodal/ltx2_video.h:315`, +`include/vllm/model_executor/models/ltx2_audio_input.h:124`, +`tests/vllm/multimodal/test_ltx2_video.cpp:7518`, +`.agents/specs/ltx25-a2v-audio-input.md:155` and `:463`, and +**`examples/ltx2_gen/main.cpp:179`**. + +That last entry is the row's own record defect and not a tenth discovery. This +paragraph said "nine" and listed **eight** until 2026-09-02; the repair covered +the eight it listed, and the ninth shipped unrepaired into a `--help` string. The +list is now enumerated by +`git grep -inE "polyphase kaiser|kaiser resampl" 6643b2bbf -- .`, which returns +exactly these nine, rather than transcribed. --- @@ -373,7 +388,7 @@ tokenizer and HuggingFace-org byte matches. | | Before | After | |---|---|---| -| `test_ltx2_vae -tc='*waveform_to_mel*'` | THREW the rate refusal at `ltx2_audio_vae.cpp:1053` | 48/48 cases, 3245/3245 assertions | +| `test_ltx2_vae -tc='*waveform_to_mel*'` | THREW the rate refusal at `ltx2_audio_vae.cpp:1053` (at `6643b2bbf`) | 48/48 cases, 3245/3245 assertions | | `test_ltx2_video -tc='*RESAMPLED*'` | THREW `'high.wav' is sampled at 44100 Hz ...` | 110/110 cases, 4876/4876 assertions | | `test_ltx2_vae -tc='*resampl*'`, section 8f | 50/54 with the EXACT INTEGER CEIL in place: `CeilAt` 65560 vs 65559 and its tail 0.256834 out, `CeilAlt` 65720 vs 65719 and its tail 0.358731 out | 54/54 |