feat(LTX25-AUDIO-RESAMPLE): resample the take instead of refusing it, and the window is hann

.agents/specs/ltx25-a2v-audio-input.md

@@ -150,11 +150,14 @@ nameable. Upstream's stage 1 is **caller-configured and guided** —
 upstream's and the trajectory is not, and no claim is made that a render here
 reproduces upstream's A2Vid output. Recorded under `## Owed`.
 
-**Out, and refused by name rather than dropped.** Resampling: upstream resamples
-with `torchaudio.functional.resample` (`ops.py:40`), an arbitrary-ratio polyphase
-kaiser resampler; this project ports only the integer-ratio hann-sinc variant,
-so a sample rate other than the encoder's is refused with the two rates in the
-message. Non-PCM16 and non-RIFF containers: upstream reads anything PyAV opens
+**Out, and refused by name rather than dropped.** Resampling was out of this row
+and is now IN the tree: row `LTX25-AUDIO-RESAMPLE`
+([#2583](https://github.com/mudler/vllm.cpp/issues/2583)) ported
+`torchaudio.functional.resample` (`ops.py:40`) and deleted the refusal this
+paragraph described. It also corrected the reason: the filter is an
+arbitrary-ratio polyphase sinc with a HANN window, not a kaiser one, because
+`resample` defaults `resampling_method="sinc_interp_hann"` and `ops.py:40`
+passes no override. Non-PCM16 and non-RIFF containers: upstream reads anything PyAV opens
 (`decode.py:252`), and no demuxer is vendored here
 (`video_api.cpp:115-121` says so explicitly). MP3/FLAC/OGG are therefore refused
 by name. `RetakePipeline` (#924) and text-to-audio.
@@ -460,8 +463,9 @@ in the waveform's own dtype (`ops.py:54`) and the encoder in the parameters'
   `AudioEncoderConfigurator.from_metadata` and `decode_audio_from_file` at
   reduced dimensions, which needs PyAV and a written fixture file, and is why it
   did not ride here.
-- **Arbitrary-ratio resampling** — refused by name; needs the polyphase kaiser
-  resampler upstream uses at `ops.py:40`.
+- ~~**Arbitrary-ratio resampling**~~ — CLOSED by row `LTX25-AUDIO-RESAMPLE`
+  ([#2583](https://github.com/mudler/vllm.cpp/issues/2583)), which ported
+  `ops.py:36-42`. The filter it names is hann-windowed, not kaiser.
 - **Compressed audio containers** — refused by name; no demuxer is vendored.
 - **`Ltx2CreateAudioLatentState` and `Ltx2ConditionAudioByReference` remain
   test-only.** Both have exactly one call site each and it is a test

docs/FEATURES.md

@@ -196,7 +196,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change.
 | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) |
 | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` and `test_ltx2_device` each gate one direction ALONE, on a forward where the other stream is PRESENT but DISABLED. Swapping the two flags is RED on both arms | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN`, which `modality_scale = 3.0` selects on every video row. On the DEVICE forward too since 2026-08-19 ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) |
 | LTX-2.5 latent upsampler `dims=2` arm | LTX-2.5 spatial upsampler checkpoint whose config sets `dims: 2` | `ltx2 the latent upsampler reproduces upstream on the dims=2 arm` (`test_ltx2_pipeline`) against the EXECUTED upstream module, plus `a dims=2 upsampler checkpoint RENDERS` (`test_ltx2_video`) through `LoadVideoEngine`/`Generate`; removing the fold reds the golden at max|diff| 0.68 while the render stays green, so the value gate and not a shape check is what holds it | No flag: `dims` is read off the checkpoint's own config, so a 2-D upsampler at `--upsampler` just works. Conv2d for `initial_conv`, both ResBlock stacks and `final_conv` (model.py:47); the frame axis folds into the batch (:86), which makes GroupNorm PER FRAME. Frame count preserved, H and W doubled, which is what both spatial call sites require. Two 2-D contradictions upstream cannot run are refused by name and EXECUTED against the real module by the golden generator: `temporal_upsample` (a Conv3d handed a folded 4-D tensor) and `rational_resampler` (five-value unpack at spatial_rational_resampler.py:41) |
-| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118) |
+| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118). ANY sample rate: the mel front-end resamples (`ops.py:36-42`, #2583) and the soundtrack returns at the FILE's rate |
 | LTX-2.5 keyframe interpolation (`KeyframeInterpolationPipeline`) | LTX-2.5 DiT + spatial upsampler | `ltx2 keyframe:` cases in `test_ltx2_pipeline` / `test_ltx2_video` via `LoadVideoEngine`+`Generate`: frame 0 APPENDS against a `ti2vid_two_stage` control, the x0 invariant on four arms, the 4096 anchor | `--pipeline-kind keyframe_interpolation`. No frame-0 special case, so `--first-frame` is guidance to interpolate FROM; stage 2's audio leaves. `--lora` REQUIRED. `--last-frame` new (#1191). CPU fixtures |
 | LTX-2.5 two-stage text/image-to-video (`TI2VidTwoStagesPipeline`) | LTX-2.5 DiT + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video` `ltx2 ti2vid:` cases through `LoadVideoEngine`+`Generate`; the x0 invariant on all FOUR arms, and the 4096 anchor read at two geometries against a res_2s control that moves | `--pipeline-kind ti2vid_two_stage`. Guided half-res stage 1 on the UNADAPTED model, plain Euler; distilled 3-sigma stage 2. `--lora` REQUIRED, no `--audio-path`; stage 1's audio leaves. CPU fixtures, Full-model run owed |
 | LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in |

examples/ltx2_gen/main.cpp

@@ -174,14 +174,16 @@ const char* Need(int argc, char** argv, int i, const char* flag) {
       "--a2v-guidance-scale is the guider\'s modality_scale; a take here rides whichever\n"
       "recipe the checkpoint resolves, in practice distilled_two_stage, whose sigmas are\n"
       "fixed. So no claim is made that this reproduces upstream\'s A2Vid output.\n\n"
-      "The WAV must already match the checkpoint: its sample rate must equal the audio\n"
-      "VAE\'s mel front-end rate and its channel count the encoder\'s in_channels.\n"
-      "Neither is converted, because upstream resamples with a polyphase kaiser\n"
-      "resampler that is not ported here and feeds the file\'s own channel count into a\n"
-      "fixed-width convolution; both mismatches are refused with both numbers, since a\n"
-      "resampled-wrong or upmixed take renders a finished clip conditioned on audio you\n"
-      "never supplied. The take must also be at least as long as the clip: upstream\n"
-      "truncates a long one and asserts on a short one, so a short one is refused.\n\n"
+      "ANY sample rate is accepted: a take whose rate differs from the audio VAE\'s\n"
+      "mel front-end rate is RESAMPLED to it, with upstream\'s own filter — the\n"
+      "gcd-reduced rational-ratio hann-windowed sinc `torchaudio.functional.resample`\n"
+      "builds at its defaults, which is what ops.py:40 calls. The CHANNEL COUNT must\n"
+      "still equal the encoder\'s in_channels and a mismatch is refused with both\n"
+      "numbers: upstream feeds the file\'s own channel count into a fixed-width\n"
+      "convolution, so up- or downmixing here would render a finished clip conditioned\n"
+      "on audio you never supplied. The take must also be at least as long as the clip:\n"
+      "upstream truncates a long one and asserts on a short one, so a short one is\n"
+      "refused.\n\n"
       "--audio-start-time seeks into the file (default 0) and --audio-max-duration caps\n"
       "how much is read (default: the clip\'s own duration). Either without\n"
       "--audio-path is refused rather than ignored. The rendered audio.wav is your own\n"

include/vllm/model_executor/models/ltx2_audio_input.h

@@ -118,13 +118,17 @@ Ltx2WavFormat Ltx2ProbeWavFormat(const std::string& bytes);
 //
 // `want_channels` is the encoder's `in_channels`; a file that declares anything
 // else is REFUSED rather than mixed, for the reason in this header's third
-// bullet. `want_sample_rate` is the mel front-end's target; a mismatch is
-// REFUSED rather than resampled, because this project ports only the
-// integer-ratio hann-sinc resampler and upstream's `ops.py:40` is an
-// arbitrary-ratio polyphase kaiser one.
+// bullet.
+//
+// THERE IS NO TARGET RATE HERE, and that is upstream's shape rather than an
+// omission: `decode_audio_from_file` reads at the file's own rate
+// (`decode.py:240-300`) and the resample happens two hops later, inside the mel
+// front-end (`ops.py:44-49`). The rendered soundtrack is this decode's output,
+// so it stays at the file's rate too (`a2vid_two_stage.py:301-303`). Row
+// LTX25-AUDIO-RESAMPLE (#2583) removed the refusal that used to sit here.
 Ltx2DecodedAudio Ltx2DecodeAudioWav(const std::string& bytes, const std::string& label,
-                                    int64_t want_channels, int64_t want_sample_rate,
-                                    double start_time, double max_duration);
+                                    int64_t want_channels, double start_time,
+                                    double max_duration);
 
 // `encode_audio` (audio_vae.py:249-274) followed by A2Vid's truncation
 // (a2vid_two_stage.py:201-202): waveform -> log-mel -> encoder -> keep at most

include/vllm/model_executor/models/ltx2_audio_vae_encoder.h

@@ -17,6 +17,7 @@
 //                                     model/audio_vae/attention.py:16-55
 //   (the audio patchifier)        <-  components/patchifiers.py:287-305
 //   (per-channel statistics)      <-  model/audio_vae/ops.py:58-75
+//   Ltx2ResampleWaveform          <-  model/audio_vae/ops.py:36-42
 //   Ltx2WaveformToLogMel          <-  model/audio_vae/ops.py:8-55
 //   Ltx2SlaneyMelFilterbank       <-  torchaudio.functional.melscale_fbanks
 //                                     (reached from ops.py:20-34)
@@ -165,22 +166,39 @@ struct Ltx2AudioProcessorConfig {
 std::vector<float> Ltx2SlaneyMelFilterbank(int64_t n_freqs, double f_min, double f_max,
                                            int64_t n_mels, int64_t sample_rate);
 
+// `AudioProcessor.resample_audio` (ops.py:36-42), which is
+// `torchaudio.functional.resample` at every default: the arbitrary
+// rational-ratio polyphase sinc resampler with `lowpass_filter_width = 6`,
+// `rolloff = 0.99` and — `ops.py:40` passes neither `resampling_method` nor
+// `beta` — a HANN window (functional.py:1441). It is NOT kaiser, which nine
+// statements in this tree asserted before row LTX25-AUDIO-RESAMPLE (#2583).
+//
+// `waveform` is [channels, samples] channel-major and each channel resamples
+// independently against one kernel, as torchaudio's packed batch does. Returns
+// `ceil(new_freq * samples / orig_freq)` samples per channel — the quotient
+// NARROWED TO f32 BEFORE the ceil, because `torch.as_tensor` of a Python float
+// takes the default dtype (functional.py:1427), which keeps one sample fewer
+// than an exact integer ceil at 180697 samples for 44100 -> 16000 and at 48102
+// of the first 60 s worth of lengths. Returns the INPUT unfiltered when the
+// rates already match (ops.py:38-39). Computed in f32, which is the dtype
+// upstream resamples in — see the note at the definition.
+std::vector<float> Ltx2ResampleWaveform(const std::vector<float>& waveform, int64_t channels,
+                                        int64_t samples, int64_t orig_freq, int64_t new_freq,
+                                        int64_t* out_samples);
+
 // `AudioProcessor.waveform_to_mel` (ops.py:44-55) at any channel count.
-// `waveform` is [channels, samples], channel-major, at `sampling_rate`.
+// `source` is [channels, source_samples], channel-major, at `sampling_rate`.
 //
-// A SAMPLE RATE THAT DOES NOT MATCH `target_sample_rate` IS REFUSED BY NAME.
-// Upstream resamples with `torchaudio.functional.resample` (ops.py:40), a
-// polyphase kaiser resampler for an arbitrary rational ratio; this project ports
-// only the integer-ratio hann-sinc variant the BWE stage needs
-// (Ltx2HannSincResampleFilter1d). Refusing is deliberate: silently treating
-// 44.1 kHz samples as 16 kHz produces audio conditioning that is pitched and
-// time-scaled wrong while every shape still checks out.
+// A SAMPLE RATE THAT DOES NOT MATCH `target_sample_rate` IS RESAMPLED, not
+// refused: `waveform_to_mel` calls `resample_audio` before the mel transform
+// (ops.py:49), so the frame count this reports is over the RESAMPLED length.
 //
 // Returns [channels, frames, mel_bins] channel-major — the layout
 // `Ltx2AudioEncoderForward` takes, which is upstream's final
 // `permute(0, 1, 3, 2)` (ops.py:55).
 std::vector<float> Ltx2WaveformToLogMel(const Ltx2AudioProcessorConfig& config,
-                                        const std::vector<float>& waveform, int64_t channels,
-                                        int64_t samples, int64_t sampling_rate, int64_t* out_frames);
+                                        const std::vector<float>& source, int64_t channels,
+                                        int64_t source_samples, int64_t sampling_rate,
+                                        int64_t* out_frames);
 
 }  // namespace vllm

include/vllm/multimodal/ltx2_video.h

@@ -310,13 +310,16 @@ inline constexpr char kLtx2ImageCrfExtra[] = "image_crf";
 // the ordinary text-to-video render.
 //
 // A 16-bit PCM RIFF/WAVE file whose channel count matches the checkpoint's audio
-// VAE encoder `in_channels` and whose sample rate matches its mel front-end.
-// Neither is converted: upstream resamples with an arbitrary-ratio polyphase
-// kaiser resampler (`ops.py:40`) this project has not ported, and it feeds the
-// file's own channel count straight into a conv that declares 2
-// (`model_configurator.py:172`). Both mismatches are refused with both numbers
-// in the message, because a resampled-wrong or upmixed-wrong take conditions the
-// render on a waveform the caller never supplied and still finishes.
+// VAE encoder `in_channels`. ANY SAMPLE RATE is accepted: the mel front-end
+// resamples it to the checkpoint's own, exactly as upstream does
+// (`ops.py:36-49`, ported by row LTX25-AUDIO-RESAMPLE, #2583). The rendered
+// soundtrack still comes back at the FILE's rate, because upstream returns the
+// caller's original take rather than a VAE round trip (`:301-303`).
+//
+// The CHANNEL count is still refused, with both numbers in the message:
+// upstream feeds the file's own count straight into a conv that declares 2
+// (`model_configurator.py:172`), so an upmixed-wrong take would condition the
+// render on a waveform the caller never supplied and still finish.
 //
 // The audio is held FROZEN through every denoise phase — upstream's
 // `ModalitySpec(frozen=True, noise_scale=0.0)` at `a2vid_two_stage.py:251-256`