Skip to content

dots3-note W7c-2: resample non-16 kHz audio via upstream's own scipy arm, because libswresample is not bit-identical to itself #2828

Description

@localai-org-maint-bot

Row: MODEL-MM-dots3-note-dots3-note-for-causal-lm

W7c-2 is the SAMPLE-RATE half of W7c. It is deliberately separate from W7c-1 (channels,
#2813) because W7c-1 is an exact mirror and this is an unavoidable recorded divergence;
keeping them apart stops one claim borrowing the other's credibility.

Why this cannot be a mirror

Upstream's chain at 9035151d6: decode with soundfile/libsndfile, PyAV/ffmpeg as fallback,
loading at sr=None (vllm/multimodal/media/audio.py:278-284, :216-258); then resample in
the data parser, MultiModalDataParser(target_sr=..., target_channels=1)
(vllm/models/dots3_note/common/processor.py:523-525) via vllm/multimodal/parse.py:695.
AudioResampler's DEFAULT method is "pyav" = libswresample (vllm/multimodal/audio.py:283,
dispatch at :305-316).

libswresample is not bit-identical to itself. Measured locally with ffmpeg 6.1.1, the same
binary and the same input, differing only in CPU dispatch:

ffmpeg -i in.wav -af aresample=16000 -c:a pcm_f32le a.wav
ffmpeg -cpuflags 0 -i in.wav -af aresample=16000 -c:a pcm_f32le b.wav
-> identical: False   ndiff 24691/32000   maxabs 9.686e-08

So a bit-exact gate against upstream's default is impossible in principle, not merely
inconvenient. Its option defaults also come from an unpinned linked binary (av appears only
in requirements/test/cpu.txt at the pin, a TEST lockfile, and PyAV bundles its own FFmpeg),
and the auto-resolved cutoff is not readable from outside the source.

This is the opposite of W6c's precedent. PIL's bicubic WAS portable because Resample.c is
~200 lines of documented filter maths over a fixed-point intermediate with exactly one answer,
and W6c gated it at 0 of 255 worst absolute difference. libswresample is float, SIMD-dispatched
and option-defaulted. W6c argues FOR porting a documented filter and AGAINST pretending
swresample is one.

The decision, already taken

Implement resample_audio_scipy (vllm/multimodal/audio.py:232-250 @ 9035151d6) and
refuse the pyav default permanently with the reason above.

This is not inventing a divergence — it is selecting another arm of upstream's own switch,
and vLLM already ships that arm in production for another model:
vllm/model_executor/models/phi4mm.py:580 @ 9035151d6 passes audio_resample_method="scipy".

The algorithm, stated exactly, because a divergence must name what it does:
scipy.signal.resample_poly at defaults over the gcd-reduced ratio — up = target//gcd,
down = orig//gcd, filter firwin(2*half_len+1, 1.0/max(up,down), window=("kaiser", 5.0))
with half_len = 10*max(up,down), scaled by up, applied by upfirdn with scipy's
n_pre_pad/n_pre_remove centring.

Measured distance from the real default, 44100 -> 16000, interior only: scipy 51.36 dB on
band-limited content, soxr 46.59, torchaudio 26.72. Record the 51.36 so nobody re-derives it.

Do NOT reuse Ltx2ResampleWaveform (#2583, ltx2_audio_vae.cpp:1151). It is a genuine
polyphase resampler and it is tempting, but it is ~25 dB FURTHER from this oracle on exactly
the band-limited content a speech encoder sees, because torchaudio's defaults are a short
kernel (lowpass_filter_width=6, hann) against swr's 32-tap kaiser-9. Say so in the spec so
the next reader does not "simplify" it back.

Gating, with no oracle

§6.4 option B holds. Follow #2583's design, which is the template:
commit scipy 1.17.1 goldens for 44100->16000, 48000->16000, 22050->16000, 8000->16000; gate
max|diff| at ~2x the measured float floor; and add a lower bound plus a difference
assertion
, because a resampler that returns its input, or returns zeros, passes a
tolerance-only gate. Add a length assertion against ceil(n * new / orig).
scipy stands to the vLLM pin exactly as torchaudio stands to the ltx-2 pin in #2583:
resample_audio_scipy is vLLM's own code, so the precedent transfers.

Scope and seam

Build it as a shared vllm::multimodal resample seam, opted into per model, never blanket.
This matters: five rows already refuse a rate mismatch and they are not all the same policy —
Parakeet's refusal is upstream-faithful (feature_extraction_parakeet.py raises rather than
resampling), while dots3's upstream resamples. dots3 opts in. Leave
audio_processor.cpp's Whisper/Voxtral refusal untouched until each model's own upstream
policy is read; one row, one model's policy.

Reachability

Production entry point ApiServer::handle_chat_completions -> InstallMultiModalChatSeam ->
MakeDots3NoteChatSeam -> RouteDots3NoteAudioWav, plus
Dots3NoteAudioProcessor::ProcessWaveform.
Two existing refusal cases must INVERT, which is the ownership test:
test_api_server_dots3_mm_forward.cpp:1389 ("a 22050 Hz WAV names the resampler refusal and
W7c") and tests/vllm/models/test_dots3_note_audio.cpp:818. A served 44.1 kHz WAV must give
the same logprobs as the same audio pre-resampled offline by scipy.

Not in scope

Containers (#2814, not this row). W8, W9, W10, W11. #2616 and #2709 keep their remaining
blockers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions