[Bug] Non-16 kHz audio is resampled on the channel axis and exported with a stale sample rate

[LauraGPT]

Summary

Non-16 kHz audio uploaded to FunClip has two normalization errors: stereo data is resampled along the channel axis before selecting the first channel, and the normalized waveform is stored with its original sample rate. This changes the waveform duration seen by ASR and the playback duration of exported audio clips.

Environment

• FunClip: current main 9e720617949e32da4c0a564d10cf5a64337fe2c2; five relevant local source files were verified against exact-main Git blobs.
• FunASR: 1.4.15 installed, but no model inference was used in this reproduction.
• OS/Python: Linux x86-64, Python 3.12 environment.
• Installation: isolated venv with the unchanged FunClip requirements and matched torch/torchaudio 2.10.0+cpu.
• Gradio: 4.44.1; librosa 0.11.0; NumPy 1.26.4; soundfile 0.14.0.
• CPU processing. No browser session, GPU or original reporter environment was tested.

Audio or video input

• Synthetic one-second waveforms at 8, 16, 44.1 and 48 kHz, mono and stereo.
• Left channel: 440 Hz sine; right channel: 880 Hz sine. These are diagnostic tones, not spoken-language accuracy samples.
• ASR: a capture-only stand-in returns fixed text/timestamps; actual VideoClipper.recog, clip, librosa and audio serialization run unmodified. No hotwords or LLM provider.
• Additional UI-component check uses actual gr.Audio().preprocess(FileData(...)) on PCM16 WAVs and postprocess() on the clip tuple. This verifies component serialization, not browser end-to-end behavior.

Steps to reproduce

Run from the repository root with the installed requirements:

import sys
import numpy as np
sys.path.insert(0, "funclip")
from videoclipper import VideoClipper

class CaptureASR:
    def generate(self, data, **kwargs):
        self.samples = len(data)
        return [{"text": "test", "raw_text": "test",
                 "timestamp": [[0, 500]]}]

for rate in (16000, 48000):
    t = np.arange(rate) / rate
    left = 0.2 * np.sin(2 * np.pi * 440 * t)
    for channels in (1, 2):
        audio = left if channels == 1 else np.stack(
            [left, 0.3 * np.sin(2 * np.pi * 880 * t)], axis=1)
        model = CaptureASR()
        clipper = VideoClipper(model)
        clipper.lang = "en"
        _, _, state = clipper.recog((rate, audio))
        (out_rate, out), _, _ = clipper.clip(
            "", 0, 0, state, timestamp_list=[[0, 8000]])
        print(rate, channels, model.samples, out_rate, len(out) / out_rate)

The explicit timestamp range selects the first 8,000 samples of the internal 16 kHz clipping timeline, so the requested output duration is 0.5 seconds. The fixed transcript deliberately removes recognition variability from this audio-processing test.

Expected behavior

Every one-second input should supply 16,000 mono samples to ASR, preserving the existing first-channel selection policy. State and returned audio should identify the normalized waveform as 16 kHz. The requested half-second clip should serialize and play for 0.5 seconds. The caller's input array should remain unchanged.

Actual behavior

Input	ASR samples	ASR duration at 16 kHz	Export rate	Half-second clip duration
16 kHz mono	16000	1 s	16000	0.5 s
16 kHz stereo	16000	1 s	16000	0.5 s
48 kHz mono	16000	1 s	48000	0.166667 s
48 kHz stereo	48000	3 s	48000	0.166667 s

The expanded eight-case direct probe also reproduces failures at 8 and 44.1 kHz: six cases violate the duration/rate contract; both 16 kHz controls pass. Input arrays remained unchanged. Actual Gradio PCM16 upload/export gives the same four-case results above.

Logs or traceback

There is no exception in these cases. The output is structurally accepted but has the wrong sample count or rate. The real Gradio serialization probe completed with exit code 0; this is a successful reproduction of incorrect behavior, not a passing regression test.

The root is in recog's normalization order and state assignment. librosa.resample uses the last axis unless specified, while Gradio supplies stereo arrays shaped (samples, channels). The code selects data[:, 0] only after this operation and stores (sr, data) without changing sr. clip slices in a 16 kHz timeline but returns that stale rate.

Screenshots or clips

The sample-count and serialized-WAV duration measurements above are the evidence; no screenshot is needed to demonstrate the rate mismatch. Recognition quality, model timestamps, video exports and speaker diarization have not been evaluated here.

Related historical context: #18 fixed the old 16 kHz assertion and had reporter confirmation; #29 addressed integer PCM conversion. Neither is the current resampling-axis/stale-rate failure, so they have not been reopened. The separate installation and multi-speaker reports #147/#139 are not claimed to have the same cause.

This report was prepared with Codex assistance. No production fix or release has been made for this report yet.

[modelpath-dev]

Seems like the audio resampling is happening incorrectly on the channel axis, which might be causing the duration mismatch. I'd check how the channel selection and sample rate assignment are handled.