Add livekit-plugins-funasr (FunASR/SenseVoice local STT)

[LauraGPT]

This PR adds livekit-plugins-funasr, a fully local speech-to-text plugin backed by FunASR and SenseVoice.

Why

SenseVoice is a non-autoregressive multilingual ASR model with strong Chinese and Cantonese support. It runs on-device without an API key and gives LiveKit agents a local alternative for Chinese, Cantonese, English, Japanese, and Korean speech.

What

• Adds FunASRSTT as a non-streaming LiveKit STT implementation.
• Combines incoming audio, resamples it to 16 kHz, converts stereo to mono, strips SenseVoice rich tags, and returns the final transcript plus detected language.
• Defers model initialization to the first recognition and runs both loading and inference off the event loop.
• Keeps lazy initialization and model.generate serialized in the worker thread, including when an awaiting request is cancelled.
• Implements the standard Plugin.download_files() hook so the default SenseVoice model can be prefetched before agent startup.
• Supports configurable model, device, language, and inverse text normalization.
• Registers the package in the root workspace, livekit-agents[funasr], and uv.lock with complete PyTorch and torchaudio runtime dependencies.
• Covers package imports, lazy loading, cancellation safety, model prefetch, resampling, and non-retryable local failures.

Usage

from livekit.plugins import funasr

stt = funasr.STT(model="iic/SenseVoiceSmall")

Prefetch the default model before starting an agent:

python -m livekit.agents download-files

CPU is the safe default. For GPU inference, install PyTorch and torchaudio builds matching the host CUDA runtime and pass device="cuda".

Verification

Validated locally on exact signed head 405afeccfb6eff6a28b58e84c303fb4e8a99aa77, merged with livekit/agents@94d1e21af3ec8918542fbbbc0f577c7534de12df:

• focused FunASR coverage: 6 passed
• adjacent STT, inference, and transcription coverage: 489 passed, 35 expected environment skips; the remaining 6 provider-matrix failures require external credentials or a separate Silero ONNX LFS artifact and do not exercise FunASR
• mypy for livekit.agents and livekit.plugins.funasr: no issues in 211 source files
• Ruff format/check, uv lock --check (378 packages), compileall, and git diff --check pass
• the plugin version, plugin dependency, and livekit-agents[funasr] extra are aligned at 1.7.1
• wheel and sdist build successfully; package metadata requires funasr>=1.3.16 and livekit-agents>=1.7.1, and both artifacts include py.typed
• relative to the prior reviewed head, plugin implementation code is unchanged; this synchronization only advances the package version and LiveKit Agents compatibility floor

The prior head also passed the exact-lock Python 3.14 unit suite (1,935 passed, 3 expected skips), full mypy (635 source files), the transcription-filter section (356/356), and real CPU inference with iic/SenseVoiceSmall. GitHub exact-head CI is complete with 9 successful checks, 3 expected path/evaluation skips, 0 failures, and 0 pending checks; Devin Review and CLA pass, and all 10 review threads remain resolved.

[LauraGPT]

Thanks for the review! Addressed the findings:

• Added the py.typed marker (PEP 561), so the package is type-checkable.
• model property now returns the configured model id instead of a hardcoded string.
• Excluded the nospeech classification label from the reported detected language (it is not a language).
• Serialized model.generate with an asyncio.Lock for thread-safety across concurrent recognitions sharing one instance.

CI is green (ruff + type-check). The transcription core (16-bit PCM -> resample to 16k -> SenseVoice -> cleaned text) was verified locally.

[LauraGPT]

Fresh FunASR/SenseVoice-side validation recheck (2026-07-07 UTC):

• PR is open, non-draft, and mergeable at head 348c46f; mergeable_state=blocked is currently the review gate.
• Thread-aware review audit shows 6 total review threads and 0 current unresolved threads. The previous AudioResampler(..., num_channels=...) discussion is resolved.
• GitHub checks visible on the head are green/skipped as expected: ruff, type-check (3.10), type-check (3.13), unit-tests, platform blockguard tests, release gate, Devin Review, and license/cla are successful; matrix/evaluation plugin jobs are skipped.
• Local focused validation on the current PR checkout:
  • git diff --check upstream/main...HEAD passed.
  • python3 -m py_compile livekit-plugins/livekit-plugins-funasr/livekit/plugins/funasr/stt.py livekit-plugins/livekit-plugins-funasr/livekit/plugins/funasr/__init__.py tests/test_plugin_funasr_stt.py passed.
  • livekit-plugins/livekit-plugins-funasr/pyproject.toml parses; package name is livekit-plugins-funasr with dependencies livekit-agents>=1.2.0, funasr>=1.1.0, and numpy.

Caveat: I did not rerun the full local pytest target in the bare ops checkout because the monorepo runtime package dependencies (livekit.api etc.) are not installed there; GitHub's own unit/type/ruff checks have already run successfully on this head.

[LauraGPT]

Updated this PR to current main and completed the packaging/runtime pass at afab30a:

• the branch is 0 commits behind and the plugin now follows the 1.6.5 release line;
• added livekit-agents[funasr], a regenerated uv.lock, complete PyTorch/torchaudio runtime dependencies, and a real package-import regression;
• local verification passes: 3 FunASR tests, full Ruff/format, mypy across 622 files, wheel/sdist build, and standalone Python 3.10 dependency resolution;
• real SenseVoiceSmall inference through STT.recognize returns the expected Chinese transcript;
• GitHub CI is complete with 9 successes, 3 expected fork skips, and no failures or pending jobs.

@tinalenguyen, would you be able to review the new local STT plugin when convenient?

[LauraGPT]

Synced this PR to current main; exact head cf327cc3c53ac1fad921fd746f745660152e6d99 is 0 commits behind and mergeable. The merge changed none of the FunASR implementation or test files.

Fresh local verification passes the 3 focused plugin regressions, Ruff and format checks, full mypy across 622 source files, uv lock --check, wheel/sdist build, and git diff --check. GitHub has also completed with all visible checks green (plus the expected fork skips) and no unresolved review threads.

@tinalenguyen, could you review the local FunASR/SenseVoice STT plugin when convenient?

[LauraGPT]

Refreshed this PR on top of current livekit/agents@e731eac and pushed head ef94cb81475c350ad6daf5119426d72082b27920.

What changed in the refresh:

• resolved the livekit-agents/pyproject.toml optional-dependency conflict by keeping the upstream >=1.6.6 plugin baseline and preserving the new funasr extra
• aligned livekit-plugins-funasr with the current release baseline: livekit-agents>=1.6.6, funasr>=1.3.16, and numba>=0.61.0 so the lock does not resolve an old Python-incompatible numba stack
• regenerated uv.lock; uv lock --check now passes

Local validation on the refreshed head:

• python -m pytest tests/test_plugin_funasr_stt.py -q -> 3 passed
• python -m pytest tests/test_plugin_funasr_stt.py tests/test_stt_context.py tests/test_inference_stt_context.py tests/test_plugin_assemblyai_stt.py -q -> 157 passed
• ruff check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• ruff format --check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• python -m compileall -q livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• git diff --check

GitHub now reports the PR mergeable again; repository CI is running on the refreshed head.

[LauraGPT]

Current-head verification for e07dd96bae706670f38ac3353127e0c2168e93d7:

• Rechecked the Devin threads with GraphQL: all review threads are resolved. The latest version-mismatch finding is fixed: livekit-agents[funasr] now requires livekit-plugins-funasr>=1.6.6, and livekit/plugins/funasr/version.py reports __version__ = "1.6.6".
• The py.typed marker exists, and local inference failures are raised as APIConnectionError(..., retryable=False).
• GitHub check rollup currently has no bad or pending checks; the PR is mergeable and only waiting for review.

Fresh focused validation on ind-gpu8:

.venv/bin/python -m pytest tests/test_plugin_funasr_stt.py -q
# 3 passed, 4 warnings

.venv/bin/ruff check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# All checks passed!

.venv/bin/ruff format --check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# 5 files already formatted

/cpfs_speech/user/zhifu.gzf/venv_uv_bootstrap/bin/uv lock --check
# exit 0, resolved 359 packages

python -m compileall -q livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
git diff --check upstream/main...HEAD

[LauraGPT]

Current-head verification for 9f995f7ee5fc4aa1a7085f49f97270217a54deac:

• GitHub check rollup is green/skipped only; the PR is mergeable and review-gated.
• Local focused plugin verification on ind-gpu8 passes:

.venv/bin/python -m pytest tests/test_plugin_funasr_stt.py -q
# 3 passed, 4 warnings

.venv/bin/ruff check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# All checks passed!

.venv/bin/ruff format --check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# 5 files already formatted

/cpfs_speech/user/zhifu.gzf/venv_uv_bootstrap/bin/uv lock --check
# resolved 364 packages, exit 0

python -m compileall -q livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
git diff --check upstream/main...HEAD

No new FunASR/SenseVoice-side blocker from this refresh.

[LauraGPT]

Hi @tinalenguyen — this new local FunASR/SenseVoice STT plugin is current with main and ready for plugin-owner review. On exact head 8cc481cbba89469cb9df9ac8d58e23ac995b8d06, all 11 GitHub checks pass; the focused suite is 3/3 under the locked minimum funasr==1.3.16; Ruff, lock, and diff checks pass; and a real 5.547-second CPU sample returns the expected Chinese transcript with language zh. I also refreshed the PR body with the exact validation evidence. Would you be able to take a look when convenient?

[LauraGPT]

@livekit/agent-devs, I synchronized this FunASR/SenseVoice plugin with current main and pushed signed head 405afeccfb6eff6a28b58e84c303fb4e8a99aa77.

Local exact-head verification is complete: the focused FunASR suite passes 6/6; adjacent STT/inference/transcription coverage passes 489 tests with 35 expected skips; Ruff, compileall, lock validation, and targeted mypy (211 source files) pass; and wheel/sdist metadata is aligned at 1.7.1 with livekit-agents>=1.7.1. The six remaining provider-matrix failures are limited to external credentials or a separate Silero ONNX LFS artifact and do not exercise FunASR.

Relative to the previously reviewed head, plugin implementation code is unchanged; only the package version and LiveKit Agents compatibility floor moved to 1.7.1. GitHub exact-head CI is now complete with 9 successful checks, 3 expected skips, 0 failures, and 0 pending checks; Devin Review and CLA pass, all 10 review threads are resolved, and the PR is mergeable. Maintainer review is the only remaining gate.

[LauraGPT]

CI update for e2b1235: ruff, both type-check jobs, all three blockguard jobs, and the release gates passed. The only failed unit test is unrelated to this plugin: tests/test_utils/test_audio_array_buffer.py::TestPerformance::test_3s_buffer_50ms_frames[48k] exceeded its 100 ms runner-dependent threshold (131.6 ms); the run otherwise had 2478 passed and 5 skipped. The fork token cannot rerun upstream Actions (403), so I have left the code unchanged rather than weakening an unrelated performance test.

[LauraGPT]

@livekit/agent-devs Rebased onto current main@d8607e6711bf9770bfd7e5476f79ab63e6a451a6 and pushed signed exact head 48e340ea2e7f2edbb55f652fadeb07a505766e18.

This refresh preserves the current LiveKit 1.8.0 release line: livekit-agents[funasr] and livekit-plugins-funasr now resolve consistently at 1.8.0.

Exact-head validation is complete: 11/11 focused FunASR plugin tests, lock validation, Ruff, and compile/diff checks locally; upstream type-check (3.10/3.13), unit-tests, ruff, Release gate, and all three platform blockguard checks are green. The PR is mergeable with resolved review threads and ready for maintainer review.