feat(stems): DiCoSe stem splitter — engine + music-page split (by Hamza Qayyum) - #59
Merged
Conversation
…za Qayyum)
Vendors Hamza Qayyum's DiCoSe.wgsl stem-separation runtime (BS-RoFormer +
one-step consistency-distilled refinement, raw WebGPU WGSL, MIT) at
packages/dicose, following the packages/acestep pattern:
- npm-workspace-ified package.json (pnpm pin dropped, scripts npm-compatible,
author credit retained); vitest suite stays runnable (34 passed, 1
artifact-dependent skip).
- vite build converted to library mode emitting dist/index.js +
dist/worker.js + d.ts; vite dev still serves the upstream demo, so the
browser harnesses under scripts/ are unchanged.
- Two minimal source patches: checkSupport() exported from the package index
(capability gating for hosts), and a DiCoSeClientOptions.createWorker seam
so a consumer bundling the prebuilt dist can supply the dedicated worker
(the internal new URL("./worker.ts", import.meta.url) default remains).
- Weights are never committed: public/model/ gitignored (623 MB f16 package
is hosted; production origin wired in a follow-up commit). Converter venv,
model cache, and generated wav artifacts excluded from vendoring.
- public/Mixture_audio_1.wav symlink replaced with the real file (CI rejects
tracked symlinks).
Upstream repo currently ships no LICENSE text (package.json declares MIT);
Hamza will supply it — noted in THIRD-PARTY-LICENSES.md in a follow-up.
New engine kind — audio in, multiple audios out:
- core/types.ts grows a minimal source-separation contract: StemAudio /
SeparationInput (full-band, stereo-capable — explicitly NOT the 16 kHz mono
ASR decode) and SeparationEngine with decodeFile() + separate().
- src/engines/stem-dicose/: page-side wrapper over the vendored
DiCoSeWorkerClient. checkSupport() gating (shader-f16 + subgroups + fixed
32-wide subgroups + 1 GiB storage binding), site-owned worker entry via the
package's createWorker seam (carries the same no-referrer fetch patch as
the acestep worker), weight/chunk progress mapped onto the shared
LoadProgress/TranscribeProgress shapes. Fast deterministic output mode by
default; { outputMode: "refined" } reaches the full CD path.
- Registry entry (category "analysis", heavy): HF availability probe +
models-local/dicose local-weights-first via localWeightDir; production
weights default to the FluidInference HF mirror.
- Playground run path: separation engines decode the picked file at native
rate/stereo through the engine and render one player + WAV download per
stem (drums/bass/other/vocals + derived instrumental); pcmToWav gains an
optional right channel for stereo downloads.
Root typecheck, production build, prettier, and interface conformance green.
Suno-style follow-up on the /music result panel: a Split stems action next to Download WAV lazy-loads the stem-dicose engine, runs fast-mode DiCoSe on the generated WAV (decoded through the vendored full-band decoder), and renders the five stems — drums / bass / other / vocals / derived instrumental — as players with per-stem stereo WAV downloads. - Progress reuses the generation progress panel: byte-accurate weight download (623 MB, first use only), then chunk-level separation progress from the worker client. - Resource discipline: the separator holds a ~625 MB GPU weight buffer, so the engine is disposed whenever the result panel resets (new generation) and on pagehide; stem object URLs are revoked with it. Generate and Split are mutually exclusive while either runs. - Styling matches the page's design language (result-actions/stems-panel additions live in the existing fluidaudio-web block of the acestep demo stylesheet).
…t, CI job - README: stem-dicose row (verified fast-mode ~3x RT on a 30 s / 48 kHz song), Split stems paragraph in the music section, dicose gate commands, packages/dicose in the layout map, and Hamza Qayyum credited for the DiCoSe.wgsl port alongside ace-step-1.5.wgsl. - THIRD-PARTY-LICENSES.md: DiCoSe.wgsl entry (MIT per package.json; upstream LICENSE text pending from the author) with karchkha/DiCoSe checkpoint provenance. - ci.yml: dicose job mirroring the acestep one (dicose:check + dicose:test; contract tests self-exclude without the gitignored weight package). - Other Audio Models page copy now mentions stem separation.
The Split stems path now probes the vite models-local middleware (models-local/dicose) before falling back to the HF default, matching the registry entry's behavior — localhost iteration shouldn't re-download the 623 MB package that is already on disk. localWeightDir is exported from the engine registry for that.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
fluidaudio-web | e534ec8 | Aug 24 2026, 04:47 PM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrates Hamza Qayyum's DiCoSe.wgsl stem splitter: a raw WebGPU WGSL port of DiCoSe (BS-RoFormer + one-step consistency-distilled refinement, arXiv 2412.06965, karchkha/DiCoSe checkpoints) — vendored as a workspace package and wired up as a new engine kind plus a Suno-style Split stems feature on the music page.
What's here
packages/dicose— vendored DiCoSe.wgsl runtime (same pattern aspackages/acestep): npm-workspace-ified, vite build converted to library mode (dist/index.js+dist/worker.js+ d.ts;vite devstill serves the upstream demo so its browser harnesses are untouched), vitest suite runnable (dicose:test, 34 passed / 1 artifact-dependent skip). Two minimal source patches:checkSupport()exported from the package index, and aDiCoSeClientOptions.createWorkerseam so the host bundles the dedicated worker (which also carries the no-referrer fetch patch, like the acestep worker). Weights are never committed (public/model/gitignored).stem-dicoseengine (src/engines/stem-dicose/) — new engine kind, audio → multi-audio:core/types.tsgrows a minimalSeparationEnginecontract (decodeFile()+separate()over full-band, stereo-capable PCM — explicitly not the 16 kHz mono ASR decode). Registered under Other Audio Models (category: "analysis", heavy) with an HF availability probe and models-local local-weights-first. Fast deterministic mode by default;{ outputMode: "refined" }reaches the full CD path (constructor opt, unused by the UI for now).pcmToWavgains an optional right channel.dicoseCI job mirroring the acestep one.Weights
623 MB f16 package + 1.25 MB manifest hosted at
FluidInference/fluidaudio-web/dicose; verified byte-identical to the local converter output (manifest byte-compare + full-bin length + prefix compare; the loader additionally validates streamed length against the manifest). Requiresshader-f16+subgroups+ fixed 32-wide subgroups + a 1 GiB storage binding —checkSupport()gates all of it with a readable error.Verified (headless Chrome 151, M5 Pro, localhost)
tsc, productionnpm run build, prettier, interface conformance,dicose:check/dicose:testall green.Licensing
DiCoSe.wgsl is MIT per its
package.json; the upstream repo does not yet ship the LICENSE text — Hamza will supply it, and it will land atpackages/dicose/LICENSE(noted in THIRD-PARTY-LICENSES.md). Authorship is credited inpackages/dicose/package.jsonand the README, likepackages/acestep. Model weights retain the karchkha/DiCoSe MIT license.🤖 Generated with Claude Code