Native C++ inference for the GAME singing-voice-to-MIDI model, built on
ggml. Runs on CPU, Metal (default on
Apple Silicon), CUDA, or Vulkan. Drop-in replacement for
python infer.py extract with no Python dependency at runtime.
This work grew out of KCKT0112/GAME-ggml (native C++/ggml GAME inference — CPU/Metal/CUDA/Vulkan, MIT). We build on the model, the operator set, and the weight layout first ported there, and retain the GAME PyTorch original as the parity reference.
The Electron UI in ui/ follows the design language of
KCKT0112/web-game (Web Generative
Adaptive MIDI Extractor) — light theme, drop zone, status dots, 88-key piano
roll, log console. That project ships no license (all rights reserved by
default), so no code was taken from it: every file under ui/ was authored
in this repository. See NOTICE.md.
- End-to-end CLI — WAV in, MIDI/TXT/CSV out (mirrors Python
extract). - Small footprint — ~50 MB GGUF for the 1.0-medium checkpoint, ~50 M params.
- Fast startup — Metal binary-archive patch keeps first-run latency under a
second on Apple Silicon. (The patch is kept in-repo at
cmake/patches/ggml-metal-binary-archive.{patch,md}; see the.mdfor design and re-apply notes.) - Third-party integration — clean PIMPL C++ API;
add_subdirectoryand linkgame_ggml::game_ggml. - Parity-tested — full pipeline output matches the PyTorch reference bit-for-bit when the same RNG numbers are injected.
waveform (44100 Hz mono)
│
▼
MelExtractor (pocketfft STFT + librosa-compatible mel)
│ mel [T, 80]
▼
Encoder (EBFBackbone, 4 layers, dim=128)
│ x_seg, x_est each [T, 128]
▼
D3PM loop (1 step default; --nsteps 8 for higher quality)
├─ remove_mutable_boundaries (stochastic)
├─ Segmenter (EBFBackbone, 8 layers; noise/time/lang embeddings)
└─ decode_soft_boundaries (local-max)
│ regions [T] + N
▼
Estimator (JEBFBackbone, 4 layers; joint attention, mixed RoPE)
│ pool_logits [N, 257]
▼
Gaussian-blurred pitch decode → notes (offset, duration, pitch, voiced)
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DGAME_GGML_BUILD_TESTS=ON
cmake --build build -jOptions:
| Option | Default | Meaning |
|---|---|---|
GAME_GGML_METAL |
ON (Apple only) |
Enable Metal backend |
GAME_GGML_CUDA |
OFF |
Enable CUDA backend |
GAME_GGML_VULKAN |
OFF |
Enable Vulkan backend |
GAME_GGML_BUILD_CLI |
ON |
Build game_ggml_cli |
GAME_GGML_BUILD_TESTS |
OFF |
Build GoogleTest suite |
pip install -r scripts/requirements.txt
python scripts/convert_pt_to_gguf.py \
--model-dir GAME-pt-1.0-medium \
-o game_medium.ggufThe script reads model.pt + config.yaml + lang_map.json from the given
directory and writes a single GGUF file containing all 671 tensors (FP32) and
74 metadata KV pairs.
Inspect the result:
./build/bin/game_ggml_cli inspect game_medium.gguf./build/bin/game_ggml_cli extract input.wav \
-m game_medium.gguf \
--output-formats mid,txt,csv \
--output-dir out/ \
--tempo 120 \
--seed 42| CLI flag | Python equivalent |
|---|---|
-m / --model |
-m |
-l / --language |
-l (takes numeric id — use inspect to see the mapping) |
--output-formats |
--output-formats |
--output-dir |
--output-dir |
--tempo |
--tempo |
--seg-threshold |
--seg-threshold |
--seg-radius |
--seg-radius (in frames) |
--est-threshold |
--est-threshold |
--t0 / --nsteps |
--t0 / --nsteps (ggml defaults to --nsteps 1; Python defaults to 8) |
--seed |
(new) — 0 pulls a random seed from the OS |
--pitch-format |
--pitch-format |
--round-pitch |
--round-pitch |
--rng-replay <path> |
(new) — replay D3PM random numbers from a file for bit-exact parity with PyTorch |
Measured on Apple M4 (macOS, 16-core Apple Silicon), 3 runs per side under
/usr/bin/time -l, fresh subprocess each run (so peak RSS is clean). Input:
28.wav — 216.0 s mono, resampled to 44.1 kHz.
Both sides consume the exact same random-number stream via --rng-replay
(see scripts/align_demo.py), so the note lists are bit-exact aligned.
min mean max
PyTorch wall (s) 5.99 6.27 6.71 (MPS via Lightning)
ggml wall (s) 3.05 3.06 3.06 (Metal, default binary)
Speedup 2.05 ×
PyTorch peak RSS 980.8 MB 981.2 MB 981.7 MB
ggml peak RSS 334.1 MB 334.4 MB 334.8 MB
Memory ratio 2.93 ×
PyTorch notes: 471 ┐
ggml notes: 471 ├── matched 1-to-1, max |Δpitch| = 0.000 semitone
┘
Real-time factor: 70.6× (ggml) vs 34.4× (PyTorch).
min mean max
PyTorch wall (s) 17.73 18.05 18.65
ggml wall (s) 9.11 9.28 9.62
Speedup 1.94 ×
Real-time factor drops to 23.3× (ggml) / 11.97× (PyTorch), but segmentation quality is marginally higher (+9 notes recovered in this clip).
Run with GAME_GGML_PROFILE=1 to print a per-chunk breakdown. Numbers below
use --nsteps 8 so the segmenter share is visible:
encoder ~0.17 s (~16%) waveform → x_seg/x_est (mel + spec_proj + 4× EBF)
segmenter ~0.79 s (~78%) x_seg → boundaries (8× D3PM sampling steps)
estimator ~0.06 s (~ 6%) x_est + regions → notes (4× JEBF + joint attn)
Segmenter dominates because D3PM loops it --nsteps times. Default 1
keeps it cheap; bump to 4 or 8 for higher quality at linear cost.
For --nsteps > 1, a cross-step DBCache is on by default on every
backend (threshold 0.25, front blocks 1, warmup 1): when the segmenter's
front-block residual between consecutive D3PM steps is below the
threshold, the tail blocks are skipped and the previous step's tail delta
is reused — a near-lossless approximation (~0.2–0.3 cents pitch drift, no
note-count change in the ablation) that cuts nsteps=8 segmenter wall time
roughly in half.
On GPU backends (Vulkan/Metal/CUDA) it is also on by default: the
device-side cache decision (B) removed the per-step host round-trip that
used to regress quantized-weight graphs, so the same 0.25 threshold
applies everywhere. Set --cache-threshold 0 to disable.
- Tuning:
--cache-threshold <float>(auto / 0 = off / 0.25...),--cache-fn-blocks <int>,--cache-warmup <int>. - Robustness knobs (borrowed from cache-dit/edge-dit.cpp): a reuse WINDOW
(
--cache-window-start/end, fractions of the loop — first/last steps stay full compute), UCache-style accumulated-error gate (--cache-error-decay+--cache-error-limit), and a consecutive-hit cap (--cache-max-continuous). All default to the per-step-threshold behavior. - With
--nsteps 1the cache is automatically disabled even if a threshold is set, keeping the fused single-graph path with zero overhead. --cache-bn-blocks(recompute the last N tail blocks on a hit) is implemented but not recommended: the extra host round-trip between the middle and back slices reintroduces the fused-vs-split fp drift and regressed note count (33→32 on the test clip); keep 0.
Local measurements (10s clip, seed 42; RTX 2070 Vulkan vs CPU AVX2; all configs produce identical note output):
| EP | weights | nsteps=1 | nsteps=8 |
|---|---|---|---|
| CPU | Q8_0 | 5.9s | 14.0s（DBCache on） |
| Vulkan (warm) | F32 | 0.39s | 0.65s |
| Vulkan (warm) | Q8_0 | 0.42s | 1.09s |
- GPU backends are 10–20× faster than CPU here once warm — use
-full(F32) packs when a discrete GPU is available; Q8 is a close second if RAM is limited. - CPU benefits from Q8 for memory, at essentially the same speed — use the
-q8pack on CPU. - Cold start: the first inference on a GPU compiles shaders (Vulkan/Metal).
Metal uses an in-repo binary-archive PSO cache; Vulkan now persists a
disk-backed
VkPipelineCache(cmake/patches/ggml-vulkan-pipeline-cache.*,GGML_VK_PIPELINE_CACHE_PATH, default under the user cache dir), so later launches load precompiled PSOs instead of recompiling — no reliance on driver-level caches.
Packages for OpenUtau are distributed as .oudep archives (a zip of
game_ggml_cli + game_medium.gguf + config.json + oudep.yaml) from the
GitHub Releases. Install
the package for your platform; OpenUtau unpacks it itself.
| OpenUtau build | Package to install |
|---|---|
| Current (new serve-API protocol) | game_ggml-<platform>.oudep or -q8 — from the latest release (e.g. v0.1.3). This package speaks the current serve protocol (src/cli/main.cpp, game_ggml_cli serve). |
| Early (old API spec) | old_game_ggml-<platform>.oudep — only shipped on the v0.1.0 release. |
If your (older) OpenUtau build fails to talk to the engine, you are on the old API spec — download the
old_-prefixed.oudepfrom v0.1.0, not the newest release.
Platforms: windows-x64-vulkan, linux-x64-vulkan,
macos-arm64-metal, macos-x64-metal (old-prefixed set).
# 1. Resample to 44.1 kHz / mono (if not already)
python3 -c "
import librosa, soundfile as sf
y, _ = librosa.load('28.wav', sr=44100, mono=True)
sf.write('/tmp/28_44100.wav', y, 44100, subtype='PCM_16')"
# 2. Capture PyTorch's D3PM RNG stream (also produces a reference MIDI)
python3 scripts/align_demo.py /tmp/28_44100.wav \
-m GAME-pt-1.0-medium/model.pt \
-g game_medium.gguf \
--cli build/bin/game_ggml_cli \
-l zh -o /tmp/align_out
# 3. Run the 3-per-side subprocess-isolated benchmark
python3 scripts/benchmark_align.py /tmp/28_44100.wav \
-m GAME-pt-1.0-medium/model.pt \
-g game_medium.gguf \
--cli build/bin/game_ggml_cli \
--rng /tmp/align_out/align_rng.bin \
-l zh -o /tmp/bench_out --runs 3add_subdirectory(path/to/game.cpp)
add_executable(my_app main.cpp)
target_link_libraries(my_app PRIVATE game_ggml::game_ggml)// main.cpp
#include <game_ggml/model.h>
#include <vector>
int main() {
auto model = game_ggml::Model::load("game_medium.gguf");
std::vector<float> waveform = /* ... load 44100 Hz mono ... */;
game_ggml::InferParams params;
params.language = 4; // from lang_map: { "zh": 4 }
params.seed = 42;
// DBCache (segmenter cross-step reuse; affects nsteps>1 only).
// -1 = auto (0.25 on all backends); 0 = off; >0 = explicit threshold.
params.db_cache_threshold = 0.25f;
params.db_cache_fn_blocks = 1;
params.db_cache_warmup = 1;
// Optional robustness knobs (defaults match --cache-* CLI flags above):
// params.db_cache_window_start = 0.0f; params.db_cache_window_end = 1.0f;
// params.db_cache_err_decay = 0.0f; params.db_cache_max_cont = 0;
// params.db_cache_bn_blocks = 0;
auto result = model.infer(waveform.data(), waveform.size(), params);
for (const auto & n : result.notes) {
if (!n.voiced) continue;
printf(" %.2fs + %.2fs : %.2f\n",
n.offset_seconds, n.duration_seconds, n.pitch_midi);
}
// Cache hit/miss counters for this inference (0/0 when cache disabled).
printf(" dbcache hits=%d misses=%d\n",
result.db_cache_hits, result.db_cache_misses);
}The public header <game_ggml/model.h> uses PIMPL; consumers never transitively
include any ggml header.
See examples/external_consumer/ for a minimal
standalone CMake project that builds against the library.
ctest --test-dir build --output-on-failureThe suite has 37 tests covering:
- Backend initialisation
- GGUF I/O round-trip
- Every op (RMSNorm, Linear, LayerScale, Embedding, GLU-FFN, CgMLP, RoPE in all three modes, Attention, PAC, EBF block)
- Encoder / Segmenter / Estimator end-to-end vs PyTorch reference dumps
- D3PM 8-step loop bit-exact with injected RNG (tolerates ≤ 2/100 boundary flips from Metal FP32 drift)
- Mel spectrogram vs
lib.feature.mel.StretchableMelSpectrogram - Slicer (short-clip + split-on-silence)
- MIDI writer (SMF type-0 structure)
- Text writers (TXT + CSV, note-name formatting)
- Full pipeline bit-exact E2E
Reference dumps are generated by python scripts/dump_reference.py --category all.
Dumps are gitignored — regenerate them as part of CI.
- 44100 Hz mono WAV only — other sample rates raise
InvalidWav. Resampling is deliberately out-of-scope to keep the footprint small. - FP32 and Q8_0 weights — the converter emits FP32 GGUF by default, or Q8_0
via
--quant-config(see CIprepare-model). All backends optimise for the quantized layout; results are near-lossless vs FP32. - Only the shipped
1.0-mediumconfig branch is supported. The estimator rejectssplitattention, learned pool merger,region_token_num > 1, anduse_region_bias=trueat load time with a clearNotImplementedmessage. - Batch size 1 per inference call — matches
infer.py extract. For parallel streams hold multipleModelinstances. - Metal FP32 precision — expected ~1e-3 per matmul; at boundary decoding this can flip one frame out of every few hundred vs the CPU reference.
Everything is fetched at configure time by CMake; nothing is vendored. Source
trees live under build/_deps/<name>-src/ after the first configure.
| Dependency | Version pin | License | SPDX identifier |
|---|---|---|---|
| ggml | v0.19.0 tag (temporary anchor — see AGENT.md "ggml version gate") |
MIT | MIT |
| pocketfft | commit 32424d20 on cpp branch |
BSD-3-Clause | BSD-3-Clause |
| dr_libs | commit 243e26ff on master |
Public Domain / MIT-0 (dual) | Unlicense OR MIT-0 |
| GoogleTest | v1.14.0 tag (tests only) |
BSD-3-Clause | BSD-3-Clause |
Each upstream LICENSE file is preserved under build/_deps/<name>-src/LICENSE*
after download. To update a dependency, change its GIT_TAG (or ggml's
URL/URL_HASH) in cmake/Dependencies.cmake and reconfigure.
MPL-2.0 — Mozilla Public License 2.0.
This project grew out of KCKT0112/GAME-ggml (MIT) and retains openvpi/GAME (MIT) as its parity reference. Those MIT grants are irrevocable and are not superseded by this repository's license — portions derived from them remain available under MIT as well.
Redistributions must also carry the upstream license notices listed in the table above; see NOTICE.md for the full provenance record.