Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion BUILDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,15 @@ build/bin/game_ggml_cli extract input.wav \

# Serve mode (for OpenUtau integration)
build/bin/game_ggml_cli serve game_medium.gguf
# Then write binary request frames to stdin (see src/cli/main.cpp for protocol)
# Then write binary request frames to stdin (see src/cli/main.cpp for protocol).

# API spec: the serve protocol (VRES request frames, notes-JSON responses)
# lives in src/cli/main.cpp (`serve` command). Packages in releases from
# v0.1.1+ speak this current protocol.
#
# > Early OpenUtau builds use the OLD API spec: install the `old_`-prefixed
# > .oudep from the v0.1.0 release (README → "OpenUtau integration"), not the
# > latest package. Do not remove the v0.1.0 old assets — old OpenUtau needs them.
```

## CUDA compatibility and CI scope
Expand Down
51 changes: 35 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,10 @@ waveform (44100 Hz mono)
## Build

```bash
cmake -S ggml_backend -B ggml_backend/build \
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DGAME_GGML_BUILD_TESTS=ON
cmake --build ggml_backend/build -j
cmake --build build -j
```

Options:
Expand All @@ -77,10 +77,10 @@ Options:
## Convert a PyTorch checkpoint

```bash
pip install -r ggml_backend/scripts/requirements.txt
python ggml_backend/scripts/convert_pt_to_gguf.py \
pip install -r scripts/requirements.txt
python scripts/convert_pt_to_gguf.py \
--model-dir GAME-pt-1.0-medium \
-o ggml_backend/assets/game_medium.gguf
-o game_medium.gguf
```

The script reads `model.pt` + `config.yaml` + `lang_map.json` from the given
Expand All @@ -90,14 +90,14 @@ directory and writes a single GGUF file containing all 671 tensors (FP32) and
Inspect the result:

```bash
./ggml_backend/build/bin/game_ggml_cli inspect ggml_backend/assets/game_medium.gguf
./build/bin/game_ggml_cli inspect game_medium.gguf
```

## Run inference

```bash
./ggml_backend/build/bin/game_ggml_cli extract input.wav \
-m ggml_backend/assets/game_medium.gguf \
./build/bin/game_ggml_cli extract input.wav \
-m game_medium.gguf \
--output-formats mid,txt,csv \
--output-dir out/ \
--tempo 120 \
Expand Down Expand Up @@ -228,6 +228,25 @@ configs produce identical note output):
launches load precompiled PSOs instead of recompiling — no reliance on
driver-level caches.

## OpenUtau integration (.oudep)

Packages for OpenUtau are distributed as `.oudep` archives (a zip of
`game_ggml_cli` + `game_medium.gguf` + `config.json` + `oudep.yaml`) from the
[GitHub Releases](https://github.com/KakaruHayate/game.cpp/releases). Install
the package for your platform; OpenUtau unpacks it itself.

| OpenUtau build | Package to install |
|---|---|
| **Current** (new serve-API protocol) | `game_ggml-<platform>.oudep` or `-q8` — from the **latest** release (e.g. `v0.1.3`). This package speaks the current serve protocol (`src/cli/main.cpp`, `game_ggml_cli serve`). |
| **Early** (old API spec) | `old_game_ggml-<platform>.oudep` — only shipped on the [`v0.1.0` release](https://github.com/KakaruHayate/game.cpp/releases/tag/v0.1.0). |

> If your (older) OpenUtau build fails to talk to the engine, you are on the
> old API spec — download the **`old_`-prefixed** `.oudep` from v0.1.0, not the
> newest release.

Platforms: `windows-x64-vulkan`, `linux-x64-vulkan`,
`macos-arm64-metal`, `macos-x64-metal` (old-prefixed set).

## Reproducing the benchmark

```bash
Expand All @@ -238,25 +257,25 @@ y, _ = librosa.load('28.wav', sr=44100, mono=True)
sf.write('/tmp/28_44100.wav', y, 44100, subtype='PCM_16')"

# 2. Capture PyTorch's D3PM RNG stream (also produces a reference MIDI)
python3 ggml_backend/scripts/align_demo.py /tmp/28_44100.wav \
python3 scripts/align_demo.py /tmp/28_44100.wav \
-m GAME-pt-1.0-medium/model.pt \
-g ggml_backend/assets/game_medium.gguf \
--cli ggml_backend/build/bin/game_ggml_cli \
-g game_medium.gguf \
--cli build/bin/game_ggml_cli \
-l zh -o /tmp/align_out

# 3. Run the 3-per-side subprocess-isolated benchmark
python3 ggml_backend/scripts/benchmark_align.py /tmp/28_44100.wav \
python3 scripts/benchmark_align.py /tmp/28_44100.wav \
-m GAME-pt-1.0-medium/model.pt \
-g ggml_backend/assets/game_medium.gguf \
--cli ggml_backend/build/bin/game_ggml_cli \
-g game_medium.gguf \
--cli build/bin/game_ggml_cli \
--rng /tmp/align_out/align_rng.bin \
-l zh -o /tmp/bench_out --runs 3
```

## Using as a third-party library

```cmake
add_subdirectory(path/to/GAME/ggml_backend)
add_subdirectory(path/to/game.cpp)

add_executable(my_app main.cpp)
target_link_libraries(my_app PRIVATE game_ggml::game_ggml)
Expand Down Expand Up @@ -305,7 +324,7 @@ standalone CMake project that builds against the library.
## Tests

```bash
ctest --test-dir ggml_backend/build --output-on-failure
ctest --test-dir build --output-on-failure
```

The suite has 37 tests covering:
Expand Down
50 changes: 34 additions & 16 deletions README_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,10 @@ waveform (44100 Hz mono)
## 构建

```bash
cmake -S ggml_backend -B ggml_backend/build \
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DGAME_GGML_BUILD_TESTS=ON
cmake --build ggml_backend/build -j
cmake --build build -j
```

选项:
Expand All @@ -67,25 +67,25 @@ cmake --build ggml_backend/build -j
## 转换 PyTorch checkpoint

```bash
pip install -r ggml_backend/scripts/requirements.txt
python ggml_backend/scripts/convert_pt_to_gguf.py \
pip install -r scripts/requirements.txt
python scripts/convert_pt_to_gguf.py \
--model-dir GAME-pt-1.0-medium \
-o ggml_backend/assets/game_medium.gguf
-o game_medium.gguf
```

脚本读取指定目录下的 `model.pt` + `config.yaml` + `lang_map.json`,写出单个 GGUF(含全部 671 个 tensor(FP32)与 74 个 metadata KV)。

检查结果:

```bash
./ggml_backend/build/bin/game_ggml_cli inspect ggml_backend/assets/game_medium.gguf
./build/bin/game_ggml_cli inspect game_medium.gguf
```

## 运行推理

```bash
./ggml_backend/build/bin/game_ggml_cli extract input.wav \
-m ggml_backend/assets/game_medium.gguf \
./build/bin/game_ggml_cli extract input.wav \
-m game_medium.gguf \
--output-formats mid,txt,csv \
--output-dir out/ \
--tempo 120 \
Expand Down Expand Up @@ -188,6 +188,24 @@ GPU 后端(Vulkan/Metal/CUDA)同样默认**开启**:设备侧缓存判定
- CPU 用 `-q8` 包(省内存、速度基本持平)。
- 冷启动:GPU 首次推理要编译 shader(Vulkan/Metal)。NVIDIA 驱动会跨进程缓存,仅首次调用多耗几秒。

## OpenUtau 集成(.oudep)

供 OpenUtau 使用的插件以 `.oudep` 压缩包分发(zip,内含 `game_ggml_cli` +
`game_medium.gguf` + `config.json` + `oudep.yaml`),见
[GitHub Releases](https://github.com/KakaruHayate/game.cpp/releases)。按你的
平台安装即可,OpenUtau 会自动解包。

| OpenUtau 版本 | 应安装的包 |
|---|---|
| **新版**(新的 serve-API 协议) | `game_ggml-<平台>.oudep` 或 `-q8`——取**最新** Release(如 `v0.1.3`)。该包使用当前的 serve 协议(`src/cli/main.cpp`,`game_ggml_cli serve`)。 |
| **早期**(旧 API 规范) | `old_game_ggml-<平台>.oudep` —— 仅存于 [`v0.1.0` Release](https://github.com/KakaruHayate/game.cpp/releases/tag/v0.1.0)。 |

> 如果(较旧的)OpenUtau 连不上引擎,说明你走的是旧 API 规范——请下载
> **`old_` 前缀**的 `.oudep`(v0.1.0),而不是最新 Release 里的包。

平台:`windows-x64-vulkan`、`linux-x64-vulkan`、
`macos-arm64-metal`、`macos-x64-metal`(old 前缀集合)。

## 复现基准测试

```bash
Expand All @@ -198,25 +216,25 @@ y, _ = librosa.load('28.wav', sr=44100, mono=True)
sf.write('/tmp/28_44100.wav', y, 44100, subtype='PCM_16')"

# 2. 捕获 PyTorch 的 D3PM RNG 流(同时产出参考 MIDI)
python3 ggml_backend/scripts/align_demo.py /tmp/28_44100.wav \
python3 scripts/align_demo.py /tmp/28_44100.wav \
-m GAME-pt-1.0-medium/model.pt \
-g ggml_backend/assets/game_medium.gguf \
--cli ggml_backend/build/bin/game_ggml_cli \
-g game_medium.gguf \
--cli build/bin/game_ggml_cli \
-l zh -o /tmp/align_out

# 3. 运行 3-per-side 子进程隔离基准
python3 ggml_backend/scripts/benchmark_align.py /tmp/28_44100.wav \
python3 scripts/benchmark_align.py /tmp/28_44100.wav \
-m GAME-pt-1.0-medium/model.pt \
-g ggml_backend/assets/game_medium.gguf \
--cli ggml_backend/build/bin/game_ggml_cli \
-g game_medium.gguf \
--cli build/bin/game_ggml_cli \
--rng /tmp/align_out/align_rng.bin \
-l zh -o /tmp/bench_out --runs 3
```

## 作为第三方库使用

```cmake
add_subdirectory(path/to/GAME/ggml_backend)
add_subdirectory(path/to/game.cpp)

add_executable(my_app main.cpp)
target_link_libraries(my_app PRIVATE game_ggml::game_ggml)
Expand Down Expand Up @@ -261,7 +279,7 @@ int main() {
## 测试

```bash
ctest --test-dir ggml_backend/build --output-on-failure
ctest --test-dir build --output-on-failure
```

套件共 37 个测试,覆盖:后端初始化、GGUF I/O round-trip、每个算子(RMSNorm、Linear、LayerScale、Embedding、GLU-FFN、CgMLP、三种模式 RoPE、Attention、PAC、EBF block)、Encoder/Segmenter/Estimator 端到端 vs PyTorch 参考 dump、D3PM 8-step 注入 RNG 逐位一致(容忍 Metal FP32 漂移引起的 ≤2/100 边界翻转)、mel 频谱 vs `lib.feature.mel.StretchableMelSpectrogram`、Slicer、MIDI 写入器、TXT/CSV 文本写入器、全流水线逐位 E2E。
Expand Down
8 changes: 4 additions & 4 deletions examples/external_consumer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ examples/external_consumer/

```bash
# From the repo root:
cmake -S ggml_backend/examples/external_consumer -B /tmp/consumer
cmake -S examples/external_consumer -B /tmp/consumer
cmake --build /tmp/consumer -j
/tmp/consumer/my_app ggml_backend/assets/game_small.gguf /path/to/input.wav
/tmp/consumer/my_app game_small.gguf /path/to/input.wav
```

## What it illustrates
Expand All @@ -29,10 +29,10 @@ cmake --build /tmp/consumer -j
broader format support.

The model's backend (CPU / Metal / CUDA / Vulkan) is inherited from the parent
`ggml_backend/` build configuration. To override, pass the relevant flags
`your build configuration. To override, pass the relevant flags
when you configure:

```bash
cmake -S ggml_backend/examples/external_consumer -B /tmp/consumer \
cmake -S examples/external_consumer -B /tmp/consumer \
-DGAME_GGML_METAL=OFF # CPU-only build
```
22 changes: 11 additions & 11 deletions scripts/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# `ggml_backend/scripts/`
# `scripts/`

Python helpers that complement the C++ binaries. All scripts assume they are
run from the repo root and have `torch`, `numpy`, `gguf`, `pyyaml`,
Expand All @@ -15,28 +15,28 @@ run from the repo root and have `torch`, `numpy`, `gguf`, `pyyaml`,
## Typical workflow

```bash
pip install -r ggml_backend/scripts/requirements.txt
pip install -r scripts/requirements.txt

# one-time: produce the GGUF for the C++ backend
python ggml_backend/scripts/convert_pt_to_gguf.py \
python scripts/convert_pt_to_gguf.py \
--model-dir GAME-pt-1.0-small \
-o ggml_backend/assets/game_small.gguf
-o game_small.gguf

# each time you touch a C++ op: regenerate the reference dumps so tests pass
python ggml_backend/scripts/dump_reference.py --category all --out ggml_backend/tests/data
python scripts/dump_reference.py --category all --out tests/data

# one-off: verify bit-exact alignment on a real clip
python ggml_backend/scripts/align_demo.py /path/to/your.wav \
python scripts/align_demo.py /path/to/your.wav \
-m GAME-pt-1.0-small/model.pt \
-g ggml_backend/assets/game_small.gguf \
--cli ggml_backend/build/bin/game_ggml_cli \
-g game_small.gguf \
--cli build/bin/game_ggml_cli \
-l zh -o /tmp/align_out

# one-off: speed + memory benchmark (reuses align_rng.bin from align_demo)
python ggml_backend/scripts/benchmark_align.py /path/to/your.wav \
python scripts/benchmark_align.py /path/to/your.wav \
-m GAME-pt-1.0-small/model.pt \
-g ggml_backend/assets/game_small.gguf \
--cli ggml_backend/build/bin/game_ggml_cli \
-g game_small.gguf \
--cli build/bin/game_ggml_cli \
--rng /tmp/align_out/align_rng.bin \
-l zh -o /tmp/bench_out --runs 3
```
Expand Down
Loading