Merge ggml-spacemit into mtmd-backend - #39
Merged
alex-spacemit merged 20 commits intoSep 17, 2026
Merged
Conversation
- Add GGML_SPACEMIT option to ggml/CMakeLists.txt - Register Spacemit backend in ggml-backend-reg.cpp - Add ggml-spacemit.h public API header - Create ggml-spacemit/ directory with CMakeLists.txt - Copy kernel files from ggml-cpu/spacemit/ (independent copy, not a reference) to make this backend self-contained Backend implementation (ggml-spacemit.cpp, session, opnode, fusion) will be added in subsequent commits.
Add the main backend implementation following the ggml-hexagon pattern:
- spacemit-opnode.h: opnode struct with fusion support (mirrors htp-opnode)
- spacemit-session.h: session struct (hardware metadata + graph cache,
no spert::Stream -- created per graph_compute)
- spacemit-kernels.h: kernel dispatch declarations
- ggml-spacemit.cpp: full backend implementation
* Buffer management reusing spine_mem_pool (alloc/free)
* Buffer type interface (alignment 64, is_host=true)
* Op fusion: 4 patterns (RMS_NORM+MUL, MUL_MAT+ADD, QKV merge, FFN merge)
* graph_compute: creates spert::Stream per forward (RAII),
guarded by #if __has_include(<spert.hpp>)
* supports_op: MUL_MAT, MUL_MAT_ID, RMS_NORM, ADD, UNARY, ROPE, SOFT_MAX,
view ops
* Device interface and registry following hexagon pattern
Assisted-by: Hermes Agent
- Add thirdparty/spine-runtime as a git submodule - spine-runtime CMakeLists.txt is included via add_subdirectory - spert.hpp is now #included directly (no #if __has_include guard) - CMakeLists.txt: include FindSMTIME.cmake for IME1/IME2 detection - CMakeLists.txt: pass full -march flags to both spert and ggml-spacemit - Verified: cross-compiles for riscv64 with spacemit toolchain (libggml-spacemit.a + libspert.so.0.6.0 produced) Assisted-by: Hermes Agent
Kernel dispatch is not yet implemented (Phase 2). Claiming only view ops (NONE/RESHAPE/VIEW/PERMUTE/TRANSPOSE) lets the scheduler route compute ops to CPU, avoiding segfaults from uninitialized tensor data. Also fix CMakeLists.txt to inline IME detection instead of depending on GGML_CPU_RISCV64_SPACEMIT flag. Verified on K3 (k3-dev-006): Qwen3-0.6B Q8_0: pp128=80.94 t/s, tg32=11.30 t/s Qwen3-1.7B Q4_0: pp128=43.81 t/s, tg32=10.24 t/s Assisted-by: Hermes Agent
Replace per-op TODO stub with a single stream.launch that processes all nodes in the graph as one SPMD kernel: - Create spert::Stream per graph_compute (RAII) - Copy node pointers to a flat array for CC core access - Each CC core (ith/nth from program_id/grid_dim) iterates all nodes - Dispatch MUL_MAT via tensor_traits->compute_forward, fallback to ggml_compute_forward_mul_mat - Dispatch RMS_NORM/ADD via spacemit RVV kernels - Dispatch ROPE/UNARY/SOFT_MAX via ggml-cpu generic ops - ctx->sync() grid barrier between ops preserves graph dependencies - fut.sync() waits for completion before returning Currently supports_op only claims view ops (NONE/RESHAPE/VIEW/PERMUTE/ TRANSPOSE) to avoid segfaults when compute kernels run on CC cores. The host-side ggml_compute_forward_* functions may not be safe to call from CC core context (threadpool/thread-local dependencies). This will be addressed in Phase 2 by writing CC-core-native kernel wrappers. Verified on K3: pp64=90.86 t/s (Qwen3-0.6B Q8_0) Assisted-by: Hermes Agent
- Remove ime_env.cpp from CMakeLists.txt (no longer compiled) - Provide global_spine_env_info in ggml-spacemit.cpp, initialized from: * spert::backend_info() for arch_id, vlen, num_cores (fallback) * GGML_SPACEMIT_WORKERS env var for num_cores (takes priority) * SPACEMIT_MEM_BACKEND / SPACEMIT_DISABLE_TCM env vars - Force GGML_CPU_RISCV64_SPACEMIT=OFF when GGML_SPACEMIT=ON - spine_core_info::get_spine_core_info returns empty (no /proc/cpuinfo) Verified on K3: GGML_SPACEMIT_WORKERS=4: pp64=93.73 t/s default (8 cores): pp128=95.13 t/s, tg32=25.72 t/s Assisted-by: Hermes Agent
- graph_compute uses ggml_graph_plan + ggml_graph_compute (standard ggml-cpu threadpool pipeline) instead of spert::Stream::launch - init_tensor calls ggml_riscv64_spacemit_get_optimal_repack_type() to set tensor->extra with proper tensor_traits - set_tensor calls ggml_riscv64_spacemit_repack_tensor() for data repack - buffer_type reuses ggml_backend_cpu_riscv64_spacemit_buffer_type() from ime.cpp so extra_buffer_type::get_tensor_traits() can match - Added get_tensor implementation (was nullptr, caused segfault) - Added fallback memcpy in set_tensor when no traits - Define GGML_USE_CPU_RISCV64_SPACEMIT on ggml-cpu so ggml_graph_compute uses spacemit thread launch path - Disable TCM in spert mode (no perfer_core_ids) - Export extern C functions with visibility(default) - supports_op claims MUL_MAT, RMS_NORM, ADD, ROPE, SOFT_MAX, UNARY, etc. Verified on K3 (Qwen3-0.6B Q4_0, -t 8): pp128 = 265.96 t/s, tg128 = 6.63 t/s Assisted-by: Hermes Agent
Replace ggml_graph_compute (CPU threadpool) with spert::Stream::launch: - Each CC core gets ith/nth from ctx->program_id/grid_dim - ctx->sync() grid barrier between ops (replaces ggml_barrier) - threadpool=nullptr: ggml_barrier calls made conditional in ime.cpp - tcm_buffer=nullptr: forward_mul_mat falls through to Path C (no TCM) - Dispatch: ggml_cpu_extra_compute_forward for MUL_MAT (tensor_traits), then manual switch for ADD/ROPE/SOFT_MAX/UNARY/CONCAT/GET_ROWS/etc Verified on K3 (Qwen3-0.6B Q4_0): pp128 = 108.86 t/s, tg128 = 2.78 t/s Assisted-by: Hermes Agent
…rrier - Allocate TCM from spert shared memory via ctx->alloc_shared() in the launch lambda, stored into tls_context.tcm_buffer - forward_mul_mat now takes Path A/B (TCM) instead of Path C - Phase 1->2 barrier: when threadpool==nullptr, use ctx->sync() via tls_context.spert_ctx (stored spert::Context pointer) - Device get_memory reports host memory so scheduler can assign tensors to SPACEMIT backend - Added TCM buffer accessor wrappers (set/get) and spert_ctx setter with visibility(default) for cross-TU access - Added #include <spert.hpp> to ime.cpp for Context type Verified on K3 (-sm none, Qwen3-0.6B Q4_0): TCM=393216 bytes, Path A/B, pp128=107.80, tg128=2.81 Assisted-by: Hermes Agent
- SPACEMIT device now has its own buffer type (separate from CPU's ggml_backend_cpu_riscv64_spacemit_buffer_type) with its own extra_buffer_type context - ggml_buft_is_spacemit() helper matches both CPU_RISCV64_SPACEMIT and SPACEMIT buft names so get_tensor_traits works for both - get_alloc_size delegates to CPU spacemit buft for repacked size - Added get_tensor to SPACEMIT buffer interface - Factory function ggml_spacemit_create_extra_buffer_type() to avoid cross-TU class visibility issues - No longer requires -sm none; scheduler auto-assigns to SPACEMIT Verified on K3 (default scheduler, Qwen3-0.6B Q4_0): pp128=113.69, tg128=2.82 Assisted-by: Hermes Agent
- ggml_spacemit_compute_forward: directly calls SPACEMIT's own extra_buffer_type::get_tensor_traits->compute_forward - No ggml_cpu_extra_compute_forward, no ggml_compute_forward_* fallback - graph_compute launch lambda: single ggml_spacemit_compute_forward call - extra_buffer_type: public inheritance + public access specifier - Removed unused ops.h/binary-ops.h includes from ggml-spacemit.cpp Assisted-by: Hermes Agent
Add ROPE, SET_ROWS, and SWIGLU kernels, fix FLASH attention VLEN capability detection, reuse the spert stream, and preserve cross-core synchronization correctness.
Protect stream creation and launch with the device-session mutex so multiple backend handles share the single hardware stream without a C++ data race.
Keep spert::Stream scoped to graph_compute because its lifetime owns hardware resources. Retain only a host mutex in the device session to serialize exclusive hardware use across backend handles.
Address MR review by simplifying architecture detection, using ggml logging, growing a session-owned workspace only when needed, and reusing per-core SPACEMIT contexts.
* ggml-spacemit: fix --no-mmap correctness by returning false from buffer_type_is_host The buffer_type is_host callback returned true, which made the model loader treat the SPACEMIT weight buffer as plain host memory. In the --no-mmap path (llama_model_loader::load_all_data), that shortcut reads tensor data directly into cur->data via file->read_raw, bypassing ggml_backend_tensor_set and therefore the buffer's set_tensor callback. Since SPACEMIT weights are physically repacked inside set_tensor (ggml_riscv64_spacemit_repack_tensor), skipping it left the weights in the original GGUF layout while the IME/RVV kernels expect the repacked layout, producing garbage output with --no-mmap. Returning false routes the --no-mmap load through ggml_backend_tensor_set so repack runs, matching the mmap path. Verified on K3/A100 with Qwen3-0.6B-Q4_0 --no-mmap: output is now coherent. * ggml-spacemit: add SILU and SOFT_MAX ops to eliminate CPU fallback * ggml-spacemit: extend F16 support for CONCAT/GET_ROWS, add CPY type-convert, ROPE freq_factors * ggml-spacemit: add F16xF32 MUL_MAT RVV kernel for dense weight matmul * ggml-spacemit: add general CONT kernel for F32/F16 non-contiguous tensors * ggml-spacemit: add YaRN ROPE support (ext_factor != 0) * ggml-spacemit: add IMROPE/MROPE support in forward_rope_impl * ggml-spacemit: add scalar kernels for missing Qwen3.5 ops (MUL_MAT F32, L2_NORM, PAD, TRI, etc.) * ggml-spacemit: implement GATED_DELTA_NET and SSM_CONV kernels (fix Qwen3.5 garbled output) * ggml-spacemit: migrate fallback operators and use worker shared buffer * ggml: add trace profiling support * ggml-spacemit: optimize rope fallback with RVV * ggml-spacemit: use row copies for contiguous fallback * ggml-spacemit: use legacy rope path for small F32 shapes * ggml-spacemit: trim masked flash attention tiles * ggml-spacemit: restore fused GDN RVV path * build(spacemit): use prebuilt spine-runtime Assisted-by: Codex * args: refactor mlock/mmap/directio into load-mode (ggml-org#20834) --------- Co-authored-by: cailinxi <cailinxi@spacemit.com>
…ate (spacemit-com#37) MUL_MAT direct path (IME reads B straight from DRAM, decode only): restrict to q4_0 and drop the q8_0 arm together with its gemm_n <= 2*NB_COLS*64 crossover. Paired A/B on K3 shows q8_0 has no winning shape: Qwen3.5-35B dense q8_0 decode GEMVs all fall under the old crossover and still measure 21.8-26.7 GB/s direct vs 28.9-30.8 staged through TCM (tg128 6.71 -> 6.91 with direct disabled), while q4_0 gains from direct (Qwen3-0.6B tg128 57.45 -> 59.05, +2.8%; pp unchanged). 35B carries no physical q4_0 weights after repack, so the gate leaves it on the staged path entirely. GDN decode: add a fused single-pass row update used for n_tokens <= 4. State rows are independent within a token, so the reference's memcpy + decay/dots pass + update pass (3 reads, 3 writes of the state per head) collapse into one load/store pair per row, with k/q/decay held in vector registers across the row loop; decode reads the input state and writes the output state directly with no intermediate copy. Kernel time at n_tokens == 1 drops 424.6 -> 216.0 us; paired A/B tg128 7.58 -> 7.93 (+4.6%), pp128 unchanged. Chunks with more tokens keep the reference two-pass form: their state working copy is L2-resident so the extra passes already run at the bandwidth floor, and the fused form loses there to register spills and the store-at-row-end reduction dependency stalling row loads on the in-order cores (measured crossover ~8 tokens; chunks are 64, or 16 with kda). Co-authored-by: Claude Code <noreply@anthropic.com>
Port the core-pinning behavior of the old ggml-cpu/spacemit spert adapter:
spine_env_info parses SPACEMIT_PERFER_CORE_ID (comma-separated physical
CC-core ids) into perfer_core_ids and caps num_cores by the pool size;
graph_compute then creates the spert Stream with
StreamConfig{n_cores, core_ids} so decoder tiles are pinned to the
requested CC cores instead of the default all-core grant, which collided
with the ONNX encoder EP cores.
Verified on K3 board (multi-asr 4+4: enc 8-11 / dec 12-15): during
decode only cores 12-15 are busy and 8-11 stay idle.
Co-Authored-By: Claude Code <noreply@anthropic.com>
github-actions
Bot
added
documentation
alex-spacemit
merged commit Sep 17, 2026
67c4e59
into
spacemit-com:mtmd-backend
11 of 14 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.