Skip to content

ggml-spacemit: type-gate direct GEMV to q4_0, fuse GDN decode row update - #37

Merged
alex-spacemit merged 1 commit into
spacemit-com:agent/ggml-spacemit-backendfrom
co-seven:agent/ggml-spacemit-fix
Sep 15, 2026
Merged

alex-spacemit merged 1 commit into
spacemit-com:agent/ggml-spacemit-backendfrom
co-seven:agent/ggml-spacemit-fix

Conversation

@co-seven

Copy link
Copy Markdown
Collaborator

MUL_MAT direct path (IME reads B straight from DRAM, decode only): restrict to q4_0 and drop the q8_0 arm together with its gemm_n <= 2NB_COLS64 crossover. Paired A/B on K3 shows q8_0 has no winning shape: Qwen3.5-35B dense q8_0 decode GEMVs all fall under the old crossover and still measure 21.8-26.7 GB/s direct vs 28.9-30.8 staged through TCM (tg128 6.71 -> 6.91 with direct disabled), while q4_0 gains from direct (Qwen3-0.6B tg128 57.45 -> 59.05, +2.8%; pp unchanged). 35B carries no physical q4_0 weights after repack, so the gate leaves it on the staged path entirely.

GDN decode: add a fused single-pass row update used for n_tokens <= 4. State rows are independent within a token, so the reference's memcpy + decay/dots pass + update pass (3 reads, 3 writes of the state per head) collapse into one load/store pair per row, with k/q/decay held in vector registers across the row loop; decode reads the input state and writes the output state directly with no intermediate copy. Kernel time at n_tokens == 1 drops 424.6 -> 216.0 us; paired A/B tg128 7.58 -> 7.93 (+4.6%), pp128 unchanged. Chunks with more tokens keep the reference two-pass form: their state working copy is L2-resident so the extra passes already run at the bandwidth floor, and the fused form loses there to register spills and the store-at-row-end reduction dependency stalling row loads on the in-order cores (measured crossover ~8 tokens; chunks are 64, or 16 with kda).

MUL_MAT direct path (IME reads B straight from DRAM, decode only):
restrict to q4_0 and drop the q8_0 arm together with its
gemm_n <= 2*NB_COLS*64 crossover. Paired A/B on K3 shows q8_0 has no
winning shape: Qwen3.5-35B dense q8_0 decode GEMVs all fall under the
old crossover and still measure 21.8-26.7 GB/s direct vs 28.9-30.8
staged through TCM (tg128 6.71 -> 6.91 with direct disabled), while
q4_0 gains from direct (Qwen3-0.6B tg128 57.45 -> 59.05, +2.8%; pp
unchanged). 35B carries no physical q4_0 weights after repack, so the
gate leaves it on the staged path entirely.

GDN decode: add a fused single-pass row update used for n_tokens <= 4.
State rows are independent within a token, so the reference's memcpy +
decay/dots pass + update pass (3 reads, 3 writes of the state per head)
collapse into one load/store pair per row, with k/q/decay held in
vector registers across the row loop; decode reads the input state and
writes the output state directly with no intermediate copy. Kernel time
at n_tokens == 1 drops 424.6 -> 216.0 us; paired A/B tg128 7.58 -> 7.93
(+4.6%), pp128 unchanged. Chunks with more tokens keep the reference
two-pass form: their state working copy is L2-resident so the extra
passes already run at the bandwidth floor, and the fused form loses
there to register spills and the store-at-row-end reduction dependency
stalling row loads on the in-order cores (measured crossover ~8
tokens; chunks are 64, or 16 with kda).

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions github-actions Bot added the ggml label Sep 14, 2026
@alex-spacemit
alex-spacemit merged commit 4e782bc into spacemit-com:agent/ggml-spacemit-backend Sep 15, 2026
9 of 10 checks passed
alex-spacemit added a commit that referenced this pull request Sep 17, 2026
* feat: scaffold ggml-spacemit backend structure

- Add GGML_SPACEMIT option to ggml/CMakeLists.txt
- Register Spacemit backend in ggml-backend-reg.cpp
- Add ggml-spacemit.h public API header
- Create ggml-spacemit/ directory with CMakeLists.txt
- Copy kernel files from ggml-cpu/spacemit/ (independent copy,
  not a reference) to make this backend self-contained

Backend implementation (ggml-spacemit.cpp, session, opnode, fusion)
will be added in subsequent commits.

* feat: implement ggml-spacemit backend

Add the main backend implementation following the ggml-hexagon pattern:

- spacemit-opnode.h: opnode struct with fusion support (mirrors htp-opnode)
- spacemit-session.h: session struct (hardware metadata + graph cache,
  no spert::Stream -- created per graph_compute)
- spacemit-kernels.h: kernel dispatch declarations
- ggml-spacemit.cpp: full backend implementation
  * Buffer management reusing spine_mem_pool (alloc/free)
  * Buffer type interface (alignment 64, is_host=true)
  * Op fusion: 4 patterns (RMS_NORM+MUL, MUL_MAT+ADD, QKV merge, FFN merge)
  * graph_compute: creates spert::Stream per forward (RAII),
    guarded by #if __has_include(<spert.hpp>)
  * supports_op: MUL_MAT, MUL_MAT_ID, RMS_NORM, ADD, UNARY, ROPE, SOFT_MAX,
    view ops
  * Device interface and registry following hexagon pattern

Assisted-by: Hermes Agent

* feat: add spine-runtime as submodule, make it a hard dependency

- Add thirdparty/spine-runtime as a git submodule
- spine-runtime CMakeLists.txt is included via add_subdirectory
- spert.hpp is now #included directly (no #if __has_include guard)
- CMakeLists.txt: include FindSMTIME.cmake for IME1/IME2 detection
- CMakeLists.txt: pass full -march flags to both spert and ggml-spacemit
- Verified: cross-compiles for riscv64 with spacemit toolchain
  (libggml-spacemit.a + libspert.so.0.6.0 produced)

Assisted-by: Hermes Agent

* fix: supports_op returns false for compute ops in Phase 1

Kernel dispatch is not yet implemented (Phase 2). Claiming only view ops
(NONE/RESHAPE/VIEW/PERMUTE/TRANSPOSE) lets the scheduler route compute
ops to CPU, avoiding segfaults from uninitialized tensor data.

Also fix CMakeLists.txt to inline IME detection instead of depending
on GGML_CPU_RISCV64_SPACEMIT flag.

Verified on K3 (k3-dev-006):
  Qwen3-0.6B Q8_0: pp128=80.94 t/s, tg32=11.30 t/s
  Qwen3-1.7B Q4_0: pp128=43.81 t/s, tg32=10.24 t/s

Assisted-by: Hermes Agent

* feat: graph_compute launches entire graph via spert::Stream

Replace per-op TODO stub with a single stream.launch that processes all
nodes in the graph as one SPMD kernel:

- Create spert::Stream per graph_compute (RAII)
- Copy node pointers to a flat array for CC core access
- Each CC core (ith/nth from program_id/grid_dim) iterates all nodes
- Dispatch MUL_MAT via tensor_traits->compute_forward, fallback to
  ggml_compute_forward_mul_mat
- Dispatch RMS_NORM/ADD via spacemit RVV kernels
- Dispatch ROPE/UNARY/SOFT_MAX via ggml-cpu generic ops
- ctx->sync() grid barrier between ops preserves graph dependencies
- fut.sync() waits for completion before returning

Currently supports_op only claims view ops (NONE/RESHAPE/VIEW/PERMUTE/
TRANSPOSE) to avoid segfaults when compute kernels run on CC cores.
The host-side ggml_compute_forward_* functions may not be safe to call
from CC core context (threadpool/thread-local dependencies). This will
be addressed in Phase 2 by writing CC-core-native kernel wrappers.

Verified on K3: pp64=90.86 t/s (Qwen3-0.6B Q8_0)

Assisted-by: Hermes Agent

* refactor: replace ime_env.cpp with spert::backend_info()

- Remove ime_env.cpp from CMakeLists.txt (no longer compiled)
- Provide global_spine_env_info in ggml-spacemit.cpp, initialized from:
  * spert::backend_info() for arch_id, vlen, num_cores (fallback)
  * GGML_SPACEMIT_WORKERS env var for num_cores (takes priority)
  * SPACEMIT_MEM_BACKEND / SPACEMIT_DISABLE_TCM env vars
- Force GGML_CPU_RISCV64_SPACEMIT=OFF when GGML_SPACEMIT=ON
- spine_core_info::get_spine_core_info returns empty (no /proc/cpuinfo)

Verified on K3:
  GGML_SPACEMIT_WORKERS=4: pp64=93.73 t/s
  default (8 cores): pp128=95.13 t/s, tg32=25.72 t/s

Assisted-by: Hermes Agent

* feat: migrate full compute pipeline from ggml-cpu/spacemit

- graph_compute uses ggml_graph_plan + ggml_graph_compute (standard
  ggml-cpu threadpool pipeline) instead of spert::Stream::launch
- init_tensor calls ggml_riscv64_spacemit_get_optimal_repack_type()
  to set tensor->extra with proper tensor_traits
- set_tensor calls ggml_riscv64_spacemit_repack_tensor() for data repack
- buffer_type reuses ggml_backend_cpu_riscv64_spacemit_buffer_type()
  from ime.cpp so extra_buffer_type::get_tensor_traits() can match
- Added get_tensor implementation (was nullptr, caused segfault)
- Added fallback memcpy in set_tensor when no traits
- Define GGML_USE_CPU_RISCV64_SPACEMIT on ggml-cpu so ggml_graph_compute
  uses spacemit thread launch path
- Disable TCM in spert mode (no perfer_core_ids)
- Export extern C functions with visibility(default)
- supports_op claims MUL_MAT, RMS_NORM, ADD, ROPE, SOFT_MAX, UNARY, etc.

Verified on K3 (Qwen3-0.6B Q4_0, -t 8):
  pp128 = 265.96 t/s, tg128 = 6.63 t/s

Assisted-by: Hermes Agent

* feat: graph_compute uses spert::Stream::launch for CC core compute

Replace ggml_graph_compute (CPU threadpool) with spert::Stream::launch:
- Each CC core gets ith/nth from ctx->program_id/grid_dim
- ctx->sync() grid barrier between ops (replaces ggml_barrier)
- threadpool=nullptr: ggml_barrier calls made conditional in ime.cpp
- tcm_buffer=nullptr: forward_mul_mat falls through to Path C (no TCM)
- Dispatch: ggml_cpu_extra_compute_forward for MUL_MAT (tensor_traits),
  then manual switch for ADD/ROPE/SOFT_MAX/UNARY/CONCAT/GET_ROWS/etc

Verified on K3 (Qwen3-0.6B Q4_0):
  pp128 = 108.86 t/s, tg128 = 2.78 t/s

Assisted-by: Hermes Agent

* feat: use spert ctx->alloc_shared for TCM, ctx->sync for in-kernel barrier

- Allocate TCM from spert shared memory via ctx->alloc_shared() in
  the launch lambda, stored into tls_context.tcm_buffer
- forward_mul_mat now takes Path A/B (TCM) instead of Path C
- Phase 1->2 barrier: when threadpool==nullptr, use ctx->sync()
  via tls_context.spert_ctx (stored spert::Context pointer)
- Device get_memory reports host memory so scheduler can assign
  tensors to SPACEMIT backend
- Added TCM buffer accessor wrappers (set/get) and spert_ctx setter
  with visibility(default) for cross-TU access
- Added #include <spert.hpp> to ime.cpp for Context type

Verified on K3 (-sm none, Qwen3-0.6B Q4_0):
  TCM=393216 bytes, Path A/B, pp128=107.80, tg128=2.81

Assisted-by: Hermes Agent

* feat: independent SPACEMIT buffer type, no longer needs -sm none

- SPACEMIT device now has its own buffer type (separate from CPU's
  ggml_backend_cpu_riscv64_spacemit_buffer_type) with its own
  extra_buffer_type context
- ggml_buft_is_spacemit() helper matches both CPU_RISCV64_SPACEMIT
  and SPACEMIT buft names so get_tensor_traits works for both
- get_alloc_size delegates to CPU spacemit buft for repacked size
- Added get_tensor to SPACEMIT buffer interface
- Factory function ggml_spacemit_create_extra_buffer_type() to avoid
  cross-TU class visibility issues
- No longer requires -sm none; scheduler auto-assigns to SPACEMIT

Verified on K3 (default scheduler, Qwen3-0.6B Q4_0):
  pp128=113.69, tg128=2.82

Assisted-by: Hermes Agent

* refactor: remove ggml-cpu fallback, use SPACEMIT own dispatch

- ggml_spacemit_compute_forward: directly calls SPACEMIT's own
  extra_buffer_type::get_tensor_traits->compute_forward
- No ggml_cpu_extra_compute_forward, no ggml_compute_forward_* fallback
- graph_compute launch lambda: single ggml_spacemit_compute_forward call
- extra_buffer_type: public inheritance + public access specifier
- Removed unused ops.h/binary-ops.h includes from ggml-spacemit.cpp

Assisted-by: Hermes Agent

* 1

* refactor spacemit backend ops

* feat(spacemit): keep Qwen3 decode graph on CC cores

Add ROPE, SET_ROWS, and SWIGLU kernels, fix FLASH attention VLEN capability detection, reuse the spert stream, and preserve cross-core synchronization correctness.

* fix(spacemit): serialize the shared persistent stream

Protect stream creation and launch with the device-session mutex so multiple backend handles share the single hardware stream without a C++ data race.

* fix(spacemit): release stream resources after graph compute

Keep spert::Stream scoped to graph_compute because its lifetime owns hardware resources. Retain only a host mutex in the device session to serialize exclusive hardware use across backend handles.

* refactor(spacemit): reuse session compute state

Address MR review by simplifying architecture detection, using ggml logging, growing a session-owned workspace only when needed, and reusing per-core SPACEMIT contexts.

* Agent/ggml spacemit backend update (#36)

* ggml-spacemit: fix --no-mmap correctness by returning false from buffer_type_is_host

The buffer_type is_host callback returned true, which made the model
loader treat the SPACEMIT weight buffer as plain host memory. In the
--no-mmap path (llama_model_loader::load_all_data), that shortcut reads
tensor data directly into cur->data via file->read_raw, bypassing
ggml_backend_tensor_set and therefore the buffer's set_tensor callback.

Since SPACEMIT weights are physically repacked inside set_tensor
(ggml_riscv64_spacemit_repack_tensor), skipping it left the weights in
the original GGUF layout while the IME/RVV kernels expect the repacked
layout, producing garbage output with --no-mmap.

Returning false routes the --no-mmap load through ggml_backend_tensor_set
so repack runs, matching the mmap path. Verified on K3/A100 with
Qwen3-0.6B-Q4_0 --no-mmap: output is now coherent.

* ggml-spacemit: add SILU and SOFT_MAX ops to eliminate CPU fallback

* ggml-spacemit: extend F16 support for CONCAT/GET_ROWS, add CPY type-convert, ROPE freq_factors

* ggml-spacemit: add F16xF32 MUL_MAT RVV kernel for dense weight matmul

* ggml-spacemit: add general CONT kernel for F32/F16 non-contiguous tensors

* ggml-spacemit: add YaRN ROPE support (ext_factor != 0)

* ggml-spacemit: add IMROPE/MROPE support in forward_rope_impl

* ggml-spacemit: add scalar kernels for missing Qwen3.5 ops (MUL_MAT F32, L2_NORM, PAD, TRI, etc.)

* ggml-spacemit: implement GATED_DELTA_NET and SSM_CONV kernels (fix Qwen3.5 garbled output)

* ggml-spacemit: migrate fallback operators and use worker shared buffer

* ggml: add trace profiling support

* ggml-spacemit: optimize rope fallback with RVV

* ggml-spacemit: use row copies for contiguous fallback

* ggml-spacemit: use legacy rope path for small F32 shapes

* ggml-spacemit: trim masked flash attention tiles

* ggml-spacemit: restore fused GDN RVV path

* build(spacemit): use prebuilt spine-runtime

Assisted-by: Codex

* args: refactor mlock/mmap/directio into load-mode (ggml-org#20834)

---------

Co-authored-by: cailinxi <cailinxi@spacemit.com>

* ggml-spacemit: type-gate direct GEMV to q4_0, fuse GDN decode row update (#37)

MUL_MAT direct path (IME reads B straight from DRAM, decode only):
restrict to q4_0 and drop the q8_0 arm together with its
gemm_n <= 2*NB_COLS*64 crossover. Paired A/B on K3 shows q8_0 has no
winning shape: Qwen3.5-35B dense q8_0 decode GEMVs all fall under the
old crossover and still measure 21.8-26.7 GB/s direct vs 28.9-30.8
staged through TCM (tg128 6.71 -> 6.91 with direct disabled), while
q4_0 gains from direct (Qwen3-0.6B tg128 57.45 -> 59.05, +2.8%; pp
unchanged). 35B carries no physical q4_0 weights after repack, so the
gate leaves it on the staged path entirely.

GDN decode: add a fused single-pass row update used for n_tokens <= 4.
State rows are independent within a token, so the reference's memcpy +
decay/dots pass + update pass (3 reads, 3 writes of the state per head)
collapse into one load/store pair per row, with k/q/decay held in
vector registers across the row loop; decode reads the input state and
writes the output state directly with no intermediate copy. Kernel time
at n_tokens == 1 drops 424.6 -> 216.0 us; paired A/B tg128 7.58 -> 7.93
(+4.6%), pp128 unchanged. Chunks with more tokens keep the reference
two-pass form: their state working copy is L2-resident so the extra
passes already run at the bandwidth floor, and the fused form loses
there to register spills and the store-at-row-end reduction dependency
stalling row loads on the in-order cores (measured crossover ~8
tokens; chunks are 64, or 16 with kda).

Co-authored-by: Claude Code <noreply@anthropic.com>

* ggml-spacemit: bind stream cores via SPACEMIT_PERFER_CORE_ID

Port the core-pinning behavior of the old ggml-cpu/spacemit spert adapter:
spine_env_info parses SPACEMIT_PERFER_CORE_ID (comma-separated physical
CC-core ids) into perfer_core_ids and caps num_cores by the pool size;
graph_compute then creates the spert Stream with
StreamConfig{n_cores, core_ids} so decoder tiles are pinned to the
requested CC cores instead of the default all-core grant, which collided
with the ONNX encoder EP cores.

Verified on K3 board (multi-asr 4+4: enc 8-11 / dec 12-15): during
decode only cores 12-15 are busy and 8-11 stay idle.

Co-Authored-By: Claude Code <noreply@anthropic.com>

---------

Co-authored-by: Alex <jinghui.huang@spacemit.com>
Co-authored-by: cailinxi <cailinxi@spacemit.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants