English | 简体中文
The standard MoE-FFN forward flow is:
AllToAll-Dispatch → GMM1(up_proj) → SwiGLU → GMM2(down_proj) → AllToAll-Combine
Each operator is a separate NPU kernel, with substantial HBM traffic between operators as intermediate activations are written and read back. With Expert Parallelism (EP), AllToAll communication and GMM computation usually execute serially, creating substantial idle time.
Measured results on Ascend A3 with a DeepSeek-V3-scale MoE workload:
| Metric | Measurement |
|---|---|
| Average Cube MAC utilization | ~54%, significantly below theoretical peak |
| Critical-path Vector execution time | ~18%, for operations such as SwiGLU while Cube cores are idle |
| AllToAll share of end-to-end time | ~17%, with 61% hidden by GEMM and 39% exposed |
| Mutual Cube/Vector idleness | The two core types alternate; only one type works at any instant |
Root cause: GroupedGEMM uses only Cube cores, leaving Vector cores idle. SwiGLU runs on Vector cores while Cube cores idle. Host-driven AllToAll leaves both AIC and AIV unable to compute during communication gaps.
Multicore MoE-FFN fuses the five operators into one kernel, executed concurrently by AIC (AI Cube) and AIV (AI Vector) cores:
- AIV cores handle AllToAll dispatch/combine and SwiGLU.
- AIC cores handle GMM1 and GMM2 matrix multiplication.
- Shared memory and event semaphores coordinate the pipeline across cores for fine-grained compute-communication overlap.
Key benefits:
- A single kernel reduces host launches and synchronization boundaries, overlapping communication, GMM, and Vector computation by tile.
- Deep compute-communication overlap improves hardware utilization.
- Applies to forward and backward for end-to-end training.
In multi-device AllToAll, simultaneous sends from multiple ranks to one destination cause congestion and expose tail latency. Rank-Aware Tile Reordering (RATR) changes AllToAll tile order to distribute destination traffic over time and avoid converging bursts. It changes no computation: only AllToAll tiles in vector_task_indexs are reordered. RATR and static schedules are generated offline in RuntimeConfig, adding no runtime overhead.
Backward computation is fused similarly:
AllToAll-Dispatch → ┌─ GMM1(act_grad) ──────────────────────┐
└─ GMM4(w2_grad) │
▼
SwiGLU-grad → GMM2(gate_grad) → ┌─ AllToAll-Combine
└─ GMM3(w1_grad)
GMM1 runs concurrently with GMM4, and AllToAll-Combine with GMM3, further reducing idle time.
The framework uses the following layers for the AIC/AIV hardware on Ascend A3/A2 NPUs:
| Layer | Python module | C++ module | Responsibility | Excludes |
|---|---|---|---|---|
| Graph | scheduler/graph.py |
— | Operator topology, split rules, shape inference | Operator-specific parameters |
| Runtime | scheduler/config.py scheduler/scheduler.py |
— | TaskDesc generation, event graphs, RATR optimization | Framework/compiler details |
| Task | tasks/ |
— | Per-operator FillConfig strategies | Core scheduling logic |
| Module | modules/<op>/ |
— | Concrete DAGs and CLI data generation | C++/framework code |
| Scheduler | — | ops/runtime/ |
Generic AIC/AIV event synchronization primitives | Compute kernel implementations |
| Operator | — | ops/<op_name>/ |
Operator-specific kernels and KernelWorker loops | Shared cross-operator logic |
| Torch binding | torch/ |
torch/csrc/ |
Torch operator registration | Graph and scheduling logic |
multicore/
├── __init__.py
├── _loader.py # Locate payload, validate OPP, and load the framework adapter on first use.
│
├── scheduler/ # Python compiler core, without operator specialization.
│ ├── __init__.py
│ ├── config.py # TaskSplitValue + init_task_split_value
│ │ # All ctypes structs, including RuntimeConfigC.
│ ├── graph.py # ComputeGraph / OperatorNode / TensorSpec
│ │ # SplitSpec / OpType / propagate_splits()
│ ├── builder.py # build_runtime_config(graph, tsv, rank_id)
│ ├── scheduler.py # revise_task_queue() / revise_gmm_task_queue_bwd()
│ └── allocator.py # Automatic param_position allocation (reserved).
│
├── tasks/ # Task type registration and fill strategies.
│ ├── __init__.py
│ ├── task_base.py # FillConfig ABC: fill(cfg, op, tsv).
│ ├── alltoall.py # AllToAllFillConfig(dispatch / combine)
│ ├── gmm.py # GmmFillConfig variants.
│ ├── swiglu.py # SwiGLUFillConfig + SwiGLUGradFillConfig
│ ├── utils.py # advance_tsv_* / add_terminate / add_dynamic_data
│ ├── tiling.py # Global tiling byte registry.
│ └── registry.py # FILL_CONFIG_REGISTRY dictionary.
│
├── modules/ # Operator modules: graph definitions and CLI data generation.
│ └── mega_moe/
│ ├── forward/
│ │ ├── graph.py # build_forward_graph(tsv, ...) → ComputeGraph
│ │ ├── gen_runtime_data.py # CLI: --tp/--ep/... generates *.bin files.
│ │ └── tiling_tables.py # GMM/SwiGLU tiling parameter tables.
│ └── backward/
│ ├── graph.py # build_backward_graph(tsv, ...) → ComputeGraph
│ ├── gen_runtime_data.py
│ └── tiling_tables.py
│
├── ops/ # C++ CANN Ascend C implementations.
│ ├── runtime/ # Shared layer copied into each op_kernel/runtime/ during builds.
│ │ ├── runtime_config.hpp # Shared constants and structs: TaskDesc / EventDesc / TaskType.
│ │ └── worker_kernel.h # KernelWorkerBase<Derived> CRTP base class.
│ │
│ ├── hyper_mega_moe/ # MoE FFN forward operator.
│ │ ├── op_host/ # Host-side shape inference, tiling, and APIs.
│ │ └── op_kernel/ # Device side.
│ │ ├── hyper_mega_moe.cpp # Kernel entry point including worker_kernel.cpp.
│ │ ├── hyper_mega_moe_tiling_key.h
│ │ ├── worker_kernel.cpp # Forward KernelWorker specialization: tiling@[23], event@[24].
│ │ │ # #include "runtime/worker_kernel.h"
│ │ ├── swi_glu/ # Isolated build tree only: copied from ops-nn.
│ │ ├── grouped_matmul/ # Isolated build tree only: copied from ops-transformer.
│ │ ├── put_mem_signal/
│ │ │ └── put_mem_signal_kernel.cpp # Self-contained DTYPE_DISPATCH_TARGET instantiation.
│ │ └── runtime/ # Copied from ops/runtime/ by the source assembler.
│ │ ├── runtime_config.hpp
│ │ └── worker_kernel.h
│ │
│ └── hyper_mega_moe_grad/ # MoE FFN backward operator, structured as above.
│ ├── op_host/
│ └── op_kernel/
│ ├── hyper_mega_moe_grad.cpp
│ ├── hyper_mega_moe_grad_tiling_key.h
│ ├── worker_kernel.cpp # Backward KernelWorker specialization: tiling@[30], event@[31].
│ ├── swi_glu/
│ ├── swi_glu_grad/
│ ├── grouped_matmul/
│ ├── put_mem_signal/
│ │ └── put_mem_signal_kernel.cpp # Self-contained DTYPE_DY instantiation.
│ └── runtime/ # Copied during build.
│ ├── runtime_config.hpp
│ └── worker_kernel.h
│
├── torch/ # CMakeLists.txt + csrc/ + ops.py
└── shmem/ # Private Torch one-sided communication and build support.
Three core abstractions:
- TensorSpec: Describes a tensor's shape, dtype, param_position, split_dim, is_dynamic, and transpose.
- SplitSpec: Declares operator splitting through split_inputs, task_num_fn, and split_output_dims.
- ComputeGraph: DAG manager whose
propagate_splits(tsv)computes each node's task_num and split_dim in topological order.
TaskSplitValue encapsulates topology hyperparameters (tp, ep, seq_size, all_expert_num, top_k) and runtime counters.
RuntimeConfigC defines a 64-byte common header. scheduler/runtime.py creates a subclass per graph, appending task,
queue, and event arrays sized to actual capacity. Host and device share the same dense layout; TaskDesc is 576 bytes.
All graphs use this structure regardless of size, without legacy-format branches, magic, or version fields.
Each file implements one operator task type through FillConfig.fill(cfg, op, tsv), writing task/event relationships into
RuntimeConfigC's all_tasks[], task index arrays, and all_event_num_triggers[].
Registration in registry.py:
FILL_CONFIG_REGISTRY: Dict[OpType, Type[FillConfig]] = {
OpType.ALLTOALL: AllToAllFillConfig,
OpType.GMM: GmmFillConfig,
OpType.SWIGLU: SwiGLUFillConfig,
OpType.SWIGLU_GRAD: SwiGLUFillConfig,
}Adding an operator requires only one dictionary entry, without changing core scheduling logic.
| File | Responsibility |
|---|---|
runtime_config.hpp |
Shared structures: RuntimeHeader, TaskDesc, EventDesc, TaskType, and accessors |
worker_kernel.h |
KernelWorkerBase<Derived> CRTP base with Process(), WaitForDependency(), and TriggerEvent() |
Key point: ops/runtime/ is not compiled directly. The source assembler
copies the complete runtime/ directory into
hyper_mega_moe/op_kernel/runtime/ and hyper_mega_moe_grad/op_kernel/runtime/ in an isolated temporary source tree, allowing each operator's
worker_kernel.cpp to access shared definitions through #include "runtime/worker_kernel.h". Neither repository business sources nor upstream dependencies
are modified in place.
Each operator retains its own KernelWorker because its hard-coded input_list indices differ:
| mega_moe (forward) | mega_moe_grad (backward) | |
|---|---|---|
| Tiling parameters | input_list[23] |
input_list[30] |
| Event counters | input_list[24] |
input_list[31] |
| GMM workspace | input_list[11] |
input_list[25] |
| SwiGLU grad workspace | — | input_list[26] |
| Specific compute kernel | TASK_SWI_GLU |
TASK_SWI_GLU_GRAD |
User (Python)
↓ build_forward_graph(tsv)
ComputeGraph [modules/mega_moe/forward/graph.py]
↓ propagate_splits(tsv)
Each OperatorNode's task_num / split_dim is populated automatically.
↓ build_runtime_config(graph, tsv, rank_id)
↓ → Polymorphic FillConfig.fill() calls.
↓ → RATR / GMM interleaving optimization.
RuntimeConfigC (ctypes) → Serialize to runtime_config_rank_*.bin.
↓
Read by C++ worker_kernel.
↓
KernelWorker::Process() main scheduling loop.
AIC (even blocks) → cube_task_indexs → GMM.
AIV (odd blocks) → vector_task_indexs → SwiGLU/AllToAll.
Event synchronization: WaitForDependency / TriggerEvent.
dispatch ──► up_proj (GMM1) ──► swiglu ──► down_proj (GMM2) ──► combine
| Operator | Type | Core | Description |
|---|---|---|---|
| dispatch | AllToAll | AIV | Route tokens to ranks by expert |
| up_proj | GroupedMatmul | AIC | [tokens, hidden] × [E, hidden, intermediate*2] |
| swiglu | SwiGLU | AIV | silu(gate) * up, producing [tokens, intermediate] |
| down_proj | GroupedMatmul | AIC | [tokens, intermediate] × [E, intermediate, hidden] |
| combine | AllToAll | AIV | Gather expert outputs back to their original ranks |
| Operator | split_value | Core | task_num calculation |
|---|---|---|---|
| dispatch | 128 | AIV | all_expert_num × (per_expert_seq_to_other // 128) |
| up_proj | 4096 | AIC | num_cube_cores × single_rank_expert_num |
| swiglu | 128 | AIV | (per_expert_seq // 128) × single_rank_expert_num |
| down_proj | 4096 | AIC | num_cube_cores × single_rank_expert_num |
| combine | 128 | AIV | all_expert_num × (per_expert_seq_to_other // 128) |
| File | Parameter position | Description |
|---|---|---|
up_proj_tiling.bin |
pos 17 | GMM1 tiling; size = num_cube_cores × tiling_entry_size |
swiglu_tiling.bin |
pos 18 | SwiGLU tiling; size = (2×num_cube_cores+1) × entry_size |
down_proj_tiling.bin |
pos 19 | GMM2 tiling |
See §6.1 for generation.
dispatch ──► act_grad (GMM1) ──────────────────────────────────────────────────────────┐
└── w2_grad (GMM4) │
▼
swiglu_grad ──► gate_grad (GMM2) ──► combine
└── w1_grad (GMM3)
| Operator | Mapping | Inputs | Outputs |
|---|---|---|---|
| dispatch | AllToAll | dy (input gradient) |
dispatch_target (dispatched gradient) |
| act_grad (GMM1) | dispatch_target @ w2.T |
dispatch_target, w2 |
act_grad_y (activation gradient) |
| w2_grad (GMM4) | hidden.T @ dispatch_target |
hidden (forward SwiGLU output), dispatch_target |
hidden_dw (W2 gradient) |
| swiglu_grad | SwiGLU backward | act_grad_y, gate (forward up_proj_y) |
grad_gate (SwiGLU gradient) |
| gate_grad (GMM2) | grad_gate @ w1.T |
grad_gate, w1 |
gate_dx (pre-combine gradient) |
| w1_grad (GMM3) | permute_out.T @ grad_gate |
permute_out (forward dispatch_target), grad_gate |
gate_dw (W1 gradient) |
| combine | AllToAll | gate_dx |
grad_x (combined gradient) |
-
act_grad (GMM1) and w2_grad (GMM4) overlap: Both depend on dispatch output
target. GMM1 usesadvance_mode="cube_only"to advance task counts without events; GMM4 usesadvance_mode="cube_custom"to advance events centrally and avoid double counting. -
combine and w1_grad (GMM3) overlap: Both depend on gate_grad (GMM2). Combine uses
advance_mode="vector_only"; GMM3 usescube_customfor events.
| File | Parameter name | Description |
|---|---|---|
act_grad_tiling.bin |
act_grad_tiling |
GMM1 backward tiling (dispatch_target @ W2.T) |
gate_grad_tiling.bin |
gate_grad_tiling |
GMM2 backward tiling (grad_gate @ W1.T) |
w1_grad_tiling.bin |
w1_grad_tiling |
GMM3 W1-gradient tiling |
w2_grad_tiling.bin |
w2_grad_tiling |
GMM4 W2-gradient tiling |
swiglu_grad_tiling.bin |
swiglu_grad_tiling |
SwiGLU backward tiling |
See §6.2 for generation.
TaskSplitValue encapsulates hardware topology parameters and runtime counters:
from hyper_parallel.core.multicore.scheduler.config import TaskSplitValue
tsv = TaskSplitValue(
tp=4, # Tensor parallel degree.
ep=4, # Expert parallel degree, equal to the NPU count.
seq_size=8192, # Global sequence length.
all_expert_num=32, # Total expert count.
top_k=8, # Top-K routing count.
)Key derived read-only properties:
| Property | Calculation | Meaning |
|---|---|---|
single_rank_expert_num |
all_expert_num // ep |
Experts owned by each rank |
seq_all |
seq_size × ep × top_k // tp |
Total tokens after TP partitioning and Top-K expansion |
per_rank_seq |
seq_all // ep |
Tokens processed by each rank |
per_expert_seq |
seq_all // top_k |
Tokens in one Top-K routing slice |
per_expert_seq_to_other |
seq_all // (ep × top_k) |
Tokens per destination rank and Top-K slice |
all_event_num |
Computed automatically | Total global events |
ComputeGraph.propagate_splits(tsv) computes each operator's task_num in topological order:
1. Reset all TensorSpec.split_dim = -1 and split_num = 1.
2. For each operator in topological order:
a. If split_inputs is None (source operator), task_num = task_num_fn(tsv).
b. Otherwise, if every (input_idx, dim) satisfies op.inputs[input_idx].split_dim == dim:
→ task_num = task_num_fn(tsv) [split]
Otherwise task_num = 1 [unsplit]
c. Mark output tensors: out.split_dim = split_output_dims[i] when task_num > 1.
Shared TensorSpec: Producers write split_dim to a shared TensorSpec object and consumers read it directly, propagating split information through the graph automatically.
Forward propagation example:
dispatch (source) → target.split_dim = 0
up_proj [(0,0)] → target.split_dim == 0 ✓ → gmm1_y.split_dim = 0
swiglu [(0,0)] → gmm1_y.split_dim == 0 ✓ → swiglu_out.split_dim = 0
down_proj [(0,0)] → swiglu_out.split_dim == 0 ✓ → gmm2_y.split_dim = 0
combine [(1,0)] → gmm2_y.split_dim == 0 ✓
split_inputs must select inputs with upstream producers inside the graph, not external leaf tensors. Leaf split_dim resets to -1 on every propagate_splits call, so using a leaf as the trigger leaves task_num permanently at 1.
# Correct: inputs[1]=target is produced by dispatch and connected within the graph.
w2_grad: split_inputs=[(1, 0)] # inputs[1] = target
# Incorrect: inputs[0]=hidden is an external leaf whose split_dim remains -1.
w2_grad: split_inputs=[(0, 0)] # task_num always equals 1.Multicore MoE-FFN synchronizes cores through event semaphores:
AllToAll dispatch: Per-expert triggering.
trigger_event = pre_event_num + (task_index // expert_group_size) + 1
Each expert group's completed dispatch task triggers an event. GMM consumes at expert granularity without waiting for all tokens to be sent.
AllToAll combine: Global triggering.
trigger_event = all_event_num (one globally unique event)
Inter-GMM event coordination in backward:
| Operator | advance_mode | Description |
|---|---|---|
| act_grad (GMM1) | cube_only |
Advance task count only, without advancing events |
| w2_grad (GMM4) | cube_custom |
Advance events centrally, accounting for GMM1 |
| combine | vector_only |
Advance task count only |
| w1_grad (GMM3) | cube_custom |
Advance events centrally |
| Operator type | Recommended split_dim | task_num_fn | Notes |
|---|---|---|---|
| GroupedMatMul (GMM) | 0 (M dimension, expert parallelism) | expert_num × (seq_per_expert // TILE_M) |
TILE_M must align with GMM block_size |
| AllToAll (dispatch) | 0 (token dimension) | total_tokens // token_tile |
Per-expert events for dispatch; global events for combine |
| AllToAll (combine) | 0 | total_tokens // token_tile |
Depends on all down_proj tiles completing: N:1 fan-in |
| SwiGLU / Add | 0 | Upstream GMM task_num (1:1) | Must match upstream task_num to avoid event misalignment |
| Rule | Description |
|---|---|
| Monotonicity | Event counters only increase, preventing deadlock |
| 1:1 dependency | Each completed producer tile triggers the corresponding consumer tile |
| N:1 fan-in | Multiple upstream tiles trigger one downstream tile; set trigger_count = N |
| 1:N fan-out | One upstream triggers multiple downstream tiles, each with a separate event ID |
| Nonoverlapping IDs | Keep dispatch, combine, and GMM event ID ranges strictly separate |
RuntimeConfig is the binary Multicore MoE-FFN scheduling configuration, containing each task's dependency events, trigger events, and tensor address offsets. It is generated independently per rank.
The header's first five uint32 fields are task_num, num_workers, task_capacity, event_capacity, and ready_event.
A ready_event of 0 means no cross-rank readiness is required. The next three uint32 fields are diagnostic_mode,
aic_profile_record_capacity, and aiv_profile_record_capacity, followed by completion_event and protocol_version,
The final 24 bytes provide alignment. diagnostic_mode=0 means ordinary execution, =1 profiling, and =2 L1 dump.
protocol_version=0 means push and =1 means pull; pull's completion_event
waits for all participating workers, followed by cross-rank confirmation that source storage is safe to reuse.
Task capacity covers compute tasks, terminate tasks, queue length, and the largest referenced task ID, aligned to 16, without a fixed task-count ceiling.
Event capacity is at least 1024 and covers unfused graphs, ready/completion events, and atomic-write tails. Layout size is
128 + 588 × task_capacity + 20 × event_capacity + group_list_bytes bytes and must be addressable with uint32 byte offsets.
Online plans and offline tools share serialize_runtime_config(). Device code validates protocol versions, storage, and index bounds.
The ordinary event region occupies event_capacity × 4 bytes; when EP > 1, it is followed by 2 × (EP + 1) × 64 bytes of persistent ready/completion state.
Each call clears only ordinary events. Persistent state is initialized once on first use, followed by a host barrier.
Grouped-matmul scratch for each AIC is 128-byte aligned, with per-core stride ceil(local_experts × 8 / 128) × 128 bytes.
group_list_bytes = max(4096, 24 × per_core_stride + 128) includes base-alignment slack, preventing full-cache-line writeback from overwriting neighboring cores.
There is no longer a fixed limit of 16 experts per rank. Host allocation, serialization, and device offsets use actual expert counts.
Up to 16 experts retain the original layout size; larger counts expand scratch as needed, subject to uint32 offsets and available memory.
# Run from the hyper-parallel repository root, containing the hyper_parallel/ package.
# Direct python gen_runtime_data.py fails with ImportError because of relative imports.
python -m hyper_parallel.core.multicore.modules.mega_moe.forward.gen_runtime_data \
--tp 4 --ep 4 \
--seq_size 8192 \
--all_expert_num 32 \
--top_k 8 \
--hidden_size 7168 \
--intermediate_size 2048 \
--num_cube_cores 24 \
--output_dir /path/to/fwd_dataOutput files:
<output_dir>/
├── up_proj_tiling.bin # GMM1 tiling, shared across ranks.
├── swiglu_tiling.bin # SwiGLU tiling, shared across ranks.
├── down_proj_tiling.bin # GMM2 tiling, shared across ranks.
├── all_event_counters.bin # Ordinary event counters and persistent ready state; requires symmetric memory.
├── gmm_workspace.bin # 256 MiB of zeros, shared across ranks.
├── runtime_config_input_rank_0.bin # Rank 0 scheduling configuration.
├── runtime_config_input_rank_1.bin
└── ...
You can also invoke it directly from Python:
from hyper_parallel.core.multicore.modules.mega_moe.forward.gen_runtime_data import build_config_for_rank
from hyper_parallel.core.multicore.modules.mega_moe.forward.graph import build_forward_graph
from hyper_parallel.core.multicore.scheduler.config import TaskSplitValue
from hyper_parallel.core.multicore.scheduler.runtime import serialize_runtime_config
tsv = TaskSplitValue(tp=4, ep=4, seq_size=8192, all_expert_num=32, top_k=8)
graph = build_forward_graph(tsv, dispatch_sv=128, up_proj_sv=4096,
swiglu_sv=128, down_proj_sv=4096, combine_sv=128,
hidden_size=7168, intermediate_size=2048)
graph.propagate_splits(tsv)
cfg = build_config_for_rank(graph, tsv, rank_id=0) # Returns RuntimeConfigC.
data = serialize_runtime_config(cfg)python -m hyper_parallel.core.multicore.modules.mega_moe.backward.gen_runtime_data \
--tp 4 --ep 4 \
--seq_size 8192 \
--all_expert_num 32 \
--top_k 8 \
--hidden_size 7168 \
--intermediate_size 2048 \
--num_cube_cores 24 \
--output_dir /path/to/bwd_dataOutput files:
<output_dir>/
├── act_grad_tiling.bin # GMM1 backward tiling, shared across ranks.
├── gate_grad_tiling.bin # GMM2 backward tiling.
├── w1_grad_tiling.bin # GMM3 W1-gradient tiling.
├── w2_grad_tiling.bin # GMM4 W2-gradient tiling.
├── swiglu_grad_tiling.bin # SwiGLU backward tiling.
├── all_event_counters.bin # 4096 uint8 zeros, 4 KB, shared across ranks.
├── gmm_workspace.bin
└── runtime_config_input_rank_<i>.bin
The Multicore entry point prepares dependencies, private SHMEM, per-SoC vendors, vendor merging, and the Torch adapter. The repository root entry point only invokes component scripts and packages successful artifacts into the complete wheel. Build commands, environment requirements, caches, and payload paths are documented in Build and Delivery. CANN custom OPP activation for source and installed-wheel environments is documented in the HyperParallel Installation Guide.
Each SoC's CANN build generates both HyperMegaMoe and HyperMegaMoeGrad. The component validates new and cached vendors,
then revalidates the four aclnnHyperMegaMoe* APIs, kernel objects/metadata, SONAMEs, and runtime dependencies after merging.
Private SHMEM libraries are checked for dedicated SONAMEs. All shipped host shared libraries are checked for ELF architecture, secure linking properties, absence of
RPATH/RUNPATH, and dependency boundaries. set_env.bash configures library search paths before Python starts.
The Torch adapter is a CMake-built dispatcher shared library loaded by torch.ops.load_library(). It has no
PyInit_* entry point and links neither libpython nor libtorch_python. At runtime it locates its bundled vendor relative to the component and loads it by absolute path.
set_env.bash configures vendor OPP and library search paths according to the CANN custom OPP contract.
SHMEM's _lifecycle.py manages native initialization, bootstrap, references, and caller-plan-driven runtime restart.
MegaMoe's heap_manager.py manages capacity, workspace layout, heap budgets, and growth diagnostics; spec.py supplies capacity formulas,
workspace.py manages physical buffers and graph leases, and execution-resource objects in module.py handle binding and closing.
The SHMEM lifecycle does not read expert capacity or workspace layouts. See the
SHMEM documentation for configuration and restart rules.
The component README owns usage, input constraints, capacity, and lifecycle documentation. This section records only
the forward/backward native ABI invoked internally by MegaMoeExperts.
multicore_ops.mega_moe(
# AllToAll dispatch
dispatch_target, # [tokens, hidden] bf16 AllToAll destination buffer; in-place output.
dispatch_target_off, # [all_expert_num] int64 Per-expert remote write offset on target PE.
dispatch_src, # [tokens, hidden] bf16 Send source data.
dispatch_src_off, # [all_expert_num] int64 Per-expert local read offset.
dispatch_size, # [all_expert_num] int32 Elements sent per expert.
# GMM1(up_proj)
up_proj_weight, # [E, hidden, intermediate*2] bf16 W1 weights.
up_proj_glist, # [E] int64 Per-expert cumulative token counts.
up_proj_y, # [tokens, intermediate*2] bf16 GMM1 in-place output.
swiglu_out, # [tokens, intermediate] bf16 SwiGLU in-place output.
# GMM2(down_proj)
down_proj_weight, # [E, intermediate, hidden] bf16 W2 weights.
down_proj_glist, # [E] int64 Per-expert cumulative token counts.
down_proj_y, # [tokens, hidden] bf16 GMM2 in-place output.
# AllToAll combine
combine_target, # [tokens, hidden] bf16 AllToAll combine destination; in-place output.
combine_target_off, # [all_expert_num] int64 Per-expert remote write offset.
combine_src_off, # [all_expert_num] int64 Per-expert local read offset.
combine_size, # [all_expert_num] int32 Elements sent per expert.
# Configuration tensors.
gmm_workspace, # [256*1024*1024] uint8 GMM workspace, 256 MiB.
up_proj_tiling, # uint8 GMM1 tiling,from gen_runtime_data.py
swiglu_tiling, # uint8 SwiGLU tiling
down_proj_tiling, # uint8 GMM2 tiling
runtime_config, # uint8 Per-rank schedule from gen_runtime_data.py.
all_event_counters, # uint8 Event synchronization counters in symmetric memory.
# Scalars.
rank_id, # int Current rank.
ep, # int Expert parallel degree.
expert_num, # int Global expert count (all_expert_num).
hidden_size, # int Hidden dimension.
seq_size, # int Global sequence length.
)Returns: Nothing; outputs are written into caller-provided Torch tensors.
multicore_ops.mega_moe_grad(
# AllToAll dispatch for gradients.
dispatch_target, # [tokens, hidden] bf16 AllToAll destination; in-place output.
dispatch_target_off, # [all_expert_num] int64 Per-expert remote write offset.
dy, # [tokens, hidden] bf16 Input gradient, AllToAll send source.
dispatch_src_off, # [all_expert_num] int64 Per-expert local read offset.
dispatch_size, # [all_expert_num] int32 Elements sent per expert.
# act_grad: GMM1 backward, dispatch_target @ W2.T.
hidden, # [tokens, intermediate] bf16 Cached forward SwiGLU output.
hidden_dw, # [E, intermediate, hidden] bf16 W2 gradient; in-place output.
w2, # [E, intermediate, hidden] bf16 W2 weights, forward down_proj_weight.
act_grad_y, # [tokens, intermediate] bf16 GMM1 backward in-place output.
# swiglu_grad
gate, # [tokens, intermediate*2] bf16 Forward up_proj_y, cached SwiGLU input.
grad_gate, # [tokens, intermediate*2] bf16 SwiGLU gradient in-place output.
# gate_grad: GMM2 backward, grad_gate @ W1.T.
w1, # [E, hidden, intermediate*2] bf16 W1 weights, forward up_proj_weight.
gate_dx, # [tokens, hidden] bf16 GMM2 backward in-place output.
grad_x, # [tokens, hidden] bf16 AllToAll combine in-place output.
# AllToAll combine for gradients.
combine_target_off, # [all_expert_num] int64 Per-expert remote write offset.
combine_src_off, # [all_expert_num] int64 Per-expert local read offset.
combine_size, # [all_expert_num] int32 Elements sent per expert.
# w1_grad(GMM3)、w2_grad(GMM4)
permute_out, # [tokens, hidden] bf16 In-place intermediate buffer for W1 gradients.
gate_dw, # [E, hidden, intermediate*2] bf16 W1 gradient; in-place output.
group_list, # [E] int64 Per-expert cumulative token counts.
# Configuration tensors.
act_grad_tiling, # uint8 GMM1 backward tiling.
gate_grad_tiling, # uint8 GMM2 backward tiling.
w1_grad_tiling, # uint8 GMM3 W1-gradient tiling.
w2_grad_tiling, # uint8 GMM4 W2-gradient tiling.
swiglu_grad_tiling, # uint8 SwiGLU backward tiling.
gmm_workspace, # [256*1024*1024] uint8 GMM workspace.
swiglu_grad_workspace, # [64*1024*1024] uint8 SwiGLU backward workspace.
runtime_config, # uint8 Per-rank scheduling configuration.
all_event_counters, # uint8 Event synchronization counters in symmetric memory.
# Scalars.
rank_id, ep, expert_num, hidden_size, seq_size,
)Step 1 — Extend OpType in scheduler/graph.py:
class OpType(Enum):
ALLTOALL = "alltoall"
GMM = "gmm"
SWIGLU = "swiglu"
SWIGLU_GRAD = "swiglu_grad"
RMSNORM = "rmsnorm" # New entry.Step 2 — Add matching TaskType entries in scheduler/config.py and C++ runtime_config.hpp, then
implement FillConfig in tasks/rmsnorm.py. Use SwiGLUFillConfig as the Vector task reference:
- Construct task and input/output descriptions with
TaskDescCandTensorDescC. - Write tasks into
cfg.all_tasksand indices into the correspondingcube_task_indices,vector_task_indices, ormix_task_indices. - Set
dependent_event,trigger_event, andall_event_num_triggers, then updatecfg.task_index_num. - Advance task and event counts through the
advance_tsv_*helpers intasks/utils.py.
Step 3 — Register in tasks/registry.py:
from .rmsnorm import RMSNormFillConfig
FILL_CONFIG_REGISTRY[OpType.RMSNORM] = RMSNormFillConfigStep 4 — Add a switch branch in C++ ops/<op>/op_kernel/worker_kernel.cpp:
case TaskType::TASK_RMSNORM:
ExecuteRMSNorm(task_desc);
break;Step 1 — Create the directory:
modules/
└── attention/
├── forward/
│ ├── graph.py
│ ├── gen_runtime_data.py
│ └── tiling_tables.py
└── backward/
└── ...
Step 2 — Build the DAG in graph.py:
from hyper_parallel.core.multicore.scheduler.graph import (
ComputeGraph, OperatorNode, TensorSpec, SplitSpec, OpType
)
from hyper_parallel.core.multicore.scheduler.config import TaskSplitValue
def build_attention_fwd_graph(tsv: TaskSplitValue, ...) -> ComputeGraph:
g = ComputeGraph()
x = TensorSpec("x", shape=[tsv.seq_size, hidden], param_position=0)
qkv_out = TensorSpec("qkv_out", shape=[tsv.seq_size, 3*hidden], param_position=1,
is_dynamic=True)
qkv_proj = OperatorNode("qkv_proj", op_type=OpType.GMM,
inputs=[x], outputs=[qkv_out],
split_spec=SplitSpec(
split_inputs=None,
task_num_fn=lambda tsv: tsv.seq_size // TILE_M,
split_output_dims=[0]))
g.add_op(qkv_proj)
return gStep 3 — Implement the CLI, following modules/mega_moe/forward/gen_runtime_data.py:
def main():
args = parse_args()
tsv = TaskSplitValue(tp=args.tp, ep=args.ep, ...)
graph = build_attention_fwd_graph(tsv)
graph.propagate_splits(tsv)
for rank_id in range(args.tp * args.ep):
cfg = build_config_for_rank(graph, tsv, rank_id)
write_bin(cfg, f"{args.output_dir}/runtime_config_rank_{rank_id}.bin")Step 1 — Create the operator directory:
ops/
└── attn_fwd/
├── op_host/
│ ├── attn_fwd_def.cpp
│ ├── attn_fwd_infershape.cpp
│ └── attn_fwd_tiling.cpp/h
└── op_kernel/
├── attn_fwd.cpp
├── tiling_key.h
├── worker_kernel.cpp
└── put_mem_signal/
└── put_mem_signal_kernel.cpp
Step 2 — Include the shared scheduler in attn_fwd.cpp:
#include "kernel_operator.h"
#include "worker_kernel.cpp"
#include "tiling_key.h"
__global__ __aicore__ void multicore_attn_fwd(...) {
GM_ADDR input_list[] = { ..., runtime_config, ..., tiling, all_event_counters };
uint32_t idx = GetBlockIdx();
worker_kernel(idx, runtime_config, input_list);
}Step 3 — Include the shared runtime in worker_kernel.cpp:
#include "kernel_operator.h"
#include "runtime/worker_kernel.h" // The build copies ops/runtime/ into op_kernel/runtime/.
class KernelWorker : public KernelWorkerBase<KernelWorker> {
public:
// input_list indices follow this operator's parameter layout.
static constexpr uint32_t TILING_IDX = N; // Replace with the actual index.
static constexpr uint32_t EVENT_IDX = M;
__aicore__ inline void ExecuteComputeKernel(TaskDesc task_desc) {
switch (task_desc.task_type) {
// Add this operator's cases.
}
}
};Step 4 — Instantiate put_mem_signal/put_mem_signal_kernel.cpp with the operator's dtype:
#include "kernel_operator.h"
#include "shmem.h"
template <typename T>
class PutMemSignalKernel { /* ... */ }; // Same as the existing implementation.
extern "C" inline __aicore__ void put_mem_signal_kernel(...) {
PutMemSignalKernel<DTYPE_YOUR_TYPE> op; // Replace with the operator's dtype macro.
op.Init(...);
op.Process();
}Step 5 — Register the operator in the source assembler so source assembly copies the shared
runtime/:
_HYPER_OPERATORS = ("hyper_mega_moe", "hyper_mega_moe_grad", "hyper_attn_fwd")| Symptom | Possible cause | Investigation |
|---|---|---|
TaskDesc event_id out of bounds |
Incorrect all_event_num calculation | Check the all_event_num property formula in scheduler/config.py |
| Different task_num in two bin files | Incorrect SplitSpec task_num_fn | Print each operator's task_num after propagate_splits() |
| AIV hang (infinite wait) | N:1 fan-in trigger_count is too large | Verify FillConfig trigger_count equals the actual upstream tile count |
| GMM workspace out of bounds | TILE_M/N too large | Check workspace budgeting in tiling_tables.py |
Missing runtime/worker_kernel.h |
New operator absent from isolated source assembly | Check the operator list in assemble_multicore_source.py and MULTICORE_SOURCE_ASSEMBLY_FAILED logs |
Python ImportError: scheduler |
Running outside the package root | Run with python -m from the hyper-parallel/ root |