Skip to content

Latest commit

 

History

History
250 lines (195 loc) · 12.6 KB

README.md

File metadata and controls

250 lines (195 loc) · 12.6 KB

High-performance functions

hyper_parallel.components.functional provides reusable high-performance functions for Ascend NPU. Some interfaces also expose an explicit eager reference backend; fused calls do not automatically fall back.

from hyper_parallel.components.functional import rms_norm

output = rms_norm(x, weight)

RoPE functions keep the Transformers call contract:

from hyper_parallel.components.functional import apply_rotary_pos_emb

query, key = apply_rotary_pos_emb(query, key, cos, sin)

Use apply_rotary_pos_emb_interleave for Transformers models whose source attention calls the interleaved variant.

MoE functions

The reusable MoE interfaces include:

  • grouped_matmul: NPU grouped matrix multiplication with backward support.
  • moe_token_permute: reorder routed tokens into expert-major order.
  • moe_token_unpermute: restore expert outputs to token-major order.
  • swiglu: fused NPU SwiGLU used by grouped experts.
from hyper_parallel.components.functional import (
    moe_token_permute,
    moe_token_unpermute,
)

permuted_tokens, sorted_indices = moe_token_permute(tokens, expert_indices)
expert_outputs = experts(permuted_tokens, tokens_per_expert, routing_weights)
output = moe_token_unpermute(expert_outputs, sorted_indices, routing_probs)

In this example, experts is an instance of hyper_parallel.components.modules.GroupedExperts. It contains only local expert weights and computation; routing, expert-parallel communication, and shared experts remain outside the module. hyper_parallel.components.modules.SharedExpert provides the parameter-owning dense MLP used as a shared expert.

Auxiliary-loss functions

hyper_parallel.components.functional also exports aux_loss_auto_scale and set_aux_loss_scale for auxiliary-loss gradient injection.

DeepSeek V3.2 DSA

The DSA interfaces are dsa_indexer, dsa_sparse_attention, dsa_kl_loss, and dsa_dense. The CANN backend targets CANN 9.2.0 and requires its matching cann_ops_transformer extension and native operators. Sparse/Dense KL use the fused torch_npu.npu_{sparse,dense}_lightning_indexer_grad_kl_loss APIs; teacher probabilities and scalar loss are computed inside CANN. Dense indexer LSE returns the native max/sum pair. Indexer and sparse-attention backward use cann_ops_transformer; attention forward uses torch_npu.npu_sparse_flash_attention.

DeepSeek V4.1 fused operators

The V4.1 adapters use the optional cann_ops_transformer PyTorch extension:

Component Required entry points
Indexer lightning_indexer (LI V2), lightning_indexer_metadata when required by the device
Attention sparse_flash_mla, sparse_flash_mla_grad, sparse_flash_mla_metadata
Indexer KL sparse_lightning_indexer_kl_loss_grad, sparse_lightning_indexer_kl_loss_grad_metadata

Installation

Follow the official ops-transformer guides:

  1. Build and install the native operator package.
  2. Build and install the PyTorch extension.

Both the native kernels and the Python extension are required. Load the CANN and custom-operator environment as described in the guides before training. HyperParallel imports the standard cann_ops_transformer package; use the standard wheel build (bash build.sh --torch_extension), without --ops or --vendor_name, which can change the Python package name at the tested revision.

Tested versions

Dependency Tested configuration
CANN Toolkit 9.0.0
Python / PyTorch / torch-npu 3.11 / 2.10.0 / 2.10.0
ops-transformer source 0b08075141913daeaf1e856faf7480ed393c8949
Python extension cann_ops_transformer 1.0.0, built from the source revision above

The source revision identifies the tested operator interfaces; the wheel version alone does not identify them.

Configuration

Enable the three adapters together through the V4.1 replacement:

plan_overrides:
  - match: "model.layers.*.self_attn"
    replace:
      kind: module
      source_type: hyper_parallel.models.deepseek_v41.modeling_deepseek_v41.DeepseekV41Attention
      implementation:
        _target_: >-
          hyper_parallel.components.modules.shared_compressed_dsa_attention.SharedCompressedDSAAttention
        use_fused_ops: true

false (default) uses the Torch reference implementation without this extension. true requires the fused entry points used by the model's execution path and raises on missing entries or unsupported inputs, without automatic fallback. The fused path supports ordinary causal CSA with compression ratio 1 or 2; candidate-pool selection is not supported.

SiTU and block attention residuals

These shared functions support backend="eager" (default, Torch reference with FP32 intermediates) and backend="ascendc" (standalone CANN operators). SiTU uses ops-nn; block attention residuals use ops-transformer.

from hyper_parallel.components.functional import situ_packed, block_attention_residuals

activated = situ_packed(packed_gate_up, beta=4.0, linear_beta=25.0, backend="ascendc")
residual = block_attention_residuals(
    prefix, history, query_weight, rms_weight, norm_eps=1e-5, backend="ascendc",
)

situ(gate, up, ...) accepts separate projections of the same shape, dtype and device; situ_packed accepts [..., 2H] with gate channels followed by up channels. Generic SiTU defaults are beta=1 and linear_beta=None; K3 supplies 4 and 25. linear_beta=None or linear_beta <= 0 disables saturation of the up projection in both backends, following the ops-nn contract.

Block attention residuals take prefix: [T,H], history: [T,N,H], projection: [H] or [1,H], and norm_weight: [H]. The model supplies norm_eps, learned parameters and block/history scheduling. The wrapper promotes mixed input dtypes differentiably and reshapes [H] queries to [1,H] for native backward. Both fused functions support first-order gradients; empty inputs/history retain connected gradients through tensor operations without launching a native kernel.

Eager reference math

The eager functions implement the SiTU and block attention-residual formulas in Torch. Their model reference is Moonshot AI's published Kimi K3 text implementation: SituAndMul.forward for SiTU and _apply_attn_res for block attention residuals. The public functions accept tensors directly instead of model modules.

SiTU computes beta * tanh(gate / beta) * sigmoid(gate) * up, with optional up saturation. The nonpositive linear_beta convention follows ops-nn; it extends the model reference's None convention.

Block attention residuals normalize residuals for scoring, then mix the original residual values. The local implementation uses elementwise multiplication and reduction for the weighted sum; the model reference uses a batched matrix multiply. They implement the same formula, but floating-point reduction order can differ. Block boundaries and history updates remain the calling model's responsibility.

CANN dependencies and errors

Install both the native operators and the matching standalone Python extension:

Function Python package and required callable exports Native build selection (--ops)
SiTU cann_ops_nn: situ_glu, situ_glu_grad situ_glu,situ_glu_grad
Block attention residuals cann_ops_transformer: block_attention_residuals, block_attention_residuals_backward block_attention_residuals,block_attention_residuals_grad

Follow the installation instructions in ops-nn and ops-transformer, building for the target CANN version and SoC. Load CANN and both custom-operator environments before training. Keep the canonical Python package names above. The revisions below compile the PyTorch/C++ bridge on first use, so a compiler, Ninja and CANN headers are required; use a separate TORCH_EXTENSIONS_DIR when changing toolchains.

The wrappers load these optional packages only when a nonempty fused path is selected and check that both training entry points are callable. Import or dynamic-library loading failures raise an informative ImportError with the original cause preserved.

  • Missing Python package or API: install the matching extension with both forward and backward.
  • Missing libraries or undefined symbols: check PyTorch/torch-npu/CANN compatibility, ASCEND_CUSTOM_OPP_PATH and LD_LIBRARY_PATH for both native packages.
  • Native input, compilation and execution errors propagate unchanged. The dispatcher and upstream entry points validate their device, shape and dtype contracts; the wrapper does not duplicate those checks or silently select eager.

Evaluated versions

The standalone operators were built with CANN 9.1.0-beta.3 and PyTorch/torch-npu 2.10.0 at these source revisions:

Upstream Source revision
ops-nn b7af75e68cc07c0912f9e020bce639e53de60700
ops-transformer a041b05638a5a3f94b79c97ce196a331738a56a1

Linear attention backends

backend="ascendc" for local KDA/GDN and pure Ulysses requires fla-npu, installed with matching Python bindings, stable launcher and native operator libraries. KDA and GDN use the same package and public fla_npu.ops.ascendc namespace. The shared qualification revision is unmodified source commit 131fd3e937bd7607e372cf80c1022a80ed5fb853. A filtered build must include both model families and their required operators; a KDA-only wheel does not qualify GDN, even if the Python exports exist. Do not combine bindings or operator libraries from different revisions. Install the wheel appropriate to the target hardware and CANN environment using the upstream installation guide.

KDA local, Ulysses and state CP share chunk_kda_fwd_prepare save mode, chunk_fwd_h, chunk_kda_fwd_finalize, chunk_kda_bwd_prepare, chunk_gated_delta_rule_bwd_dhu and chunk_kda_bwd_finalize. The adapter owns raw-input gradients and converts sequence-major inputs to the head-major native stage layouts. Beta sigmoid backward uses Torch's aten.sigmoid_backward with the FP32 beta_eff already saved by Prepare. The native finalize sigmoid-backward flag stays disabled on every device; the adapter does not inspect device names or require an FLA beta helper. Raw beta logits are not retained for backward. Native operators validate their own hardware support; this common adapter does not imply every device has completed numerical or performance qualification.

GDN requires chunk_gated_delta_rule_fwd and chunk_gated_delta_rule_bwd with layout, cu_seqlens and chunk_indices. Normalization and effective gate/beta remain in the caller's autograd graph. Both wrappers use per-invocation save mode and reject unsupported external state-gradient requests. KDA layer CP and GDN pure Ulysses use the same backend for causal convolution; Ascend C additionally requires causal_conv1d and causal_conv1d_bwd with the SiLU derivative included.

KDA state P2P, AllGather, grouped AllGather/P2P and recursive doubling additionally require native CP summary/merge APIs. RD shares the existing transport and reset planner while scanning native S/M forward and native G/N in reverse, without the Triton forward-transfer cache. The validated native base is the same shared revision above, built for the target device. This is a main-source build, not a claim that the v26.9.1 release wheel exports the required CP/staged backward interfaces. The adapter converts backward-summary gk and local dHT to model dtype as required by its public ABI. Communication and summary apply remain FP32. These conversions introduce rounding relative to FP32-input extensions; no local native patches are required. Older releases without the independent stages or CP summary exports are not compatible with this path. These protocols share the same local KDA stages and can be combined with Ulysses. Forward exchanges use S/M, while backward produces fresh G/N instead of caching M. See KDA context parallelism for configurations, supported shapes and the state-buffer lifetime contract.