English | 简体中文
Release date: 2026-06-30
HyperParallel v1.0.0 is the project's first official release, marking the transition from rapid iteration to stable delivery. This release delivers the complete feature set of a distributed parallel acceleration library optimized for Ascend SuperPoDs, spanning cluster-level distributed parallelism and on-chip multicore parallelism, data parallelism and multidimensional hybrid parallelism, and acceleration for both training and inference.
DTensor Distributed Tensors
- DTensor fundamentals: a distributed tensor abstraction encapsulating a local shard, DeviceMesh, and Placements
- DTensor redistribution: on-demand redistribution across mesh dimensions, with cache optimization (
compact_str + rank_id) - DeviceMesh: multidimensional device-topology management;
init_device_meshcreates meshes of arbitrary shapes - Layout: tensor-to-mesh placement mappings for declaratively defining parallel strategies
manual_seed: distributed random-seed control with PyTorch parityinit_parameters/init_empty_weights/init_on_device: sharded-parameter initialization utilities
HSDP/FSDP Data Parallelism
fully_shard: fully shard parameters, gradients, and optimizer states to reduce per-device memory usageHSDPModule: hybrid sharded data-parallel module wrapperhsdp_sync_stream: gradient-synchronization stream management- HSDP Overlap: full overlap mode for concurrent communication and computation
set_gradient_scaling_factor: configurable gradient scaling factor- Hybrid-parallel parameter sharding: FSDP streams correctly handle mixed parallel parameters in TP+DP scenarios
- MindSpore parameter-hook migration and fixes
Shard Distributed Operators and Tensor Parallelism
shard_module: declarative parallel-strategy interface usingsharding_planto configure input, output, and parameter placementscustom_shard: integrate custom parallelism into the DTensor parallel workflowDFunction: base class for custom distributed autograd functions, integrated with DTensor dispatchparallelize_value_and_grad: parallelized value and gradient computationSkipDTensorDispatch: bypass DTensor dispatch in gradient hooks to operate directly on local tensors- Distributed operator registration through a YAML registry and Python implementations
- Distributed operators including AllGatherMatmul / MatmulReduceScatter (MC2), RotaryPositionEmbedding, and more
- MindSpore replicate allreduce overlap
Tensor Parallelism (Declarative TP Styles)
ColwiseParallel: column-wise shardingRowwiseParallel: row-wise shardingSequenceParallel: sequence parallelismPrepareModuleInput/PrepareModuleOutput/PrepareModuleInputOutput: module input/output preparation hooksparallelize_module: declarative TP application interface supporting fnmatch glob patterns- Loss Parallel: parallel loss computation for TP training
Expert Parallelism
ExpertParallel: standard all-to-all EP, with each rank holdingnum_experts / ep_degreelocal expertsTensorParallel(within EP): TP-only weight shardingExpertTensorParallel: two-dimensional EP+TP parallelism on a 2-D[ep, tp]mesh- MoE building blocks:
FeedForward,GroupedExperts,TokenChoiceTopKRouter, andMoE - Load balancing:
expert_bias,tokens_per_expert, and auxiliary load-balance loss (MoEAuxLossAutoScaler) - Decoupled MoE+EP token dispatch: token permutation and dispatch separated from forward
- Safe state across PP+EP micro-batches:
DispatchContextprevents instance-sharing issues
Context Parallelism
ContextParallel: basic context parallelismAsyncContextParallel: asynchronous context parallelism- DSA family:
DSAIndexerContextParallel/AsyncDSAIndexerContextParallelDSAIndexerLossContextParallel/AsyncDSAIndexerLossContextParallelDSASparseAttentionContextParallel/AsyncDSASparseAttentionContextParallel
- TP DTensor local rewrapping support
Pipeline Parallelism
PipelineStage: pipeline-stage wrapper supporting dx/dw computationSchedule1F1B: 1F1B schedulingScheduleInterleaved1F1B: interleaved 1F1B scheduling (VPP)ScheduleGPipe: GPipe schedulingMetaStep/MetaStepType/BatchDimSpec: scheduling-unit abstractions- PP+FSDP support:
MetaStepintegrates FSDP metasteps for parameter sharding within PP stages - PP communication/computation overlap (
overlap_b_f):- Composite B/F steps (
OVERLAP_B_F) CommComputeOverlaptwo-thread coordinatorHookCoordinatorCOMM-first rendezvous- Concurrent EP A2A and computation
- PP P2P prefetch:
overlap_p2p=True, batched P2P transport (duplex default under overlap_b_f) - forward-side P2P prefetch via
batch_isend_irecv
- Composite B/F steps (
- PP Activation Swap: activation swapping for pipeline parallelism
- Variable-layer + mixed-recompute under overlap_b_f
- SAPP-PPB: automatic Pipeline Parallelism Balancing module
- Batch-size validation: the MindSpore backend rejects batch sizes not divisible by
micro_batch_num
MoE Multicore Parallelism
- Multicore MPMD parallelism: on-chip multicore parallelism with core-level one-sided communication using memory semantics
- Multicore optimization of MoE communication/computation overlap: fuse AllToAll-Dispatch / GMM1 / SwiGLU / GMM2 / AllToAll-Combine into one kernel, with fine-grained overlap between AIC and AIV cores
- RATR (Rank-Aware Tile Reordering): AllToAll tile reordering generated offline with zero runtime overhead
Activation Checkpoint / Swap
checkpoint/checkpoint_wrapper: selective activation checkpointing (SAC)swap/swap_wrapper/swap_tensor_wrapper: activation swapping (offload to CPU and prefetch during backward)CheckpointPolicy: recomputation policy configurationSwapManager: swap manager- Coordinated configuration of activation swapping and checkpointing
- Swap fusion: fused swap-operation optimization
- Shared-function exclusion: exclude shared functions from callable overlap tracking
- LlamaFactory integration: activation recomputation and swapping support
- MindSpore support for independent recomputation scheduling
- Mpipe VLM multimodal Transpose scheduling
AdamW: standard AdamW optimizerMuon: momentum-based Muon optimizerChainedOptimizer: chained optimizer combining Muon and AdamWget_hyper_optimizer: optimizer factory functionget_hyper_lr_scheduler: learning-rate scheduler factory function- Sharded optimizers: sharded optimizer-state management integrated with FSDP/HSDP
- Gradient scaling factor support
- Enhanced gradient clipping: align
clip_grad_norm_reduction with each gradient's process group
- Distributed checkpoint saving and loading (planner, storage, and resharding)
async_staging: asynchronous staging for checkpoint savingoffline_transform: offline format conversion- Hugging Face format support
- Conversion between sharding strategies
- SAPP-ND: ND search module including memory and performance estimation
- SAPP-PPB: Pipeline Parallelism Balancing for automatically balancing pipeline-stage assignments
Symmetric Memory: one-sided communication with symmetric-memory semanticsAllGather: one-sided AllGather operationAllGatherMatmul/MatmulReduceScatter(MC2): fused communication operators
init_process_group/destroy_process_groupget_process_group_ranks/get_backendsplit_group/get_group_local_rank/mark_created_groups
- DeepSeekV3
- Qwen3 family:
- Qwen3.5-0.8B-Base
- Qwen3.5-35B-A3B-Base
- Qwen3-VL-30B-A3B-Instruct
- LlamaFactory integration: activation recomputation, swapping, and HSDP support
- MindSpore llama3 TP+FSDP demo
- Torch llama3 PP+FSDP+CP demo
- PP+EP 1F1B correctness-verification example
- MoE Expert Parallel correctness demo
LLMTrainer: LLM training frameworkVLTrainer: multimodal vision-language training framework- Callbacks:LoggingCallback, MoeMonitorCallback
parallel_dims: parallel-dimension configuration
Status update: removed after v1.0.0. The
hyper_parallel/platform/abstraction,get_platform(), and the MindSpore backend have been removed from the main branch. The library now uses native PyTorch directly. The following describes the state at the v1.0.0 release and is retained only as historical information.
- Dual-backend support: PyTorch (GPU/NPU) and MindSpore (Ascend NPU)
- Unified abstraction through
get_platform() - Platform-specific implementations:
- FSDP/HSDP: Torch and MindSpore backends
- Pipeline Parallel: Torch and MindSpore backends
- Activation Checkpoint/Swap: Torch and MindSpore backends
- Process Group:Gloo + HCCL
- Wheel packaging: Python/architecture tags; framework adapters follow the corresponding framework's CXX11 ABI contract
- Optimizer module: chained Muon+AdamW optimizer, learning-rate schedulers, and sharded optimizers
- Activation Checkpoint refactoring: new
checkpoint_wrapper/swap_wrapper/swap_tensor_wrapperinterfaces - PP enhancements: FSDP metastep integration, batched P2P transport,
overlap_b_fcommunication/computation overlap, and activation swapping for PP - Context Parallel enhancements: async CP, the DSA family (Indexer/Loss/SparseAttention), and TP DTensor local rewrapping
- New Qwen3-family model support: Qwen3.5/Qwen3.5-MoE/Qwen3-VL-MoE
- MoE+EP enhancements: decoupled token dispatch, load-balance auxiliary loss and expert-bias synchronization, and zero-overhead activation storage
- Distributed Ops:AllGatherMatmul / MatmulReduceScatter (MC2), RotaryPositionEmbedding, Detach, StopGradient
- HSDP Overlap: full overlap mode
- Loss Parallel: parallel loss computation for TP training
- gradient_scaling_factor: configurable gradient scaling factor
- Mpipe: heterogeneous scheduling for multimodal pipeline parallelism
- SAPP-PPB / SAPP-ND: automatic pipeline balancing and ND search
- LlamaFactory integration: activation optimization and HSDP
- Trainer framework: LLMTrainer and VLTrainer
- DTensor: in-place random operators return self; fixed strided-shard
full_tensorconcatenation; parameter casts preserve DTensor identity - FSDP/HSDP: fixed device-mesh construction in the FSDP meta context for
tp>1/dp=1, noncontiguous all-reduce, mixed-parallel parameter sharding, and parameter-hook migration - Shard: fixed replicated-parameter state tracking; preserve DTensor identity for in-place operators when bypassing dispatch; overlap MindSpore replicate all-reduce
- Activation Checkpoint: exclude shared functions from callable overlap tracking; fixed recomputation packing of sharded parameter views
- Pipeline: fixed the default PipelineStage device and added batch-size divisibility validation
- Context Parallel: fixed preservation of TP layouts in async CP
- Optimizer: align
clip_grad_norm_reduction with each gradient's process group - Packaging: Python/architecture wheel tags; framework adapters follow the corresponding framework's CXX11 ABI contract
- Codecheck: added docstrings in 66 files to resolve C0116 warnings
- Activation Checkpoint UT:17.9% → 95% coverage
- Shard UT/ST: increased coverage, restructured tests, and migrated to the Gloo CPU backend (206 cases)
- Context Parallel UT: CP unit-test coverage
- PP+FSDP+TP combined numerical-accuracy tests against a single-device baseline
- Fixed Process Group worker paths and improved world-group teardown safety
- Added Trainer / trainer/utils / core/utils UT and ST
Thank you to the following contributors to v1.0.0, ordered by commit count:
| Contributor | Commits |
|---|---|
| xuxinglei | 26 |
| yide12 | 20 |
| Meng107 | 14 |
| lichen666 | 12 |
| DavidFFFan | 12 |
| david-he91 | 12 |
| ch-l | 10 |
| changzherui1 / changzherui | 7 |
| 宋佳琪 | 5 |
| silkage_jiajia | 5 |
| yao_yf | 4 |
| wangleiit03 | 4 |
| hedongdong | 4 |
| zhuangqinghe | 2 |
| zhangyuguo | 2 |
| yuzhenhua666 | 2 |
| yanxr123 | 2 |
| wang_hua_2025 | 2 |
| rongyue | 2 |
| liuchongming74 / liuchongming | 2 |
| Liang_Ziyi | 2 |
| lei_xxx | 2 |
| jinxiaoxian | 2 |
| guangpengz | 2 |
| Cui-yshoho / celiaccui | 2 |
| zhangbuxue | 1 |
| yuzhenhua | 1 |
| SHUYUAN6 | 1 |
| sargerasking | 1 |
| qhzhuang11111111 | 1 |
| pengshuyuan | 1 |
| niujunhao | 1 |
| liuyanwei / liu-yanwei6 | 1 |
| huilan_li / huilan li | 1 |
| duchengyi | 1 |
| buxue | 1 |
| bj-wang1 / bj-wang | 1 |
| alpha-junh | 1 |
Total: 187 commits, 1040 files changed, 103628 insertions, 23160 deletions
| PR | Description |
|---|---|
| !460 | feat: add activation recompute & swap support for LlamaFactory integration |
| !524 | feat: SAPP-PPB for Pipeline Parallelism Balancing |
| !622 | [feature][pipeline_parallel] pp support fsdp |
| !627 | [feature][fully_shard] support set_gradient_scaling_factor |
| !631 | merge feat-sapp-nd-pr1 into master |
| !635 | full overlap for hsdp |
| !636 | merge test-gloo into master |
| !637 | swap_fusion |
| !639 | fix(shard): skip unsharded replicate params |
| !640 | merge feat/moe-load-balance into master |
| !647 | feat(moe): zero-overhead activation storage via internal score weighting |
| !649 | [Add Test]: add UT and ST for trainer / trainer/utils / core/utils |
| !656 | feat: add distributed ops AllGatherMatmul and MatmulReduceScatter (MC2) |
| !657 | bugfix: add CMake pre-build support for mindspore custom_ops |
| !659 | feat(dtensor): add DTensor manual_seed (PyTorch parity) |
| !660 | feat(context-parallel): add DSA context parallel style |
| !661 | feat: add distributed op RotaryPositionEmbedding with UT and MindSpore ST |
| !663 | feat(context_parallel): add async colossal cp coverage |
| !664 | refactor ms pipeline_stage |
| !665 | test: improve UT coverage for core/shard and custom_ops/experimental |
| !667 | Restructure multicore parallel code |
| !668 | merge bugfix_strided_shard_master into master |
| !669 | feat(gate-doctor): autonomous PR gate doctor skill |
| !670 | feat(examples): MindSpore llama3 TP+FSDP demo and local embedding |
| !671 | feat: add pipeline parallel comm/compute overlap (B/F + lazy a2a) |
| !673 | disable_dispatch |
| !675 | merge master into master |
| !677 | test: add MindSpore PP+EP+overlap PoC test |
| !678 | merge dmodule_protocols_part1 into master |
| !680 | refactor(test): overhaul shard ST test structure and file naming |
| !682 | refactor(moe+ep): decouple token permutation and dispatch from forward |
| !683 | fix: avoid recompute packing sharded param views |
| !684 | [task][fully_shard] improve fully_shard ut cov rate |
| !685 | feat: register Detach and StopGradient distributed ops in element_wise_ops.yaml |
| !687 | feat(moe): add auxiliary load-balance loss, distributed expert_bias synchronization, and sequence_partition_group |
| !689 | fix(dtensor): keep parameter casts as dtensor |
| !690 | multicore megamoe rename w1_grad and w2_grad; add test cases |
| !691 | migrate distributed op |
| !693 | fix: package sapp nd memory estimation |
| !694 | perf(pipeline): use platform.empty for recv buffer allocation |
| !695 | refactor: add dist-op-analysis & dist-op-dev skills; simplify dev workflow docs |
| !697 | refactor: preserve tensor identity in activation swap for keep_graph |
| !699 | merge feat-sapp-nd-pr2 into master |
| !700 | merge pp-swap into master |
| !703 | fix(dtensor): size dim_map by tensor rank in _calc_shard_info |
| !704 | fix: add loss parallel support for tensor parallel training |
| !705 | merge pp-swap into master |
| !706 | feat(examples): add Llama3 PP demos and unify training steps |
| !707 | [fix][fully_shard] adjust prefetch, unshard position |
| !710 | [feature][pipeline_parallel] pipelinestage support dx,dw compute |
| !713 | test(context-parallel): add cp unit coverage |
| !720 | feat(examples): add MoE Expert Parallel correctness demo |
| !723 | fix(shard): overlap MindSpore replicate allreduce |
| !724 | test(torch): add PP+FSDP+TP composite accuracy tests vs single-card baseline |
| !725 | bug_fix |
| !726 | feat: support tp dtensor local rewrap for context parallel |
| !729 | feat(pipeline): overlap_bf_recompute_fixes |
| !730 | merge fix-grad-norm into master |
| !734 | test: add comprehensive UT for core/activation_checkpoint (17.9% → 95% coverage) |
| !737 | feat(example): add PP(1F1B)+FSDP2+CP demo for Llama3 on Torch backend |
| !738 | chore: make agent tooling assistant-agnostic for any AI coding assistant |
| !740 | feat(moe): add PP+EP 1F1B correctness verification example |
| !741 | ms_support_recompute_independent_schedule |
| !744 | fix fsdp hook clear |
| !749 | merge fix/ms-fsdp-param-hook-migration into master |
| !752 | fix(init_weights): preserve parameter attributes when moving to target device |
| !754 | fix(activation_checkpoint): exclude shared functions from callable overlap tracking |
| !755 | merge sapp_ut_coverage1 into master |
| !756 | fix(dtensor): return self from MindSpore in-place random ops |
| !757 | [shard] Preserve DTensor identity for in-place ops on the dispatch bypass |
| !759 | [bugfix] fix FSDP device-mesh construction in the meta context for tp>1/dp=1 and noncontiguous compat all-reduce errors |
| !766 | test: fix process_group worker path and avoid double world-group teardown |
| !767 | fix dcp level st |
| !768 | feat(pipeline_parallel): forward-side P2P prefetch for overlap_b_f via batch_isend_irecv |
| !770 | fix dcp level st |
| !771 | fix(fsdp2): use assign=True when loading full state dict to fix optimizer device check on torch >= 2.8 |
| !775 | fix(packaging): tag wheel per python/arch and unify cxx11 abi per backend |
| !777 | fix(codecheck): add missing docstrings for remaining C0116 warnings |
| !779 | fix(dtensor): refine MindSpore random op inplace dispatch |
| !782 | merge fix/ms-param-shape-error into master |
| !783 | muon, Hp |
| !786 | fix(pipeline): default PipelineStage device to current accelerator |
| !787 | fix(pipeline): reject batch size not divisible by micro_batch_num on MindSpore |
| !788 | test: reduce NPU level0 ST gate load with gloo level0 coverage |
| !790 | [feature][pipeline_parallel] torch backend pp support FSDP metastep |
| !795 | merge fix_named_parameters into master |
| !803 | fix: remove duplicate empty decorator in sparse_grad_kl_loss test |
| !806 | [bugfix] fix reduce type |
| !808 | Add MHC pre clamp sinkhorn custom op |
- MindSpore CP: DSA-family async CP supports only the PyTorch backend; MindSpore uses no-op placeholders
- DTensor: centric communication and cross-mesh redistribution remain unimplemented
- EP: the basic dropless workflow, communication/computation overlap, hot expert migration, and replicas of heavily used experts remain unimplemented
- PP: ZBV and SeqPP scheduling, and assigning different device counts to individual PP stages, remain unimplemented
- HyperOffload: SAS/SPO/SAC user configuration interfaces, offload with memory semantics, and automatic strategy generation remain unimplemented
- AutoParallel: Fast-Tuner and PARADISE are currently demo features under continued optimization
- One-sided communication: AllToAll, AllReduce, ReduceScatter, and low-precision communication with high-precision accumulation remain unimplemented
To upgrade from v0.2.0 to v1.0.0:
- Activation Checkpoint API changes: new
checkpoint_wrapper/swap_wrapper/swap_tensor_wrapperinterfaces; use these in place of the older APIs - Optimizer module: new
Muon/ChainedOptimizer/get_hyper_optimizer/get_hyper_lr_schedulerinterfaces can be integrated directly into training - PP API extensions:
PipelineStageadds dx/dw computation and FSDP metastep support;ScheduleInterleaved1F1Baddsoverlap_b_f/overlap_p2pparameters - CP API extensions: new
AsyncContextParalleland DSA-family interfaces - Wheel packaging changes: wheel filenames include Python ABI and host architecture tags; native adapters must match the target framework's CXX11 ABI contract
- Report an issue
- Contribute code: join the Parallel Training System SIG
- License: Apache 2.0