Skip to content

Latest commit

 

History

History
61 lines (47 loc) · 3.65 KB

index.md

File metadata and controls

61 lines (47 loc) · 3.65 KB

HyperParallel Documentation

English | 简体中文

Documentation for the distributed parallel acceleration library optimized for Ascend SuperPoDs.

Getting Started

Document Description
Installation Guide Source builds, dependencies, and environment configuration

Feature Guides

Document Description
HSDP / FSDP Data Parallelism fully_shard, HSDPModule, overlap, and gradient scaling
TP Tensor Parallelism ColwiseParallel, RowwiseParallel, parallelize_module, and Loss Parallel
PP Pipeline Parallelism PipelineStage, schedules, overlap_b_f, PP+FSDP, P2P prefetch, and Mpipe
CP Context Parallelism ContextParallel, AsyncContextParallel, and DSA variants
KDA Context Parallelism Configuration and memory usage for KDA P2P, AllGather, Ulysses, and hybrid strategies
KDA Recursive Doubling and Ulysses Hybrid CP RD state scans, TND reset pruning, and transition caching
Linear Attention TND Context Parallelism KDA / GDN document boundaries, configuration, state exchange, and numerical semantics
EP Expert Parallelism ExpertParallel, MoE building blocks, and load balancing
Activation Checkpoint / Swap checkpoint_wrapper, swap_wrapper, swap_tensor_wrapper, and combined configuration
Optimizer AdamW, Muon, ChainedOptimizer, and learning rate schedulers
Chunk Loss Chunked output projection and cross-entropy for large vocabularies; Trainer and Qwen3-MoE integration
DCP Distributed Checkpoints Saving/loading checkpoints, asynchronous staging, and offline conversion
Automatic Parallelism SAPP-ND and SAPP-PPB
LlamaFactory + HyperParallel SFT Training SFT configuration, launch, numerical validation, and performance validation
Model Integration Validation Adapter declarations, structural checks, module parity, numerical validation matrices, and evidence reports

API Reference

Document Description
API Reference Complete interface documentation organized by feature
DFunction Custom distributed autograd functions for Torch

FAQ and Troubleshooting

Document Description
FAQ and Troubleshooting Installation, runtime, communication, memory, and debugging

Contributing

Document Description
Development Environment Environment setup and dependencies
Testing Standards Test framework, markers, and distributed tests
Release Process Version releases and documentation updates

Other Documentation

Document Description
Release Notes v1.0.0 Version history
PP overlap_b_f Design Detailed compute-communication overlap design
TP Styles Tensor parallel style documentation
Agent Workflow AI agent workflow reference