| HSDP / FSDP Data Parallelism |
fully_shard, HSDPModule, overlap, and gradient scaling |
| TP Tensor Parallelism |
ColwiseParallel, RowwiseParallel, parallelize_module, and Loss Parallel |
| PP Pipeline Parallelism |
PipelineStage, schedules, overlap_b_f, PP+FSDP, P2P prefetch, and Mpipe |
| CP Context Parallelism |
ContextParallel, AsyncContextParallel, and DSA variants |
| KDA Context Parallelism |
Configuration and memory usage for KDA P2P, AllGather, Ulysses, and hybrid strategies |
| KDA Recursive Doubling and Ulysses Hybrid CP |
RD state scans, TND reset pruning, and transition caching |
| Linear Attention TND Context Parallelism |
KDA / GDN document boundaries, configuration, state exchange, and numerical semantics |
| EP Expert Parallelism |
ExpertParallel, MoE building blocks, and load balancing |
| Activation Checkpoint / Swap |
checkpoint_wrapper, swap_wrapper, swap_tensor_wrapper, and combined configuration |
| Optimizer |
AdamW, Muon, ChainedOptimizer, and learning rate schedulers |
| Chunk Loss |
Chunked output projection and cross-entropy for large vocabularies; Trainer and Qwen3-MoE integration |
| DCP Distributed Checkpoints |
Saving/loading checkpoints, asynchronous staging, and offline conversion |
| Automatic Parallelism |
SAPP-ND and SAPP-PPB |
| LlamaFactory + HyperParallel SFT Training |
SFT configuration, launch, numerical validation, and performance validation |
| Model Integration Validation |
Adapter declarations, structural checks, module parity, numerical validation matrices, and evidence reports |