Repository navigation
Conversation
|
xianliang xu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
|
Hi @xuxl1008, thanks for your contribution! This pull request is currently waiting for the CLA Assistant check to pass before the CI process can continue. Please follow the signing link shown by CLA Assistant on this PR. If you have already signed the CLA but the check is still failing, please make sure that all email addresses used in your commits are added to and verified on your GitHub account. After updating your email settings or signing the CLA, please click Recheck in the CLA Assistant comment/check so the status can be refreshed. Once the CLA Assistant check passes, CI will continue automatically. |
f0070d8 to
c935e4f
Compare
c935e4f to
2aaf004
Compare
2aaf004 to
65ffd80
Compare
Paired: GitHub #1564 ↔ GitCode !1936
Chunked MoE training with bounded overlap
Long-sequence MoE training can exhaust memory through routed-token expansion and expert activation reconstruction. This PR adds opt-in
MoE(chunk_size=C)training with complete first-order gradients, plus optionalchunk_overlap="forward","backward", or"both". The default remains synchronous ("none"). Chunking trades reconstruction and communication for a smaller active expert working set; overlap spends additional temporary memory to recover some of that time.Implementation
U = G @ W2for probability and activation gradients, avoiding FC2 reconstruction solely for the probability gradient. Include a CPU autograd reference.Validation
The overlap follow-up passes:
chunk_overlap=both, attention checkpointing, resharding and two AdamW updates. This is functional integration, not a convergence or released-scale result.Earlier follow-ups recorded 93 relevant CPU checks and NPU receive-map boundary/gradient checks; they were not all rerun for this scheduling-only change. The local aggregate
arg_markchecker still flags four hardware-independent UT methods, whereas the repository's specific UT rule says they normally have no hardware marks. No artificial hardware restrictions or suppressions were added to hide this discrepancy. Markdownlint is unavailable locally. Remote CI and maintainer review remain outstanding.Complete-step measurements
Small random Qwen3-MoE decoder, L8/H1024/I512/E128/top-k8, local N8192, C2048; identical grouped expert/permutation/fused RMSNorm backends. Includes loss, backward and AdamW with learning rate zero, two warmups and five measured steps, without a profiler. Report the median of per-step slowest-rank times and the maximum device memory peaks.
Relative to synchronous ChunkMoE, elapsed time falls by 4.19% / 6.47%, with 67.44 / 126.38 MiB additional allocated memory. EP4 runs none then both; EP8 reverses the order. Timing sources are archived by hash; the final follow-up only extracts configuration/lifetime/test helpers, preserving tensor operations and collective order.
These are single-node Ascend 910B observations. Local tokens and model dimensions stay fixed between EP4 and EP8, so global tokens double and local experts halve: this is a holdout with fixed tokens per rank, not a strong-scaling comparison. These measurements compare schedules on ChunkMoE; they do not establish superiority over the best checkpoint baseline or native Tutel MoE. Forward/backward benefits must not be added arithmetically.
Memory and supported boundaries
Layer inputs and low-dimensional routing plans remain live across layers and scale with sequence length. Each receive inverse map adds 8 bytes per received route. Expert-only checkpointing retains dispatched expert inputs across layers; whole-MoE checkpointing has a different retention/reconstruction tradeoff. Comparisons must include those lifetimes and complete multilayer peaks.
Overlap adds current-layer send/receive/reorder workspaces, not a high-dimensional activation cache retained for every layer. Budget by actual owner receives, including skew, rather than average
top_k * C. Future X can delay current dX and communication can compete with GEMMs; enabling overlap is not guaranteed to help.main_grad, TP, quantized experts or graph capture.Review checklist