Skip to content

feat: add chunked MoE training with bounded overlap and parent FSDP - #1564

Open
xuxl1008 wants to merge 1 commit into
mindspore-ai:masterfrom
xuxl1008:feat/chunk-moe-training
Open

xuxl1008 wants to merge 1 commit into
mindspore-ai:masterfrom
xuxl1008:feat/chunk-moe-training

Conversation

@xuxl1008

@xuxl1008 xuxl1008 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Paired: GitHub #1564 ↔ GitCode !1936

Chunked MoE training with bounded overlap

Long-sequence MoE training can exhaust memory through routed-token expansion and expert activation reconstruction. This PR adds opt-in MoE(chunk_size=C) training with complete first-order gradients, plus optional chunk_overlap="forward", "backward", or "both". The default remains synchronous ("none"). Chunking trades reconstruction and communication for a smaller active expert working set; overlap spends additional temporary memory to recover some of that time.

Implementation

  • Reuse the common router, shared expert, original expert IDs, EP ownership, and grouped SwiGLU backend. Preserve common-MoE parameter identity and state-dict layout. A custom autograd function consumes routing plans in reverse order; backward does not repeat top-k selection or count exchange. Token statistics and auxiliary loss are aggregated once per original forward.
  • Use a minimal NPU SwiGLU VJP with private FP32 weight-gradient accumulation. Reuse U = G @ W2 for probability and activation gradients, avoiding FC2 reconstruction solely for the probability gradient. Include a CPU autograd reference.
  • Add bounded, call-local scheduling. Forward processes pairs, overlapping the second dispatch and first combine with expert work. Backward prefetches one future block's X after current probabilities and upstream gradients are ready, overlapping expert reconstruction/VJP. Pending sends retain their storage until waited, exceptional exits drain pending work, and all ranks validate schedule agreement before payloads. Reuse the already flattened layer input and capture each invocation's backward mode. No global prefetch table or additional cross-layer expert activation cache.
  • Add an explicit DeepSeek-V3 adapter, attention-only checkpoint policy, recipe, and real-decoder training example. Preserve FP32 sigmoid routing, correction-bias/group selection, normalization, route scale, and shared-expert semantics. Convert packed gate/up weights before optimizer creation. Forward the overlap setting through the adapter and example CLIs.
  • Support FSDP around the whole MoE or its parent with EP inside. Reject independently wrapped experts/router/shared children whose forward hooks the executor bypasses.
  • Improve the shared EP receive permutation for ordinary execution and ChunkMoE: reuse token/probability mappings and inverse-gather restoration/backward. Optionally construct known NPU receive maps of at least 8192 rows directly with Triton-Ascend; retain Torch for smaller/unknown receives, other devices, or a missing optional backend. No EP-degree-specific dispatch rule. Source-side top-k reduction is unchanged; kernel failures are surfaced.

Validation

The overlap follow-up passes:

  • CPU complete-gradient, routing/statistics, frozen-parameter, noncontiguous-input, short-tail, two-pending-microbatch, multilayer and optimizer checks, plus DeepSeek adapter tests: 6 tests in the final targeted rerun. All four schedule values are exercised.
  • Four NPU launchers covering EP4, EP2 with expert replicas, the ordinary score-after path, and parent FSDP2 x EP2 with real updates. EP8 separately covers original IDs, full gradients, shared/auxiliary branches, empty receiving owners and short tails. Schedule disagreement is rejected before payload exchange.
  • The real small random DeepSeek example with parent FSDP2 x EP2, chunk_overlap=both, attention checkpointing, resharding and two AdamW updates. This is functional integration, not a convergence or released-scale result.
  • Pylint, complexity, spelling, style, agent-catalog and local-documentation-link checks. The ST launcher remains backend-free at collection time. The final lifecycle-only refactor was rechecked with CPU/four-card tests and the DeepSeek example.

Earlier follow-ups recorded 93 relevant CPU checks and NPU receive-map boundary/gradient checks; they were not all rerun for this scheduling-only change. The local aggregate arg_mark checker still flags four hardware-independent UT methods, whereas the repository's specific UT rule says they normally have no hardware marks. No artificial hardware restrictions or suppressions were added to hide this discrepancy. Markdownlint is unavailable locally. Remote CI and maintainer review remain outstanding.

Complete-step measurements

Small random Qwen3-MoE decoder, L8/H1024/I512/E128/top-k8, local N8192, C2048; identical grouped expert/permutation/fused RMSNorm backends. Includes loss, backward and AdamW with learning rate zero, two warmups and five measured steps, without a profiler. Report the median of per-step slowest-rank times and the maximum device memory peaks.

EP Schedule Step ms Peak allocated MiB Peak reserved MiB
4 none 539.12 3701.51 4170
4 both 516.51 3768.96 4326
8 none 579.09 2424.85 3668
8 both 541.60 2551.23 3632

Relative to synchronous ChunkMoE, elapsed time falls by 4.19% / 6.47%, with 67.44 / 126.38 MiB additional allocated memory. EP4 runs none then both; EP8 reverses the order. Timing sources are archived by hash; the final follow-up only extracts configuration/lifetime/test helpers, preserving tensor operations and collective order.

These are single-node Ascend 910B observations. Local tokens and model dimensions stay fixed between EP4 and EP8, so global tokens double and local experts halve: this is a holdout with fixed tokens per rank, not a strong-scaling comparison. These measurements compare schedules on ChunkMoE; they do not establish superiority over the best checkpoint baseline or native Tutel MoE. Forward/backward benefits must not be added arithmetically.

Memory and supported boundaries

Layer inputs and low-dimensional routing plans remain live across layers and scale with sequence length. Each receive inverse map adds 8 bytes per received route. Expert-only checkpointing retains dispatched expert inputs across layers; whole-MoE checkpointing has a different retention/reconstruction tradeoff. Comparisons must include those lifetimes and complete multilayer peaks.

Overlap adds current-layer send/receive/reorder workspaces, not a high-dimensional activation cache retained for every layer. Budget by actual owner receives, including skew, rather than average top_k * C. Future X can delay current dX and communication can compete with GEMMs; enabling overlap is not guaranteed to help.

  • CPU reference or BF16 NPU grouped SwiGLU, score-after, no autocast; evenly assigned experts over one-dimensional EP. Local token count, C, E, top-k and overlap mode must agree across the communication domain.
  • Complete first-order gradients, frozen parameters and normal microbatch accumulation; no repeated backward on one forward, higher-order chunk gradients, external main_grad, TP, quantized experts or graph capture.
  • Parent FSDP support does not imply the generic Trainer's nested expert-FSDP wrapping is supported. Keep ChunkMoE outside outer decoder/MoE checkpoint regions; use independent attention checkpointing.
  • No dispatch0/router1 pipeline yet: both forward plans are prepared before the payload window. No deeper queues, dynamic expert migration, adaptive policy or joint KDA execution.

Review checklist

  • Public usage guide, API defaults, DeepSeek recipe and runnable examples.
  • Complete gradients, original routes, empty owners, short tails and the supported FSDP boundary.
  • Shared receive optimizations apply to ordinary/checkpoint and chunk execution alike.
  • Measure combined forward/backward overlap on multilayer EP4 and EP8 probes, including additional buffers.
  • Remote CI and maintainer review.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


xianliang xu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@mindspore-ci-bot

Copy link
Copy Markdown
Contributor

Hi @xuxl1008, thanks for your contribution!

This pull request is currently waiting for the CLA Assistant check to pass before the CI process can continue.

Please follow the signing link shown by CLA Assistant on this PR. If you have already signed the CLA but the check is still failing, please make sure that all email addresses used in your commits are added to and verified on your GitHub account.

After updating your email settings or signing the CLA, please click Recheck in the CLA Assistant comment/check so the status can be refreshed.

Once the CLA Assistant check passes, CI will continue automatically.

@xuxl1008
xuxl1008 force-pushed the feat/chunk-moe-training branch from f0070d8 to c935e4f Compare November 13, 2026 09:08
@xuxl1008 xuxl1008 changed the title feat: add chunked MoE training with reusable routing plans feat: add chunked MoE training with DeepSeek and parent FSDP support Nov 13, 2026
@xuxl1008
xuxl1008 force-pushed the feat/chunk-moe-training branch from c935e4f to 2aaf004 Compare November 14, 2026 14:56
@xuxl1008 xuxl1008 changed the title feat: add chunked MoE training with DeepSeek and parent FSDP support feat: add chunked MoE training with bounded overlap and parent FSDP Nov 15, 2026
@xuxl1008
xuxl1008 force-pushed the feat/chunk-moe-training branch from 2aaf004 to 65ffd80 Compare November 15, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants