Skip to content

feat(compile): support DeepSeek V4.1 packed static graph training - #1434

Merged
github-actions[bot] merged 1 commit into
mindspore-ai:masterfrom
honghaowang25:packing_v41
Nov 5, 2026
Merged

github-actions[bot] merged 1 commit into
mindspore-ai:masterfrom
honghaowang25:packing_v41

Conversation

@honghaowang25

@honghaowang25 honghaowang25 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Prepare packed metadata before graph capture and replay document tensors independently of compression. Preserve standard WikiText loading and expose router metrics as graph outputs.

Validate 40-layer debug FP32 FSDP2 training for 30 steps with dynamic shapes, packing enabled, and Compressor/Indexer disabled.

What type of PR is this?

/kind feature

What does this PR do / why do we need it:

Enable DeepSeek V4.1 packing-only text training through HyperParallel's standard WikiText data pipeline and joint-graph compiler.

Packed document boundaries and document counts vary between batches. Reading real boundary values during capture, or treating document-dependent Python metadata as constants, prevents correct graph capture and replay. Router metrics also need explicit graph outputs because replay does not update Python module attributes.

This PR:

  • Prepares validated packing geometry before capture and registers a stable pytree representation containing token-shaped metadata tensors.
  • Reuses document IDs for MoE sequence balancing, with empty capacity slots excluded from the existing loss average.
  • Uses the same detached Tensor-valued last_mean_aux_loss getter in eager and graph modes, returns router metrics through loss_dict, and converts them to Python floats in the graph-external logging callback.
  • Adds a WikiText FSDP2 configuration and a launcher that accepts external model, tokenizer, dataset, and optional Engram resources, without requiring a bundled debug preparation script.

Validation:

  • 44 focused unit tests passed, including changing-length/boundary graph replay and packing loss/gradient checks.
  • The packing-only implementation completed 30-step eager and joint-graph runs with a 40-layer, randomly initialized FP32 debug model, FSDP2, and dynamic=true for the graph run.
  • After simplifying the public launcher, an external-resource FSDP2 eager smoke test completed 2 steps successfully.

Scope: CP=TP=EP=1; Compressor, Indexer, candidate pool, and Indexer KL are disabled. Compression ratios must be configured in the external model config to match its layer count. These results establish functional execution, not full-model eager/graph FSDP numerical parity or fused DSA performance.

Which issue(s) this PR fixes:

Fixes #1437

Code review checklist [illustration]:

  • whether to verify the function's return value (It is forbidden to use void to mask the return values of security functions and self-developed functions. C++ STL functions can be masked if there is no problem)
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid (if there is no test case, please explain the reason)
  • Whether the API change is involved
  • Whether official document modification is involved

@honghaowang25

Copy link
Copy Markdown
Contributor Author

/retest

@honghaowang25

Copy link
Copy Markdown
Contributor Author

/check-pr

@mindspore-ai-bot

Copy link
Copy Markdown

Gate trigger failed: please fix the /check-pr findings and obtain the pr-check-pass label before retrying.

@honghaowang25

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1371 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1371
Check_AI_Keyword FAILURE 1371

@honghaowang25

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1372 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1372
Smoke_CPU_X86_CentOS_Level0 FAILURE 1372
coverage FAILURE 1372

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ❌ FAILURE 73%/87% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1428
Check_Pylint FAILURE 1428

@mindspore-ai-bot

Copy link
Copy Markdown

@honghaowang25, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck.

@honghaowang25

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1429 is running. Please wait a moment... (Link)

Prepare packing tensor inputs only at the graph boundary. Register a
graph-specific pytree type while preserving ordinary eager FSDP input
objects, boundaries and compression caches. Expose router metrics through
a unified Tensor getter and retain the external-resource WikiText launcher.

Emit router metrics through shared postforward, with writers detaching
aux losses at the source so the getter stays a pure on-device mean.
Dispatch the micro-step-end lifecycle through BaseTrainer and TextTrainer
wrappers, with an explicit loss mapping in the callback contract. Logging
averages reporting microbatches while trainer loss mappings retain generic
summation. Cover the getter's scalar-free contract plus lifecycle dispatch
order and failed microsteps.
@honghaowang25

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1434 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ✅ SUCCESS 74%/72% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- shard 177.432s 300.0s
- activation_memory 117.43512s 120.0s
- auto_models 93.585s 150.0s
- optimizer 238.872s 300.0s
- pipeline_parallel 551.615s 600.0s
- swap_optimizer 294.776s 300.0s
- accuracy 80.242s 120.0s
- init_weights 68.426s 118.75s
- fully_shard 304.201s 600.0s
- context_parallel 75.914s 120.0s
- expert_parallel 159.97063s 200.0s
- multicore 110.64325s 120.0s
- distributed_checkpoint 76.741s 120.0s
- tensor_parallel 215.006s 300.0s
- clip_grad 70.97s 120.0s
- dtensor 67.725s 120.0s
- process_group 57.465s 90.0s
- llama3_examples 58.71s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ✅ SUCCESS 1434

@github-actions
github-actions Bot enabled auto-merge November 5, 2026 07:19

@liuchongming74 liuchongming74 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions
github-actions Bot merged commit 7b1ea07 into mindspore-ai:master Nov 5, 2026
10 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support DeepSeek V4.1 packed WikiText training with joint-graph compilation

7 participants