Skip to content

fix(compile): advance contiguous gradient inputs with reduce-scatter - #1309

Merged
kisnwangg merged 1 commit into
mindspore-ai:masterfrom
honghaowang25:ut_fix
Oct 27, 2026
Merged

kisnwangg merged 1 commit into
mindspore-ai:masterfrom
honghaowang25:ut_fix

Conversation

@honghaowang25

@honghaowang25 honghaowang25 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Paired: GitHub #1309 ↔ GitCode !1848

What type of PR is this?

/kind bugfix

What does this PR do / why do we need it:
Keep NPU reduce-scatter inputs contiguous while emitting schema-backed backward preparation owned by each reduction. Move private preparation with its launch so layout conversion no longer pins communication at the graph tail. Preserve dependency, effect and storage-release boundaries.

Add regressions for private and shared preparation, ownership, mutations, alias releases and repeated scheduling. Retain strict loss/gradient checks and the requirement for real reduce-scatter movement.

Which issue(s) this PR fixes:

Fixes #1311

Code review checklist [illustration]:

  • whether to verify the function's return value (It is forbidden to use void to mask the return values of security functions and self-developed functions. C++ STL functions can be masked if there is no problem)
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid (if there is no test case, please explain the reason)
  • Whether the API change is involved
  • Whether official document modification is involved

@honghaowang25

Copy link
Copy Markdown
Contributor Author

/check-pr

@honghaowang25

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1143 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1143
Check_Lizard FAILURE 1143

@mindspore-ai-bot

Copy link
Copy Markdown

@honghaowang25, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck.

@honghaowang25

Copy link
Copy Markdown
Contributor Author

/retest

Keep NPU reduce-scatter inputs contiguous while emitting schema-backed
backward preparation owned by each reduction. Move private preparation
with its launch so layout conversion no longer pins communication at the
graph tail. Preserve dependency, effect and storage-release boundaries.

Add regressions for private and shared preparation, ownership, mutations,
alias releases and repeated scheduling. Retain strict loss/gradient checks
and the requirement for real reduce-scatter movement.

Validation: 68 FSDP/overlap tests on Torch 2.9.0 CPU; 247 compile tests on
the current runtime; CI Pylint 3.3.7 rates both production files 10.00/10.
@honghaowang25

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1144 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ✅ SUCCESS 72%/97% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- activation_memory 116.3705s 120.0s
- shard 162.83s 300.0s
- auto_models 93.41s 150.0s
- optimizer 233.696s 300.0s
- expert_parallel 137.11762s 120.0s
- multicore 109.74s 120.0s
- pipeline_parallel 516.776s 600.0s
- accuracy 76.261s 120.0s
- swap_optimizer 216.521s 300.0s
- distributed_checkpoint 73.814s 120.0s
- fully_shard 277.669s 600.0s
- clip_grad 79.06s 120.0s
- context_parallel 70.99s 120.0s
- tensor_parallel 211.369s 300.0s
- dtensor 68.325s 120.0s
- init_weights 65.271s 118.75s
- process_group 61.489s 90.0s
- llama3_examples 56.864s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ✅ SUCCESS 1144

@kisnwangg
kisnwangg merged commit 7188126 into mindspore-ai:master Oct 27, 2026
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Contiguous gradient preparation prevents FSDP reduce-scatter overlap

3 participants