Skip to content

feat(rl): add colocated asynchronous GRPO training - #1359

Merged
github-actions[bot] merged 1 commit into
mindspore-ai:masterfrom
Zjunyuan2026:feat/fully-and-colocate-async-grpo
Oct 31, 2026
Merged

github-actions[bot] merged 1 commit into
mindspore-ai:masterfrom
Zjunyuan2026:feat/fully-and-colocate-async-grpo

Conversation

@Zjunyuan2026

@Zjunyuan2026 Zjunyuan2026 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Paired: GitHub #1359 ↔ GitCode !1870

/kind feature

Summary

Add colocated asynchronous GRPO for Qwen3 dense models, allowing actor training and vLLM rollout to share the same devices.
Generation and optimization alternate on those devices, while a separate CPU producer manages sampling and agent execution.

Colocated execution

  • Prefetch complete GRPO candidate groups through bounded admission credits and result buffers.
  • Pause rollout admission before an update and settle active model calls or preserve partial generation output.
  • Put the inference engine to sleep before training uses the shared devices.
  • Publish updated actor weights through IPC, commit the new policy version, and resume unfinished sampling work.
  • Support internal single-turn sampling and external multi-turn agent workflows through the existing producer and task interfaces.
  • Require single-node FSDP training with training TP=1 and one rollout replica whose TP matches the training world size, with CPU offload and resharding enabled.

Policy correctness and recovery

  • Keep sampled log probabilities, behavior-policy identities, and per-call policy segments attached to the original actions.
  • Bound policy lag and retain complete candidate groups when selecting training batches.
  • Keep off-policy correction disabled by default; metrics_only reports rollout-to-anchor differences without changing the objective.
  • Provide optional decoupled correction with a frozen actor anchor, detached token-level importance weights, and configurable weight clipping.
  • Allow group-level rejection and same-policy refill before an optimizer update.
  • Apply correction weights to the policy objective while preserving reference-KL regularization.
  • Drain sampling before checkpoint snapshots and commit source, buffer, and policy-version state alongside the model checkpoint.
  • Coordinate the shared agent gateway when handing sampling over to evaluation.

Configuration and observability

  • Add GSM8K and Search-R1 recipes for the colocated execution path.
  • Validate topology, sampling capacity, probability settings, and supported runner combinations before runtime initialization.
  • Report stage timings, optimizer progress, sample age, behavior versions, and correction diagnostics.

Related issue

Related to #1213.

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/check-pr

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1235 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1235
Check_Pylint FAILURE 1235

@mindspore-ai-bot

Copy link
Copy Markdown

@Zjunyuan2026, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck.

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/check-pr

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1236 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ❌ FAILURE 21%/0% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1236
UT_TEST FAILURE 1236
coverage FAILURE 1236

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/check-pr

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1240 is running. Please wait a moment... (Link)

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/check-pr

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1250 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ✅ SUCCESS 72%/0% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- shard 189.046s 300.0s
- auto_models 107.908s 150.0s
- fully_shard 368.315s 600.0s
- pipeline_parallel 678.086s 600.0s
- accuracy 104.202s 120.0s
- tensor_parallel 235.623s 300.0s
- activation_memory 124.47062s 120.0s
- swap_optimizer 332.536s 300.0s
- optimizer 296.429s 300.0s
- context_parallel 93.96s 120.0s
- expert_parallel 154.91475s 120.0s
- clip_grad 109.209s 120.0s
- init_weights 96.761s 118.75s
- dtensor 98.326s 120.0s
- distributed_checkpoint 92.969s 120.0s
- process_group 93.423s 90.0s
- multicore 128.46188s 120.0s
- llama3_examples 78.832s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ✅ SUCCESS 1250

@github-actions
github-actions Bot enabled auto-merge October 31, 2026 01:11
auto-merge was automatically disabled October 31, 2026 01:26

Head branch was pushed to by a user without write access

@Zjunyuan2026
Zjunyuan2026 force-pushed the feat/fully-and-colocate-async-grpo branch from 8dbdaef to 11495e3 Compare October 31, 2026 01:26
@github-actions
github-actions Bot enabled auto-merge October 31, 2026 01:27
@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/check-pr

@Zjunyuan2026

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1281 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ✅ SUCCESS 72%/0% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- activation_memory 115.662s 120.0s
- shard 164.626s 300.0s
- auto_models 93.771s 150.0s
- pipeline_parallel 521.902s 600.0s
- fully_shard 289.286s 600.0s
- optimizer 230.523s 300.0s
- init_weights 68.84s 118.75s
- swap_optimizer 270.761s 300.0s
- accuracy 75.786s 120.0s
- clip_grad 66.971s 120.0s
- distributed_checkpoint 68.797s 120.0s
- multicore 110.31375s 120.0s
- tensor_parallel 208.9s 300.0s
- expert_parallel 137.25325s 120.0s
- context_parallel 64.975s 120.0s
- dtensor 68.367s 120.0s
- process_group 58.029s 90.0s
- llama3_examples 61.266s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ✅ SUCCESS 1281

@github-actions
github-actions Bot merged commit 74c9a42 into mindspore-ai:master Oct 31, 2026
7 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants