NeMo RL Software Architecture Update: PR Tracking

[anwithk]

NeMo RL Evolution: PR Tracking

This issue consolidates the open and in-flight PRs across the NeMo RL evolution workstreams. It is meant as a single index so we can track status in one place. PRs are grouped by workstream, with cross-workstream PRs noted where relevant.

Data Plane

Owner: Zhiyu Li

• feat: data plane transfer queue integration #2439: Data Plane initial integration
• test(data_plane): pin async-RL filter flow + example BaseRolloutFilter #2593: Async RL example (Data Plane with Simple Storage)
• test: add TQ nightly coverage set (simple + mooncake_cpu backends) #2616: Nightly tests coverage (Data Plane with Simple Storage)
• [Feat]: Add RayStorageClient to support the RDT feature of Ray TransferQueue/TransferQueue#167: RayRDT with NIXL in TransferQueue

Router Replay (R3)

Owner: Zeyu Zhou

• feat: add router replay (R3) support #2590: R3 PR with docs and wandb report
• Add R3 router replay support #2465: R3 base PR

FT, Distillation

Owner: Pranav Prashant Thombre

• feat(distillation): add TransferQueue support for On-Policy Distillation #2580: FT / Distillation PR
• feat(data-plane): add TQ fault tolerance APIs #2492: Fault tolerance support in Data Plane for the Async controller

Refit

Owner: Songlin Jiang

Core refactor series (blocks the delta and RDMA refit work):

• feat: [NemoRL] Introduce WeightSynchronizer ABC with IPC/HTTP/NCCL transports #2466: WeightSynchronizer ABC with IPC/HTTP/NCCL transports (merged)
• feat: Wire WeightSynchronizer into algorithm layer, replacing inline refit logic #2467: wire synchronizer into GRPO and distillation
• feat: Unify policy phase transtions and offload methods #2470: unify policy phase transitions (PolicyTrainerInterface)
• feat: Introduce generation backend registry and decouple algorithms #2475: generation backend registry and create_generation() factory (also Generation)

Delta Weight Transfer

• feat: add delta-compressed collective refit #2444: delta weight transfer (perf optimization and data-race fix)

P2P RDMA based refit

• feat: add checkpoint-engine refit interface and integrate NIXL #2608: checkpoint engine interface with NIXL backend PoC

NCCL hierarchical API

Owner: Youngeun Kwon

• feat: NCCL Xfer based refit integration #2413: NCCL hierarchical API / RDMA based refit

AsyncRL

Owners: Akash Mehra, Yuki Huang

Train pump (3-PR async-GRPO split-API stack):

• feat(algorithms): SingleController streaming train_pump (split-API consumer) #2700: SingleController streaming train_pump (split-API consumer)
• feat(policy): split-API train-step state machine on DTensor v1/v2 #2692: DTensor v1/v2 split-API and PolicyTrainerActor
• feat(megatron): split-API train-step state machine on MegatronPolicyWorker #2683: Megatron split-API train-step state machine

Rollout and per-prompt streaming:

• feat: add AsyncRolloutManager for native async per-prompt rollouts #2566: native async rollout path (also Generation)
• refactor: unify per-prompt rollout to RolloutManager #2567: unify rollout paths into a unified factory (also Generation)
• feat: add AsyncNemoGymRolloutManager for gym per-prompt rollouts #2528: gym rollout path (also Gym and Generation)
• refactor: refactor async utils #2448: refactor async utils
• feat: support staleness-window in ReplayBufferNew #2458: staleness-window support
• REP: Embedded Storage Backend for GCS Fault Tolerance ray-project/enhancements#64: SingleController/HeadNode fault tolerance RFC
• REP: Ray Active-Passive Head Architecture ray-project/enhancements#65: SingleController/HeadNode fault tolerance RFC

Single Controller:

• feat: single controller e2e #2819: support SingleController e2e run

Gym

Owners: Ananth Subramaniam, Hemil Desai

• Add sandbox API and mini SWE agent 2 OpenSandbox evaluation Gym#1368: Sandbox API large PR (to be split into smaller parts)
• feat: add AsyncNemoGymRolloutManager for gym per-prompt rollouts #2528: gym rollout path (shared with AsyncRL)

Generation

• NVIDIA-NeMo/RL [Don't merge] feat(grpo): add NRL_GEN_BENCHMARK_SKIP_TRAINING for async generation #2930: Skip training for generation benchmarking

Reference links (full URLs)