[faresobeid]

Router-replay ids and sampling masks are per-token arrays, so they grow with batch × context × layers × top-k. Inline, they travel as base64 through the API server, env servers and orchestrator, then once per CP rank over ZMQ. On multi-node runs (including disaggregated P/D) this PR passes them by handle: the inference server writes them to a file on the shared filesystem and only small segment descriptors travel. Single-node, eval and frozen-source rollouts stay inline.

What

• Switch: orchestrator.payload_root. rl sets it to <run dir>/payloads for multi_node deployments with router replay or sampling-mask capture, and defaults trainer.data.num_workers = 1.
• Inference: a train rollout's request carries extra_args.payload_dir = <payload_root>/v<k> (k = dispatch policy version). serving_tokens.py writes routing rows (uint8/uint16, as vLLM picks) and the CSR sampling mask to one fsynced file per request and returns payload segments (PayloadSegment(field, file, offset, pos, rows, dtype, shape)) instead. Mask ids are bit-packed at ceil(log2(vocab_size)) bits (18 for 150k vocabularies), with per-row start indices so any row range can be read.
• Routed layers only: routed_layers(hf_config) lists the layers with a learned router. Dense MLP layers (first_k_dense_replace, mlp_layer_types, num_dense_layers, mlp_only_layers), Nemotron-H Mamba/attention layers and DeepSeek-V4 hash layers are left out. The server applies it to routing rows (by handle and inline). The trainer scatters the rows back with zero rows for the other layers, which is what vLLM leaves there. The DeepSeek-V4 trainer model routes hash layers by token id instead of replaying them.
• P/D: both engines get payload_dir and each writes its own file. The router merges the two segment lists (router#60): prefill's routing, then decode's clipped at prefill's coverage; prefill's 1-token mask segment is dropped.
• Orchestrator: clips segments to each sample (verifiers#2765), shifts them through packing and truncation, and never mixes inline and by-handle samples in a micro batch.
• Trainer: each rank reads only its CP chunk, in the data worker, grouped by file with 32 pread threads; stale or short reads retry, then raise.
• Retention: a shipped batch's files are deleted once inference applies the policy that trained on it; dropped or discarded work deletes its files immediately; an orphan sweep removes v<k> older than every live rollout and older than step - max_off_policy_steps. A fresh run wipes the root; a resumed run keeps it (fix(orchestrator): drop restored work that points at deleted by-handle payload files #4044 drops restored work whose files are gone).

Cost

Rows per token (routed layers only): Qwen3-30B-A3B 384 B, GLM-4.5-Air 360 B (−2.2 %), GLM-5.3 600 B (−3.8 %), DeepSeek-V4-Flash 240 B (−7 %), Nemotron-3-Nano 138 B (−56 %). Masks about 4 B per completion token + 2.25 B per kept id, −35 to −39 % vs 4 B per kept id on GPU runs below.

Measurement	Inline	By handle
GLM-4.5-Air, batch 128, 1+1 nodes (real run): orchestrator RSS / ZMQ wire per step	6.9 GB / 720 MB	3.2 GB / 70 MB
512 samples, 2.3M tokens, DP 8, CP 4 (offline): held / peak / egress	1.2 GB / +3.3 GB / 3.9 GB	0.29 GB / +0.57 GB / 0.33 GB
Nemotron-3-Super rows, 16 × 30k tokens, CP 8 (CPU bench): pack+encode / egress	3.9 s / 15 GB	0.3 s / 0.13 GB

Step time on GLM-4.5-Air was unchanged: at that size the gain is memory. Large-scale step time and shared-filesystem throughput have not been measured yet.

Correctness (GPU, Qwen3-30B-A3B TP1, VLLM_BATCH_INVARIANT=1, router replay + masks)

• Colocated (vLLM 0.31): tokens equal; every routing/mask segment equals inline; 140/140 trainer micro batches (CP 1/2/4) bitwise equal.
• P/D (vLLM 0.32, 1P1D NIXL, router built from router#60): 160 turns run 4 ways (P/D and colocated × inline and by handle), including concurrency 10, 22 decode preemptions and uint16 rows. Tokens, routing and masks bitwise equal in all modes; P/D rows equal colocated rows; 126/126 trainer micro batches bitwise equal. P/D response size 12–48 MB → 0.06–0.44 MB.
• Packed mask ids and routed-layer rows (vLLM 0.32, compared against full-row servers from before the change): Qwen3-30B-A3B (colocated and 1P1D) and GLM-4.5-Air TP2 (dense layer 0), 40 turns each. After unpacking and expansion, routing equals the full rows (200/200 and 120/120 checks), masks are bitwise equal (160/160 and 80/80), and trainer micro batches at CP 1/2/4 are bitwise equal (28/28 each). Mask bytes in the payload files: −35 % (Qwen), −39 % (GLM). Layer maps were also checked on CPU, with real HF configs and prime-rl trainer configs, for DeepSeek-V4-Flash (3 hash layers), Nemotron-3-Nano (23 of 52 layers routed) and GLM-5.3.

Tests

pytest -m "not gpu" tests/unit/orchestrator tests/unit/test_configs.py tests/unit/inference tests/unit/transports tests/unit/train/rl: 410 passed; router#60 cargo test --lib routed_experts: 6 passed; ruff clean. Docs: docs/inference.md § Payloads by handle.

Before merge

• verifiers#2765 and renderers#170 merge; re-pin deps/verifiers (now 410e84e8) and deps/renderers (now 4163c89).
• router#60 merges and ships in a vllm-router release; bump the vllm-router pin (v0.2.3). Without it, P/D turns carry only decode's routing and the orchestrator drops it with a warning.
• Whichever of this PR and feat: exact score centering over the sampling mask (arXiv:2609.20807) #3794 (score centering) lands second writes sampling_mask_logprobs by handle too.

Upstream: vllm-project/vllm#57968 (open) — Mooncake keys-only R3, an alternative design for the routing half (not a dependency).

---

Note

Medium Risk
Touches the full train data path (inference → orchestrator → trainer) and shared-filesystem retention; mistakes could drop or misread routing/mask data and bias RL, though extensive unit tests cover parity with inline mode.

Overview
Multi-node router replay and sampling-mask capture now spill per-token payloads to a shared filesystem instead of shipping huge inline arrays through HTTP, env servers, orchestrator, and ZMQ.

The rl entrypoint sets orchestrator.payload_root (typically <run dir>/payloads) when deployment is multi_node and routed experts or sampling masks are enabled, and defaults trainer.data.num_workers = 1 so payload reads can overlap training. Train rollouts pass extra_args.payload_dir = <root>/v<policy version>; vLLM serving_tokens fsyncs one file per request and returns payload segment descriptors. A new transports/payload layer handles CSR bit-packed masks, routed_layers-only routing rows, clip/pack in the orchestrator, CP-sharded pread in the trainer, and lifecycle GC (per-batch delete after policy apply, immediate cleanup on drops, version sweeps).

Inline payloads remain for single-node, eval, and clients without payload_dir. Docs add Payloads by handle.

^{Reviewed by Cursor Bugbot for commit d1f029e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 72319a0. Configure here.}