[mikasenghaas]

Summary

Router-replay rows, the score-centering top-k head and sampling masks travel today from the env servers to the orchestrator inline, and the orchestrator packs them into the trainer batch. At high concurrency with long contexts, the orchestrator node then holds terabytes of per-token arrays that only the trainer reads. Now they never leave the env servers inline: the train env servers keep them, and references are the only way they reach the trainer. There is no config and no inline path:

• Env servers keep the arrays (transports/payload.py). vLLM still returns them inline, so the env server keeps full control over the response. When a train episode finishes, each env-server worker (verifiers' set_episode_hook, PayloadHook.keep) encodes every trainable branch's arrays (encode_branch_arrays), keeps the blob, serves it over HTTP (GET <url>/<id>, a stdlib threaded server per worker on the env server's bind host, loopback by default, DELETE to free). It leaves the URL and, per branch, the blob id, the kept fields and the routing shape in trace.info["prime_rl_payload"]. Env-server workers import no torch through this path. Episode streams never carry the arrays (verifiers#2934).
• The orchestrator packs references, never arrays. A sample carries a PayloadSlice (url, id, kept token count, fields); truncation shortens the count. A micro batch lists one reference per sequence, so samples with and without arrays pack together; samples with and without routing do not (orchestrator/trajectories.py, orchestrator/batch.py). Padding dummies reference only routing rows.
• Trainer ranks fetch and assemble. Before each step, every rank fetches its micro batches' blobs in parallel and assembles them (pack_token_arrays, pad_token_arrays: cut each to its sample, concatenate in packing order, pad; a sample without arrays gets empty rows), so the tensor conversion is unchanged (trainer/rl/data.py, transports/payload.py).
• Strict accounting, no sweep. Every kept array is deleted exactly once (DELETE <url>/<id>), by whoever holds it last: the trainer, once every rank fetched a step's arrays (after the world all-reduce that agrees on failed fetches, one rank per data rank deletes); the orchestrator, for whatever it receives but never ships: the train sink for filtered and dropped traces and groups, untrained branches, pruned samples, samples that staleness masking voids, and batches and queues discarded at shutdown; the eval sink for every eval episode; the dispatcher for episodes that returned after their group was dropped; the env server, for episodes it kept that never reached the orchestrator, including runs the client abandoned after the reply (PayloadHook.discard, called by verifiers' delivery contract).
• Missing arrays fail loudly. Each fetch retries before giving up, and bypasses HTTP proxies. A step where no micro batch's arrays can be fetched anywhere stops the run. A micro batch whose arrays still cannot be fetched is dropped from the loss on every rank that trains it (one all-reduce), never trained on zeros, and counted in payload/dropped_micro_batches (summed over data ranks). Its routing keeps the model's shape and repeats real rows, so the forward pass still routes like real data.
• Every env server keeps its arrays; eval deletes them. prime-rl's env-server entrypoint always installs the hook, with no config. Eval never trains, so EvalSink.add deletes the arrays of every eval episode it receives, in the orchestrator and in the standalone eval entrypoint (orchestrator/eval_sink.py).
• Env-server replicas on every inference node. On multi-node SLURM runs with inference nodes, every inference node runs a replica of every launcher-managed env server, bound to the node address. Each replica publishes its address to envs/<split>/<name>.<node>.address. The orchestrator connects to all replicas and sends each run to the replica with the fewest runs in flight (orchestrator/envs.py). So env-server memory and CPU spread over all inference nodes, even for a single env. The launcher sets orchestrator.env_server_replicas to the number of inference nodes.

Builds on PrimeIntellect-ai/verifiers#2934 (merged): episode streams without training arrays, and set_episode_hook with its keep/discard contract. deps/verifiers points at its merge commit on verifiers main (60290d436).

Breaking

• Env server placement. On multi-node SLURM runs with inference nodes, launcher-managed env servers now run as one replica per inference node instead of one server next to the orchestrator. Env-server logs move to logs/envs/<split>/<name>.<node>.log. No option restores the old placement. To pin a source's env server elsewhere, run it yourself and point the source at it with serve.address.
• Train env servers must run prime-rl's env-server entrypoint. Episode streams no longer carry training arrays, so a train source served any other way (serve.address pointing at a plain verifiers server) delivers none. Runs that need sampling masks or top-k heads stop with an error saying so.
• Per-server limits apply per replica. serve.max_concurrent, the worker pool and sandbox limits of a launcher-managed env server now apply to each replica, so a source's total rises with the number of inference nodes. Migration: divide a limit that must hold run-wide by the number of inference nodes.
• Persisted batches can't be replayed. With the filesystem rollout transport, a step file now holds references, and the trainer deletes the arrays once it has fetched them. A trainer that re-reads an old step file after a restart finds the arrays gone and drops every micro batch (payload/dropped_micro_batches).
• The sampling_support/size metric is removed. Traces reach the orchestrator without sampling masks, so nothing feeds it.

Verification

Qwen3-30B-A3B-Thinking on i3_math with router replay (48 layers × 8 experts, uint8) and IcePop score centering (top-128 head), batch 128 × group 8, up to 1,024 rollouts in flight, 16k-token completions, H200.

Inline (main) vs by reference, 1 trainer + 1 inference node each, same config otherwise, 10 steps, measured on an earlier revision of this branch:

	inline	by reference
Reward (mean of steps 1–10)	0.427	0.388
Mismatch KL	4.0e-4	4.0e-4
Step time (orchestrator / trainer)	381 s / 380 s	378 s / 375 s
Orchestrator peak RSS	14.5 GiB	2.7 GiB
Dropped micro batches	–	0

Per-step reward varies by ±0.07 at this batch size, so the reward gap is within about two standard errors; the trainer sees byte-identical arrays (unit test below) and the mismatch KL is unchanged.

Env-server replicas, 2 trainer + 4 inference nodes, 27 steps each. one env server vs one replica per inference node:

	one env server	replica per inference node
Env-server peak RSS	90.6 GiB on one node	25.6–25.9 GiB on each of 4 nodes
Peak memory used, inference nodes	220 GiB on the env-server node, 122–164 GiB elsewhere	144–158 GiB on every node
Orchestrator peak RSS	3.2 GiB	3.3 GiB
Payload fetch per trainer step (mean / max)	1.26 s / 2.30 s	0.74 s / 2.20 s
Reward (mean of steps 1–27)	0.398	0.415
Mismatch KL (mean)	3.9e-4	3.9e-4
Step time (orchestrator / trainer)	118 s / 117 s	124 s / 122 s
Dropped micro batches	0	0

The replica run continued to its 60-step limit and exited cleanly. Each replica's memory peaks early (≤ 25.9 GiB) and stays flat at ~22–23 GiB, so eager deletes keep up. Over 60 steps: no dropped micro batches, payload fetch 0.67 s mean, orchestrator peak RSS 3.3 GiB, reward mean 0.417. In both runs, most env-server memory is the math runtime's per-episode subprocesses and the pool workers; the kept arrays are about 10 GiB in total.

Transport, measured on an earlier revision of this branch: deltas into the orchestrator 8.8 GiB → 120 MiB per step, trainer batch 2.6 GiB → 78 MiB per step.

Tests: tests/unit/transports/test_payload.py: real verifiers traces go through the env-server hook, trace_to_samples, packing (truncation, padding, multi-sample bins) and the HTTP fetch, and each fetched micro batch holds exactly its branches' encoded arrays in packing order. Samples with and without routing never share a micro batch. Explicit heads, masks with sampler logprobs, plain masks and a sample without arrays survive truncation, mixed packing, padding and the trainer's tensor conversion. Also covered: the routing of a micro batch whose fetch failed (truncated sample included), a keep that fails midway, a branch whose arrays don't encode, the bind host (wildcard included), and the store and discard. tests/unit/orchestrator/test_train_sink_payloads.py: the train sink deletes exactly the arrays of filtered, dropped, stale, pruned and discarded samples and leaves only the shipped ones; the eval sink deletes those of eval episodes; the dispatcher deletes those of episodes that returned after their group was dropped. tests/unit/orchestrator/test_env_replicas.py: least-loaded routing over replicas. pytest -m "not gpu" tests/unit after merging main: 1003 passed; after the last fixes, the affected suites (orchestrator, transports, trainer, configs, eval) pass 498.

🤖 Generated with Claude Code

---

Note

High Risk
Changes the training data path (HTTP payload fetch, strict delete accounting, micro-batch drop on fetch failure) and multi-node env-server placement; misconfiguration or network issues can silently drop training signal or fail steps.

Overview
Training arrays (router replay, sampling masks, top-k heads) are kept on env servers and passed by reference instead of riding inline through the orchestrator and rollout transport. The env-server entrypoint installs PayloadHook, which encodes branch arrays over HTTP and stamps trace.info["prime_rl_payload"]; samples carry PayloadSlice, the packer threads payloads through micro batches, and trainer ranks fetch blobs before each step then release them after a DP-wide sync. Dropped or never-shipped arrays are deleted from train/eval sinks, the dispatcher on cancel, and when fetches fail micro batches are zeroed out of the loss (with payload/dropped_micro_batches).

Multi-node runs spread env-server load: orchestrator.env_server_replicas (auto-set to inference node count) launches one env server per inference node; the orchestrator connects to all replicas and routes episodes least-loaded first. SLURM templates move env-server startup off the orchestrator node; address files use <name>.<replica>.address.

Orchestrator batching no longer inlines slice/pack logic for routed experts and masks—that moves to transports/payload.py. Live monitor deltas keep full delta dicts; the sampling_support/size metric is removed.

^{Reviewed by Cursor Bugbot for commit d65bfc0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ef6d678. Configure here.}