[faresobeid]

What

Under disaggregated P/D, stitch by-handle payload segments the same way the router stitches inline routed_experts.

prime-rl's /inference/v1/generate can return per-choice payload: [{field, file, offset, pos, rows, dtype, shape}, ...] instead of inline routed_experts / sampling_mask. The segments point at rows in a shared file. Without this PR, the P/D merge leaves them unchanged, so the response has no prefill routing for the prompt.

Merge rule (merge_routed_payload_in_json, called from merge_routed_experts_for_pd next to the inline merge)

A routed_experts segment covers the absolute positions [pos, pos + rows). The prime-rl writer anchors segments at the end: pos = prompt_len + len(completion_token_ids) - 1 - rows. On vLLM 0.31 P/D a decode instance emits rows from its first forward (prompt_len - 1), so its rows overlap the end of prefill's.

• Only field == "routed_experts" segments are touched. Other fields, such as sampling_mask, keep the decode response's segments, as the inline path does.
• Prefill routed segments come from prefill choices[0] and are kept as-is. Every decode choice gets them.
• cut = max(pos + rows) over the prefill routed segments.
• Decode routed segments keep only rows at positions >= cut:
  • a segment with pos + rows <= cut is dropped;
  • a segment with pos < cut is clipped: offset += (cut - pos) * row_bytes, rows -= cut - pos, pos = cut, where row_bytes = itemsize(dtype) * prod(shape).
• Merged payload = prefill routed segments, then the decode segments in their original order.
• Decode has routed segments but prefill has none: error, like the inline merge.
• Neither side has routed segments: no change. The router never reads payload files.
• prefill_has_routed_experts also counts routed payload segments, so a streaming P/D request with them is refused, as for inline routing.

Dependencies

• feat: pass router-replay ids and sampling masks by handle on multi-node runs prime-rl#3902: the by-handle writer, with the end-anchored segment positions above.
• feat: pass router-replay ids and sampling masks by handle under disaggregated P/D prime-rl#4156 turns by-handle on under P/D; it needs a release with this PR and a vllm-router pin bump.

Tests

• Unit tests in routed_experts_merge.rs: prepend + drop of decode rows before cut + sampling_mask pass-through, and the no-op case.
• cargo test, cargo +1.95.0 clippy --all-targets -- -D warnings, cargo +1.95.0 fmt --check.
• GPU (prime-rl#4156): 1P1D NIXL, Qwen3-30B-A3B, vLLM 0.32, wheel built from this branch merged with main. 80 two-turn episodes (160 requests per mode) (concurrency 1 and 10, decode preemption, uint8 and uint16 rows): by-handle routing and masks are bitwise equal to the inline P/D merge and to a colocated server.

Note

Add payload-segment merging for by-handle routed_experts in PD router

• Adds merge_routed_payload_in_json in routed_experts_merge.rs. It copies prefill routed_experts payload segments into every decode choice, drops decode rows already covered by prefill, and trims and re-offsets partially overlapping decode segments using dtype and shape metadata.
• Extends prefill_has_routed_experts to detect routed_experts entries in the first choice's payload segment list, not just inline data.
• VllmPDRouter::merge_routed_experts_for_pd in vllm_pd_router.rs now runs both the inline and payload-segment merges, combining their results and propagating either error.
• Adds tests for segment merge, no-op when no routed payload exists, and error on overflowing metadata.
• Risk: invalid or incompatible payload metadata (negative or non-integer pos/rows/offset, unsupported dtypes, overflow in segment_end or segment_row_bytes) now returns an error from merge_routed_experts_for_pd instead of being ignored.

^{Macroscope summarized 38b7135.}

---

Note

Medium Risk
Changes P/D response assembly for inference generate payloads; bad or mismatched segment metadata now fails the merge instead of returning an unmerged decode body.

Overview
Adds by-handle P/D stitching for routed_experts when responses use choices[].payload file segments instead of inline routed_experts.

New merge_routed_payload_in_json prepends prefill routed_experts segments onto each decode choice, computes a cut from prefill coverage, and drops or trims overlapping decode segments (adjusting offset, pos, and rows from dtype/shape). Non-routed_experts payload fields stay on the decode side. prefill_has_routed_experts now treats routed payload segments like inline data (including streaming refusal). VllmPDRouter::merge_routed_experts_for_pd runs the existing inline merge and this payload merge, OR-ing success and surfacing errors from either path.

Unit tests cover overlap trimming, no-op when no routed segments, and overflow on segment metadata.

^{Reviewed by Cursor Bugbot for commit 38b7135. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This adds a new production integration for file-backed routed-experts payloads, including segment clipping, offset rewriting, and altered streaming/error behavior in P/D responses. Existing paths are preserved, but the external payload contract and customer-visible response assembly warrant human review.

Notes:

• Diff unchanged. Approvability was decided on eligibility alone.

No code changes detected at 38b7135. Prior analysis still applies.

^{You can add or adjust custom eligibility rules. Learn more.}