[faresobeid]

Stacked on #3902 (by-handle payloads). Parked: only needed for TRR / score centering by handle. Not for review now.

What

Each by-handle per-token field is one line in PAYLOAD_FIELDS (transports/payload.py), so Total Router Recall weights and score-centering mask logprobs can travel by handle when those features come back, with no further reader or packing changes:

PAYLOAD_FIELDS = {
    "routed_experts": PayloadField(0, "inputs"),
    "routed_expert_weights": PayloadField(0.0, "inputs"),
    "sampling_mask": PayloadField(-1, "labels", ragged=True),
    "sampling_mask_logprobs": PayloadField(-math.inf, "sequence", ragged=True),
}

Each entry gives the fill value and the window a trainer rank reads on text micro batches: its CP chunk of inputs (routing), its CP chunk shifted onto labels (masks), or the whole sequence at the tokens' own positions (values the loss reads after the CP gather). Multimodal micro batches read every field whole, as before.

• Ragged fields: PayloadField.ragged marks sampling_mask and sampling_mask_logprobs. feat: pass router-replay ids and sampling masks by handle on multi-node runs #3902 stores masks as CSR; this generalizes that layout to any ragged field (int32 values for an int fill, float32 for a float fill).
• Reader (read_field / data._read_payload, called from prepare_micro_batch): dtype-aware (float fields as float32, int fields as int32) and window-driven. Each tensor comes back under the TensorMicroBatch key of the same name and replaces only that inline field.
• Orchestrator: the last-token row repeat and the drop-on-gap in trajectories._sample_payload apply to every inputs field; dummy batches keep only inputs fields; prepare_sample's cut-out placeholder covers every non-inputs field.

When the features come back

• Score centering (feat: exact score centering over the sampling mask (arXiv:2609.20807) #3794): it unpacks (ids, logprobs) from the mask in _PackedOutputs. The by-handle writer then adds one line next to the mask:

  arrays.append(("sampling_mask_logprobs", prompt_len, (mask.counts, logprobs)))

  The trainer gets sampling_mask_logprobs over the whole sequence; positions without logprobs are -inf (the inline path uses -9999; both give zero centering weight).
• TRR (no open prime-rl PR; feat(rl): add experimental Total Router Recall #3509 is closed): write the fp32 weights as routed_expert_weights next to uint8 ids (5 B per layer and top-k slot instead of 8 B for int32 [ids | weight bits]), and concatenate them back into the layout MoE.forward expects after the read. Under P/D the router would stitch this field like routed_experts.

Tests

• test_payload_by_handle_matches_inline now also covers routing weights (against TRR's inline int32 [ids | weight bits] layout) and CSR mask logprobs (whole-sequence window), with CP 1 and 2. test_inline_and_by_handle_samples_keep_their_masks (from feat: pass router-replay ids and sampling masks by handle on multi-node runs #3902) still passes on the table-driven reader.
• pytest -m "not gpu" tests/unit/orchestrator tests/unit/test_configs.py tests/unit/inference tests/unit/train tests/unit/utils with feat: pass router-replay ids and sampling masks by handle on multi-node runs #3902's pinned verifiers/renderers on the path: 586 passed. ruff check, ruff format --check: clean.