fix(megatron): preserve HybridEP packed-input alignment

[seonjinn]

What

Guarantee aligned token counts for HybridEP when sequence packing is enabled.

Why

Packed lengths can differ across expert-parallel ranks. Without one alignment owner, HybridEP collectives can consume different token counts and silently corrupt policy logprobs or hang.

How

• Keep the existing one-time NeMo-RL pre-padding path when explicitly enabled.
• Otherwise enable Megatron-Core per-layer uneven-input padding automatically.
• Support both Megatron-Core padding field names and fail during setup if neither exists.

Test

• Lyris GB200 targeted MCore tests: 5 passed.
• Qwen3.5-35B-A3B, 8 nodes x 4 GB200, Async/NCCL Reshard, BF16 training + MXFP8 FlashInfer TRTLLM rollout, HybridEP: 20/20 steps completed.
• Generation KL error: mean 0.00136, range 0.00095-0.00215.
• Steady state, steps 2-20: E2E 19.07 s, generation 1.98 s, policy training 6.74 s, policy/reference logprobs 5.82 s, refit 1.10 s.
• W&B run

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[seonjinn]

Root-cause validation completed on GB200.

The broken HybridEP run used sequence packing without either NeMo-RL pre-padding or Megatron-Core uneven-input padding. Its generation KL error was already 0.8066 at step 1, before any optimizer update or refit, and averaged 0.7985. This isolates the failure to training-side policy logprobs rather than the refit path.

With this change, the same Qwen3.5-35B-A3B Async/NCCL Reshard configuration completed 20/20 steps. Generation KL error averaged 0.00136 with a 0.00095-0.00215 range.

• Broken run
• Fixed 20-step run

[seonjinn]

Self-review completed on head 1c8cfa42.

• Reviewed the HybridEP padding decision, setup integration, unit coverage, and documentation.
• Confirmed that exactly one component owns packed-input alignment: explicit NeMo-RL pre-padding when enabled, otherwise Megatron-Core per-layer padding.
• Confirmed compatibility with both known Megatron-Core field names and a fail-loud setup error when neither field exists.
• Targeted GB200 tests passed, and the Qwen3.5 Async/NCCL Reshard run completed 20/20 steps with generation KL error mean 0.00136.
• No blocking correctness, compatibility, security, or deadlock issues were found.

[seonjinn]

/ok to test 5caf656