build: bump vllm to 0.29.0

[yfw]

What does this PR do ?

Bumps vLLM from 0.25.1 to 0.29.0 (released 2026-09-09), which moves the whole environment to torch 2.13.0 (torchvision 0.28.0, flashinfer 0.6.18, nvidia-cutlass-dsl 4.6.2, tilelang 0.1.12, llguidance 1.7.x). Supersedes #4058 (the 0.28.0 bump): every commit from that branch is carried over unchanged, plus the two 0.28 → 0.29 commits at the top. Draft until the nightlies on both SKUs have been compared against main.

0.28.0 → 0.29.0 delta that matters here (checked against the 0.29.0 wheel metadata and source):

• Dependencies: only flashinfer-python/-cubin/-jit-cache 0.6.16.post3 → 0.6.18 (cu130 wheels for both arches) and a new unconditional instanttensor>=0.1.9; transformers>=5.10.4 is already satisfied by the locked 5.12.1.
• Every API the 0.28 fixes touch (make_fp8_moe_kernel signature, is_quantized_kv_cache / kv_cache_uses_per_token_head_scales, RoutedExperts.load_weights name resolution, mm_device_do_normalize) is unchanged in 0.29.0.
• Six of the seven NeMo-RL source patches apply verbatim to the 0.29.0 files (dry-run against the released sources). The seventh, the RayExecutorV2 TCPStore port patch, is obsoleted by upstream [Bugfix] Avoid TCPStore port collision for co-located non-DP Ray engines vllm-project/vllm#53666 / #50969 — see below.
• 0.29.0 does not include [Bugfix] Make mm_device_do_normalize encoder-cudagraph safe vllm-project/vllm#55370 (encoder-cudagraph fix for on-device image normalization); we default that path off anyway.

Changes

Bullets are prefixed with the commit that made the change.

Dependencies (pyproject.toml / uv.lock)

• 83fa485 — vLLM 0.25.1 → 0.28.0 from PyPI (default cu130 wheels for both arches), torch 2.11.0 → 2.13.0, torchvision 0.26.0 → 0.28.0, flashinfer-python/-cubin/-jit-cache 0.6.13 → 0.6.16.post3, nvidia-cutlass-dsl[cu13] 4.5.2 → 4.6.2 in the vllm extra, tilelang <0.1.13, llguidance>=1.7.0,<1.8.0. flash-attn switches from the pinned cu13torch2.10 GitHub wheels to flash-attn==2.8.1 (sdist) because no 2.8.x wheel exists for torch ≥ 2.11.
• 2e0bdb8 — lockfile regenerated for the Docker uv version.
• 9d8f808 — nvidia-cutlass-dsl-libs-base is no longer excluded. The exclusion worked around cutlass#3259 (the 4.5.x libs-base and libs-cu13 wheels overwrite 180 shared files, 99 with different content). 4.6 split the wheels cleanly: libs-base ships the entire Python cutlass DSL and libs-cu13 only the _cutlass_ir.cu13 extension, zero overlapping files (verified by unzipping 4.6.0 and 4.6.2). flashinfer 0.6.16+'s gdn_prefill.py imports cutlass.cute at module load and vLLM 0.28+'s qwen_gdn_linear_attn.py imports gdn_prefill eagerly, so with the exclusion in place every Qwen3.5 worker and every Blackwell vLLM worker died with ModuleNotFoundError: No module named 'cutlass.cute' (17 nightly tests plus an external 64-node super run). The mcore extra's own nvidia-cutlass-dsl==4.5.2 pin moves to 4.6.2 so the training venv does not inherit the 4.5.x overlapping pair (flashinfer 0.6.8.post1 accepts >=4.4.2). The only fork still on 4.5.0 is trtllm (via tensorrt-llm), which is unusable on torch 2.13 anyway — see Known issues.
• 9d8f808 — flash-attn source build bounded with MAX_JOBS=4 via [tool.uv.extra-build-variables]. flash-attn's setup.py sizes its parallelism from the host's free memory (not the cgroup) and cpu_count()//2, with --threads 4 per nvcc. Every nemo-ci build attempt on both architectures died with ResourceExhausted after dozens of Killed "$CICC_PATH/cicc" (6 attempts, 22–63 kills each). 4×4 built cleanly on aarch64 (2h47m) and x86 (~2h, alongside TE and flash-mla).
• 9d8f808 — numpy override capped at <2.5 (resolves 2.4.6). vLLM 0.28+ pins numba==0.65.0, whose metadata requires numpy<2.5; the numpy>=2.1.0 override forced 2.5.1 past it and numba refused to import in the vLLM venv ("Numba needs NumPy 2.4 or less"). Not fatal by itself — vLLM's optional-module probe catches it — but it left a broken numba in the venv and polluted every trace.
• 9d8f808 / 7696b89 — ModelOpt c3b913b9 → 5dde396bd (Aug 4). The old vLLM plugin registers a quant module on fused_moe.layer.FusedMoE, which 0.28 removed. ModelOpt swallows that AttributeError inside import_plugin, leaving RowParallelLinear/ColumnParallelLinear/… registered but the module gone from sys.modules; NeMo-RL's explicit re-import in vllm_quant_patch.py re-runs the decorators and every fakequant nvfp4 test died with AssertionError: RowParallelLinear already registered!. 5dde396bd ported the plugin to RoutedExperts. It is pinned there deliberately and must not move past 2ded5aba3 (ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550): that commit introduces GroupedQuantizer, which breaks the Megatron fakequant DDP hooks and real-quant dist-ckpt loading (found when the first 0.28 pass ran on ModelOpt main, 613e5e8b).
• 576589f — vLLM 0.28.0 → 0.29.0, flashinfer-python/-cubin/-jit-cache 0.6.16.post3 → 0.6.18. instanttensor scoped to x86_64 via an override: 0.29 lists it unconditionally, but it is the optional load_format="instanttensor" safetensors loader, only publishes x86_64 wheels (cp310–cp314), and its sdist compiles libaio, liburing and boost with setuptools. vLLM imports it lazily inside instanttensor_weights_iterator and NeMo-RL never selects that load format, so the wheel is installed on x86_64 and skipped on aarch64 (GB200) instead of being built from source in the SBSA image.
• Lock diff beyond marker churn (cumulative vs main): numpy 2.5.1 → 2.4.6, cutlass-dsl 4.5.2 dropped, libs-base 4.5.0/4.6.0/4.6.2 and libs-core 4.6.0/4.6.2 added, modelopt 0.46.0.dev86 → 0.47.0.dev25+g5dde396bd, vllm 0.29.0, flashinfer 0.6.18, instanttensor 0.1.9 (x86_64 only). uv lock --check passes on the pristine parent and on the new lock; tests/unit/test_dependency_pins.py passes.

vLLM source patches / worker (patches.py, vllm_worker.py)

• 83fa485 — _patch_vllm_shm_broadcast_bind_retry re-targeted: 0.28+ binds the MessageQueue remote socket to port 0 directly (no probe/bind race any more) but ignores VLLM_PORT; the patch restores reserved-band selection with bind retries so engine sockets stay out of the ephemeral range. Still applies verbatim on 0.29.0.
• b863e80 — _patch_vllm_ray_executor_v2_tcpstore_port recognises the upstream fix. vLLM 0.29 ships [Bugfix] Avoid TCPStore port collision for co-located non-DP Ray engines vllm-project/vllm#53666 and #50969: RayWorkerV2.create_dist_init_method binds the torch.distributed TCPStore itself on a kernel-assigned port and holds the socket (self._dist_init_store = store) until init_process_group reuses it, so the probe/bind window the patch closed on 0.25–0.28 is gone and _select_tcpstore_port no longer exists. Without this change the patch found no anchor and logged "Engines spanning nodes may fail with EADDRINUSE" on every worker start. It now logs at info and leaves the file alone; the port-arithmetic unit tests skip when the installed vLLM carries the upstream marker, and a new test pins the no-warn/no-edit behaviour against a 0.29-style source. The reserved VLLM_PORT band still governs the MessageQueue and API-server ports.
• 8b6709e — VllmInternalWorkerExtension.synchronize_device renamed to synchronize_sparse_refit_device. vLLM 0.29 adds WorkerBase.synchronize_device ([Bugfix][DP] Synchronize the device on pause completion vllm-project/vllm#52914, pause-completion path) and WorkerBase.init_worker asserts that a worker extension never shadows a Worker attribute, so every vLLM engine on the first 0.29 nightly died at init with AssertionError: Worker class <vllm.v1.worker.gpu_worker.Worker> already has an attribute synchronize_device, which conflicts with the worker extension class. The extension method only drained peers before the sparse-delta refit removed staged batch files; its RPC caller and test follow the rename. An AST scan of both extension classes against 0.29's Worker/WorkerBase shows no other overlap, and a new vllm-marked unit test mirrors vLLM's collision predicate so the next upstream name clash fails in CI rather than on a GPU job.
• da2f1dc — ErrorResponse imported from vllm.entrypoints.serve.engine.protocol. vLLM 0.29 moved openai/engine/protocol.py out of the openai package ([Frontend] Move engine/protocol.py out openai folder vllm-project/vllm#54492); VllmAsyncGenerationWorker.post_init_async imported it from the old path when bringing up the OpenAI-compatible server, so every recipe that serves generation over HTTP (Gym / swe1 recipes: qwen3_30ba3b_thinking_swe1_16n8g_megatron_cp2_r3_async_gym, qwen3_1_7b_1n8g_megatron_super_swe1) died on pass 3 with ModuleNotFoundError: No module named 'vllm.entrypoints.openai.engine'. Every other name the async worker imports from vLLM still resolves in 0.29.0 and the serving classes it subclasses changed only for this move. A new vllm-marked test walks nemo_rl/models/generation/vllm with ast, collects all 86 from vllm… import statements (mostly lazy, method-local imports no unit test executes) and resolves them against the installed vLLM.
• 9a4a747 — Tied-embedding aliases dropped before load_weights. vLLM 0.29's AutoWeightsLoader (Fix weight tying vllm-project/vllm#51665) skips a tied alias such as lm_head.weight and then asserts that its canonical model.embed_tokens.weight was loaded in the same load_weights call. Refit streams weights in transport-sized batches, so the two routinely land in different calls and every tied-embedding model refit from a DTensor/automodel policy died at its first weight update on pass 3 (ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_noncolocated{,_async}, dapo_gemma4_e2b_it_1n8g_fsdp2_automodel; the llama3.2-1b DTensor recipes were still running) with ValueError: 'lm_head.weight' was skipped because it is tied to 'model.embed_tokens.weight' … was not found in the checkpoint. The alias never loads anything, so it is now dropped up front using vLLM's own _get_tied_embedding_params and the model's hf_to_vllm_mapper (so Gemma-style renamed parameters are covered), on both the batched refit and the native reload_weights path; the MTP drafter still sees the unfiltered stream. Unit-tested; a vllm-marked test pins the helper's existence.
• f6993c8 — mm_device_do_normalize defaults to False. vLLM 0.28+ (#50411) calls the HF image processor with do_rescale=False, do_normalize=False and re-applies both on the GPU in the vision tower's dtype. The policy normalizes the same images on the CPU in fp32 through the same processor, and the nightly probability-error checks assume identical inputs on both sides. Verified on CPU with transformers 5.12.1: the fused path is exact in fp32 but differs by up to 0.019 (≈1.7 % of the pixel std) in bf16. On the second 0.28 pass this brought the Qwen2.5-VL clevr DTensor recipe back to main-level metrics (token_mult_prob_error 1.019, gen_kl 0.0008, vs 198 / 0.14 before). Opt back in with policy.generation.vllm_kwargs.mm_device_do_normalize=true.

fp8 generation (quantization/fp8.py)

• 1734158 — make_fp8_moe_kernel lost its layer kwarg in 0.28 (0.25 forwarded it only to the FlashInfer TRTLLM experts). Both call sites updated; the unit test's expected kwargs follow.
• 1734158 — process_weights_after_loading_kv re-mirrored on 0.28's BaseKVCacheMethod.process_weights_after_loading: the attention layer no longer has calculate_kv_scales (dynamic scales are a KV-cache dtype now, kv_cache_uses_per_token_head_scales), the static-scale branch keys off is_quantized_kv_cache, and the host _k_scale_cpu/_v_scale_cpu copies are refreshed on refit. Parameters are still kept so refit can update them. Unchanged in 0.29.0.

ModelOpt fakequant refit (nemo_rl/modelopt/models/generation/)

• cfcdccb — per-expert input_quantizer._amax routed around vLLM's expert loader. The Megatron side exports one input_quantizer._amax per expert projection (…experts.16.up_proj.input_quantizer._amax); the vLLM ModelOpt MoE module owns a single fused quantizer per projection group (w13_input_quantizer / w2_input_quantizer). vLLM 0.25 loaded these through the model-level parameter dict, which NeMo-RL patches to include quantizer buffers. 0.28+'s AutoWeightsLoader hands every experts.* name to RoutedExperts.load_weights, which rewrites it with the expert mapping and resolves the result with a single getattr — a dotted buffer name cannot resolve that way and every fakequant nvfp4 MoE refit died with AttributeError: Layer … has no parameter 'w13_input_quantizer._amax'. New vllm_quant_moe_amax.route_moe_input_quantizer_amax applies the same rewrite vLLM does, walks the dotted path on the expert module, fans the values in with max, and hands the remaining weights to vLLM's loader. Unit-tested (3 tests); the loader contract is identical in 0.29.0.

Megatron (nemo_rl/models/megatron/)

• b549eab — packed HybridModel MTP forwards get placeholder position IDs. Megatron-LM f96158a9f (#6912, pulled in by the Megatron-Bridge bump ci: Bump Megatron-Bridge to 5ed9799  #4022 on Sept 6) asserts mtp_inputs.input_ids is not None for packed sequences whose position_ids are None; this failed on main's nightlies Sept 7–9 too (grpo-nemotron3-super-120BA12B-16n8g-megatron and the external 64-node super run). Not a vLLM issue; carried here because the nightlies cannot be compared without it. Regression tests included.

Distributed (collectives.py)

• 3558f0a — all_gather_into_tensor → all_gather_single (torch 2.13 API).

Docker (docker/Dockerfile, 3rdparty/TensorRT-LLM-workspace/_backend.py)

• 168aaac — TRT-LLM wheel mirrored into the hermetic image layer. nemo-ci runs docker buildx rm/create around every build job, so the trtllm-wheel-cache-* cache mount that the hermetic stage compiles the wheel into lives for exactly one build, while the hermetic layer is shared through the registry cache. A rebuild that changes only NeMo-RL source (pass 2 below) therefore reused the layer on a fresh builder, re-executed the release stage, and died on every runner at [release 5/8] with TRT-LLM cached wheel is required but was not found … TRTLLM_REQUIRE_CACHED_WHEEL=1 (4 images × 3 attempts, identical). The hermetic stage now sets TRTLLM_WHEEL_CACHE_MIRROR_DIR=/opt/trtllm_wheels so the backend also writes the content-addressed wheel into the layer, the release stage rsyncs that mirror back into the (possibly empty) mount before the trtllm venv prefetch, and the backend skips the mirror copy when the file already exists so the release layer no longer duplicates the wheel. Final image content is unchanged (/opt/trtllm_wheels already held this wheel). Costs one TRT-LLM recompile on the next build; afterwards source-only rebuilds work on any runner. Independent of the vLLM bump and can be split out if preferred.

Known issues not fixed here

Issue	Tests	Status / proposed fix
sglang-kernel ABI vs torch 2.13 — sgl_kernel/.../common_ops.abi3.so: undefined symbol _ZNK2at10TensorBase14const_data_ptr…	all 6 sglang recipes (3 per SKU)	sglang-miles (3003d70f) pins torch==2.11.0 and sglang-kernel==0.4.5; sglang 0.5.19 moved to torch==2.13.0 and sglang-kernel==0.4.6.post1. Needs a sglang-miles rebase; a speculative pin bump is not safe because a failed uv sync --extra sglang fails the entire image build. Note main's sglang tests also fail today on torch.multiprocessing.reductions._rebuild_cuda_tensor_original.
TRT-LLM ABI vs torch 2.13 — tensorrt_llm/libs/libth_common.so: undefined symbol _ZNR5torch7Library4_def… in TrtllmAsyncGenerationWorker.__init__	both GB200 trtllm recipes	No TRT-LLM release supports torch 2.13.0 (1.3.0rc26 requires torch<=2.13.0a0).
w4a16 NVFP4 real-quant refit — ModelOpt layerwise reload is incomplete for 23 layer(s): …routed_experts: 134701312/179601664 elements	grpo-nanov3-30ba3b-4n4g-megatron-qa-nvfp4-w4a16-real (GB200)	Exactly ¾ of each expert tensor arrives, so the streamed w4a16 layout no longer matches 0.28+'s RoutedExperts. The w4a4 sibling passes. Needs an audit of the real-quant refit against the new MoE weight layout. main Sept 8 ran to completion and failed only on golden metrics.
Qwen3-Omni audio-MCQ hang — vlm_grpo-qwen3-omni-30ba3b-audiomcq-4n8g-megatron.v1, gloo recv timeout (1800 s) inside VllmGenerationWorker.generate	1	Reproduced on both 0.28 passes; passes on main. 0.28+ runs external-DP engines in lockstep (execute_dummy_batch on idle ranks), and this is the only recipe where one DP rank finishes a step with no work while another still generates. Not yet root-caused.
GB200 DTensor step-time regression — gemma3 1n4g +12 %, llama 1n4g a few %	GB200 DTensor recipes	x86 step times unchanged; not yet profiled.
nixl-cu13 1.3.0 nixl_ep extension built against torch 2.11 — nixl_ep_cpp.cpython-313-…so: undefined symbol _ZN3c104impl3cow23materialize_cow_storage…	none	Non-fatal: vLLM's has_nixl_ep() probe catches it and disables the optional nixl_ep all2all backend, but the traceback appears in every vLLM worker log and nemo-ci's error extractor picks it as the "cause" of unrelated failures. Bump nixl to a torch-2.13 build in a follow-up.
grpo-nemotron3-super-120BA12B-8n4g-megatron (GB200) — Unquantized FlashInfer TRTLLM refit does not yet support a co-trained MTP drafter	1	Pre-existing (main Sept 8 identical). Blackwell picks the FlashInfer TRTLLM BF16 MoE backend, and #3545 rejects co-trained MTP on that path. Either add moe_backend: triton to the recipe's vllm_kwargs (supported since 0.25) or extend _weight_update_lifecycle to include the drafter's TRTLLM modules in the layerwise reload. Separate PR.
New in 0.29, to watch — FlashInfer all-reduce is now the default TP all-reduce path (VLLM_ALLREDUCE_USE_FLASHINFER default 0 → 1 in envs.py)	TP > 1 recipes	Nothing observed yet (no 0.29 run so far). Opt-out is VLLM_ALLREDUCE_USE_FLASHINFER=0 in vllm_cfg.env_vars if a TP recipe regresses.

Notes for reviewers

• The cutlass change removes a [tool.uv] exclusion that was load-bearing at 4.5.x. I verified the 4.5.2 overlap (180 files) and the 4.6.x non-overlap (0 files) directly from the PyPI wheels, and uv pip install of the overlapping 4.5.2 pair does not error — it silently picks a winner per file — which is why the mcore pin had to move to 4.6.2 as well.
• instanttensor is the one place this PR relies on a uv override marker to drop a dependency on one arch. If someone wants load_format="instanttensor" on GB200 later, the sdist would need an aarch64 wheel or a toolchain in the SBSA image.
• mm_device_do_normalize=False changes a vLLM default for every multimodal recipe. It restores the 0.25 behaviour byte-for-byte on the processor side; the cost is the CPU normalize vLLM's PR was avoiding.
• ModelOpt is pinned to a specific commit, not main, with a "do not move past" note in pyproject.toml; the reason (GroupedQuantizer) is in the 7696b89 message.
• Unit tests run here: tests/unit/test_dependency_pins.py, test_vllm_quant_moe_amax.py, test_vllm_modelopt_real_quant_config.py, the new test_vllm_tcpstore_port.py::test_patch_recognizes_upstream_fix_and_leaves_source_alone (executed outside the vllm marker filter). test_vllm_fp8_quantization.py and the vllm-marked port tests need the rebuilt image (no vLLM in the host venv).

Test plan

1. Full nightly suite on both SKUs against the PR head, compared per test with the same-day main nightly. Results table will be appended in the format used on build: bump vllm to 0.28.0 #4058 / build: bump vllm 0.20.0 → 0.25.1 #3280 once the pipelines finish.
2. Root-cause every PR-only failure; the 0.28 pass-2 classification is the starting point.
3. DSv3 perf and the disabled.txt qwen3.5 entries: not yet run.

Results — nightly

Pass 1 — PR head b863e80, nemo-ci#67220071 (tests: H100 67285306, GB200 67285294). Images built uncached in 142 min (x86) / 177 min (sbsa); traces confirm vllm 0.29.0, flashinfer 0.6.18, torch 2.13.0, ModelOpt 0.47.0.dev25+g5dde396bd, instanttensor present on x86 only and no source build on sbsa. Every vLLM-based recipe failed at engine init on the synchronize_device assertion above (~12 min per job), so this pass only yields results for the non-generation recipes (SFT/DPO/RM/distillation-without-vLLM) and confirms the two known ABI failures (sglang-kernel). 17 H100 jobs additionally bounced at git checkout on the corrupted Ag1sxjDZu/10 / 5wSiZLJL2/7 workspaces and were retried (some up to three times). Once the assertion was confirmed systematic, the 113 remaining vLLM-generation jobs in this pass were cancelled to free the clusters for pass 2; the 31 non-generation and sglang jobs (SFT/DPO, both SKUs) were left to finish and count as this PR's results for those recipes.

Pass 2 — PR head 8b6709e (adds the synchronize_device rename), nemo-ci#67287391. Never produced an image: all four builds failed at [release 5/8] on the missing TRT-LLM cached wheel described under Docker above (12 attempts across 9 different runners). Cancelled.

Pass 3 — PR head 168aaac (adds the wheel mirror), nemo-ci#67293195. The hermetic TRT-LLM step recompiles once (≈80 min on arm in pass 1); the uv layers stay cached. Compared against the Sept 9 main nightly nemo-ci#66919718 (H100 68 pass / 32 fail, GB200 26 pass / 14 fail) and the 0.28 PR's second pass nemo-ci#67131233 (H100 85 / 15, GB200 25 / 15).

Pass 1's kept non-generation jobs so far: 15 H100 + 2 GB200 passed; failures are all pre-existing golden misses that fail identically on main (dpo_qwen2_5_math7b_1n8g_megatron_fused_linear_logprobs accuracy 0.406 < 0.5, dpo_nanov3_30B3AB_1n8g_fsdp8ep8_automodel_v2 loss/step-time, dpo_nanov3_30B3AB_1n4g_fsdp4ep4_automodel step-time, dpo_mistral_nemo_instruct_2407_1n8g_fsdp2tp8_actckpt_long loss 0.7087 < 0.70) plus the known sglang-kernel ABI failures on both SKUs.

Pass 3 interim (02:50 PT): vLLM engines start (extension injected cleanly), eagle3, r3_async_single_controller and the SFT yarn recipe already pass. Two more 0.29 regressions found and fixed: da2f1dc (ErrorResponse import, Gym/swe1/mopd recipes) and 9a4a747 (tied-embedding alias assertion at refit, every tied-embedding DTensor/automodel recipe). Other failures so far: offline HF cache misses (nanov3.5, qwen3-8-27b), the sglang ABI break, pre-existing moonlight / qwen3.5-automodel-ep16 failures. gemma3_1b_it_1n8g_fsdp2tp1 and nanov3_30BA3B_2n8g_megatron_lora failed early without a driver excerpt in the job trace and are still being root-caused.

Per-test table will be appended when pass 3's test children finish.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.