Skip to content

fix(qwen3.5): freeze vision modules in text-only recipes - #4077

Draft
seonjinn wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
seonjinn:sna/freeze-qwen35-vision-text-recipes-20260910
Draft

fix(qwen3.5): freeze vision modules in text-only recipes#4077
seonjinn wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
seonjinn:sna/freeze-qwen35-vision-text-recipes-20260910

Conversation

@seonjinn

@seonjinn seonjinn commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Problem

Qwen3.5 text-only Megatron recipes instantiate the vision modules but did not freeze them in the shared base recipe. Text batches never execute those modules, so DDP saw only 307 of 640 parameters in one gradient bucket and failed when the next training step reset the bucket.

Change

Freeze the vision model and projection in the shared Qwen3.5 text recipe before DDP and optimizer setup. BF16, FP8, and MXFP8 child recipes now inherit the same trainable parameter set. Remove the duplicate FP8-only settings.

Test

  • Regression test before the fix: 1 failed, 2 passed (freeze_config missing)
  • GB200 regression test after the fix: 4 passed
  • Qwen3.5 Async NCCL Reshard, MXFP8 training (fp8_param=true) + MXFP8 rollout, FlashInfer TRTLLM: 2/2 steps completed; generation KL error was 0.0012 at both steps
  • 20-step follow-up is queued

W&B: https://wandb.ai/nvidia/nemo-rl-mxfp8-training/runs/oz68ga7p

Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: seonjinn <sna@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant