fix(minimax-h3): keep Qwen3VL vision pos-embed interpolation on the compute device

[lstein]

The crash (rig queue item 10352)

modeling_qwen3_vl.py:783: hidden_states = hidden_states + pos_embeds
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Hit once when two simultaneous MiniMax H3 jobs ran and a Ref2VA reference image sent a prompt through the text encoder's vision tower; never reproduced since.

Root cause

Qwen3VLVisionModel.fast_pos_embed_interpolate (transformers 5.5.4, line 707) derives its working device from self.pos_embed.weight.device and builds all of its tensors there. Under partial loading, nn.Embedding is autocast-wrapped (CustomEmbedding), so that weight may legitimately sit on the CPU while the model computes on CUDA. The per-op autocast casts weights to each input's device — which keeps every normal layer on the compute device — but this function manufactures its own inputs on the weight's device, so the whole interpolation lands on the CPU and the add in forward dies.

That's why it's intermittent: it needs (a) the 27 GB TE partial-loaded under VRAM pressure with visual.pos_embed.weight specifically among the CPU-resident keys, and (b) an image in the prompt. A fully loaded TE can never hit it (autocast disabled, weight resident). Same anti-pattern class as diffusers invoke-ai#9373 ("don't infer the compute device from current param residency").

Fix

Minimal, idempotent, class-level patch (mirroring contiguous_attention), installed by both TE load paths:

• wrapped forward records the compute device from its hidden_states input (the very tensor pos_embeds is added to);
• fast_pos_embed_interpolate runs unchanged, and only its result is moved to the recorded device.

Weight resident → exact upstream behavior (move is a no-op). Weight on CPU → the small interpolation (a few thousand rows) runs on CPU and one tensor crosses the bus.

Testing

• 5 new tests (baseline equality on CPU, device recording, result move validated against meta, unset-attr behavior, idempotence); full tests/backend/minimax_h3/ suite passes (159).
• Adversarial fresh-context review: clean. Its two advisories: (1) rot_pos_emb's inv_freq looks like a sibling pattern but is a non-persistent buffer, which _move_non_persistent_buffers_to_device() unconditionally pins to the compute device on every load pass — not offloadable, safe; (2) Krea2's Qwen3VLEncoderLoader doesn't install the patch, but its encoder path is text-only today (vision tower never runs) — if Krea2 ever grows image prompts, its loader should call apply_qwen3vl_vision_pos_embed_device_patch() too.

Rig test

Hard to trigger on demand (needs the TE partial-loaded with that specific weight off-device). Running image-conditioned Ref2VA prompts while both GPUs are busy is the closest recipe; mostly this rides along until the stars align again — at which point it should just work instead of failing the job.

🤖 Generated with Claude Code

https://claude.ai/code/session_014xFbHnmFTLsHE9e1PAdvme