ADd wan2 modular video to video

[lucasruan1618]

What does this PR do?

This PR adds video-to-video support for Wan2.1 through the Diffusers modular pipeline.

The new WanVideoToVideoBlocks workflow:

• encodes an input video into Wan VAE latents;
• selects the denoising schedule from strength;
• adds noise to the encoded video at the selected timestep;
• denoises with a Wan2.1 T2V transformer; and
• decodes the result back into video frames.

The implementation supports both official Wan2.1 T2V checkpoints:

• Wan-AI/Wan2.1-T2V-1.3B-Diffusers
• Wan-AI/Wan2.1-T2V-14B-Diffusers

Wan2.1 I2V checkpoints are not compatible because their transformers use a different image-conditioning input interface.

Example

from pathlib import Path

import torch

from diffusers import UniPCMultistepScheduler, WanVideoToVideoBlocks
from diffusers.utils import export_to_video, load_video


MODEL_ID = "Wan-AI/Wan2.1-T2V-14B-Diffusers"
INPUT_VIDEO_URL = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/hiker.mp4"
OUTPUT_PATH = Path("wan_modular_video_to_video_14b.mp4")

pipe = WanVideoToVideoBlocks().init_pipeline(MODEL_ID)
pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16})
pipe.update_components(
    scheduler=UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=3.0)
)
pipe.to("cuda")

video = load_video(INPUT_VIDEO_URL)
videos = pipe(
    video=video,
    prompt="Change the video style to anime, with hand-drawn cel shading, expressive details, and vivid colors",
    negative_prompt="Bright tones, overexposed, static, blurred details, subtitles, low quality",
    height=480,
    width=720,
    strength=0.7,
    num_inference_steps=25,
    generator=torch.Generator(device="cuda").manual_seed(42),
    output="videos",
)

export_to_video(videos[0], str(OUTPUT_PATH), fps=8)

For lower-memory GPUs, replace pipe.to("cuda") with pipe.enable_model_cpu_offload().

Testing

• Adds modular pipeline loading and memory tests.
• Adds output parity coverage against the native WanVideoToVideoPipeline.
• Manually validated Wan-AI/Wan2.1-T2V-14B-Diffusers at 480x720 on a 48 GB RTX 6000 Ada GPU.
• The manual run produced 49 frames at 8 FPS without CPU offloading.

Self-review

Verdict: READY

Resolved blocking issues

1. CUDA batch consistency. WanVideoVaeEncoderStep now forwards the pipeline generator to encode_vae_image, preserving per-sample VAE encoding when a generator list is supplied. The intermediate video_latents are now identical between single and batched execution. The test uses the existing Wan-specific 2e-3 tolerance for small CUDA differences introduced by later model and decoder operations. (src/diffusers/modular_pipelines/wan/encoders.py:621)
2. Invalid spatial dimensions. The VAE encoder now rejects heights and widths that are not divisible by 16 before preprocessing or denoising. Regression coverage checks invalid height and invalid width independently. (src/diffusers/modular_pipelines/wan/encoders.py:604)

Non-blocking issues

1. Clarify the pre-generated latents API. The assembled pipeline advertises optional latents, but video and video_latents remain required. A latent-only call fails with Required input 'video' is missing, while passing both performs an unnecessary VAE encode before ignoring video_latents. Confirm the intended modular API with the reviewer. (src/diffusers/modular_pipelines/wan/before_denoise.py:645)
2. Add user-facing documentation. This introduces public WanVideoToVideoBlocks and WanVideoToVideoModularPipeline APIs without a corresponding usage page or example under docs/ or examples/. Per .ai/review-rules.md, new public behavior should update usage documentation.
3. Use the standard mapping helper. _wan_v2v_map_fn always returns one class and can use _create_default_map_fn("WanVideoToVideoModularPipeline"), as prescribed for checkpoint variants in .ai/modular.md. (src/diffusers/modular_pipelines/modular_pipeline.py:110)

Dead-code analysis

Path	Status	Reason
wan/encoders.py: WanVideoVaeEncoderStep	Used	Included by WanVideoToVideoBlocks; produces video_latents.
wan/before_denoise.py: video-to-video timestep and latent steps	Used	Included by WanVideoToVideoCoreDenoiseStep; their outputs feed the Wan denoising loop.
wan/modular_blocks_wan_v2v.py: blocksets	Used	Exported publicly and selected by WanVideoToVideoModularPipeline.
modular_pipeline.py: _wan_v2v_map_fn	Used but redundant	Referenced by MODULAR_PIPELINE_MAPPING; the standard constant mapping helper can replace it.

No likely-dead inference path was found. This PR adds pipeline blocks rather than a new model, so there is no new model forward path to trace.

Validation performed

• pytest tests/modular_pipelines/wan/test_modular_pipeline_wan.py -k VideoToVideo -q: 22 passed, 1 skipped.
• Ruff on all ten changed source and test files: passed.
• utils/check_forward_call_docstrings.py: passed.
• utils/modular_auto_docstring.py on the new blockset: up to date.
• Public import smoke test and git diff --check: passed.

Fix before submitting: none. Leave for the actual review: agree on latent-only behavior and documentation scope.

Before submitting

• Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
  • Did you read the Coding with AI agents guide?
  • Did you run the self-review skill on the diff?
  • Did you share the final self-review notes in the PR description or a comment?
• Did you read the contributor guideline?
• Did you read our philosophy doc?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link if applicable.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Are you the author (or part of the team) of the model/pipeline?

Who can review?

• Pipelines and models: @yiyixuxu and @asomoza

[github-actions]

Hi @lucasruan1618, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an issue (or gets the no-issue-needed label), you can ignore this message — it stays here as a comment, but it no longer applies.