Fix HiDiffusion ratio overrides and SDXL T2 behavior

LICENSE-HiDiffusion.txt

@@ -1,7 +1,9 @@
 HiDiffusion - License notice
 
 Original project: https://github.com/megvii-research/HiDiffusion
-Vendored from:    https://github.com/monofy-org/HiDiffusion
+Intermediate fork: https://github.com/monofy-org/HiDiffusion
+Vendored implementation baseline:
+    https://github.com/Teriks/dgenerate/tree/d83b839033cc22c5101fb0f987bd4eb2de3d5d12/dgenerate/extras/hidiffusion
 Vendored under:   invokeai/backend/hidiffusion/
 
 ================================================================================
@@ -14,6 +16,18 @@ and at:
 
     https://www.apache.org/licenses/LICENSE-2.0
 
+The dgenerate distribution includes the following NOTICE for its HiDiffusion
+fork:
+
+    This code is vendored from: https://github.com/monofy-org/HiDiffusion
+
+    This repository is a fork which implements some fixes to HiDiffusion.
+
+    It further modifies HiDiffusion so that apply_hidiffusion accepts a
+    torch.Generator object, enabling deterministic images based on a seed.
+
+    HiDiffusion is here: https://github.com/megvii-research/HiDiffusion
+
 ================================================================================
 MODULE KEYS (BSD 3-Clause License)
 ================================================================================
@@ -59,7 +73,9 @@ LOCAL MODIFICATIONS
 The following changes were applied when integrating HiDiffusion into InvokeAI:
 
 * The code was vendored under the invokeai.backend.hidiffusion package.
-* apply_hidiffusion() was extended to accept a torch.Generator, enabling
-  deterministic image generation from a seed.
 * HiDiffusion patching was integrated with InvokeAI's model loading and
   generation lifecycle.
+* Automatic model- and resolution-specific ratios can be overridden per
+  invocation without mutating the vendored global preset tables.
+* Inpainting, guidance, adapter, and cached-model lifecycle behavior was
+  adapted to InvokeAI's denoising pipelines.

docs/src/content/docs/features/hidiffusion.mdx

@@ -17,6 +17,7 @@ Learn more: https://github.com/megvii-research/HiDiffusion
 3. In the **Advanced** grid, enable **HiDiffusion** and optionally adjust the two sub‑toggles and ratios:
    - **HiDiffusion: RAU‑Net**
    - **HiDiffusion: Window Attention**
+   - **HiDiffusion: Automatic Ratios**
    - **HiDiffusion: T1 Ratio**
    - **HiDiffusion: T2 Ratio**
 
@@ -28,12 +29,27 @@ Learn more: https://github.com/megvii-research/HiDiffusion
 
 - **HiDiffusion: Window Attention**: Enables windowed attention blocks. This can boost local texture/detail, but may slightly affect global coherence in some prompts.
 
-- **HiDiffusion: T1 Ratio**: Controls when HiDiffusion switches into its mid‑stage behavior. Lower values switch earlier; higher values preserve global structure longer.
+- **HiDiffusion: Automatic Ratios**: Uses HiDiffusion's original discrete model- and resolution-specific T1/T2 presets behind the scenes. SDXL uses the 2048 preset while either latent dimension is below the 4096 reference threshold, and the 4096 preset only when both dimensions reach it. The manual sliders retain their values while automatic ratios are enabled, so disabling automatic ratios restores the previous manual settings.
 
-- **HiDiffusion: T2 Ratio**: Controls when HiDiffusion switches into its late‑stage behavior. Higher values keep window attention active longer and can sharpen local detail.
+- **HiDiffusion: T1 Ratio**: Controls how long the primary RAU-Net stage remains active. At extreme resolutions, this is the later of the two RAU-Net cutoffs. Lower values switch earlier.
+
+- **HiDiffusion: T2 Ratio**: Manually controls the end of the additional early RAU-Net stage. For ordinary SDXL generation, this also controls when the primary stage begins. When automatic ratios are enabled, the original implementation uses a fixed `8 / 50` boundary for this early stage at the 2048 preset. Excessive manual values can reduce pose and background diversity or introduce artifacts. T2 cannot exceed T1 and does not control window attention.
+
+The T1/T2 input names are retained for API compatibility with the upstream implementation's code keys. T2 is the earlier boundary and T1 is the later boundary; do not interpret the names as chronological order. With automatic ratios at the SDXL 2048 preset, the additional stage runs for the first 8 of every 50 denoising steps, the primary stage then runs until T1, and the ordinary UNet is used afterward. Inpainting follows the upstream non-aggressive schedule and does not use the additional early stage.
+
+For img2img and inpainting, these phases retain their positions in the full denoising schedule. InvokeAI clips them to the portion selected by denoising strength instead of restarting HiDiffusion at the first remaining step. If denoising begins after T1, RAU-Net is not applied.
+
+### Manual ratio constraint
+
+When setting the ratios manually, **T2 must be less than or equal to T1**. T2 controls an additional resolution-reduction stage that depends on the primary T1 stage. Allowing T2 to remain active after T1 has ended would produce an invalid RAU-Net stage order and can cause severe structural artifacts.
+
+The UI limits the maximum T2 value to the current T1 value. If T1 is reduced below the current T2 value, the UI also reduces T2 to match it. Workflows or API requests that explicitly provide `T2 > T1` are rejected by the backend instead of being silently modified.
 
 ## Tips
 
-- Try **1536–2048 px** for the clearest benefits (SDXL).
+- Start around **1536–2048 px** for SDXL. Automatic ratios do not interpolate: the upstream 4096 preset is selected only when both image dimensions reach 4096 px (512 latent pixels). Test intermediate and non-square sizes explicitly before increasing either ratio manually.
+- Ordinary SDXL generation uses the upstream staged schedule: the additional RAU-Net path is active first, the primary path takes over at the early boundary, and the ordinary UNet takes over at T1. Inpainting does not use the initial aggressive stage.
+- HiDiffusion is applied only to the SDXL base denoise stage, not the optional refiner stage.
+- Window Attention falls back to ordinary global attention when a feature-map dimension cannot be divided into its 2x2 window layout. This avoids detail loss from resizing the feature map solely for window partitioning.
 - If results look worse, disable **Window Attention** first, then RAU‑Net.
 - Effects vary by scheduler and model; compare with the same seed for a fair test.

invokeai/app/invocations/denoise_latents.py

@@ -16,7 +16,7 @@
 from diffusers.schedulers.scheduling_tcd import TCDScheduler
 from diffusers.schedulers.scheduling_utils import SchedulerMixin as Scheduler
 from PIL import Image
-from pydantic import field_validator
+from pydantic import field_validator, model_validator
 from torchvision.transforms.functional import resize as tv_resize
 from transformers import CLIPVisionModelWithProjection
 
@@ -133,7 +133,7 @@ def get_scheduler(
     title="Denoise - SD1.5, SDXL",
     tags=["latents", "denoise", "txt2img", "t2i", "t2l", "img2img", "i2i", "l2l"],
     category="latents",
-    version="1.6.0",
+    version="1.7.0",
 )
 class DenoiseLatentsInvocation(BaseInvocation):
     """Denoises noisy latents to decodable images"""
@@ -209,15 +209,15 @@ class DenoiseLatentsInvocation(BaseInvocation):
         description=FieldDescriptions.hidiffusion_window_attn,
         title="HiDiffusion: Window Attention",
     )
-    hidiffusion_t1_ratio: float = InputField(
-        default=0.4,
+    hidiffusion_t1_ratio: Optional[float] = InputField(
+        default=None,
         ge=0,
         le=1,
         description=FieldDescriptions.hidiffusion_t1_ratio,
         title="HiDiffusion: T1 Ratio",
     )
-    hidiffusion_t2_ratio: float = InputField(
-        default=0.0,
+    hidiffusion_t2_ratio: Optional[float] = InputField(
+        default=None,
         ge=0,
         le=1,
         description=FieldDescriptions.hidiffusion_t2_ratio,
@@ -248,6 +248,16 @@ def ge_one(cls, v: Union[List[float], float]) -> Union[List[float], float]:
                 raise ValueError("cfg_scale must be greater than 1")
         return v
 
+    @model_validator(mode="after")
+    def validate_hidiffusion_ratio_order(self):
+        if (
+            self.hidiffusion_t1_ratio is not None
+            and self.hidiffusion_t2_ratio is not None
+            and self.hidiffusion_t2_ratio > self.hidiffusion_t1_ratio
+        ):
+            raise ValueError("HiDiffusion T2 ratio must be less than or equal to the T1 ratio")
+        return self
+
     @staticmethod
     def _get_text_embeddings_and_masks(
         cond_list: list[ConditioningField],
@@ -926,6 +936,9 @@ def step_callback(state: PipelineIntermediateState) -> None:
                     t1_ratio=self.hidiffusion_t1_ratio,
                     t2_ratio=self.hidiffusion_t2_ratio,
                     generator=torch.Generator(device="cpu").manual_seed(seed),
+                    is_inpainting_task=self.denoise_mask is not None,
+                    denoising_start=self.denoising_start,
+                    denoising_end=self.denoising_end,
                 )
             )
 
@@ -1157,6 +1170,9 @@ def _lora_loader() -> Iterator[PatchSpec]:
                     t1_ratio=self.hidiffusion_t1_ratio,
                     t2_ratio=self.hidiffusion_t2_ratio,
                     generator=torch.Generator(device="cpu").manual_seed(seed),
+                    is_inpainting_task=self.denoise_mask is not None,
+                    denoising_start=self.denoising_start,
+                    denoising_end=self.denoising_end,
                 )
                 if self.hidiffusion
                 else nullcontext()

invokeai/app/invocations/fields.py

@@ -152,8 +152,15 @@ class FieldDescriptions:
     hidiffusion = "Apply HiDiffusion (RAU-Net + MSW-MSA) for higher-resolution denoising"
     hidiffusion_raunet = "Apply HiDiffusion RAU-Net blocks"
     hidiffusion_window_attn = "Apply HiDiffusion window attention blocks"
-    hidiffusion_t1_ratio = "Override HiDiffusion early switch threshold (T1 ratio)"
-    hidiffusion_t2_ratio = "Override HiDiffusion late switch threshold (T2 ratio)"
+    hidiffusion_t1_ratio = (
+        "Override the duration of HiDiffusion's primary RAU-Net stage (upstream code key T1_ratio). "
+        "At extreme resolutions this is the later of the two RAU-Net cutoffs."
+    )
+    hidiffusion_t2_ratio = (
+        "Override the duration of HiDiffusion's additional extreme-resolution RAU-Net stage (upstream code key "
+        "T2_ratio). This is the earlier cutoff when both stages are active and cannot exceed T1; excessive values "
+        "can reduce composition diversity or introduce artifacts."
+    )
     scheduler = "Scheduler to use during inference"
     positive_cond = "Positive conditioning tensor"
     negative_cond = "Negative conditioning tensor"

invokeai/app/invocations/metadata_linked.py

@@ -624,7 +624,7 @@ class LatentsMetaOutput(LatentsOutput, MetadataOutput):
     title=f"{DenoiseLatentsInvocation.UIConfig.title} + Metadata",
     tags=["latents", "denoise", "txt2img", "t2i", "t2l", "img2img", "i2i", "l2l"],
     category="metadata",
-    version="1.2.0",
+    version="1.3.0",
 )
 class DenoiseLatentsMetaInvocation(DenoiseLatentsInvocation, WithMetadata):
     def invoke(self, context: InvocationContext) -> LatentsMetaOutput: