[Feature] Tiled Diffusion #1440

Activity

1. Green-Sky

   added

   enhancementNew feature or requestNew feature or request

   on Apr 18, 2026
2. stduhpf commented on Apr 18, 2026

   

   stduhpf

   on Apr 18, 2026

   Contributor

   More actions

   It's not too hard to implement in a naive way, I've done it a couple times with older versions of sd.cpp already, but I felt it had too many issues to be worth making a PR.

   For example when prompting for a normal portrait, you get things like that:

   

   (of course it's less noticable when doing img2img, but still there)

   Maybe there's a way to offset the positional encodings to mitigate these issues though, but I haven't figured that out.

   😄React with 😄4Reacted by Erik Scholz, Jeremy Lee Shields, agsoler and rene-descartes2021
3. Green-Sky commented on Apr 18, 2026

   

   Green-Sky

   on Apr 18, 2026

   ContributorAuthor

   More actions

   It's not too hard to implement in a naive way, I've done it a couple times with older versions of sd.cpp already, but I felt it had too many issues to be worth making a PR.

   q.q

   Maybe there's a way to offset the positional encodings to mitigate these issues though, but I haven't figured that out.

   Rope scaling? Maybe, but we would get "sd stretch-marks" again.

   ---

   I hacked together a bashscript using image-magick that tiles the image in 2x2 and upscales it. It (tries to) use a mask where there is overlap to keep the image aligned. I have only really tried sdxl yet.
   I have yet to achieve an actual improvement using sdxl. But it might just be the model or parameters.

   #! /usr/bin/env bash
   set -e
   
   tile_pad=128
   
   # input image size
   #W=1024
   W=$(magick identify -format '%w' $1)
   #H=1024
   H=$(magick identify -format '%h' $1)
   
   echo "W:$W H:$H tile_pad:$tile_pad"
   
   # copy working copy
   cp $1 tmp_input.png
   
   #scale=2
   
   ## HACK: first downscale to 50%, to hide vae/diffusion/quant artifacts
   ## actually better to do upscale first, then downscale back to og after
   #magick tmp_input.png -resize 50% tmp_input.png
   #(( W/=2 ))
   #(( H/=2 ))
   
   # 2x
   magick tmp_input.png -resize 200% tmp_input.png
   # w and h are now tile size (without pad)
   
   
   # splint into 2x2 tiles
   magick tmp_input.png -crop 2x2+$((tile_pad*2))+$((tile_pad*2))@ tmp_tile.png
   
   # TODO: add overlap in addition to padding
   
   # gen masks
   # keep in mind that w/h are already 0.5x
   
   # mask off right+bottom
   magick -size $((W))x$((H)) canvas:white -background black -gravity Southeast -splice $((tile_pad))x$((tile_pad)) tmp_tile_mask-0.png
   # mask off left+bottom
   magick -size $((W))x$((H)) canvas:white -background black -gravity Southwest -splice $((tile_pad))x$((tile_pad)) tmp_tile_mask-1.png
   # mask off right+top
   magick -size $((W))x$((H)) canvas:white -background black -gravity Northeast -splice $((tile_pad))x$((tile_pad)) tmp_tile_mask-2.png
   # mask off left+top
   magick -size $((W))x$((H)) canvas:white -background black -gravity Northwest -splice $((tile_pad))x$((tile_pad)) tmp_tile_mask-3.png
   
   echo "tiles and masks generated."
   
   sd(){
   	result/bin/sd-cli -m models/cyberrealisticXL_v80-q8_0.gguf --cfg-scale 3 --steps 36 --scheduler exponential --sampling-method dpm++2m -n "lowres, bad quality, worst quality, artistic error, wrong hand, wrong foot, bad anatomy, bad feet, bad reflection, bad perspective, bad arm, bad proportions, bad hands, bad leg, watermark, signature, dated, artist name, copyright name, sketch, censored, facial mark, scan" --diffusion-fa --offload-to-cpu --vae-tiling --vae-tile-size 3x3 --preview proj "$@"
   }
   
   prompt='An ultra-realistic 8k HDR editorial photograph. Sharp focus, 8K resolution, detailed texture.'
   strength="0.35"
   
   sd -p "$prompt" --strength $strength -i tmp_tile-0.png --mask tmp_tile_mask-0.png -o tmp_tile_diff-0.png
   echo "-----------------------"
   sd -p "$prompt" --strength $strength -i tmp_tile-1.png --mask tmp_tile_mask-1.png -o tmp_tile_diff-1.png
   echo "-----------------------"
   sd -p "$prompt" --strength $strength -i tmp_tile-2.png --mask tmp_tile_mask-2.png -o tmp_tile_diff-2.png
   echo "-----------------------"
   sd -p "$prompt" --strength $strength -i tmp_tile-3.png --mask tmp_tile_mask-3.png -o tmp_tile_diff-3.png
   echo "-----------------------"
   
   echo "diffusion done, merging..."
   
   # TODO: blended merging
   #magick \( tmp_tile_diff-0.png tmp_tile_diff-1.png +smush -$((tile_pad*2)) \) \( tmp_tile_diff-2.png tmp_tile_diff-3.png +smush -$((tile_pad*2)) \) -smush -$((tile_pad*2)) result.png
   
   # alt crop
   magick tmp_tile_diff-0.png -gravity Northwest -extent $((W))x$((H)) +repage tmp_tile_diff_crop-0.png
   magick tmp_tile_diff-1.png -gravity Northeast -extent $((W))x$((H)) +repage tmp_tile_diff_crop-1.png
   magick tmp_tile_diff-2.png -gravity Southwest -extent $((W))x$((H)) +repage tmp_tile_diff_crop-2.png
   magick tmp_tile_diff-3.png -gravity Southeast -extent $((W))x$((H)) +repage tmp_tile_diff_crop-3.png
   
   magick \( tmp_tile_diff_crop-0.png tmp_tile_diff_crop-1.png +smush 0 \) \( tmp_tile_diff_crop-2.png tmp_tile_diff_crop-3.png +smush 0 \) -smush 0 result.png
   

   Also there is no blending, so the seam is very visible.

   input:
   
   (ernie-image is great)

   output:
   
4. Green-Sky commented on Apr 28, 2026

   

   Green-Sky

   on Apr 28, 2026

   ContributorAuthor

   More actions

   Maybe there's a way to offset the positional encodings to mitigate these issues though, but I haven't figured that out.

   I'm more interested in this now. I was looking into zimage generation and you can tell the quality degrades below q8_0 significantly in the fine details. So tiling would be cool here.

   Also I noticed an annoyance. When you define an input image to renoise and then diffuse, it is an extra copy of the image in vram. So 512x768 gains an extra 271.12 MiB. At that point I might as well just generate at a higher resolution instead.
5. wbruna commented on Apr 28, 2026

   

   wbruna

   on Apr 28, 2026

   Last edited by wbruna

   Contributor

   More actions

   Also I noticed an annoyance. When you define an input image to renoise and then diffuse, it is an extra copy of the image in vram. So 512x768 gains an extra 271.12 MiB. At that point I might as well just generate at a higher resolution instead.

   @Green-Sky , see if this helps: #1465
6. Green-Sky commented on Apr 29, 2026

   

   Green-Sky

   on Apr 29, 2026

   ContributorAuthor

   More actions

   Also I noticed an annoyance. When you define an input image to renoise and then diffuse, it is an extra copy of the image in vram. So 512x768 gains an extra 271.12 MiB. At that point I might as well just generate at a higher resolution instead.

   @Green-Sky , see if this helps: #1465

   This helped, thanks!

   🎉React with 🎉1Reacted by Wagner Bruna
7. Green-Sky commented on May 1, 2026

   

   Green-Sky

   on May 1, 2026

   ContributorAuthor

   More actions

   Maybe there's a way to offset the positional encodings to mitigate these issues though, but I haven't figured that out.

   I'm more interested in this now. I was looking into zimage generation and you can tell the quality degrades below q8_0 significantly in the fine details. So tiling would be cool here.

   What if, instead of normal overlap at the tile border, we have like every second latent beyond the current tile. RoPE should encode the absolute position of those sparse latents, meaning it can see some things of the whole image.

   Basically Non-uniform pixel density. Also can be less then "every second latent". Also does not need to be uniform either, it can get less and less the further away it is from the current tile.
8. Green-Sky commented on May 7, 2026

   

   Green-Sky

   on May 7, 2026

   Last edited by Green-Sky

   ContributorAuthor

   More actions

   This is now less relevant, since the compute graph can now be split to limit concurrent vram usage thanks to #1476 🚀

   edit: in a way this is "tiled compute"