feat: cross-stage offload modes and layer-streaming for low-VRAM GPUs - #1477

Closed

fszontagh wants to merge 100 commits into

leejet:masterfrom

fszontagh:feature/vram-offloading-v2

Commits on Mar 4, 2026

feat: add dynamic VRAM offloading for large models
fszontagh
committed
6bc4025
feat(cli): add --offload-mode option for dynamic VRAM offloading
fszontagh
committed
d2f2836
fix: prevent SEGV when GPU reload fails during offload
fszontagh
committed
ef40d61
fix: offload cond_stage before LoRA application when using offload mode
fszontagh
committed
57a12c0
Fix LoRA + offload VRAM conflict with retry mechanism
fszontagh
committed
e6ea65e
Use memory-based offload for LoRA instead of disk reload
fszontagh
committed
c42b659
Fix GPU memory leak in GGMLRunner destructor
fszontagh
committed
aa51517
Add smart VAE offload with dry-run VRAM estimation
fszontagh
committed
7839a54
Implement smart VRAM-based offload decisions
fszontagh
committed
b454eb1
Add configurable reload_diffusion option for post-generation behavior
fszontagh
committed
1febdbe
Add CLI options for all offload configuration settings
fszontagh
committed
f324b03
Add granular tensor offloading infrastructure
fszontagh
committed
af8f5fa
Fix layer registration: use tensor map instead of raw GGML context
fszontagh
committed
cb82950
Skip bulk GPU allocation in layer_streaming mode + improve tensor tracking
fszontagh
committed
d3b989d
Fix streaming mode: add to_backend conversion and skip_param_offload
fszontagh
committed
55a837a
Add layer streaming support for MMDiT/SD3
fszontagh
committed
d109d00
Add layer streaming support for WAN video models
fszontagh
committed
5235117
Add layer streaming support for QwenImage and ZImage
fszontagh
committed
b013674
Add ref_latents support to Flux streaming (WIP)
fszontagh
committed
26c5e12
Fix Flux streaming: use coarse-stage approach like UNet
fszontagh
committed
7a8dd53
Fix layer_streaming mode: force T5 offload before diffusion
fszontagh
committed
2037666
Add layer streaming support for Anima model
fszontagh
committed
07c08e1
Fix AnimaConditioner offloading and layer_streaming state consistency
fszontagh
committed
8272591
Add offload_streaming_layers to free GPU memory before VAE decode
fszontagh
committed
a1b486a
Implement true per-layer streaming for QwenImage
fszontagh
committed
258bb14
Fix TRUE per-layer streaming: defer compute buffer free until after output read
fszontagh
committed
c3e52a6
Add pre-VAE-encode offloading to prevent OOM during image encoding
fszontagh
committed
c66f0d0
Implement async layer prefetching for layer streaming mode
fszontagh
committed
ebb8ddb
Add async prefetching to all TRUE per-layer streaming models
fszontagh
committed
142013c
Fix CFG causing redundant model loading in layer streaming mode
fszontagh
committed
582acb3
Fix ZImage TRUE per-layer streaming: load refiner layers before refiner stage
fszontagh
committed
e220c67
Disable broken TRUE per-layer streaming for ZImage, fall back to normal compute
fszontagh
committed
be36ea0
Add comprehensive debug logging for ZImage TRUE per-layer streaming
fszontagh
committed
10ee7a6
Add extensive debugging for ZImage TRUE per-layer streaming
fszontagh
committed
e546fa6
Remove misleading input buffer check after GGML compute
fszontagh
committed
7e59edb
Clean up ZImage TRUE per-layer streaming debug code
fszontagh
committed
2ad9c8c
Reduce verbose DEBUG logging in layer streaming
fszontagh
committed
6fd7efa
Fix t_emb buffer aliasing in ZImage TRUE per-layer streaming
fszontagh
committed
88fd0b2
Fix non-streaming offload modes (cond_only, cond_diffusion, aggressive) OOM
fszontagh
committed
117f647

Commits on Mar 6, 2026

Deduplicate streaming code into GGMLRunner and align style with upstream
fszontagh
committed
98d7f6c
Add VRAM offloading documentation
fszontagh
committed
1ad143c

Commits on Apr 4, 2026

Merge upstream/master: sd::Tensor migration, webp, spectrum caching
fszontagh
committed
f865af4

Commits on Apr 27, 2026

Merge upstream/master: hires fix, more samplers, tokenizer split
fszontagh
committed
8c3ad49

Commits on Apr 29, 2026

Fix layer streaming dispatch lost during upstream merge
fszontagh
committed
77127d8
Avoid 256 MB scratch alloc per streaming dispatch
fszontagh
committed
39fca39
Honour streaming_prefetch_layers in z_image streaming loop
fszontagh
committed
0da04f1
Honour streaming_prefetch_layers across all DiT/UNet runners
fszontagh
committed
b759cd2
Reuse compute buffer across z_image streaming layers
fszontagh
committed
b705b36
Skip empty MultiLoraAdapter when no LoRAs target a model
fszontagh
committed
7114e8c

Commits on May 1, 2026

Merge remote-tracking branch 'upstream/master' into feature/vram-offloading
fszontagh
committed
0509ad9

Commits on May 4, 2026

Commits on May 5, 2026

Commits on May 6, 2026

Commits on May 7, 2026

Park VAE on CPU pinned under layer_streaming
fszontagh
committed
427f35a

Commits on May 8, 2026

Evict streaming residency on sampling failure
fszontagh
committed
cb9e77b

Commits on May 15, 2026

Merge upstream/master into feature/vram-offloading-v2
fszontagh
committed
e36ad7b
Add layer_streaming support for HiDream O1
fszontagh
committed
dab2c52

Commits on May 16, 2026

Wire HiDream O1 through per-component offload flags
fszontagh
committed
ceac8dd

Commits on May 17, 2026

Merge upstream/master into feature/vram-offloading-v2
fszontagh
committed
9467013
Add layer streaming executor header
fszontagh
committed
83702cc
Address code review on executor header
fszontagh
committed
17d2fb5

Commits on May 18, 2026

Implement LayerStreaming::run_streaming (per-layer path)
fszontagh
committed
1dde96b
Address code review on executor implementation
fszontagh
committed
254be83
Add start_layer_idx parameter for chunk-K skip
fszontagh
committed
9ec29b6
Drop default args on run_streaming output params
fszontagh
committed
443d552
Add SDCPP_STREAM_PROFILE instrumentation to executor
fszontagh
committed
8656f1d
Migrate HiDream O1 to shared streaming executor
fszontagh
committed
5080cfe
Migrate Z-Image to shared streaming executor
fszontagh
committed
db71012
Migrate Qwen Image to shared streaming executor
fszontagh
committed
67a529c
Migrate Flux to shared streaming executor
fszontagh
committed
51937da
Migrate WAN to shared streaming executor
fszontagh
committed
a587226
Migrate Anima to shared streaming executor
fszontagh
committed
52af33d
Migrate MMDiT (SD3) to shared streaming executor
fszontagh
committed
f13cdd4
Migrate UNet (SDXL/SD1.x) to shared streaming executor
fszontagh
committed
9832118
Remove dead per-model streaming helpers
fszontagh
committed
756b73c

Commits on May 22, 2026

Commits on May 26, 2026