Repository navigation
[Feature] Ideogram4 Qwen3-VL image conditioning #1679
Description
Feature Summary
Effectively providing editing capabilities.
Detailed Description
The embedder/conditioner for ideogram4 is Qwen3-VL, which is capable of taking in images, and technically videos. While ig4 might not be trained for that specifically, the llm was frozen during training, meaning it will work anyway.
I see --llm_vision exists for other models already, so that would be the thing needing extension.
prompt template: Picture {{ image_count.value }}: {% endif -%} <|vision_start|><|image_pad|><|vision_end|>
Activity
stduhpf commentedon Jun 18, 2026
I've already tried (not really hard) to implement it, looks like vision for Qwen3-VL has already been implemented in #1485, but the tensor naming in the llama.cpp mmproj is not recognised, probably because Hidream comes as an all-in-one checkpoint.
stduhpf commentedon Jun 21, 2026
@Green-Sky Looks like most of the required work was done with #1688
Green-Sky commentedon Jun 21, 2026
Comfy-Org/ComfyUI#14567
This looks like something similar.
stduhpf commentedon Jun 21, 2026
That's completely different, it's not based on vision, but uses RoPE offsets, like the edit models. I'm surprised it can work.
Green-Sky commentedon Jun 21, 2026
I guess I got confused by the img to ref stuff. Still fun.
stduhpf commentedon Jun 21, 2026
It does not turn Ideogram into a proper edit model, but it works much better than just the visual conditionning approach.
Green-Sky commentedon Jun 21, 2026
My guess is that pretaining contained some editing to make the editing finetune straight forward.
stduhpf commentedon Jun 21, 2026
It behaves a bit like an instruct-pix2pix model with img_cfg too high, adhering way too closely to the reference image.
Green-Sky commentedon Jun 23, 2026
https://github.com/BitPoet/ComfyUI-bitpoet-IG4Inpaint
https://huggingface.co/BitPoet/Ideogram4-Inpaint-LoRA
Someone did an editing lora. Not sure which one this is at first glance, it talks a lot about conditioning, but it looks more like ref latents too me.