[BUG] SDXL diffusers-directory loading broken on master: version re-detection + "te." prefix collision in unordered_map

[A-Panic]

[BUG] SDXL diffusers-directory loading broken on master: version re-detection + "te." prefix collision in unordered_map

Transparency note: this issue was drafted by Hermes Agent (an AI assistant by Nous Research, running the GLM model by Z.ai) at the request of its user, based on a locally reproduced and debugged failure. All reproduction steps, debug output and proposed patches below come from the user's local build and have been manually verified.

Bug Description

Loading an SDXL model from a diffusers-format directory (sd-cli -m /path/to/sdxl-diffusers-dir) fails on master (master-841, commit 6b3edaa) with model metadata validation failed. Two independent bugs in src/model_loader.cpp / src/name_conversion.cpp combine to break it.

Bug 1 — get_sd_version() re-runs on already-converted tensor names

ModelLoader::convert_tensors_name() (model_loader.cpp:200-212) calls get_sd_version() on the original diffusers names (unet.down_blocks.*, te.1.text_model.*), correctly detects VERSION_SDXL (is_unet=1 is_xl=1 has_multiple_encoders=1), and renames tensors to original layout (model.diffusion_model.input_blocks.*, cond_stage_model.1.transformer.*).

But version_ is never cached, so init() (stable-diffusion.cpp:904) calls get_sd_version() again, this time over the already-converted names. In the second pass:

• te.1.* no longer exists (renamed to cond_stage_model.1.transformer.* — which is matched only if is_unet is already true when the iteration reaches it, see Bug 2's ordering note)
• detection falls through to token_embedding_weight.ne[0] == 768 → SD 1.x

The wrong version then selects the wrong expected-tensor set → 518 validation errors (model.diffusion_model.input_blocks.1.1.norm.weight not in model metadata), exit.

Debug output (patched build, two calls):

get_sd_version: is_unet=1 is_xl=1 has_multiple_encoders=1 token_emb_ne0=768   <- inside convert_tensors_name()
get_sd_version: is_unet=1 is_xl=0 has_multiple_encoders=0 token_emb_ne0=768   <- init(), after conversion
Version: SD 1.x

Bug 2 — te.1. diffusers prefix collides with te. rule in the unordered prefix map

When bug 1 is worked around, the Conditioner then reports 517 missing tensors: cond_stage_model.1.transformer.text_model.embeddings.token_embedding.weight not in model metadata.

Root cause in convert_tensor_name() (name_conversion.cpp): ModelLoader::init_from_diffusers_file() loads text_encoder_2/ with prefix te.1., but the SDXL prefix_map has no te.1. key — only {"te.", "cond_stage_model.transformer."} and {"text_encoder.2.", "cond_stage_model.1.transformer."}.

The map is std::unordered_map<std::string, std::string>, so iteration order is arbitrary. For a tensor named te.1.text_model.embeddings...:

• starts_with("te.1.text_model...", "te.") is true (the literal . of te. matches the 1's preceding dot: te.1 = te + .1)
• if the unordered map happens to visit the "te." entry before any more-specific rule, the tensor becomes cond_stage_model.transformer.1.text_model... — wrong; the conditioner expects cond_stage_model.1.transformer.text_model...

Observed conversion (patched build, debug print):

pre-convert name: te.1.text_model.embeddings.token_embedding.weight -> cond_stage_model.transformer.1.text_model.embeddings.token_embedding.weight

Steps to Reproduce

1. Get any SDXL model in diffusers layout (e.g. SG161222/RealVisXL_V4.0 or a local juggernautXL_v9 split into unet/ vae/ text_encoder/ text_encoder_2/)
2. ./bin/sd-cli -M img_gen -m /path/to/diffusers-dir --vae-tiling --steps 2 -W 512 -H 288 -p test -o out.png -v
3. Observe model metadata validation failed

Expected Behavior

SDXL diffusers directories load like other diffusers layouts (SD1.5 works, Flux/Wan with split files work).

Actual Behavior

Version: SD 1.x (bug 1) → wrong tensor set → 518 model metadata validation failed errors; with bug 1 patched, 517 Conditioner model tensor ... not in model metadata errors (bug 2).

Environment

• stable-diffusion.cpp master-841 (6b3edaa)
• Linux (openSUSE Tumbleweed), AMD RX 9060 XT, ROCm/HIP backend (SD_HIPBLAS=ON, gfx1200)
• Also confirmed independent of backend: failure happens before any GPU work

Proposed Fixes

Both fixes are minimal and verified working locally:

Fix 1 — cache the detected version in convert_tensors_name() and early-return in get_sd_version():

// model_loader.cpp
void ModelLoader::convert_tensors_name() {
    SDVersion version = (version_ == VERSION_COUNT) ? get_sd_version() : version_;
    if (version_ == VERSION_COUNT) {
        version_ = version;  // cache: re-detection over converted names gives a wrong result
    }
    ...
}

SDVersion ModelLoader::get_sd_version() {
    if (version_ != VERSION_COUNT) {
        return version_;
    }
    ...
}

Fix 2 — resolve the diffusers loader prefixes before the unordered map lookup in convert_tensor_name():

// name_conversion.cpp, before replace_with_prefix_map(name, prefix_map)
// ("te.1.x".starts_with("te.") is TRUE — the map's plain "te." rule can win nondeterministically)
if (starts_with(name, "te.1.")) {
    name = "cond_stage_model.1.transformer." + name.substr(5);
} else if (starts_with(name, "te.2.")) {
    name = "cond_stage_model.1.transformer." + name.substr(5);
}

With both patches, SDXL diffusers-dir loading works end to end: Version: SDXL, 0 validation errors, generate_image completed (30 steps 1024x576 in ~60s on gfx1200).

Happy to open a PR with these two fixes if useful.