DeepSeek V4 Flash Vision: image inference fails on Metal (M4 Max)

[cropduster]

Summary

Text completion works, but image input fails with 400 invalid_request_error: "DeepSeek V4 vision inference failed".

Setup

• Hardware: Apple M4 Max, 128 GiB unified memory
• Backend: Metal
• Commit: b0982a1 (includes fc8bf3c "Add DeepSeek V4 Flash vision support")
• Main model: DeepSeek-V4-Flash-Vision-Exp-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out.gguf (145.27 GiB, SSD streaming)
• Encoder: DeepSeek-V4-Flash-Vision-Encoder.gguf (0.87 GiB)

What works

• --inspect loads both the main model and the vision encoder without error.
• ds4-server starts and serves text completions correctly (the model reasons and answers).

What fails

• Sending an image (PNG data URI via image_url) returns HTTP 400 with {"error":{"message":"DeepSeek V4 vision inference failed","type":"invalid_request_error"}}.

Root cause (source-level)

The failure comes from ds4_gpu_deepseek4_vision_encode (ds4_metal.m:44875) returning 0. The encoder runs a chain of ds4_gpu_glm53_matmul_bf16 + glm53_vision_dispatch_* calls (the DeepSeek vision encoder reuses the GLM-5.3 Metal kernels), and one of the stages fails silently (ok = 0 with no stderr detail).

GLM-5.3 vision works on the same machine, so the shared Metal kernels are fine — the DeepSeek-specific dispatch (deepseek4_vision_dispatch_round) appears to be where it breaks. There is a debug hook gated by DS4_DEEPSEEK4_VISION_DEBUG_PREFIX (ds4_metal.m:44854), but no per-stage error logging.

Repro

./ds4-server --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf --ssd-streaming ...
# POST /v1/chat/completions with content [{type:text},{type:image_url,image_url:{url:"data:image/png;base64,..."}}]

Thanks for ds4.

[cropduster]

Refining with additional observations after further testing on the same machine (M4 Max 128 GiB, Metal, commit 110afdd):

Text works flawlessly — the Vision-Exp MXFP4 model answers correctly via both the direct server and LiteLLM. It reasons by default (emits reasoning_content), so requests need a larger max_tokens to leave room for the final answer after thinking.

Image input fails with two distinct errors:

1. 400 "DeepSeek V4 vision inference failed" — from ds4_gpu_deepseek4_vision_encode (ds4_metal.m:44875) returning 0.
2. 500 "metal prefill failed" — the log shows ds4: gpu layer 0 ffn batch encode failed / ds4: gpu layer-major prefill layer 0 encode failed, i.e. the main model prefill fails at layer 0 FFN when the prompt contains vision embeddings.

Relevant context: the working report in #929 (Charlie-Louis) ran the Q2-based AProjQ4 model fully resident (78.6 GiB, no --ssd-streaming). My failing case is the MXFP4 artifact (145 GiB), which requires --ssd-streaming on a 128 GiB host. So this looks specific to the --vision + --ssd-streaming combination (or the MXFP4 artifact), rather than the GPU generation.

Since the resident Q2 path in #929 works, this is likely not a blocker for the recommended Q2 flow — just flagging that the MXFP4 + SSD-streaming vision path fails on Metal.

Thanks.

[cropduster]

Root cause found — supersedes my earlier hypothesis in the issue body

The dispatch chain is fine. The DeepSeek vision encoder's Metal view is destroyed by the SSD-streaming runtime remap, and a stale flag prevents re-mapping:

1. Startup with SSD streaming maps only the token embedding (ds4.c:63225, ds4_gpu_set_model_map_spans); the vision encoder map is then registered additionally via ds4_gpu_set_model_map_range (ds4.c:63413) → vision_map_ready = true. Both view sets coexist.
2. The first prefill installs runtime spans (metal_graph_install_model_spans, ds4.c:~18357) → ds4_gpu_set_model_map_spans fails its coverage check (it only counts views with the main model's map pointer), calls ds4_gpu_model_views_clear() (ds4_metal.m:11367) — clearing ALL views, including the encoder's — and rebuilds only the main-model spans.
3. The encode path re-maps lazily only if (e->metal_ready && !e->vision_map_ready) (ds4.c:63887). The flag is stale true → no re-map → ds4_gpu_wrap_model_range fails:

   ds4: Metal model range 0.87..0.87 GiB is not covered by mapped model views
   

   (0.868 GiB = exactly the encoder GGUF size) → ds4_gpu_deepseek4_vision_encode returns 0 → 400.

Reproduced live on 110afdd (M4 Max 128 GiB, Vision-Exp MXFP4, --ssd-streaming --vision): a minimal 8x8 PNG via image_url returns 400 "DeepSeek V4 vision inference failed", and the log shows the 0.87..0.87 GiB is not covered line for that request. Same log also shows the related first-prefill failure Metal model range 0.10..0.10 GiB is not covered … gpu layer 0 ffn batch encode failed — same clobber family.

Resident runs don't hit this because ds4_gpu_set_model_map_range early-returns/appends without clearing. A small fix that should work: a views generation counter bumped by ds4_gpu_model_views_clear(), checked before the lazy re-map at ds4.c:63887, so vision_map_ready gets invalidated and the existing re-registration path runs. I'm preparing a patch and will report back with verified results; happy to send a PR if you want it upstream.

[cropduster]

Patch ready — verified end-to-end. PR: #937

The fix described in my previous comment turned out to be two coupled bugs, both now addressed in PR #937:

1. Sidecar views cleared by the span remap (as analyzed above) — fixed with a model-views generation counter; the vision encode path now re-registers the sidecar whenever the generation moved.

2. Secondary bug found while verifying that fix: ds4_gpu_set_model_map_range on Metal overwrites the primary model map globals, so (re-)registering the vision map annexed the main map context — the next prefill failed with 0.10 GiB not covered on the vision sentinel tensor (layers.0.ffn.gate.bias_vl, identified via enriched error output + GGUF header parse). PR Fix DeepSeek Vision-Exp image inference with SSD streaming on Metal #937 implements the previously-declared-but-missing ds4_gpu_set_aux_model_map_range on Metal (append-only, no global update) and makes ds4_gpu_set_model_map_spans clear only its own mapping's views, preserving sidecar views.

Verification (M4 Max 128 GiB, Vision-Exp MXFP4, --ssd-streaming --vision, first request after cold start is an image):

Before (both bugs, 110afdd):

HTTP 400 "DeepSeek V4 vision inference failed"
ds4: Metal model range 0.87..0.87 GiB is not covered by mapped model views

After PR #937:

IMAGE:  200 | 37.9s   (first request after cold start, no 'not covered' lines)
IMAGE2: 200 | 56.2s | finish: stop | content: "Das Bild ist grün."   (64×64 green PNG — content verified)
TEXT:   200 | 19.8s | 17×23=391 correct | decoding ~18.6–20.3 t/s

Log check across all post-fix requests: zero not covered lines (previously 2 per image request). CUDA/ROCm get constant-zero generation stubs (they never clear views); resident runs were never affected.

Full logs, root-cause analysis, and the intermediate-failure documentation are in the PR description: #937. Happy to split the diagnostic fprintf out of the diff if you prefer it minimal.

[Charlie-Louis]

yes Xcode can now also analyze screenshots, so the server works ...

[gilbert-barajas]

Data point for the M4 Max report: the same Vision-Exp build runs image inference on an M5 Max (128 GB, macOS 27, Metal resident, no --ssd-streaming) with engine 110afdd:

• ds4-server -m DeepSeek-V4-Flash-Vision-Exp-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8.gguf --vision DeepSeek-V4-Flash-Vision-Encoder.gguf --metal --ctx 65536 --warm-weights
• healthy in ~10–25 s warm (~2 min cold), ~81 GB resident
• a 1440×900 PNG via the OpenAI image content part with thinking:{type:"disabled"}: 140 tokens in 5.9 s, and the description was correct (it named the app, the map's centre town and the coordinates on screen)
• with thinking left ON at a 200-token cap, content came back empty (all tokens in reasoning_content, finish_reason=length) — worth ruling out before reading a failure as a vision bug

Not claiming anything about M4 Max; only that the path is good on this Metal box.