ds4f-q2-q4 cannot be served resident on 128 GB DGX Spark (GB10) — global OOM during startup span preparation

[adamlawi]

Environment

• NVIDIA DGX Spark, GB10 (sm_121), 128 GB unified memory, aarch64
• Ubuntu 24.04.4, kernel 6.17.0-1029-nvidia, driver 580.173.02, CUDA 13.0.88
• ds4 built from 6747e77 with make cuda-spark, no local changes
• Models from ./download_model.sh: ds4f-q2 (80.76 GiB) and ds4f-q2-q4 (90.89 GiB)

Summary

ds4f-q2 works well. ds4f-q2-q4 cannot be started resident at all on this machine: every
attempt is killed by the kernel OOM killer during the startup tensor-span preparation phase, before
the memory plan is even printed. Context size is not the factor — it fails identically at
--ctx 100000, --ctx 16384 and --ctx 4096.

download_model.sh describes this quant as "Works on DGX Spark but loading may struggle compared
to ds4f-q2", which understates it for the 128 GB configuration.

Reproduction

./download_model.sh ds4f-q2-q4
./ds4-server -m gguf/DeepSeek-V4-Flash-Layers37-42Q4KExperts-...-fixed-0731.gguf \
  --host 0.0.0.0 --port 8888 --ctx 16384 --cors

Result: Killed (SIGKILL, exit 137). Last lines before death:

ds4: built 456 aligned CUDA artifacts (68.58 GiB) in 18.9s; expert raw residency replaced
ds4: CUDA aligned artifacts replace expert residency; leaving the 90.89 GiB model mmap unpinned
ds4: CUDA preparing model tensor mappings
ds4: CUDA loading model tensors into device cache
ds4: CUDA loading model tensors 16.01 GiB cached
ds4: CUDA prepared model tensor mappings 16.88 GiB
<killed>

What we tried (all resident attempts)

Mode	ctx	Result
ds4-bench	8321	passed, but startup span phase took 296 s
ds4 --perplexity-file	32768	OOM-killed
ds4 --perplexity-file	16384	OOM-killed
ds4 --perplexity-file	4096	OOM-killed (1882-token input)
ds4-server	100000	OOM-killed
ds4-server	16384	OOM-killed

--ssd-streaming is the only mode that starts reliably, but decode drops to ~1.2 t/s
(80 tokens in 66 s), versus 14.7 t/s for resident ds4f-q2.

The measurable difference

The startup phase that precedes the memory plan differs by 3.5x in volume and 120x in time:

Model	startup model preparation covered	time
ds4f-q2 (server and bench)	8.20 GiB of tensor spans	2.2 s
ds4f-q2-q4 (bench, survived)	28.45 GiB of tensor spans	296 s
ds4f-q2-q4 (server)	reached 16.88 GiB, then killed	—

The 296 s for a phase that takes 2.2 s with ds4f-q2 suggests the surviving run was already
thrashing at the memory ceiling; the server runs crossed it.

Likely cause

cuda/mmq/ds4_repack.cu has aligned repack builders for exactly three tensor classes, each gated on
the tensor type:

• ds4_repack_iq2_candidate — type == 16 (IQ2_XXS), .ffn_gate_exps / .ffn_up_exps
• ds4_repack_q2k_candidate — type == 10 (Q2_K), .ffn_down_exps
• ds4_repack_q8_candidate — Q8_0 dense

There is no Q4_K candidate or builder. In ds4f-q2-q4 the routed experts of layers 37–42 are
Q4_K, so they get no aligned artifact and must be pulled into the device cache in raw form. On GB10
the CPU and GPU share one 128 GB pool, so 68.58 GiB of artifacts plus ~20 GiB of raw Q4_K spans plus
the model mapping exhausts it.

--ssd-streaming reports the same split from the other side:

SSD streaming mixed-precision model: 6/43 routed layers off the slab size class
will bypass the expert cache and read experts via mapped model views

Note this appears to be a different gap from the mmq prefill one: q4k_path && n_tokens > 1u
(ds4_cuda.cu:23961) is live in this build, and prefill does work — it is just slower
(548 t/s vs 872 t/s for ds4f-q2 at 8k). So the missing piece is the aligned/SoA residency path,
not the mmq prefill kernels.

Kernel OOM record

Killed process 124531 (ds4-server) total-vm:279950520kB, anon-rss:968kB
oom-kill:constraint=CONSTRAINT_NONE,...,global_oom
Node 0 Normal free:29852kB ... active_file:32kB inactive_file:720kB

Global OOM with page cache already drained to ~0 — nothing left to reclaim. Because ds4 sets
oom_score_adj=1000 it dies first, which is good, but the same event also took down unrelated
desktop-session processes (pipewire, dbus-daemon, kactivitymanage).

Speed and quality data, for context on whether this is worth fixing

Same prompt file, 128 generated tokens per frontier, ds4-bench:

ctx	q2 prefill	q2-q4 prefill	q2 decode	q2-q4 decode
2048	790.7	459.7	17.31	13.57
4096	881.5	553.8	14.92	10.04
6144	875.7	556.2	14.83	9.59
8192	872.1	547.6	14.69	9.47

Quality on an identical 1882-token sample: ds4f-q2 ppl 11.108, ds4f-q2-q4 ppl 10.841
(the latter measured under --ssd-streaming, since resident mode OOMs; weights are identical so
the number should be unaffected).

Suggestions

1. If a Q4_K aligned repack path is not planned, consider changing the download_model.sh wording
   for ds4f-q2-q4 — on a 128 GB DGX Spark it does not "struggle to load", it cannot load resident.
2. A startup pre-flight check comparing the projected span volume against available unified memory
   would turn an OOM kill into a clear error message.

Happy to run further diagnostics on this machine — it is a stock DGX Spark and we can rebuild
and retest on request.

[GiorgioOppo]

did you se my pr #618 ?

[adamlawi]

Yes, read it carefully — thanks for the pointer. They turn out to be two different gaps, so this issue is not covered by #621, but your work is very relevant to us for a second reason.

Different tensor class. #621 touches the dense attention projections (Q8_0 → Q4_K). The OOM here comes from the routed experts of layers 37–42 in ds4f-q2-q4, which are Q4_K and match none of the aligned repack candidates in cuda/mmq/ds4_repack.cu (iq2_candidate → IQ2_XXS gate/up, q2k_candidate → Q2_K down, q8_candidate → dense Q8_0). Without an aligned artifact they are pulled into the device cache raw, which is what inflates the startup span phase from 8.20 GiB / 2.2 s to 28.45 GiB / 296 s. The mmq prefill path for Q4_K (q4k_path && n_tokens > 1u) is live in this build and works — just slower — so this is the aligned/SoA residency path, not the prefill kernels.

Why #621 still matters here. Those 8.20 GiB of dense spans are exactly what your change shrinks. On a 128 GB DGX Spark the margin is much tighter than the numbers in this issue suggest: with ds4f-q2 at --ctx 200000, filling 194,681 tokens (97% of the window) leaves only 2.5 GB free at peak, and --ctx 220000 leaves 0.94 GB. Freeing ~2 GiB of resident dense weights would be material — it would plausibly move the usable ceiling from 200k to 250–300k on this class of machine, which is a different benefit from the SSD-streaming speedup you measured on the M1 Pro.

Offer to test. Your backend table lists CUDA as implemented but untested. This is a stock DGX Spark (GB10 / sm_121, CUDA 13.0.88, aarch64, 128 GB unified) with a measurement harness already set up — ds4-bench sweeps, full-window prefill with memory sampling at 2 s intervals, and --decode-consistency. Happy to run it, details in #621.

Blocker: I could not find an AProjQ4 GGUF published anywhere, and gguf-tools/deepseek4-quantize requires --hf DIR with the original safetensors, which does not fit here alongside the output (644 GB free vs ~568 GB source). If you can publish ...AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf, or describe the requant path you used to derive it from an existing GGUF, I can start immediately.