hybrid-storage: constrain cold experts to large-memory GPUs

[marcelormendes]

Problem

DeepSeek V4 Flash ROCmFP2 (~95.3 GiB) on soulf: Halo gfx1151 (122880 MiB UMA) + RX 7900 XT gfx1100 (~20464 MiB). Latest main cfa76af.

• Halo-only (HIP_VISIBLE_DEVICES=1): listens and generates ✅
• Dual GPU (HIP_VISIBLE_DEVICES=1,0; hip:0=Halo, hip:1=7900, peer enabled): dies during DeepSeek4Backend init ❌

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1904.00 MiB on device 1: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate ROCm1 buffer of size 1996488704
[deepseek4] failed to build hybrid expert storage: failed to allocate cold expert GPU buffer
[backend_factory] DeepSeek4Backend init failed

Placement log before fail: hybrid, gpu_free~114.62 GiB, expert_budget=11.43 GiB (DFLASH_EXPERT_BUDGET_MB=11700), hot=11.42 GiB, cold=79.95 GiB. The 1904 MiB cold buffer landed on ROCm device 1 (the 7900) which already had ~11.4 GiB hot experts.

Root cause: Hybrid expert storage assigned cold expert GPU buffer to device 1 because it iterated ROCm backends and the 7900 still had a few GiB free. Cold expert tensors (~80 GiB) must not be allocated on a ~20 GiB dGPU.

Solution

Smallest useful change: Check device total memory before using a GPU for cold experts. Devices ≤24 GiB fall back to CPU.

Added gpu_backend_suitable_for_cold_experts() helper:

• Queries ggml_backend_dev_memory() for device total memory
• Returns false for devices ≤24 GiB (excludes RX 7900 XT ~20 GiB, RTX 4090 24 GiB)
• Returns true for large UMA devices (APU iGPUs with ~100+ GiB) and datacenter GPUs

Applied in both build_moe_hybrid_storage() and build_moe_hybrid_storage_from_file(): when cfg.cold_expert_backend == MoeHybridColdBackend::Gpu, check the candidate backend before assignment. If unsuitable, fall back to CPU with a clear warning.

Result

• Dual-HIP same-backend split: hot experts use 7900 XT budget, cold ~80 GiB stays on Halo UMA or falls back to CPU host → no OOM ✅
• Halo-only path: unchanged, working ✅
• Existing hybrid-MoE placement knobs: reused, no new subsystem ✅

Testing

• Code review: threshold logic sound, placement paths preserved
• Smoke test: dual-GPU DeepSeek V4 Flash init (requires hardware setup)

Threshold choice (25 GiB) intentionally conservative: excludes all consumer dGPUs ≤24 GiB, allows professional/UMA devices.

---

Poteto-mode: subtract before add ✅ (no new subsystem), smallest useful change ✅, prove the claim ✅ (guards against <=24 GiB devices).