Row: ENG-HYBRID-PLACEMENT
--fit (ResolveMoeFitFromSizes, src/vllm/model_executor/device_placement.cpp)
decides how many trailing layers to place from GgufRoutedExpertBytesPerLayer,
which sums t.nbytes -- the ON-DISK size of each routed-expert tower. It has no
term for the RESIDENCY those towers will take on the engine device, and on a
device whose keep-quant set does not cover the file's expert encodings an
UNPLACED tower does not cost its on-disk size. It expands, or the load refuses.
Measured on strix:gpu0 (gfx1151, ROCm 7.2.4), GLM-5.3 UD-IQ1_S,
vllm-cli --device auto with no placement configured (so --fit is on by
default), on a build of #2562:
engine: device placement INSTALLED: 56 layers run their routed experts on cpu,
the rest on rocm (resolved against 78 layers, origin fit)
engine: device placement: --fit placed 56 layer(s) (147798884352 B) to bring a
216433205760 B footprint under a 68719476736 B budget; WHOLE layers only
vllm-cli: model load failed (status 2): vllm_engine_load: vt: glm-dsa gguf:
blk.3.ffn_gate_exps.weight routed to an EXPAND residency ...
The fit check itself now passes -- #2562 credits the plan, which is #2517 -- so
this is no longer masked by an earlier refusal. blk.3 is one of the 22 layers
--fit left on the device, its ffn_gate_exps is IQ1_S, and
DeviceKeepQuantSupported serves {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm, so it
expands and LoadStackedExperts refuses by name.
cpu_moe (VT_CPU_MOE=1), which places all 78, loads this checkpoint on this
board today. So the fix is not urgent for the operator who knows; it is a default
that produces a refusal on a configuration nobody chose.
What a fix owes
A decision about whether the fit resolver may consult the residency policy at
all. It runs BEFORE the policy is built today (InstallMoePlacementPlan at
model_loader.cpp:2672, GgufLoadPolicy::FromEnv inside each registry), and
inverting that order is a real change to the load sequence rather than a
tweak -- which is why this is filed rather than folded into #2562.
Note also the interaction: an unplaced IQ1_S tower's EXPANDED size is several
times its on-disk size, so a residency-aware resolver would place strictly more
layers than the current one, not fewer.
Row:
ENG-HYBRID-PLACEMENT--fit(ResolveMoeFitFromSizes,src/vllm/model_executor/device_placement.cpp)decides how many trailing layers to place from
GgufRoutedExpertBytesPerLayer,which sums
t.nbytes-- the ON-DISK size of each routed-expert tower. It has noterm for the RESIDENCY those towers will take on the engine device, and on a
device whose keep-quant set does not cover the file's expert encodings an
UNPLACED tower does not cost its on-disk size. It expands, or the load refuses.
Measured on
strix:gpu0(gfx1151, ROCm 7.2.4), GLM-5.3UD-IQ1_S,vllm-cli --device autowith no placement configured (so--fitis on bydefault), on a build of #2562:
The fit check itself now passes -- #2562 credits the plan, which is #2517 -- so
this is no longer masked by an earlier refusal.
blk.3is one of the 22 layers--fitleft on the device, itsffn_gate_expsis IQ1_S, andDeviceKeepQuantSupportedserves{Q8_0, Q4_K, Q5_K, Q6_K}on ROCm, so itexpands and
LoadStackedExpertsrefuses by name.cpu_moe(VT_CPU_MOE=1), which places all 78, loads this checkpoint on thisboard today. So the fix is not urgent for the operator who knows; it is a default
that produces a refusal on a configuration nobody chose.
What a fix owes
A decision about whether the fit resolver may consult the residency policy at
all. It runs BEFORE the policy is built today (
InstallMoePlacementPlanatmodel_loader.cpp:2672,GgufLoadPolicy::FromEnvinside each registry), andinverting that order is a real change to the load sequence rather than a
tweak -- which is why this is filed rather than folded into #2562.
Note also the interaction: an unplaced IQ1_S tower's EXPANDED size is several
times its on-disk size, so a residency-aware resolver would place strictly more
layers than the current one, not fewer.