fix(BACKEND-ROCM): ask the device that will RUN a placed expert tower whether it may keep its blocks

[localai-org-maint-bot]

fix(BACKEND-ROCM): ask the device that will RUN a placed expert tower whether it may keep its blocks

Row: BACKEND-ROCM-IQ-EXPERT-RESIDENCY. Spec:
.agents/specs/rocm-placed-expert-residency.md. Base 35116605b.

Fixes #2516. Fixes #2517.

GLM-5.3 UD-IQ1_S now LOADS on strix:gpu0 (gfx1151, ROCm 7.2.4), where it
previously refused twice. No token is claimed: the first forward throws in the
MLA block for a reason this row does not own and has filed as #2564.

Two predicates stopped the load, in this order, and they are sequential rather
than alternative -- fixing either alone still refuses.

The fit check ignored the plan installed 220 lines above it (#2517). With
VT_CPU_MOE=1 the load announced device placement INSTALLED: 78 layers run their routed experts on cpu and then refused on the next line needing
216433205760 bytes (201.56 GiB) — the UN-reduced figure, with every routed
expert already placed on the host. Nothing in CheckDeviceWeightFit carried any
representation of a resolved placement, so there was no path by which the placed
bytes could leave the bound.

The loader asked the wrong device (#2516). Past that refusal, the load
reached LoadStackedExperts and refused again:
blk.3.ffn_gate_exps.weight routed to an EXPAND residency. The tower is IQ1_S,
DeviceKeepQuantSupported serves {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm, so it
expanded — for a tower the plan had already sent to the CPU, whose vec_dot
table covers all six encodings this checkpoint's experts use, and whose bytes
never reach the GPU at all: GLM-5.3 reads a tower only through
GlmExpertSlice, never through ResidentWeight, and its registry prepare
hook is a no-op.

What changed

GgufLoadPolicy::device is the ENGINE's device, and its own field comment says
every device-dependent decision reads it. Hybrid placement (#2023, #2314) added
a second, narrower answer for exactly one class of tensor: RunMoePlaced hands
MoeBlock the PLACEMENT device. A keep-quant decision asks whether the device
that will execute this GEMM has a vec_dot for the encoding, so for a placed
routed-expert tower it is a question about that device. Route and PeekRoute
now resolve it through one shared ComputeDeviceFor, so the lane predicate and
the loader cannot disagree about a file (#1378). This is #1136 and #2406 one
seam further along.

It is inert in four terms, and the last one is load-bearing rather than
defensive: a never-installed plan's engine_device() is kCPU, so the placed
answer is adopted ONLY when the plan actively moves THIS tensor off the device
the plan itself was resolved for. Every load in this tree that configures no
placement routes byte for byte as before.

On CUDA it is inert even WITH a placement, and that is checkable by
inspection rather than by hope. RouteGgufTensor has exactly three
device-dependent terms. DeviceQuantGatherSupported is reached only for
kEmbeddingTable, a role this override does not touch.
DeviceKeepQuantSupported answers true for every device except ROCm, and
DeviceKeepF16Supported is dev != kROCM. So substituting kCPU for a CUDA,
Metal, Vulkan, XPU or Tenstorrent engine cannot change any answer at all; ROCm
is the one platform whose answer moves, which is the platform the defect was
measured on.

GgufStagedWeightFootprint gained a const MoePlacementPlan* (null = today's
answer, byte for byte). A placed routed-expert tensor moves into
placed_tensor_count / placed_bytes and leaves the bound, exactly as a
streamed one already did, and the refusal now says what the placement took out.

test_gguf_device_fit's IQ case asked CurrentPlatform(), which made it red on
every ROCm build for a reason the case did not state (#2516's own filing). It
now pins both devices it is about: kCPU keeps the blocks, kROCM expands
them, so the gap #1940 owns is asserted rather than stumbled over.

The box, and the two predicates re-measured on it

BOX=rc-worker-lcjhd gfx=gfx1151 rocm=7.2.4
gcnArchName=gfx1151 name=AMD Radeon Graphics
integrated=1 managedMemory=1 concurrentManagedAccess=1 pageableMemoryAccess=0
warpSize=32 multiProcessorCount=20 totalGlobalMem=68719476736
hipMemGetInfo free=68555898880 total=68719476736 (64.000 GiB)
HostMemoryIsDeviceAddressable(0) = 0   <-- the lane guard's 2nd condition
UseManagedAlloc(0)               = 1   <-- a DIFFERENT predicate

Read with hipGetDeviceProperties on the leased board rather than inherited, and
both predicates printed side by side because #2507 confounded them once.
#2515 is confirmed: pageableMemoryAccess = 0, so
HostMemoryIsDeviceAddressable is FALSE and the host-slot expert lane cannot
serve here at all. That is why this row does not repair the lane. Numbers from
gfx1100/gfx1200/gfx1201 in other ROCm specs do not transfer to this board
and none is reused.

RED, on the real artifact

Base 35116605b, HIP build, -DVLLM_CPP_HIP_ARCHITECTURES=gfx1151,
ROCm backend: ENABLED for arch(es) [gfx1151] asserted at configure, no ccache
(#2506). libvllm.so.0.0.3 sha256 ecfc903d1b70794ceaacc80d0624405985c66ca9e23b6b46427b7e0e504cbb98;
sentinel routed to an EXPAND residency = 1, ALREADY EXCLUDED from the figure
= 0. Model: the derived first shard, sha256
b3e9838651a5c279533c98390ab4bc03cf1d8c176d5be0754180f07d9ed85c01 (never the
directory).

test_gguf_device_fit on that build — 21 cases / 20 passed / 1 failed; 155
assertions / 153 passed / 2 failed, which is #2516's filing reproduced to the
number.

The binary-identity guard earned its keep, and the measurement is worth
recording. Across the base and patched builds vllm-cli has the identical
sha256 ac37fb6d11ba3c1ae7d3d931777ac32ef6fe6bee00231ba18470330740192529 — it is
a thin ABI client and nothing in it changed — while libvllm.so.0.0.3 moves
ecfc903d1b70794ceaacc80d0624405985c66ca9e23b6b46427b7e0e504cbb98 ->
b7f8b962d17fee11eb72b137efcbf6f9e7374789a2e5499edc4e5e90df385e35. A guard that
digested only the executable would have reported two different builds as one. The
source-level sentinel is the second discriminator: ALREADY EXCLUDED from the figure appears 0 times in the base .so (and 0 times in the base SOURCE, which
is checkable without a build) and 1 time in the patched one.

Three product legs, vllm-cli --device auto --prompt "The capital of France is":

Leg	Env	Result
A	(default; --fit is on)	plan INSTALLS 56 layers on cpu, next line refuses needing 216433205760 B (201.56 GiB) — the UN-reduced figure. #2517.
B	VT_CPU_MOE=1	plan INSTALLS 78 layers on cpu — every routed expert on the host — and the refusal STILL quotes 216433205760 B. This is the decisive one: with nothing left to place, the fit check is simply wrong.
C	VT_CPU_MOE=1 VT_DEVICE_WEIGHT_BUDGET_BYTES=0	past the refusal, and then: vt: glm-dsa gguf: blk.3.ffn_gate_exps.weight routed to an EXPAND residency ... at glm_moe_dsa_loader.cpp:392. #2516.

Two blockers, in that order, and C is what proves B is not the only one.
reference-tier distinct=0 in all three: the process refused before any forward,
so no leg here says anything about a kernel.

GREEN

All seven suites GREEN on the gfx1151 HIP build of the patched tree, cases and
assertions read together (a 3 failed / 0 assertions failed run reads clean if
you grep only one of them):

Suite	Cases	Assertions
test_gguf_device_fit	24 / 24	182 / 0 failed
test_gguf_keep_quant	52 / 52	10325 / 0 failed
test_device_placement	21 / 21	222 / 0 failed
test_gguf_device_fit_reach	21 / 21	100 / 0 failed
test_glm_moe_dsa_gguf_load	5 / 5	228 / 0 failed
test_expert_stream_wiring	4 / 4	882 / 0 failed
test_platform	17 / 17	133 / 0 failed

test_gguf_device_fit was 21 cases / 1 failed and 155 assertions / 2 failed on
the same box before this change.

The product legs

Leg E — VT_CPU_MOE=1 vllm-cli --model <derived shard 1> --device auto --prompt "The capital of France is" --max-tokens 8 --temperature 0. THE MODEL
LOADS. Where the base binary refused twice, the patched one gets all the way
through:

vllm-cli: loading model from .../GLM-5.3-UD-IQ1_S-00001-of-00006.gguf
[vt load] mmap+header      13.110 s
engine: device placement INSTALLED: 78 layers run their routed experts on cpu, the rest on rocm (resolved against 78 layers, origin stated)
[vt load] weights        1372.153 s
[vt load] gguf prefault spans=589 paged_in=11.620 GiB in 1037.930 s (11.5 MiB/s)
INFO auto-fit max_model_len: reduced from 1048576 to 8192 to fit the KV cache (256 blocks x 32 tokens).
engine: device placement: engine device rocm, 1 override placing to cpu
vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1)
[vt op-provider] op=4  device=5 selected=vt-native priority=0 registered=1
[vt op-provider] op=1  device=5 selected=vt-native priority=0 registered=1
[vt op-provider] op=82 device=5 selected=vt-native priority=0 registered=1

1809 tensors resolved, 11.620 GiB paged in (device=5 is ROCm, and the three
ops selected before the throw are all vt-native), the engine sizes its KV
cache, and the scheduler starts. All 228 routed-expert towers stayed
compressed, and that is proof rather than inference: LoadStackedExperts
throws by name on any tower that expands, so a load which completes at all has
kept every one of them. The 11.5 MiB/s is the CIFS share the artifact lives on
and is not a property of anything this row changed.

NO TOKEN CAME OUT, AND NONE IS CLAIMED. The first forward throws in the MLA
block:

engine-fatal: EngineCore busy loop threw: MLA block: the split A-projection path
needs vt::FusedNormRope to read the merged [kv_lora_rank + qk_rope_head_dim] row,
because a BLOCK-QUANTIZED kv_a_proj_with_mqa (q8_0) has no row slice ... This is
reachable only with VT_MLA_FUSED_NORM_ROPE=0 on a keep-quant MLA checkpoint

VT_MLA_FUSED_NORM_ROPE was not set. fused_nr
(mla_attention.cpp:550-551) also asks
vt::OpRegistered(kFusedNormRope, device), ROCm registers no such op, and
OpRegistered is deliberately a native-only probe that cannot see the reference
tier — so the message's own diagnosis is one cause short and this run is the
counterexample. Filed as #2564, which is now the single thing between this
checkpoint and a token on gfx1151. It is this spec's own W6 stop condition, and
the row stops there rather than widening into the MLA arm. reference-tier distinct=0 at the throw.

Leg F — the same command with no placement configured, so --fit is on by
default. The fit check now PASSES (this row's repair, visible as the absence of
the cannot serve line that leg A had), the plan places the trailing 56 of 78,
and the load then refuses on blk.3.ffn_gate_exps.weight — one of the 22 layers
--fit left on the device. --fit sizes a placement from ON-DISK bytes and
cannot know an unplaced tower expands here. Predicted in the spec's ## Owed
before the run, confirmed by it, and filed as #2565. cpu_moe is the
supported route on this board.

Leg G — --device cpu --max-tokens 1, the control. It loads in 8.990 s off
the now-warm page cache (gguf prefault spans=590 paged_in=12.119 GiB in 0.648 s)
and is then killed by this job's own 1800 s timeout inside the forward, having
emitted nothing. That timeout was mine and was too short: the sibling
measurement on dgx:gpu0 puts one token of this model's CPU forward at 950 s on
20 cores, and this run had 1790 s of forward. It is a timeout, not a failure,
and it is reported as one — rc=124, no what(), no throw. It is also the
resident-footprint reading this row owes: 12.119 GiB on a CPU engine and
11.620 GiB on the ROCm engine under cpu_moe, both against this board's
MEASURED 58.000 GiB hipMallocManaged ceiling and its reported 64.00 GiB pool.
Which is why #2518's 6.00 GiB of optimism cannot decide this load in either
direction, and why it is filed rather than fixed here.

Reference-tier hits: 0 on every leg, base and patched. #2505 means no
--device value names ROCm, so configuration alone cannot separate a GPU run
from a host fallback; this is the op table's own answer, and the three ops leg E
selected before the throw are device=5 selected=vt-native. Nothing here fell
back, because nothing here got far enough to need to.

Mutations

Every mutation was REBUILT before it was run — a mutation the compiler kills is
not a killed mutation — and every restore was verified by sha256 against the
pre-mutation digest of both product files.

Mutation	Build	test_gguf_device_fit
(none)	rc=0	24 cases / 24 passed; 182 assertions / 0 failed
ComputeDeviceFor never consults the plan	rc=0	rc=1 — 1 case, 1 assertion failed
the footprint never credits the plan	rc=0	rc=1 — 2 cases, 7 assertions failed
CheckDeviceWeightFit passes nullptr to the footprint	rc=0	rc=1 — 1 case, 3 assertions failed

These three ran locally against a hand-linked binary of the same test file, which
is an INSTRUMENT and is reported as one; the CMake targets are run on the box
alongside the product legs, and the reachability proof is the production entry
point rather than any of these — the base binary refuses this checkpoint and the
patched binary loads it, through vllm-cli → vllm_engine_load on the real
artifact.

A fourth mutation, deleting &vllm::ActiveMoePlacementPlan() from the ONE
production CheckDeviceWeightFit call site in model_loader.cpp and rebuilding
on the board, is queued on strix:gpu0 and will write
/workspace/glm53-iqlane/out3/job3.log. It is not needed for the claims above
and is not counted among them; it exists so the call-site deletion is measured on
the same binary the product legs used rather than on a hand-linked one.

What this does NOT do

• It does not port a ROCm i-quant vec_dot (ROCm ports zero I-quant formats, and porting a complex-vec_dot one would test whether #1910's nwarps split transfers #1940). Measured unnecessary here:
  a placed tower is executed by vt::cpu's vec_dot, and on this board an
  UNPLACED tower cannot be executed at all — GlmResidentExpertSlice refuses a
  device that is neither CPU nor host-addressable before any kernel is asked
  for.
• It does not repair the host-slot expert lane (gfx1151 reports pageableMemoryAccess=0, so the host-slot expert lane can never serve on Strix Halo -- #2507's second condition is false, not satisfied #2515), and does not need to.
• It does not move device_memory_total_bytes (ROCm's device-weight budget is hipMemGetInfo's 64.00 GiB total, but every allocation on an integrated board is managed and tops out at a measured 58.000 GiB #2518). That issue is real and
  stays open: the budget is hipMemGetInfo's 64.00 GiB while every allocation
  here is hipMallocManaged with a measured 58.000 GiB ceiling. It cannot
  decide this load — the credited footprint is far below both figures — so it
  is not fixed under a row that would have no way to gate the difference.
• It does not make --fit's resolver residency-aware (--fit sizes a placement from ON-DISK bytes, so it can leave a tower on a device whose keep-quant set cannot hold it #2565, filed from leg F).
  cpu_moe is the supported route on this board.
• It does not touch the MLA block (MLA: the split A-projection is reachable with VT_MLA_FUSED_NORM_ROPE unset, because a backend that does not REGISTER kFusedNormRope takes the same branch -- GLM-5.3 on gfx1151 #2564). That is this spec's own W6 stop
  condition and the row honours it.
• No speed number, in either direction. docs/ROCM.md:60-61 disqualifies a
  performance result from a run with a non-zero reference-tier count, and this
  model's MLA/DSA arm is eight ops short on ROCm — every one of them lands on
  that tier.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]