ROCm gfx1151: the Q4_K_M arm diverges from the CPU arm at three near-tie steps on an identical prefix

[localai-org-maint-bot]

Row: BACKEND-ROCM

The Qwen3.8-27B Q4_K_M token gate ran on gfx1151 for the first time on a board
that could finish it (#2546, rc job 85e4091e-1042-4225-9a92-6797449fddf9,
6 clean legs of 6, all byte-identical). It returns TOKEN_GATE=FAIL at 3 of 6,
and the divergences are not the post-#2534 CPU tier's 3 of 6.

Resolving every contested step to what all four sides emit separates two effects
the first-diff column mixes:

step	ROCm	CPU tier	HIP oracle	aarch64 oracle	what it is
p1/34	198	198	198	3095	our two tiers AGREE; the oracle disagrees with itself
p2/20	13	13	13	539	our two tiers AGREE; the oracle disagrees with itself
p5/32	16	16	15	16	our two tiers AGREE; the oracle disagrees with itself
p1/45	303	1521	1521	3095	our two tiers DIFFER; ROCm is wrong
p3/45	25	393	393	393	our two tiers DIFFER; ROCm is wrong
p4/14	22486	4593	22486	22486	our two tiers DIFFER; ROCm is right

This issue owns the bottom three rows. At those steps the ROCm and CPU arms
of vllm.cpp compute a different argmax over an identical prefix on an identical
artifact, with no oracle involved in the comparison. #2534 owns the CPU tier's
residual magnitude term; it does not explain this, because two of the three
prompts ROCm fails are prompts the CPU tier passes and the third fails at a
different step.

The term is not one-signed. ROCm loses two of those steps and wins the third,
and the one it wins is a step the CPU tier is convicted at.

Size of the effect

Every ROCm loss is a rank-2 near-tie. Over 288 decode steps the oracle ranks our
token 1 on 285 and 2 on 3, and nothing worse than rank 2 occurs. Nineteen steps
have an oracle top1 - top2 gap below 0.20; our arm takes the top-1 on 16 of
them. The three it loses have gaps 0.131054, 0.006284 and 0.053452. The worst is
larger than two of the three gaps the CPU tier loses, so this is a comparably
sized error landing on different steps rather than a smaller one.

What is already excluded

• The board. 6 clean legs of 6, zero GPU Hang / Memory access fault /
  HW Exception lines, all six legs byte-identical. ROCm gfx1151: hipMallocManaged on a part that cannot fault-and-recover; plain hipMalloc is 0/12 against 10/12 #2511 is fixed and landed.
• A CPU fallback. reference_tier_hits=0 with 20 device=5 op selections on
  every leg, and the job fails on a non-zero count.
• The oracle. ORACLE_REPRO=YES: today's HIP oracle reproduces the recorded
  ids on 6 of 6 prompts identically, and CHAIN_OF_CUSTODY=EXACT against the
  stock llama-completion binary.
• The tokenizer. 0 of 6 divergences.
• The binary. Built clean at 27da7787e's product tree, both landed fixes
  asserted present in source, and the #2511 marker present in the new
  libvllm.so and absent from the pre-fix one.

Next traceable step

Dump the pre-sampler logits from the ROCm tier and the portable reference tier
over the same prefix at p1/45, p3/45 and p4/14, which localises the term to a
layer rather than to a device. Note that 27da7787e withdraws host
addressability on this part, so the reference tier is no longer reachable
in-process here and the comparison needs a deliberate arm.

Not admissible from this

No speed, latency or memory number. AGENTS.md §Gates admits none until this
gate passes, and #2497 stays blocked behind it.

Evidence:
docs/bench-evidence/qwen38-27b-q4km-rocm-gfx1151-token-gate-v2-20260902.md.
Spec: .agents/specs/rocm-gfx1151-q4k-token-gate-v2.md.
Raw logs: /mnt/nas_share/rc/rocm-tokgate-strix-v2/out/rc-worker-lcjhd-20260902T155343Z/.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [claude-code]