unsloth: pin ggml-org#25863 so the ROCm nightlies stop miscomputing on AMD APUs

[danielhanchen]

Overview

Adds ggml-org#25863 to the pin set, so the ROCm and Vulkan nightlies stop shipping the AMD APU host-buffer race.

This is the one that actually reaches users. #145 carries the same change onto fork master, but the nightly tree is the upstream tag plus these pins, so a master merge alone ships nothing.

The defect

c7d87229 ("ggml-cuda : restore prop.integrated on HIP builds", ggml-org#24233) turned the integrated flag back on for HIP. CUDA keeps it off, with the comment Temporarily disabled due to issues with corrupted output. With it on, the scheduler may place a compute input in pinned host memory, where an H2D input write can race a graph that is still running.

Two reporters bisected to that exact commit independently, from different symptoms:

report	symptom	evidence
ggml-org#25992	-np 4 --kv-unified returns another slot's response verbatim	bisected to c7d87229, clean parent, gfx1151
ggml-org#27506	perplexity 7.72 to 3024 on Llama-3.2-3B from b10040	reproduced on three separate gfx1151 machines, Vulkan clean on each
ggml-org#27579, #27556	HIP wrong, Vulkan right, same commit, same box
lemonade-sdk/llamacpp-rocm#123	"various models start becoming somewhat to fully broken from b1298"	multiple gfx1151 reporters, upstream Vulkan flawless

The rule those threads converge on is that corruption tracks n_ubatch < n_batch, and output is clean whenever the whole prompt lands in a single ubatch. Making integrated runtime-switchable on an otherwise unmodified tree gives PPL 850,121 on and 72.80 off, from one binary.

Two things worth carrying into how this gets judged. One reporter ran a greedy completion canary clean for forty minutes against a backend that was numerically broken throughout, so a plausible-looking chat reply is not evidence in either direction here. The same reporter measured Vulkan 24% faster at decode on this chip, so there is no throughput argument for leaving ROCm users on the broken path while upstream deliberates.

Why this pin and not the other one

ggml-org#27311 is the better end state: a ring buffer for input tensors, making host buffers correct rather than unavailable. It is an 18-commit scheduler change and is currently CONFLICTING, so it is not pinnable today. When it lands, this pin comes out.

ggml-org#25863 is the narrow version: refuse to schedule compute out of the host buffer on HIP integrated devices, while pinned memory stays available for staging. Discrete HIP and every CUDA path are untouched, since integrated is already false there. It is measured at PPL 63.67 against a Vulkan reference of 64.10, where the base gives 872,006. Open and MERGEABLE upstream as of the pinned commit ce82541.

Verification

The pin preflight probes the merge; what I want to see is that plus green gfx1150 and gfx1151 ROCm legs.

I cannot build this here, and I would rather say so than imply otherwise: the Strix Halo runner I have has no compiler and this needs the ROCm toolchain. Once a nightly exists I can drive two checks on that runner: llama-perplexity over wikitext-2 at -b 2048 -ub 512 before and after, expecting the previous nightly to be the broken number and the new one to match a Vulkan reference, and one -np 4 --kv-unified run against ggml-org#25992's nonce-checked harness.

Drop this entry once a base tag carries the work, per the rule in _doc.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-30T11:22:25.775988Z	d1ef3ff	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[danielhanchen]

Measured evidence for this pin, from the gfx1151 runner.

llama-perplexity over wikitext-2, 24 chunks, n_ctx=4096, upstream b10631, same host, same two model files. Only the backend differs:

backend	Qwen3.8-27B UD-Q4_K_XL V2	same file, V3
Vulkan	6.3115	6.2292
ROCm	28081.5732	29732.4237

A factor of about 4,450 against the Vulkan reference, on our own published ROCm build.

The conditions are the Family A trigger:

perplexity: calculating perplexity over 24 chunks, n_ctx=4096, batch_size=2048, n_seq=1
system_info: ... | ROCm : NO_VMM = 1 | ...
[1]2277.2928,[2]7476.5070,[3]9913.5072, ... [24]28081.5732,

batch_size=2048 with llama-perplexity's default n_ubatch of 512 is n_ubatch < n_batch. Unified memory is not set in either state. The series is already at 2,277 on chunk 1, so it is wrong from the start rather than drifting. Same shape as ggml-org#27506 (7.72 to 3024 on Llama-3.2-3B), about nine times worse on this model.

A separate cell on the same runner isolates the variable directly: identical 50.9 GiB BF16 file, identical seed, unified memory absent in both states, -b 4096 throughout, and only -ub differing. With -ub 4096 the model answers correctly at 4.1 tok/s. With -ub 512 it returns an empty completion, three times out of three.

Two honesty notes. Neither perplexity cell is formally certified: a gate checking GPU placement failed because the probe parses load_tensors: lines that llama-perplexity does not emit, so the harness declined to issue a verdict. system_info confirms the ROCm backend independently, and a silent CPU fallback would have produced a number near 6.3 rather than 28,081, so the gate's concern would push the result the other way. And this measures the defect, not the fix: I still cannot build the patched tree here, so the prebuilt legs remain the first check on ggml-org#25863 itself.

[danielhanchen]

Superseded by #160, which pins #158 instead. #158 carries ggml-org#25863's change in this fork, so pinning both that PR and the upstream commit would apply the same hunks twice and fail the resolve step. Only one entry can be listed and #160 uses ours, which keeps the code reviewable here.

Nothing is lost by closing this: the same change still reaches the nightlies, via #158's pin rather than the upstream commit's.