WidenRowToF32 reads a bf16 RMSNorm gamma at an odd address on two models, and it is the second sanitize-cpu finding

[localai-org-maint-bot]

Row: FIX-UNALIGNED-CONSUMERS-2540

Spec: .agents/specs/unaligned-safetensors-consumers.md. Claimed 2026-09-02: the fix repairs the CONSUMERS of unaligned safetensors bytes, so the zero-copy borrow survives.

Was owed by .agents/specs/debtfix-glue-rank-bound-and-repack-device.md under ## Owed; ownership moved to the row above, which fixes it.

vt::cpu::WidenRowToF32 (src/vt/cpu/cpu_matmul_elem.cpp:565-583) reads a bf16 tensor
through const uint16_t*. On two models it is handed an odd address, which is
undefined behaviour and a real misaligned load on any target that does not fix it up in
hardware.

Measured on origin/main 63889449c plus the DEBTFIX branch (which touches none of
the files in either stack), Debug + -fsanitize=address,undefined,
VT_POOL_BYPASS=1. Both binaries abort under the lane's own
-fno-sanitize-recover=all and exit 1 before reporting a single doctest assertion:

src/vt/cpu/cpu_matmul_elem.cpp:577:61: runtime error: load of misaligned address
0x782f0e901907 for type 'const uint16_t', which requires 2 byte alignment
    #0 vt::cpu::WidenRowToF32(vt::DType, void const*, long, float*)  cpu_matmul_elem.cpp:577
    #1 RmsNormKernel                                                 cpu_ops.cpp:557
    #2 vt::RmsNorm(...)                                              ops.cpp:1043
    #3 FusedChainCompositeImpl                                       ops.cpp:1114
    #4 vt::FusedChainComposite(...)                                  ops.cpp:1265
    #5 vt::FusedChain(...)                                           ops.cpp:1299
    #6 vt::FusedChain(...)                                           ops.cpp:1332

and from there through a production entry point on both models:

binary	frame 7 onward
test_dots3_note_attn	Dots3NoteModel::ForwardDevice (dots3_note_device.cpp:1307) -> ForwardDots3NoteForCausalLM (dots3_note_registry.cpp:137) -> ModelRegistry::Forward (model_registry.cpp:646)
test_muse_glimmer_text	RunLayer (muse_glimmer.cpp:298)

The operand is the GAMMA, not the activation. cpu_ops.cpp:557 is
WidenRowToF32(w.dtype, w.data, h, wf.data()) — the loop-invariant RMSNorm weight,
widened once per call. Both call sites reach it with w = ResidentWeight(d, layer.input_layernorm, {H}), so the odd pointer is in what ResidentWeight hands back
for that tensor, not in any scratch buffer. Two addresses seen, ...7cd and ...907,
both odd.

What this is not. It is not the dense_device_glue.h rank overrun (#2435): that one
is fixed, and neither of these stacks contains a file that fix touched. It is a SECOND
independent reason sanitize-cpu (address,undefined) is red, in a different subsystem,
and it was hidden behind the first one because that lane aborts at the first finding and
test_qwen4_exp_layer_loop sorts earlier.

Reproduce:

cmake -S . -B build-sanitize -DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_CUDA=OFF \
      -DVLLM_CPP_SANITIZE='address,undefined'
cmake --build build-sanitize -j 2 --target test_muse_glimmer_text test_dots3_note_attn
VT_POOL_BYPASS=1 UBSAN_OPTIONS=print_stacktrace=1 ./build-sanitize/tests/test_muse_glimmer_text; echo $?
VT_POOL_BYPASS=1 UBSAN_OPTIONS=print_stacktrace=1 ./build-sanitize/tests/test_dots3_note_attn; echo $?

Owed: find where the odd w.data comes from and fix it THERE. A memcpy inside
WidenRowToF32 would silence the report and leave whatever produced an odd bf16 weight
pointer in place, which is the wrong end of the defect.

[localai-org-maint-bot]

CI on PR #2541 ran the whole 701-test suite under address,undefined and this is
BROADER than the two binaries this issue was filed from. Job
100076447357,
head 1421d8846, VT_POOL_BYPASS=1, 5168.60 sec, 99% tests passed, 7 tests failed out of 701.

Every UBSan report in the entire run is this defect. Six reports, all
src/vt/cpu/cpu_matmul_elem.cpp:577:61: runtime error: load of misaligned address ... for type 'const uint16_t', at six distinct addresses, and grep -c dense_device_glue
over the whole log is 0. Six of the seven failing tests are this issue:

#	test	note
151	test_llama_embedding_fold	new — a third model family
189	test_muse_glimmer_text	as filed
190	test_muse_glimmer_text_fallback	new
216	test_dots3_note_attn	as filed
454	test_openai_api_server	new — it reaches through the SERVER
466	test_capi	new — it reaches through the C ABI in include/vllm.h

The seventh, test_glm_moe_dsa_schedule, is unrelated (a 384-byte fixture leak, fixed
in #2541).

What this adds to the diagnosis. The issue named dots3_note and muse_glimmer
and located the operand as the RMSNorm gamma. test_capi and test_openai_api_server
reaching it means the path is live from the public C ABI and the registered server on
their default configuration, not only from a model test — so this is a production
reachability finding, not a fixture one. test_llama_embedding_fold adds a third
architecture, which argues the odd pointer is produced by something shared
(ResidentWeight / the norm-weight load) rather than by any one model's loader.

The trailing address nibbles cluster on three values across the six reports —
...907, ...893, ...7cd — all odd, which is consistent with a single producer
handing out a byte offset that is not rounded to the element size.

Still owed, unchanged: find where the odd w.data comes from and fix it THERE. A
memcpy inside WidenRowToF32 would silence six reports and leave the producer in
place.

[localai-org-maint-bot]

Broader than filed, and reachable from the public surface.

CI job 100076447357 (full 701-test suite, VT_POOL_BYPASS=1) shows six UBSan reports, all
the same signature — cpu_matmul_elem.cpp:577:61: load of misaligned address … 'const uint16_t' — across:

• test_dots3_note_attn
• test_muse_glimmer_text
• test_muse_glimmer_text_fallback
• test_llama_embedding_fold ← a third model, not in the original report
• test_openai_api_server ← the registered server
• test_capi ← the public C ABI

So this is not confined to two model tests. It is reachable from the server path and from
include/vllm.h's own surface, which raises it above a test-only nuisance.

For attribution: grep -c dense_device_glue and grep -c MakeTensor over that entire log are
both 0, so none of these is the #2435 out-of-bounds write — that one is fixed, and this is
what it had been masking. The sanitize-cpu lane compiles -fno-sanitize-recover=all and
aborts at the first finding, and test_qwen4_exp_layer_loop sorted earlier, which is why
nobody could see these until it stopped firing.

[localai-org-maint-bot]

A seventh test, and a confirmation that #2435's fix is clear of this.

CI job 100123789893 on #2541's head, after the dense_device_glue rank bound landed:

The following tests FAILED:
  151 - test_llama_embedding_fold
  189 - test_muse_glimmer_text
  190 - test_muse_glimmer_text_fallback
  216 - test_dots3_note_attn
  454 - test_openai_api_server
  466 - test_capi
  508 - test_qwen35_exl3        <- new

test_qwen35_exl3 was not in the earlier set; it arrived with main's recent EXL3 work and
carries the same signature. Counts over that whole log: 12 hits of
cpu_matmul_elem.cpp:577 (this issue), 0 of dense_device_glue (#2435, fixed), 0
LeakSanitizer lines (the 384-byte leak, fixed).

So this is now the only thing reddening sanitize-cpu, and it spans seven tests across
four models plus the registered server and the public C ABI. The lane is
continue-on-error: true, so it does not block — but it is the single reason that lane is
not green, and every PR inherits it.

[localai-org-maint-bot]

It is six tests and four entry paths, not two models — and after today it is the ONLY thing left reddening the lane

Measured from sanitize-cpu (address,undefined) on main itself at
bb78d1ee8 (run 33601071886, job 100154726648, completed 2026-09-02T08:37:24Z).
That run is the first one on main after #2541 landed, so its verdict is
current in a way the pull-request runs quoted elsewhere are not.

Eight tests failed. Grouped by signature:

Signature	Tests
cpu_matmul_elem.cpp:577 misaligned uint16_t (this issue)	test_llama_embedding_fold, test_muse_glimmer_text, test_muse_glimmer_text_fallback, test_dots3_note_attn, test_openai_api_server, test_capi
dense_device_glue.h:52 kMaxRank (#2559/#2560)	test_qwen4_exp_layer_loop
cpu_exl3_kernels.cpp:130 misaligned uint16_t (#2558)	test_qwen35_exl3

Twelve cpu_matmul_elem.cpp:577 mentions across the log — six reports, each
naming the error line and stack frame #0.

The rank-5 group is fixed at 292049691 (see #2559/#2560, both now closed), and
the 384-byte test_glm_moe_dsa_schedule leak was fixed by 8cb5434dc inside
#2541. This issue and #2558 are the entire remaining red.

Two entry paths this issue does not name

test_openai_api_server and test_capi were unattributed. They are this defect,
reached through the C ABI and the served HTTP surface rather than through a
model test:

test_capi              vllm_embed (capi/vllm_c.cpp:1432)
                         -> LLMEngine::embed -> ... -> ModelRegistry::Forward
                         -> Qwen3DenseModel::ForwardHidden -> RunLayer (qwen3.cpp:165)
                         -> vt::FusedChain -> RmsNorm -> RmsNormKernel (cpu_ops.cpp:557)
                         -> WidenRowToF32 (cpu_matmul_elem.cpp:577)

test_openai_api_server ApiServer::handle_embeddings (api_server.cpp:579)
                         -> identical chain from LLMEngine::embed down

Both go through ForwardLlamaModelEmbedding
(llama_embedding_registry.cpp:118), which is also
test_llama_embedding_fold's path. So the four distinct entry points are
ModelRegistry::Forward via the embedding registry (three tests),
muse_glimmer.cpp:298 RunLayer, dots3_note_device.cpp:1307, and
vllm_embed in the public C ABI.

The producer chain, closed

This issue established the operand is the gamma, from
ResidentWeight(d, layer.input_layernorm, {H}). The odd pointer's origin is the
borrow lever:

1. qwen3_weights.cpp:71 and muse_glimmer_weights.cpp:616 both load the gamma
   with LoadBf16Direct(get, base + "input_layernorm.weight").
2. LoadBf16Direct (include/vllm/model_executor/models/dense_weight_loaders.h:363)
   calls BorrowStTensorBytes(borrowed, t, vt::DType::kBF16, shape) and returns
   the borrow when it succeeds.
3. BorrowStTensorBytes (src/vllm/model_executor/models/qwen3_5_weights.cpp:490)
   checks size identity only and hands back t.data verbatim. A safetensors
   payload begins at 8 + header_len + sum(preceding sizes), none of which is
   required to be even, so the base can be odd. That is an ordinary file.

This is the same producer #2558 traced for suh/svh, one function apart — so
it is one defect with two consumers, not two defects.

Consequence: #2561 as it stands should close this issue as well

PR #2561's gate is in BorrowStTensorBytes and is keyed on the element width:

const size_t required_alignment = (dtype == vt::DType::kI8) ? 2 : elem;
if (reinterpret_cast<uintptr_t>(t.data) % required_alignment != 0) return false;

The gamma borrows as kBF16, so elem == 2, an odd base returns false, and
LoadBf16Direct falls through to its existing MakeOwned + memcpy, which is
64-byte aligned. Every one of the six tests above should go green on that change
without touching cpu_matmul_elem.cpp.

That makes the consumer-side repair (routing WidenRowToF32 through
vt::LoadUnaligned, the idiom cpu_layernorm.cpp:30-42 and vt::cpu::LoadF32
already carry for exactly this reason, per #627 and ea4deb203) defence in
depth rather than the fix. It is still worth having — WidenRowToF32 takes a
bare const void* and cannot know its caller's provenance, and this is the
sixth recurrence of the class in a file whose siblings were already repaired —
but it should be sequenced after #2561 so the two are not gated against each
other.

What was NOT measured here

No build was produced in this session: the host hit ENOSPC (447G volume, 504M
free) part-way through the sanitizer build, so there is no local red-before /
green-after pair. Everything above is the CI verdict on main at bb78d1ee8
plus static tracing of the tree at 292049691. The confirming run on the tip is
33613881727.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [claude-code]

[localai-org-maint-bot]

A note for whoever picks this up, measured today on #2561, which hit the same class one function away.

Do not fix this by refusing the borrow. It is the tempting shape, it looks fail-closed and safe, and it costs far more than it appears to.

#2561 added exactly that gate to BorrowStTensorBytes (refuse when t.data % elem != 0, let the caller fall through to MakeOwned + memcpy). The fallback is genuinely correct and genuinely aligned, so the change reads as sound. build-test-cpu then went red on two tests that assert the zero-copy mechanism:

tests/vllm/test_load_direct_upload.cpp:142
  CHECK( w.bytes.borrowed() ) is NOT correct!  values: CHECK( false )

tests/vllm/models/test_qwen3_5_dense_load_residency.cpp:296
  "dense load: the MODELOPT NVFP4 arm BORROWS the mapping, it does not copy"

The blast radius is the part worth recording. A safetensors payload starts at 8 + header_len, and a header length is arbitrary, so its parity is arbitrary. In the direct-upload fixture header_len is 171, giving a payload base of 179, and every tensor in the file is refused:

tensor	dtype	file offset	address	required	result
w	BF16	0	179	2	odd, refused
v	BF16	16	195	2	odd, refused
f	F32	32	211	4	211 % 4 = 3, refused

So a producer-side gate does not cost the borrow for the occasional misaligned tensor. It costs direct upload for every tensor in roughly half of all bf16 checkpoints, and three quarters of f32 ones, decided by one arbitrary number in the header.

The consumer-side repair is the one this tree already established. vt::LoadUnaligned in include/vt/unaligned.h is how the first four recurrences in .agents/specs/unaligned-safetensors-loaders.md were fixed, and it is what cpu_layernorm.cpp and vt::cpu::LoadF32 use. For this issue that means putting WidenRowToF32 (src/vt/cpu/cpu_matmul_elem.cpp:577) on the seam rather than gating the loader that feeds it.

Two other things measured while confirming this:

1. The blast radius here is wider than the title says. It is not two models. The same chain is reached through the public C ABI and the served HTTP surface, which is why test_capi and test_openai_api_server are in the failing set and not separate defects: vllm_embed (src/capi/vllm_c.cpp:1432) and ApiServer::handle_embeddings (src/vllm/entrypoints/openai/api_server.cpp:579) both reach RunLayer -> vt::FusedChain -> RmsNormKernel -> WidenRowToF32.

2. This issue and Unaligned safetensors borrow hands EXL3 suh/svh an odd base pointer; HadRowBlock faults on it (fifth recurrence of the unaligned-loaders class) #2558 share one producer. The gamma here and EXL3's suh/svh there both come out of BorrowStTensorBytes. One consumer-side pass over the seam can close both, and it is worth checking that before opening two fix branches.

[localai-org-maint-bot]

Update: the shared-producer theory in my previous comment is no longer static tracing. It is measured.

#2561's producer-side gate ran the sanitize-cpu (address,undefined) lane, and all six tests this issue owns passed on that head, alongside #2558's:

test_llama_embedding_fold        Passed
test_muse_glimmer_text           Passed
test_muse_glimmer_text_fallback  Passed
test_dots3_note_attn             Passed
test_openai_api_server           Passed
test_capi                        Passed
test_qwen35_exl3                 Passed   (#2558)

Zero misaligned address reports in the job, and zero at cpu_matmul_elem.cpp:577.

Two consequences for whoever fixes this:

1. This issue and Unaligned safetensors borrow hands EXL3 suh/svh an odd base pointer; HadRowBlock faults on it (fifth recurrence of the unaligned-loaders class) #2558 are one defect. Gating the single producer BorrowStTensorBytes closes both. They should be fixed together, in one change, rather than separately.

2. The blast-radius caution in my previous comment still stands, and is now the only open question. The producer-side gate achieves the pass list above at the cost of the zero-copy path (test_load_direct_upload and test_qwen3_5_dense_load_residency both go red, and direct upload is disabled for every tensor in any checkpoint whose 8 + header_len has the wrong parity). The consumer-side vt::LoadUnaligned repair should reproduce exactly the same pass list without that cost, and it is now a well-targeted fix rather than a speculative one, because the producer is confirmed as the single root cause.

Target sites for the consumer-side version: WidenRowToF32 (src/vt/cpu/cpu_matmul_elem.cpp:577), HadRowBlock (src/vt/cpu/cpu_exl3_kernels.cpp:130), TileWord32 (src/vt/cpu/cpu_exl3_dequant.cpp:64).