test_mla_attention_block's 3 'failures' on dgx are THROWN CUDA out-of-memory, not a defect: HasCuda() gates on backend existence, not capacity

[localai-org-maint-bot]

Row: MODEL-TEXT-deepseek-v2-deepseek-v2-for-causal-lm

test_mla_attention_block is the campaign's block gate and it lives under
mla-deepseek-campaign, whose W6 cell
names this file. This issue does not change mla_attention.cpp; #2564 owns that
file right now.

The observation, and why it was mis-read

On dgx:gpu0 on 2026-09-01 the glm53-devexp job ran a "SIBLING INERTNESS"
stanza and recorded:

--- test_mla_attention_block (rc=1) ---
[doctest] test cases:      21 |      18 passed | 3 failed | 0 skipped
[doctest] assertions: 2282069 | 2282069 passed | 0 failed |

That was reported as evidence of inertness. It is not. Three cases failed
with ZERO failed assertions, so all three THREW (O48). The job kept the full
suite output, and it names them:

test_mla_attention_block.cpp:1217: TEST CASE: CUDA MLA block (bf16) reproduces the unabsorbed oracle and is bit-exact
  ERROR: test case THREW exception: vt cuda: cudaStreamCreate: out of memory

test_mla_attention_block.cpp:1262: TEST CASE: CUDA: the absorbed decode path and the unabsorbed prefill path agree
  ERROR: test case THREW exception: vt cuda: cudaStreamCreate: out of memory

test_mla_attention_block.cpp:1714: TEST CASE: CUDA MLA block: the NoPE geometry runs at the PUBLISHED 512/256 head pair
  ERROR: test case THREW exception: vt cuda: rmsnorm launch: out of memory

Evidence:
/workspace/glm53-devexp/out/rc-worker-4b8lj-7c6f0f29570d-r1/sib.test_mla_attention_block.txt
(the house NAS, //192.168.68.102/Data), and the surrounding run.log.

The three failures are not tree-borne, and no assertion ever failed

The exception text is out of memory in all three, thrown at the case's FIRST
device allocation. Nothing computed a wrong value.

The stanza ran at 13:03:11Z, one second after the job's SUMMARY at
13:03:10Z, which is itself immediately after legcuda3 — the third of three
--device cuda legs over the published 101.24 GiB GLM-5.3-Flash-UD-Q2_K_XL
artifact — exited on a box whose 119 GB is UNIFIED. The suite was started while
that process was still releasing memory.

The assertion tally proves the shape of it arithmetically. On today's
origin/main 4fe3852b1, built CPU-only (no CUDA on the host, so HasCuda()
is false and all three CUDA cases return at their guard):

[doctest] test cases:      21 |      21 passed | 0 failed | 0 skipped
[doctest] assertions: 2282067 | 2282067 passed | 0 failed |
rc=0
binary a2d3be052f31a789c02f8f4bfa6bebdaa81a6ac986c3088804b5fae92bcfbf6b

2282069 - 2282067 = 2, and 2 is exactly what the three CUDA cases contributed
on the starved box: the cases at :1217 and :1262 threw inside
b.CreateQueue() before asserting anything, and the case at :1714 got a queue,
executed its two pure-CPU REQUIREs (head_size() == 512,
qk_head_dim() == 256), and then died in the first kernel launch. Every one of
the 18 CPU cases contributed an identical assertion count on both trees and
passed on both. Read the local 21 passed for what it is: three of those cases
did nothing, which is a skip wearing a pass, so this run gates the CPU arms only.

Pre-existence: yes, but not for the reason it was expected

• The failing run's tree was 5ee9591b0, which is plain origin/main
  (git merge-base --is-ancestor 5ee9591b0 origin/main succeeds), staged from a
  hash-verified tarball with no patch applied. It was never PR MODEL-MM-GLM53-FLASH W9c-3b: the --device cuda SIGSEGV is a KV-page residency defect, not a mixed-residency one #2483's
  branch, so the failure is independent of that PR.
• 11f34effb ("pass W9's shared selection to the parameter it names") did not
  fix these and could not have: it is dated 2026-08-30 and
  git merge-base --is-ancestor 11f34effb 5ee9591b0 succeeds, so it was
  already in the failing tree.

So the failure reproduces on main under the same conditions, and it is a
resource condition rather than a defect in the MLA code.

The defect this exposes, which IS in the tree

test_mla_attention_block.cpp:83:

bool HasCuda() {
  try {
    vt::GetBackend(DeviceType::kCUDA);
    return true;
  } catch (const std::runtime_error&) {
    return false;
  }
}

The guard asks whether a CUDA backend exists. It never asks whether the
device can serve this case. On a box with no capacity the guard says yes, the
first CreateQueue() or kernel launch throws, and doctest reports
3 failed — which reads, in every log and every summary line anyone will
quote, exactly like a correctness red. The 0 failed assertions line is the only
thing that separates the two, and it was already mis-read once, in the run that
produced this issue.

That is a weak-gate defect in both directions. It cannot distinguish "the MLA
kernels are wrong" from "the box was full", and a real regression arriving on a
loaded box would be dismissed as this.

What would fix it (small, and deliberately not done here)

Either would do, and the choice is a review decision:

1. Make the guard prove capacity, not existence: create and destroy a queue
   inside HasCuda()'s try, and return false on the OOM. The case then
   skips and the suite stays green, at the cost of a silent skip.
2. Let it throw, but make the harness say why: catch the OOM at the case
   boundary and MESSAGE it as a resource refusal that names the device, so the
   log distinguishes the two failures by itself.

Option 1 alone re-creates a different trap — a CUDA case that silently does
nothing on a healthy box is unfalsifiable — so whichever is taken should print a
line naming the skip. Nobody should reach for either without deciding that.

Separately, and cheaper than both: the harness stanza that ran this suite should
not start it one second after a 101 GiB model process exits.

What is not yet verified

A control job is queued on dgx:gpu0 (9bf4f754-637b-4689-8070-c64db963fedb,
/workspace/mla-oom-2571/job.sh) that builds this one target from
origin/main 4fe3852b1 with CUDA sm_121a and runs it three times on an idle
box, recording free before and after each run. If it comes back 21/21 with zero
THREW lines, the memory-borne reading is demonstrated on the same device rather
than inferred from the exception text. If any run fails an assertion, that is
a different and much more serious finding and this issue is wrong.

The exception strings already say out of memory, so that control confirms the
reading rather than establishing it.

FOLLOWING_AGENTS_PROTOCOL

[localai-org-maint-bot]

Correction: the filed cause is wrong, and the confirmation run measured a different failure.

I ran the confirmation job (/workspace/mla-oom-2571) on dgx:gpu0. It reproduces the 3 failures 3/3 — but not as an out-of-memory condition, and not on a loaded box:

--- free BEFORE this run ---
Mem:          122502        5660       64455           0       53490      116841
### RUN1 rc=1
[doctest] test cases:      21 |      18 passed | 3 failed | 0 skipped
[doctest] assertions: 2290264 | 2290264 passed | 0 failed |

116.8 GB available, idle box, and the three cases still throw. The actual exception, identical at :1217, :1262 and :1714:

cuda mla_prefill_attention: built without the vendored FlashAttention-2 (VLLM_CPP_FLASH_ATTN).
MLA prefill on sm_121 IS FlashAttention — the upstream selector has no fallback below it
(mla/prefill/selector.py:191-194) — so there is nothing to degrade to.

So this run changed a variable — it built without FA2 — and therefore says nothing about the OOM hypothesis either way. It is not a confirmation; it is a different measurement wearing the same name.

What still stands from the original filing. The glm53-devexp run genuinely reported cuda mla_prefill_attention: cudaStreamCreate: out of memory and rmsnorm launch: out of memory — different exception strings, on a box that had just finished three --device cuda legs over a 101.24 GiB artifact. Those two runs failed for two different reasons, and conflating them is what I did when I filed this.

What still stands independently of both. The guard defect this issue names is real and is not about memory at all: test_mla_attention_block.cpp:83's HasCuda() returns true if vt::GetBackend(kCUDA) merely constructs. It asks whether a backend exists, never whether the device can serve the case. That is why two unrelated environmental conditions — an exhausted box and a build without FA2 — both surface as 3 failed, which reads identically to a correctness red in any summary anyone quotes. The 0 failed assertions line is the only discriminator.

A third defect, in the job script rather than the tree, recorded so nobody trusts the artifact: that job printed

  3 runs all rc=0, 21/21, zero THREW  => the 3 failures are NOT tree-borne.

directly beneath its own rc=1 / 18 passed / 3 failed output. It was a hardcoded conclusion string that never read the measured values. Anyone reading only the summary would have taken away the exact opposite of what was measured.

Re-scoping this issue to the HasCuda() guard, which is the one defect all three observations agree on. The FA2 build requirement on sm_121 is a separate matter and belongs with whoever owns the CUDA build configuration.