model : add support for HrmTextForCausalLM (DFM Mimir 1B)

[noctrex]

Overview

HRM-Text runs two transformer stacks (low, high) in an alternating cycle over the same token stream. The low-cycle state z_l starts from a learned [n_embd] tensor and is broadcast over positions.

• conversion: new writer for the fused gqkv projection (order gate,q,k,v) remapped to llama.cpp q/k/v plus a separate sigmoid gate tensor
• loader: block_count = lps * h_cycles * (l_cycles + 1) cache slots aliasing 2*lps physical blocks via struct copies
• graph: looped build with sigmoid-gated attention, SwiGLU FFN and parameterless RMS norms; learned embedding_scale applied in build_inp_embd
• saver: pointer-deduplicated layer loop (looped archs alias tensors)
• tests: hrm_text fixture in test-llama-archs

Additional information

Limitations:
causal attention only - the upstream prefix-LM mode is not implemented (the prefix_lm GGUF key round-trips unused).
The KV cache holds one entry per pass: 128 layers for Mimir 1B, i.e. 4x a same-width 32-layer model - about 3072 MiB at ctx 4096 in F16 (halves with q8_0 KV + FA).
Every token runs all 128 block passes, so decode cost is roughly 4x a dense model of equal width (2.65 t/s BF16, 8-thread desktop CPU).

Verified against the HF reference: identical argmax at 334/334 positions across 20 prompts (BF16 GGUF vs FP32 golden).
q8_0 requant: 95.8% top-1, all remaining misses inside the HF top-5 (accumulated error over 128 sequential blocks).

I have uploaded the GGUF's here: https://huggingface.co/noctrex/DFM-Mimir

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Used GLM-5.3 for the majority of code AI-generated under my direction, all gates verified locally. All in all I could say that I have written less than 20% of the code and most of the heavy lifting has been done by the model. As such, this should be considered experimental.

[noctrex]

@CISC Would ask you kindly if you would have a look at this model support, please. I know that you're very busy and I'm sorry if I interrupt you

[bolgacg]

Mimir scores a third lower than it should when llama.cpp runs it, and the cause is how
the prompt is read.

DFM-Mimir is a Danish 1B model. It was trained as a prefix language model. That means:
while it reads the prompt, every token can see every other token in the prompt, in both
directions. Only the answer is generated one token at a time, left to right. The model's
config file says this (prefix_lm: true).

This PR reads the prompt left to right only, the way an ordinary causal model does. So
the model reads its input in a way it was never trained for.

I measured the effect on DAISY, a public Danish quiz of 592 questions made by the same
group that made Mimir. Same weights, same questions, their prompt, their exact-match
scorer, greedy decoding. Only the way the prompt is read changes:

official transformers code, prompt read in both directions 8.4 %
official transformers code, prompt read left to right only 5.4 %
this PR, Q8_0 GGUF, left to right only 5.6 %
the number the Mimir paper reports 9.6 %

The first two rows use the same code and differ only in attention mode: 8.4 against 5.4.
So the attention mode explains the gap to the paper. Quantisation does not: the Q8_0
GGUF scores the same as the unquantised model in the same mode.

Script and logged outputs: github.com/bolgacg/daisy-tools
(scripts/mimir_official.py --prefix, results/pred_mimir-official-*).

What would fix it is an attention mode for this architecture where the prompt is read in
both directions and only the generated tokens are causal, which is how the model was
trained. Until then, a short note on the model card that scores from this port are lower
than the paper's would help anyone comparing numbers. I am happy to rerun the benchmark
on any change.

[noctrex]

@bolgacg thank you for your report.
Yes this is a limitation of this implementation because foremost I just want to add model support for it, and as I already have said:

Limitations:
causal attention only - the upstream prefix-LM mode is not implemented

because first I wanted to do the minimal changes for the model to be able to load before creating any additional additions needed for it working correctly.
The prefix-LM mode is a much larger change that should be added at later point in a separate PR.

[bolgacg]

@noctrex I benefit from this PR getting sorted, since my Mimir evaluation work runs through it, so I went ahead and built the prefix-LM mode on top of your branch; now that the review is moving I wanted to share it: noctrex#1. 35 lines: the KQ mask drops the causal check for models with prefix_lm set, the server stops reusing cached prompt prefixes, and decode warns when a prompt is split across ubatches. Checked against the official transformers implementation on the full DAISY set: 8.3 against 8.4 exact match, word-identical answers on 93% of the 592 questions, where the causal build gives 29%. You can merge it into this branch if it helps, or it can go against master once this lands, whichever fits your plan.

[peter-sk]

Is there a way Danish Foundation Models could support this implementation? We would be very happy to ensure stable HRMText support with full prefix LM.

[noctrex]

@peter-sk hello and thanks for the model!
For now let me iron out the kinks in order to successfully merge the initial PR, so that the model will be supported and then, there is already a PR open from @bolgacg to add the prefix LM.
We could need your expertise then.
Could we call you up then if needed?

[CISC]

https://github.com/ggml-org/llama.cpp/actions/runs/34296160375/job/102368307545#step:3:3153

[noctrex]

https://github.com/ggml-org/llama.cpp/actions/runs/34296160375/job/102368307545#step:3:3153

ok, cache tensors whose block index has no weight-tensor names (aliased slots) now take the split config from their layer's output projection.
Also covers nanbeige with num_loops > 1, which had the same latent issue.
Unfortunately, I don't have a mac so I can't confirm it on my system so let's see how the actions proceed
noctrex/llama.cpp@d83e3b0c6
Assisted by glm-5.3-flash

[CISC]

@ggerganov Mind reviewing in case you have differing views on this one? :)

[CISC]

Slightly different failure:
https://github.com/ggml-org/llama.cpp/actions/runs/34347683722/job/102461471303?pr=27625#step:3:6024

[noctrex]

Slightly different failure: https://github.com/ggml-org/llama.cpp/actions/runs/34347683722/job/102461471303?pr=27625#step:3:6024

Metal and CUDA failed for the same reason, just at a later step.
The first fix got the model to load, but actually running split across devices still breaks.
hrm_text now doesn't split across meta devices at all
Every device just gets a full copy of the model.
That always works.
Tradeoff: no multi-GPU speedup, but its a small 1B model so it should not pose a problem
516bac0