add GLM-5.3-Flash (GLM5-Next) support - #27773
Conversation
github-actions
Bot
added
model
106ece6 to
9370c82
Compare
|
Hey @timkhronos great work on the PR! A few requests if possible:
Tagging @ngxson for visibility as well. I re-checked and if (1) + (2) is applied, the quants we uploaded work fine (+ the small shard-1 rewrite) and KLD / PPL are correct under this PR. Seems like a simple alias isn't possible actually :( It breaks the quants made with this PR |
|
Hmmm https://github.com/timkhronos/llama.cpp/pull/9/changes would alias the tensors but it looks a bit problematic hmmm |
|
Throwing up some performance numbers here from the lower end of consumer hardware (128GB DDR5 + 24GB VRAM (4090)). avar6 has some freshly converted imatrix quants from this PR up as of now if anyone else wants to give them a go: https://huggingface.co/avar6/GLM-5.3-Flash-BF16-gguf For the IQ3_S, I am getting roughly 300t/s prefill at 256K context and 2048 b/ub size. Generation speed starts off at around 9t/s and drops down considerably by mid window (~128K) to around 6t/s. This seems to track with the 'pooled indexer keys' issue. The model is fully coherent and seems to be working fine. I don't have PPL/KL numbers at the moment as I still need to generate a logit dump. I have noticed an interesting memory quirk, which I haven't seen before. This is the only model I have ever seen have inconsistent checkpoint sizes. As the context fills the checkpoints grow alarmingly fast in size. At ~90K they are already up to nearly 1.6GB. I don't know if this is an expected behavior for this model arch, or if this is a something which needs to be looked into. Also, something of note for you @danielhanchen which I found last night while looking over the three PRs for this arch. The vision towers between this PR and yours differ as well. This PR reuses the name clip.vision.projector_type = "glm4v" while you built a new one clip.vision.projector_type = "glm5next". Likely not much of an issue given how easy it is to regenerate mmproj files, but it will need to be delt with as well. |
|
Yes I'll re-do the vision! This is fine! @timkhronos I confirmed timkhronos#9 works fine and does not break your GGUFs. We will however need to do a cheap shard-1 update so that should be fine |
…ed up long context decode, fla, and slight MTP improvements.
|
@timkhronos I saw you changed the tensor naming - but my solution I provided was to allow everyone's quants to work - now your own ones you uploaded don't work haha. We still need to provide the shard rewrite for the naming (glm5-next) which we're fine with, but now the DeepSeek convention means you yourself have to reupload all shards or do a tensor rename inplace with a script - was this your intention? |
|
@danielhanchen Hey! I ended up going with the the indexer_compressor naming scheme, as it is closer to what's already there, and I was meaning to ask Avar to reconvert anyways, as his ggufs were made when we were missing quantization protection for some crucial tensors so they are not ideal. Your vision projectors will need reconverting though most likely, and your main model ggufs might be missing the |
|
@timkhronos Hey! I made some shard rewrites to https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF/tree/main/Shard_Rewrite for in preparation! |
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* studio: keep MTP on for GLM-5.3-Flash Auto drops embedded MTP for any MLA model, because llama.cpp's MLA/DSA MTP path duplicates the target KV and recomputes the sparse indexer every draft step. That is measured on GLM-5.2 and it holds there. glm5next does not work that way. Its NextN block is an ordinary DSA layer with its own one-layer KV cache, so it pays neither cost, and the gate was turning off a speedup rather than avoiding a slowdown. On the b10715-mix-86bd2d3 prebuilt, UD-IQ1_S on one B200 at --spec-draft-n-max 3, generation goes from 61.1 to 80.2 tok/s with draft acceptance 0.634. Name the exempt architectures instead of widening the gate, so an MLA model nobody has benchmarked still defaults to the safe side. Only the unhyphenated spelling is listed: ggml-org/llama.cpp#27773 ports the same model as "glm5-next" without building the NextN graph at all, and promoting MTP there would be wrong. Also add glm5next to the architectures whose target KV skips the MTP block. Its trunk context installs a filter returning il < n_layer() && !is_recr(il), so blk.45 gets no target KV and reserving for it only costs context. * studio: do not charge GLM-5.3-Flash for a duplicated target KV * studio: tighten comments on the GLM-5.3-Flash MTP change * studio: reserve KDA recurrent rollback copies for MTP * studio: route both rollback reserves through one helper * studio: final comment tightening on the MTP reserve change
|
@ggerganov You may want to review |
|
@timkhronos Sorry, I think we need to create a full Edit: #28173 |
| // glm5-next, complete pools of kpool consecutive positions per sequence, scored as whole pools | ||
| // Cache sequence-private complete pools and re-pool only changed pools. | ||
| uint32_t get_n_kpool (uint32_t kpool) const; // Padded pool count, where the last pool is always unused. | ||
| uint32_t get_n_kpool_new(uint32_t kpool, const llama_ubatch * ubatch) const; // Exact count of new pools. |
There was a problem hiding this comment.
Instead of passing ubatches from the graph and mutating constant state, can we initialize the kpool states upon llama_memory_hybrid_idx_context construction?
There was a problem hiding this comment.
I moved the state init out of the graph into apply(). Ubatch i's pools span tokens that only enter the cells once ubatch i applies, so I think construction time init could get a bit messy. I think apply() is the first point where the layout can be read from the real cells, and it should still precede all graph access.
| if (mem->get_mem_idx() != nullptr && mem->get_kpool() > 0) { | ||
| kpool_states.push_back(kpool_build_state(nullptr)); | ||
| } |
There was a problem hiding this comment.
It is not clear what this does when the ubatch argumetn is nullptr.
There was a problem hiding this comment.
That was building a layout only state for the full cache context, so get n kpools has something to report during graph reserve and state ops. I split it into an explicit kpool_build_layout, and kpool_build_state takes the ubatch by ref now.
| // Sequence edits invalidate cached relative pools. | ||
| bool kpool_is_dirty () const { return kpool_dirty; } | ||
| void kpool_clear_dirty() { kpool_dirty = false; } |
There was a problem hiding this comment.
This is confusing. From the code, it looks like the kpools are owned by the llama_memory_hybrid_idx_context which lives only during the processing of the logical batch. They seem to be recreated for each logical batch - is this not correct? A sequence edit cannot occur inbetween ubatches of a logical batch. Or am I missing something?
Basically, it is not clear to me what is the lifetime of the kpools - try to clarify this.
There was a problem hiding this comment.
So basically, there are two things with different lifetimes. The per ubatch pool layout (kpool_state) is owned by the context and is rebuilt every logical batch.
But the pooled key values are not per batch, but are scattered into the idx_cache, and persist across batches, so each batch repools only the pools it's ubatches touch. Sequence edits between batches shifts the pool grid, which invalidates the cached values, but without touching the layout, and that is what the flag on the memory tracks. It is cleared only once the first ubatch of the next batch succeeds. I renamed it to kpool_cache_stale and split the state builder so the layout only build is explicit. Hope it reads a bit clearer this way.
There was a problem hiding this comment.
Just in case you forgot to push the commit - it's not visible atm.
There was a problem hiding this comment.
Sorry, It should be in 8c28939 now, I misscopied my Acctoken and didn't notice it rejected.
6 participants
Overview
Add support for GLM -5.3-flash a 320B hybrid model, supporting both text and vision.
Additional information
Architecture
GLM 5.3 flash mixes 34 KDA linear layers with 11 DSA laters, with mHC and Deepseek style Moe. Most of the parts are already in llama.cpp so I reused whatever I could:
What I implemented new:
llama_memory_hybrid_dsa: recurrent state + DSA cache, cloned from hybrid ISWA.Rebased onto llama_memory_hybrid_idx instead of the earlier ISWA clone.the encoder is the same family as glmv4 with per head qk-norm, clamped Swiglu and no post conv norm. It reuses glm4v projector with a swiglu_limit key and an optional image token budget.Added as glm5v as GLM 5.3 Flash requires a different pre processing method than what glm4v uses.Tests
Limitations
Quantized GGUFs converted with this PR are available here.
Requirements